Quantifying Uncertainty in Deep Spatiotemporal Forecasting

Abstract: Quantifying uncertainty is critical to risk assessment and decision making in high stakes domains. However, prior works for deep neural network uncertainty estimation have mostly focused on point prediction. A systematic study of uncertainty quantification methods for spatiotemporal forecasting has been missing in the community. In this paper, we analyze forecasting uncertainty in spatiotemporal sequences from both the Bayesian and Frequentist point of view via statistical decision theory. We further conduct case studies to provide an empirical comparison of Bayesian and Frequentist uncertainty estimation techniques. Through experiments on traffic and COVID-19 forecasts, we conclude that, with a limited amount of computation, Bayesian methods are typically more robust in mean prediction, while Frequentist methods are more effective in estimating the confidence levels.

Reviews and author responses:

This paper proposes a comparison of deep learning (DL) uncertainty quantification (UQ) methods in spatiotemporal forecasting problems (specifically, in graph-based problems). The paper define first a metric that is used in statistics and econometrics for interval forecasts, mean interval score Then it reviews the main DL and UQ methods for temporal and spatio-temporal methods. Then it compares 6 methods, 4 from Frequentist framework and 2 from Bayesian framework, on 2 forecasting graphical problems.

I found the paper clear and interesting. This paper is almost a review paper, yet I think it has a great value to the community. The comparisons are showing interesting findings, the state-of-the-art is curated and the use of the 'new' metric is also explained. While no new method or even metric is proposed, I think this paper is worth publishing. My main concern is about the spatial part of the 'spatio-temporal forecasting'. It looks like only graph-based problems are investigated, while a grid-version problem, which are very frequent, would have been interesting.

Questions/remarks:

- In the second experiment (covid), what exactly is the point in using the encoder-decoder sequence? Why the DCRNN does not work as it is in this case? 

- Also, why using the flight mobility as graph edges? isn't the 'standard' propagation, by 'road' and in between neighboring states, neglected?

- From Figure 3, it looks like the confidence bounds often do not contain the ground truth. Do you think it is because of lack of data or because the model is here too simplistic (some key parameters are not used as input)?

small remarks:
- part 2.1: equation of MIS: define rho
- eq. 5: define A (adjacency matrix?)
- Table 2 --> Table 1 (change ref. in the text)


This paper aims to provide a systematic study on the problem of uncertainty quantification for spatial-temporal forecasting. The study reveals several interesting conclusions regarding different types of methods on the task.

Could the authors elaborate on how MIS can be used as a loss function during training? Is it by jointly optimizing both the NN parameters theta and the lower and upper confidence bound of the predictions?

It is a bit vague in Sec. 3.3 and therefore not clear what is the proposed frequentist UQ method specifically. It would be helpful to write down the equations to be precise.

The general guidelines in Sec. 3.5 make sense and might be of interest to practitioners. The main conclusion here is that Bayesian methods perform better in terms of mean predictions and frequentist methods perform better in terms of confidence intervals, which is no surprise I guess.

A large number of works on probabilistic NN and/or Bayesian deep learning for uncertainty estimation are missing [1,2,3].

It is interesting to see MC dropout actually harms the prediction accuracy and the only one method with significant improvement margin is the SG-MCMC.

For the COVID experiments, how is the graph constructed exactly? The description in Sec. 5.2.1 misses a lot of details.

It is a bit strange that Table 2 reports the prediction/forecast for the residual between the ground-truth death incident numbers and GLEAM’s predictions. Such a setting seems problematic. Why not directly predict the death incident numbers? Is it because the prediction is highly inaccurate? Either way, if this result is in the appendix, it would be better to move it to the main text. To me, even the results in the appendix (as a figure) are not useful enough. It is difficult to evaluate DeepGLEAM’s improvement.

Another question is: If one adds the predicted residual from DeepGLEAM to the GLEAM prediction, would GLEAM’s prediction accuracy be improved?

Although some results are interesting. it is unclear whether this paper is a pure study or there are actually proposed new methods.

Minor:
Page 3: Is the adjacency matrix of size P * P?
Page 6: Start of Sec. 5.1.2: Table 2 -> Table 1

[1] Natural-Parameter Networks: A Class of Probabilistic Neural Networks, NIPS 2016
[2] Feed-forward Propagation in Probabilistic Neural Networks with Categorical and Max Layers, ICLR 2018
[3] Sampling-free Epistemic Uncertainty Estimation Using Approximated Variance Propagation, CVPR 2019

----------
After Rebutal

I appreciate the authors’ response to my questions and they addressed some of my minor concerns on the clarity and presentation. The authors confirm that that their work is a pure study on UQ for forecasting. In this case, the methods evaluated is not very specific to the forecasting tasks, and the conclusions are not very surprising either. Therefore it seems there is limited technical merit from this paper. The setting of the COVID forecasting task is also a bit strange and the authors’ response does not fully address my concern. 

I would like to keep my ratings unchanged.

**Q14**: "The related work ... more references.”

**A**: We added the recommended reference in Sec. 3.3.

**Q15**: “In Table 1 ... (since you are using it with MIS).“

**A**: Thanks for the suggestion, we included the interval and MAE in Table 1 and Table 2. The related analysis was added in Sec 5.1.2 and 5.2.2. 

**Q16**: "Also, evaluating a single confidence level might not be enough. It is standard practice to evaluate on multiple confidence levels to evaluate the entire distribution."

**A**: Thanks for the suggestion. Now that we have an extra page in the main text, we have included 95% CI, 90% CI, 80% CI in the updated Table 1 and Table 2. 

**Q17**: “For the COVID-19 dataset, you have less than four months of data ... significant non-stationarity, I think there are better datasets for a fair comparison between the methods.”

**A**: We chose this dataset because it is both timely and important to understand the uncertainties of the predictions about the ongoing pandemic. We have trained our DeepGLEAM model on the residual between the physical based GLEAM model and observed death tolls to mitigate the nonstationarity. We computed the autocorrelation function which empirically demonstrates that the nonstationarity is not as significant as the original data. 

## Questions during the rebuttal period:

**Q18**: “Why do you focus on 95% confidence level?”

**A**: We focused on 95% confidence level since it is the frequently used confidence level in both traffic prediction and COVID-19 prediction.

**Q19**: “What do you mean by "MIS is non-parametric"? I can optimize MIS for a parametric model”. “"Strong assumptions on the likelihood function". What are these assumptions?”

**A**: Thank you for suggesting that we make our claims more accurate. We have revised the wording to “MIS does not require the model to be parametric”, and have specified the assumptions.

**Q20**: “What does DCRNN mean? Did you apply any hyperparameter optimization? (How did identify the values you used in DCRNN?)”

**A**: Thanks for pointing out. We added its full name “Diffusion Convolutional Recurrent Neural Network” in Sec. 5.1.1. The hyperparameter setup can be found in Appendix C.1 and D.2. 

**Q21**: “Why did you combine the MAE with the MIS? Does that change your analysis in Proposition 1 and 2?”

**A**: We combined MAE with MIS as the MIS loss function in order to simultaneously optimize the point estimate $f(x)$, upper bound $u(x)$, and lower bound $l(x)$. It does not change the analysis in Proposition 1 and 2.

### Other minor comments:
**Q22**: “When defining MIS in section 2.1, you do not introduce \rho (the confidence level).”

**A**: Thanks for pointing out. We fixed it in Sec. 2.1.

**Q23**: “"It is difficult for DNN as they often do not generate explicit likelihood outputs." -> I suggest you cite some normalizing flows papers.”

**A**: Thanks for the suggestion. We added the reference.

**Q24**: “Last paragraph before Section 3.2: You have already introduced MIS in Section 2.1. This sentence is out of place.”

**A**: Thanks for pointing out. We fixed it.

**Q25**: “Section 3.5. "Uncertainty qualification" should be "Uncertainty quantification".”

**A**: Thanks for pointing out. We fixed it.

**Q26**: “Time granularity of COVD-19 dataset is not clear (daily? weekly?)”

**A**: Thanks for pointing out, we updated the setup in Appendix D.2 to make it more clear.  

**Q27**: “You did not mention that the spline quantile function was a linear spline.”

**A**: Thanks for pointing out. We fixed it. 









Thank you for the insightful comments. We decided to split our reply into two parts due to the word limit. 

**Q1: "The objectives ... the same "uncertainties"”**

**A**: We have to take into account both bias correction and parameter calibration problems, as in real world examples, one does not have a clear cut between the two. Admittedly, different methods are targeting and excel at quantifying uncertainties from different sources. That is exactly why we conduct this experiment to test their performance on spatio-temporal data and observe the tradeoffs.

We have expanded Sec. 3.1 to include a discussion on this question. We have also pointed out that methods like quantile regression focuses on the parameter calibration problems whereas the bootstrap method can take both into account, albeit having a potentially high resampling variance.

**Q2: "The description of ... the Winkler score.”**

**A**: We agree that it does not have to be a credible interval. It can be used to evaluate confidence intervals as well, which is partly why we chose this metric. The reason that asymptotically there are 1-α of the observations outside of the “optimal” interval is because of the balance it strikes between the reward towards narrower intervals and the penalty towards excluding observations. We have modified the language in the paper accordingly. We included the Winkler score in Sec. 2.1.

**Q3: "We ﬁrst ... "unify" them.”**

**A**: We agree that we are not really trying to unify the frequentist and Bayesian philosophies. We have changed the wording accordingly.

**Q4: "Point prediction ... sequence predictions.”**

**A**: Here we do not actually mean “point estimate”. Over the time series, the “point prediction” means that the prediction is over a point, instead of over the entire time series.

**Q5: "Deterministic models -> What do you mean by "deterministic"? ”**

**A**: Deterministic model means the output of the model is determined by the parameter values and the initial conditions without stochasticity.

**Q6: "high-quality uncertainty -> This is never defined. ”**

**A**: Thank you for pointing out. We believe you mean “high-quality uncertainty quantification”. Since we used MIS to measure the uncertainty quantification, high-quality uncertainty quantification corresponds to small MIS. We added it in Sec. 3.5.

**Q7: "Accurate predictions vs better generalization.”**

**A**: To provide an explanation for these two terms, accurate predictions could be understood as low MAE/RMSE or general (empirical) risk. Better generalization is equivalent to low out-of-sample prediction risk (extrapolation error). We care about this point when we only have very limited data size. For ‘better generalization’, there are several discussions in the deep learning community both in theory by Poggio et al (2018), and in practice by Zhang et al (2016).
[1] Poggio, Tomaso, et al. "Theory IIIb: Generalization in deep networks." arXiv preprint arXiv:1806.11379 (2018). 
[2 ]Zhang, Chiyuan, et al. "Understanding deep learning requires rethinking generalization." arXiv preprint arXiv:1611.03530 (2016).

**Q8: "Confidence intervals are not prediction intervals.”**

**A**: Thanks for pointing out. This is a typo, we fixed it.

**Q9: ""Frequentist methods generally outperform Bayesian methods in conﬁdence interval." -> What does that mean? Are you referring to coverage?"**

**A**: It means the MIS of Frequentist methods is generally smaller than the MIS of Bayesian methods.

**Q10: ""One-sided quantile loss function" -> you did not define it. You can at least refer to part of the MIS expression."**

**A**: We added the quantile loss function and MIS loss function in Sec. 3.3.

**Q11: “Quantile crossing ... Bayesian philosophies?”**

**A**: In the asymptotic limit, where the size of the data set is infinite and if the data belong to the model class, we will not have the quantile crossing problem. Therefore, adding variations in both the data and the parameters help alleviate this problem. And using an ensemble of models is in itself a way to integrate across different models, reminiscent of the Bayesian approach.

**Q12: “What do you mean by "CRPS functions"... about SQ.“**

**A**: We fixed it. Gasthaus et al (2019) minimize CRPS by assuming the quantile function to be a piecewise linear spline with monotonicity. And we implemented this method.

[1] Jan Gasthaus, Konstantinos Benidis, Yuyang Wang, Syama Sundar Rangapuram, David Salinas,
Valentin Flunkert, and Tim Januschowski. Probabilistic forecasting with spline quantile function
RNNs. In The 22nd International Conference on Artificial Intelligence and Statistics, pp. 1901–
1910, 2019.

**Q13: “In Section 3.5, you say "Through both theoretical analysis and experiments, ...", where is this "theoretical analysis"?”**

**A**: This is a typo, we deleted it.















Thank you for the insightful comments. Here is our reply:

$\textbf{Q1:}$ “Could the authors elaborate on how MIS can be used as a loss function during training? Is it by jointly optimizing both the NN parameters theta and the lower and upper confidence bound of the predictions?”
“It is a bit vague in Sec. 3.3 and therefore not clear what is the proposed frequentist UQ method specifically. It would be helpful to write down the equations to be precise.”

$\textbf{A:}$ We added the loss function of MIS regression as Eqn. (6) and the loss function of quantile regression as Eqn. (7) in Sec. 3.3. We use the MIS loss function to train the NN parameters theta and optimize the point estimate $f(x)$, upper bound $u(x)$, lower bound $l(x)$. We use the quantile loss function to train the NN parameters theta and optimize three quantiles, one for upper bound, one for median, and one for lower bound. Hope this can help.

$\textbf{Q2:}$ “A large number of works on probabilistic NN and/or Bayesian deep learning for uncertainty estimation are missing.”
Answer: Thanks for pointing out. We added these works in Sec. 4.

$\textbf{A:}$ “For the COVID experiments, how is the graph constructed exactly? The description in Sec. 5.2.1 misses a lot of details.”
Answer: Thanks for the comment. We revised the text describing the graph in Sec. 5.2.1. 

$\textbf{Q3:}$ “It is a bit strange that Table 2 reports the prediction/forecast for the residual between the ground-truth death incident numbers and GLEAM’s predictions. Such a setting seems problematic. Why not directly predict the death incident numbers? Is it because the prediction is highly inaccurate? Either way, if this result is in the appendix, it would be better to move it to the main text. To me, even the results in the appendix (as a figure) are not useful enough. It is difficult to evaluate DeepGLEAM’s improvement.”
“Another question is: If one adds the predicted residual from DeepGLEAM to the GLEAM prediction, would GLEAM’s prediction accuracy be improved?”

$\textbf{A:}$ Thanks for pointing out. “Directly predicting the death number using DCRNN (Deep) suffers from low accuracy as we have limited samples.” We added this sentence in Sec. 5.2.1 for a better explanation. We also added quantitative comparisons among GLEAM, DeepGLEAM, and Deep models in Table 4 in Appendix D.3. There is a 6.6% improvement for DeepGLEAM (DCRNN) and 17% improvement for DeepGLEAM (SG-MCMC) on average from GLEAM.

$\textbf{Q4:}$ “Although some results are interesting. it is unclear whether this paper is a pure study or there are actually proposed new methods.”:

$\textbf{A:}$  The field is currently lacking a systematic study of uncertainty quantification methods for spatiotemporal forecasting problems. We did not intend to propose new methods but rather to obtain a more holistic view of different UQ methods, applied to this domain. We did an extensive experiment among frequently used frequentist and Bayesian methods on two real datasets of different sizes,  identified the metrics to quantify the performances of both Bayesian and frequentist methods, and made a useful recipe for practitioners. This recipe covers multiple cases including large data with sufficient computation, large data with limited computation, and small data. Also, we improve the accuracy of the GLEAM prediction.  



Thank you for the insightful comments. Grid-version models such as convLSTM are one of the directions we will explore in the future. 

For other questions:

$\textbf{Q1:}$ “In the second experiment (covid), what exactly is the point in using the encoder-decoder sequence? ”

$\textbf{A:}$ Thanks for pointing out, we updated the DeepGLEAM setup in Appendix D.2 to make it more clear  “The encoder reads as input a $7 \times 50 \times 4$ tensor that comprises the daily residuals between the observed death number and the GLEAM forecasts for the $50$ US states over $4$ different prediction horizons. It encodes the information in $7$ hidden layers. 
The decoder produces forecasts of the weekly residuals between the true death number and the GLEAM forecasts for each state for the following $4$ weeks. We perform autoregressive weekly death predictions (from one week ahead to four weeks ahead).”

$\textbf{Q2:}$ “Why the DCRNN does not work as it is in this case?”

$\textbf{A:}$ Review 1 also points this out. “Directly predicting the death number using DCRNN (Deep) suffers from low accuracy as we have limited samples.” We added this sentence in Sec. 5.2.1 for explanation. We also added quantitative comparisons among GLEAM, DeepGLEAM, and Deep models in Table 4 in Appendix D.3. 

$\textbf{Q3:}$ “why using the flight mobility as graph edges? isn't the 'standard' propagation, by 'road' and in between neighboring states, neglected?”: 

$\textbf{A:}$ GLEAM already explicitly considers short-range commuting and long-range traveling while simulating the spread of the COVID-19 pandemic. Therefore, in DeepGLEAM, we deemed sufficient to introduce air traffic data to capture the linking existing between the outbreak dynamics of the different states. 

$\textbf{Q4:}$ “From Figure 3, it looks like the confidence bounds often do not contain the ground truth. Do you think it is because of lack of data or because the model is here too simplistic (some key parameters are not used as input)?” 

$\textbf{A:}$ In theory, the confidence interval does not have to contain the ground truth. Thanks to Review 4’s comment, we separate MIS into the interval and the coverage,  where the interval is the length between upper and lower bounds and the coverage is the penalty for confidence bounds not containing the truth. In order to get a better MIS, we need to consider both interval and coverage. So it is more like a tradeoff. 

$\textbf{Q5:}$ “eq. 5: define A (adjacency matrix?)”: 

$\textbf{A:}$ Yes, A is an adjacency matrix (see Sec. 3.2: “We also have spatial information represented as an adjacency matrix A.”)

$\textbf{Q6:}$ “part 2.1: equation of MIS: define rho, Table 2 --> Table 1 (change ref. in the text)”

$\textbf{A:}$ Thanks for pointing out. We fixed it.



## Summary

The paper considers the problem of uncertainty quantification in spatio-temporal forecasting with deep neural networks. Both frequentist and Bayesian methods are compared on two datasets (traffic and COVID-19). Recipe for practitioners is given based on data size and computational budget.

## Reasons for score

Unfortunately, I vote for rejection. The work is not clearly motivated with respect to the state-of-the-art. There is a clear lack of rigor in the terminology used, with misleading claims. There are multiple issues in the experiments. After reading the paper, many questions and motivations related to design choices remain unanswered. More details are given below.

## Strengths

- Overall, the paper is easy to read.
- Uncertainty quantification in Deep forecasting models is an important problem for optimal decision making. Spatio-temporal data makes the problem harder.
- Multiple uncertainty quantification methods are compared (both frequentist and bayesian).

## Weaknesses

- The objectives are not clearly stated. The authors never really define uncertainty quantiﬁcation, which has different meanings depending on the context. In fact, there are multiple sources of uncertainties (aleatoric, epistemic, etc). Which one are you trying to capture? Furthermore, some of the methods considered do not capture the same "uncertainties" (e.g. quantile regression vs bootstrap).

- There is a clear lack of rigor in the terminology used. Certain claims are too strong or misleading.

1. The description of the Mean Interval Score is misleading. You say: "It rewards narrower credible intervals and penalizes intervals that do not include the observations". It does not have to be a credible interval. Furthermore, note that the "optimal" interval at level $\alpha$ should have $1 - \alpha$ of the observations outside the interval. The MIS tries to find the interval with the right coverage. You should also mention the Winkler score.

2. "We ﬁrst unify uncertainty quantiﬁcation from Frequentist and Bayesian perspectives into a single framework". I find this statement a bit too strong. Although expression (1) and (2) might be useful to understand the two philosophies, they do not really "unify" them.

3. "Point prediction" has a strong meaning in statistics/ML. We often contrast point prediction and probabilistic prediction (or forecasts). Why do you mean by "prior works for deep neural network uncertainty estimation have mostly focused on point prediction"?  Also, in the following sentence, you contrast point prediction and sequence prediction: "Unfortunately, most existing UQ methods deal with point predictions instead of sequence predictions."

Other notions that need clarification/definition:
- Deterministic models -> What do you mean by "deterministic"?
- high-quality uncertainty -> This is never defined.
- Accurate predictions vs better generalization
- Confidence intervals are not prediction intervals
- "Frequentist methods generally outperform Bayesian methods in conﬁdence interval." -> What does that mean? Are you referring to coverage?


4. The paragraph "MIS and quantile regression" is poorly written
- "One-sided quantile loss function" -> you did not define it. You can at least refer to part of the MIS expression.
- Quantile crossing and remedies are poorly explained. Why an ensemble model would solve the quantile crossing problem? Why does it combine the frequentist and the Bayesian philosophies? 
- What do you mean by "CRPS functions"? Also, Gasthaus et al (2019) did not approximate the CRPS (a closed-form expression was derived).
- What is Spline Quantile regression (SQ) ? I guess you are referring to Gasthaus et al (2019). You should make this clear, and say more about SQ.


5. In Section 3.5, you say "Through both theoretical analysis and experiments, ...", where is this "theoretical analysis"? 

6.  The related work on "Uncertainty quantification" is weak. Given that this is the main topic of the paper, this section should be extended with more references. See some references below. Also, how this work fits in the literature is not clear.

- Danijel Kivaranovic, Kory D. Johnson, Hannes Leeb. Adaptive, Distribution-Free Prediction Intervals for Deep Networks. Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics, PMLR 108:4346-4356, 2020.
- Tagasovska, Natasa, and David Lopez-Paz. "Single-model uncertainties for deep learning." Advances in Neural Information Processing Systems. 2019.
- Pearce, Tim, et al. "High-quality prediction intervals for deep learning: A distribution-free, ensembled approach." International Conference on Machine Learning. 2018.

7. There are multiple issues in the experiments.

In Table 1, the comparison between forecasting methods is unfair. In fact, optimizing MIS and evaluating on MIS will tend to win. This is also observed in Table 2 with the quantile loss. Instead, you should also report interval length and coverage, i.e. split the MIS into two components. This might give you more insights into the differences between the methods. You should also report MAE (since you are using it with MIS).

Also, evaluating a single confidence level might not be enough. It is standard practice to evaluate on multiple confidence levels to evaluate the entire distribution (see below).

"SQ method uses CRPS as the loss function, which could be unfair for comparison using MIS metric". It can be shown that the CRPS is the integral of the MIS metric for all confidence levels. So, it is fair to compare them if you compute MIS for all levels.

For the COVD-19 dataset, you have less than four months of data, and you try to forecast the following 4 weeks. I find it too ambitious. Is COVID-19, the right dataset to use? With very few data points and significant non-stationarity, I think there are better datasets for a fair comparison between the methods. See the papers you have cited in the "related work" section.


## Questions during the rebuttal period

Please address and clarify the issues raised above including the following additional questions:

- Why do you focus on 95% confidence level?

- What do you mean by "MIS is non-parametric"? I can optimize MIS for a parametric model.

- "Strong assumptions on the likelihood function".  what are these assumptions?

- What does DCRNN mean? Did you apply any hyperparameter optimization? (How did identify the values you used in DCRNN?)

- Why did you combine the MAE with the MIS? Does that change your analysis in Proposition 1 and 2?


# Other minor comments

- When defining MIS in section 2.1, you do not introduce \rho (the confidence level).
- "It is difficult for DNN as they often do not generate explicit likelihood outputs." -> I suggest you cite some normalizing flows papers.
- Last paragraph before Section 3.2: You have already introduced MIS in Section 2.1. This sentence is out of place.
- Section 3.5. "Uncertainty qualification" should be "Uncertainty quantification".
- Time granularity of COVD-19 dataset is not clear (daily? weekly?)
- You did not mention that the spline quantile function was a linear spline.









Meta-review: This paper investigates methods for producing and evaluating interval forecasts rather than point forecasts. The authors focus on spatio-temporal forecasts whose interval accuracy is measured with the Mean Interval Score.

Pros:

Uncertainty quantification is an important topic that is often ignored in the ML literature where the focus remains on point predictions. The COVID-19 example the authors use is a clear example of the need for such methods: the CDC-sponsored COVID forecast hub mandates interval forecasts.

Cons:

In the words of one reviewer "This paper is almost a review paper", a statement with which I fully concur. Neither the evaluation metric (MIS) nor the methods for generating intervals are novel. There is no particular effort to argue "why MIS and not weighted interval score" or "why these methods". If it were a review paper with more comprehensive coverage of the relevant state-of-the-art (a deficiency mentioned by multiple reviewers), it may make a more positive impression. As is, the story is mainly incomplete. To resubmit at another venue, I would suggest the authors either (a) state clearly what is new here in this paper or (b) make it a review paper that more comprehensively evaluates and discusses current state of the art. The statements in the last paragraph of page 1 read more like the authors are shooting for (b) than (a): "we conduct a systematic study", "we investigate". Finally, the evaluations undertaken here do not really make use of (or correct for) the spatio-temporal task. Presumably some locations/time-periods are more difficult than others. So simply averaging over everything (as in Table 1) doesn't take the structure into account. It is likely better to examine relative accuracy to strawman forecaster or perhaps use a random effects model.

Decision: withdrawn-rejected-submissions