Probability Paths and the Structure of Predictions over Time
Authors: Zhiyuan Jerry Lin
Hao Sheng
Sharad Goel
Venue: NeurIPS 2021 Poster (2021)
Keywords: Probability Path
Bayesian Time Series Modeling
Evolving Prediction
Martingale

Abstract: In settings ranging from weather forecasts to political prognostications to financial projections, probability estimates of future binary outcomes often evolve over time. For example, the estimated likelihood of rain on a specific day changes by the hour as new information becomes available. Given a collection of such probability paths, we introduce a Bayesian framework -- which we call the Gaussian latent information martingale, or GLIM -- for modeling the structure of dynamic predictions over time. Suppose, for example, that the likelihood of rain in a week is 50%, and consider two hypothetical scenarios. In the first, one expects the forecast to be equally likely to become either 25% or 75% tomorrow; in the second, one expects the forecast to stay constant for the next several days. A time-sensitive decision-maker might select a course of action immediately in the latter scenario, but may postpone their decision in the former, knowing that new information is imminent. We model these trajectories by assuming predictions update according to a latent process of information flow, which is inferred from historical data. In contrast to general methods for time series analysis, this approach preserves important properties of probability paths such as the martingale structure and appropriate amount of volatility and better quantifies future uncertainties around probability paths. We show that GLIM outperforms three popular baseline methods, producing better estimated posterior probability path distributions measured by three different metrics. By elucidating the dynamic structure of predictions over time, we hope to help individuals make more informed choices.

Probability Paths
and the Structure of Predictions over Time
Zhiyuan Jerry Lin"
Hao Sheng
Sharad Goel Harvard University sgoel0hks harvard. edu
Facebook zylin@fb. com
Stanford University
haoshengOcs. stanford.
Abstract
In settings ranging from weather forecasts political prognostications to financial projections, probability estimates of future binary outcomes often evolve over time. For example, the estimated likelihood of rain on specific day changes by the hour as new information becomes available. Given collection of such probability paths, we introduce Bayesian whaneworrkwhich we call the Gaussian latent information martingale, or GLIM-for modeling the structure of dynamic predictions over time. Suppose, example, that the likelihood of rain in a week is 50%, and consider two hypothetical scenarios. In the first, one expects the forecast be equally likely become either 25% or 75% tomorrow; in the second, one expects the forecast to stay constant for the next several days. A time-sensitive decision-maker might select a course of action immediately in latter scenario, but may postpone their decision in the former, knowing that new information is imminent. We model these trajectories by assuming predictions update according latent process of information flow, which is inferred from historical data. In contrast to general methods for time series analysis, this approach preserves important properties of probability paths such the martingale structure and appropriate amount of volatility and better quantifies future uncertainties around probability paths. We show that GLIM outperforms three popular baseline methods, producing better estimated posterior probability path distributions measured by three different metrics. By elucidating the dynamic structure of predictions over time, we hope to help individuals make more informed choices.
Introduction
Probabilistic predictions of future binary outcomes are ubiquitous in real-world statistical and machine
learning problems, ranging from election modeling [Erikson and Wlezien 2012 Shirani- et al.)
2018 Gelman et al.] 2020 to weather forecasting |Esteves et al.)
2019 to assessing mortality
risk Teres et al.) 1987 Malmberg et al. 1997 Mlrtinez-Allrio al. 1999 Yan al. 2020 risk
assessment tools in criminal justice applications [Chouldechova] 2017 Lin et al. 2020 Offen such probabilistic predictions are not static, rather evolve over time as more information becomes available. For instance, the estimated likelihood candidate wins election updates with every new poll that is conducted, and the estimated chance rain on a given future day changes by the hour as new metcorological data become available.
In many of these domains, a large body of work aims to provide accurate, real-time forecasts that account for the very latest information. However, significantly less attention has been paid to understanding how the predictions themselves evolve over time. If there's 's 35% chance of rain one week from now, what can say about tomorrow's prediction of rain on that same day? In expectation, tomorrow's 's prediction must be the same as today's- were not, we would be better
"Research conducted when the author Stanford University.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Summer
Winter
100%
Probability
75%
50%
25%
0%
0
3
4
5
2
3
6
7
Days
Figure I: Dynamic forecasts of rainfall across several Australian cities during the summer and winter, starting one week before the target date. All forecasts initially indicate approximately 35% chance of rain. But the forecast trajectories- which we call probability paths-have higher volatility the summer than in the winter.
off updating today's prediction to match our expectation tomorrow's forecast. In other words, such
sequence of probabilistic predictions should be martingale [Augenblick and Rabin 2021 Foster and Stine 2021, Gelman al.) 2020 Taleb et al., [2019 Taleb] 2018 Ely et al. 20155. But aside from satisfying this martingale property, the forecast trajectories vary widely from setting the next.
Consider, for example, the collection of dynamic weather forecasts presented in Figure The plot shows the evolution of 7-day precipitation predictions for a of Australian cities [Williams] 2011 Young and Young 2018), where we have disaggregated the predictions by season (summer or winter) and restricted those instances for which the 7-day forecast starts at approximately 35%.
visually apparent, the mean prediction on any later day is still approximately 35%, expected. But the summer forecasts are considerable more volatile than those the winter. the winter months (June-August), the prediction is unlikely to change substantially until immediately before the target date. In contrast, the summer months (December-Februaae the prediction is likely to oscillate considerably-eittee up down-with each passing day.
These patterns have immediate consequences for seneisensi decision makers [Ferguson] 2004 Tsitsiklis and Van Roy 1999 Chow and Robbins 1963]. As a simple example, suppose one is planning weekend picnic, with preference both sunshine and for sending out invitations as soon possible. If one knows that a weather forecast is unlikely to change in the immediate future, one might opt to simply send out the invitations and hope for the best. Alternatively, knowing that more information is imminent- as reflected by forecasts that are likely to soon change one might choose wait another day before deciding to hold the party.
In this paper, we develop a Bayesian framework to investigate- and predict-the structure of dynamic probabilistic forecasts: the Gaussian latent information martingale, GLIM. In contrast existing techniques for time-series modeling, our approach is tailored to the specific properties of evolving probabilistic forecasts, which we call probability path. Most importantly, our approach satisfies the martingale property described above, with the expected value of future predictions along the path equal to the current prediction. Our model is trained directly on historical paths, and learns the heterogeneous, covariate-dependen structure of forecasts. As result, it is rich enough to predict the type of structure illustrated Figure We show that this approach often outperforms existing methods for quantifying uncertainty for evolving time series forecast, ostensibly because past general-purpose methods do not leverage the idiosyncratic properties of probability paths. As real-time forecasting becomes increasingly popular, we hope this work helps researchers and decision makers investigate and act on these dynamic predictions.
Also known stream of beliefs, streamflow forecast, evolution of forecasts under different contexts.

---

2
Background
In this section, we formally define the problem we aim to address and introduce related literature pertaining to this topic.
2.1 Problem Setup
Suppose we are interested in a binary outcome YT E {0, 1} that will either occur or not at some fixed future time T and the current time is f 0. For example, may be interested in whether it will rain T= 7 days from today. At time O, we observe an initial probabilistic point estimate Yo- = yo about the final outcome (e.g.. from static weather forecast model), as well collection of covariates X (e.g., current and recent metcorological measurements). Our goal, then, is to accurately estimate how the predictions [=] of the event in question will evolve over time, described by posterior distribution of {Y) given all the information we have observed at 0.
While this problem might seem to be straightforward time series modeling application at first
sight, as shown by statisticians and economists AAgenblick and Rabin 2021 Foster and Stine 2021
Gelman et al. 2020 Taleb al. 2019 Taleb 2018 Ely al., 2015], J. such evolving probabilistic predictions need to have an expectation equal to the initial prediction (i.e., being a martingale
starting from yo) and shall not display excessive or insufficient movement (i.e., appropriate amount
of volatility as we will explain later in Section 4. putting additional constraints on the shape of
predicted probability path distributions. Indeed, without any additional information, the only sensible posterior distributions of future predictions (Y]== are ones with the mean equal to yo [Gelman
al. 2020
Taleb 2018 the current observed probabilistic point estimation.
is worth noting that in this problem, unlike in many common time series modeling tasks, we are not interested in producing a sequence of point estimates for the future probability path {Y) as model that always outputs the current best probabilistic prediction yo for all future (Y) is arguably the best point estimates we could possibly give without observing additional information. Instead, here we wish to provide an accurate estimate of the entire posterior probability path distribution conditioning on having observed all information available
O. As we will explain in detail in Section 4 the quality of the estimated distribution can be assessed via examining some of the properties probability path distribution should possess such as its posterior mean calibration, expected volatility, and credible interval coverage at different significance levels.
2.2 Related Work
In this subsection, we briefly review three classes of methods relevant to the problem of estimating the posterior distribution of probability paths (Y)T
Martingale model of forecast evolution. The model most relevant to our setting is probably the martingale model of forecast evolution, or MMFE, originally developed by Graves et al. [1986] and Heath and Jackson [1994) estimate the evolution of production forecasts in supply chain optimization. In order to estimate how the forecasts of the production of a commodity at a future time T evolve for = 0 to T, learning from historical data, MMFE directly estimates the covariance matrix of increments between consecutive time points. With the learned covariance matrix, at t = 0 we are able to simulate arbitrary number of possible scenarios (hence the posterior distribution of forecast evolution) of how the production predictions for ( = T will be at time / = o,.... T. Although MMFE was originally invented few decades ago, there are many recent successful applications and improvements of MMFE in wide range of fields such as energy management [Xu et al. 2019).
reservoir operation [Zhao et al. 2013 [011]]. stock replacement [Boulaksil] 20166), and production planning (Aley et al.) 2015]. Finally, recent work such as Augenblick and Rabin [202] and Foster and Stine [021] are studying martingale structures of probability paths, but they usually focus on developing diagnostic metries for existing predictions instead of constructing predictive frameworks that capture probability path heterogeneity like GLIM.
Probabilistic time series models. When it comes to modeling time series data, probability path or not, arguably the most widely known class of models probabilistic time series models. Famous examples include the autoregressive (AR), autoregressive moving average (ARMA), and autoregres- sive integrated moving average (ARIMA) models [Brockwell and Davis 2016). The autoregressive
structure and the estimated Gaussian noise term allow us to simulate future time series evolution
3

---

through recursive sampling. If we additionally model the noise terms themselves as a function of noise terms from previous time steps, we have a stochastic volatility model. For example, when we assume the variance of the error terms or innovation-follon
AR structure, we have a ARCH [Engle model. Similarly, assuming ARMA structure of error terms give the GARCH (Bollersiev
1982
1986 model. More recent development on stochastic volatility models include GARCHX [Sucarrat
2020 Francq and Thieu 2019 Engle and Patton 2007]. Augmented ARCH [Bera et al. 1992]. and
the Gaussian Copula Process Volatility (GCPV) model [Wilson and Ghahramani 2010). which all
allow for more complex structure of estimated variance over time and some of them grant the use of covariate information. Despite their vast success in numerous applications, most of these classical probabilistic time series models are primarily designed to model a single time series, rather than to model the multiple, independent time series in our motivating examples. Further, while some of these approaches may still be applied to model probability paths with some modifications, they are not designed to fully leverage the distinctive characteristics of probability paths, such as their martingale property, volatility level, or their finite time horizon (meaning the predictions must converge to O or 1 at fixed time T).
Bayesian deep sequence models. Rapid development of deep learning models over the past decade has offered new avenue of modeling time series data. Specifically, modern deep-learning based recurrent neural networks such as LSTM [Hochreiter and Schmidhuber 1997, Goodfellow et al. 2016] and DeepAR (Salinas al. 2019] can not only model and predict time series data, but also naturally incorporate the information of covariates into their predictions. There is a stream of recent work (Wen et al.) 2017 Eisenach et al.] 2020) Rangapuram et al. 2018) particularly focusing providing better calibrated future time series predictions. Moreover, recent advancement in Bayesian deep learning has made uncertainty estimation possible in addition point predictions through random regularization [Srivastava et al.) 2014 Gal and Ghahramani] 2015 Kingma et al.] 2013) or
weights distribution [Blundell et al. 2015). J. In spite of their significant model capacity, typical deep sequence models suffer from the same problem as classical probablistic time series models: they not constrained by the martingale property in this bounded probability path setting. As we will show in Section 4 deep models tend to become overly conservative when it comes to uncertainty estimation and do not conform with the martingale requirement.
In summary, all of these techniques are powerful methods for modeling general time-series data, but none are tailored the idiosyneratic properties of probability paths. In Section 4 we will evaluate GLIM against representative baseline models from each of the three classes of models and illustrate how GLIM produce posterior future probability path samples with more accurate uncertainty estimation and better mean calibration.
3 Modeling Probability Paths
3.1 Gaussian latent information martingale
Now, we formally introduce the Gaussian latent information martingale, or GLIM, as a framework to model and predict probability paths. To model a probability path (Y)To we introduce corre- sponding sequence of latent variables { where the scalar Z, intuitively represents the amount of information received between time and t. Receiving positive information (i.e., positive Z,) makes the ultimate outcome Yr more likely, while receiving negative information makes the ultimate outcome less likely.
Now, we assume the latent variables {2)T- follow a multivariate Gaussian distribution with mean zero and T T covariance matrix Σ- Σ(X, f). where the covariance matrix can depend on both the observed covariates and a parameter 0. Finally, we model (Y)To by assuming predictions along the probability path can be expressed in terms of the following conditional probabilities:
Y, = Pr 7
Zi
0
Z1
Z
where y is a fixed constant.
Eq. (T) is closely related to the latent variable formulation of logistic regression models, but we additionally condition on the latent variables. This expression has several attractive properties for modeling probability paths. First, Y, is naturally constrained to lie between 0 and 1. Second, Yr is

---

guaranteed to be either 0 or 1, since once we condition on Zi..... Zr, there is no randomness left in the expression. Finally, and most importantly, Y satisfies the martingale property, as shown below. Proposition 1. For (Y) satisfying Eq.
① we have B| Yo. Yol = Y.
SK<T.
The proof is given by Appendix A.3 in the supplement material. Given collection of probability paths, our goal is to infer the parameter 6, as we describe next.
3.2 Model inference
We take Bayesian approach to fit GLIM defined above. The main difficulty is deriving tractable expression for the corresponding likelihood function. so, we first introduce some notation.
Given any 0 < t < T, we divide the covariance matrix Σ into four quadrants as follows:
T
}
Σ
Σ1
22
}
T-t
For t O, we further define Σg E. For arbitrary matrix M, use the notation M to denote the (i,j) entry in the matrix. Similarly, for an arbitrary vector we use the notation V(t) to denote the i-th entry in the vector.
Now, using this notation, Theorem I presents recursive formula for computing the (log) probability density function for our Gaussian latent information martingale with parameter 0. That expression can in turn be used to infer the parameters of the model from the observed data via maximum likelihood estimation or. alternatively. fully Bayesian inference.
Theorem 1. Consider probability path yuu.... Vr, with associated covariate vector X. For parameter and covariance matrix {X, f), suppose Y, is given by the GLIM model, defined in Eq. TT. Then there is unique y such that Yo yo:
7 = + (vo)
(1)))
(2)
3
where & is the CDF of the standard normal distribution. Further, the log probability density, function the path ,,, BT under the model is given by
① (vt) fu,)
(Φ ( (yx))
log [() - E,);
Σ log ā.
27?
2
yr log(yr-)) + (1 yr) log(1 1 yr vr-1),
where jt and ōg are computed according to the following four-step procedure:
1. For < < T, calculate E and a' according to the following expressions (and set
E E):
E'= E - - (Ei)-
a' = 1 IT -1
2. For 1st T, iteratively compute 24:
Ens Elann (vr) E1=1 a )2 -Y
24
1 1+a
3. For 1 St T. calculate μt (and set pd 0):
5

---

x
Z
NO 2)
Σ
1
T
Figure 2: The directed graphical model under consideration of GLIM. Solid lines denote the generative model. Dashed lines denote the identification of ス from realized values of Y; this induction also requires knowledge of Σ and :1, 1t but those arrows omitted in the diagram for simplicity. The parameters 0 of the covariance matrix E are learned from the data.
4. Finally, for 1 1st< T, calculate jt and oz
P + - (1 + a (0) 2 + (1 (0)
( + aQr))
Viid (1))
Dis E(a.s
Figure _ shows graphical representation of the model and inference process. proof of Theorem is given in Appendix of the supplementary material. The key observation is that given realized values of .i..... Y,, together with the parameters of the underlying data- gettraaeneatingg process, one can compute the implied values of the latent variables Z1, Zr. Using change of variables, one can then compute the probability density function itself.
Much of the notational complexity in Theorem stems from the fact that the latent variables can have non-trivial correlation structure. This flexibility allows our model better capture the patterns of real-world data. However, in some cases, it is sufficient to assume the latent variables are independent, which in turn considerably simplities our expression of the density.
Corollary 1. Suppose the latent variables {ZYT [1=1 independent, with diagonal covariance matrix E having diagonal entries of oz,... ... .7. Then the expressions for jar and ō, in Theorem have the following simplified form:
Lizt o?
((U-1)))
ō:
2i=r+1 o?
List
A proof of Corollary is given in Appendix A.2 of the supplementary material.
Given the expression for the likelihood derived in Theorem [ there multiple ways carry out model inference. For example, one could sweep over the parameter space to maximize the likelihood of the observed probability paths under the model. Here instead apply Bayesian approach, putting weakly informative prior on 0, and then approximating its posterior distribution via Hamiltonian Monte Carlo (HMC), implemented Stan [Carpenter et al. (017).
Without further constraints, is not fully identified by the data, since multiplying all of the latent variables in Eq. 4 by positive constant does not affect the sign of the relevant expression. Thus, in our applications below, we constrain the scale of the latent variables by requiring Var = of 1.
From fitted GLIM model, it is straightforward to draw probability path from the posterior distribution over paths. We do in three steps. First, we draw value of from the inferred posterior.

---

Next, we draw the vector of latent variables (,1....,2r) from the multivariate normal distribution (O E(X, 0)). Finally, compute values of ........8 yr according to Eq. (T). As shown in the supplementary material, the value of Eq. ① can be computed analytically, yielding:
7
+ 3; M()
Vt = +
(3)
E (1,))
for 1 where 7, p², and E are defined as Theorem only y -' 2 2220 0.
For = we have Yr if and
Experiments
We now explore the efficacy of GLIM through series of experiments on two real-world datasets. We have additionally included simulation study on synthetic dataset in Appendix Bo demonstrate GILM's empirical finite sample behavior. E(X,0) 0) with autoregressive structure and heteroskedastic variance. Specifically, we set
our experiments, we use a covariance matrix
E(e)) 0,0 o:, where the variance time is of ts(
In this setup, the covariance matrix E is parameterized by 9 = (p, The first parameter, P. controls the correlation between latent variables, with p = O corresponding to independence. The second parameter, 3, controls how the latent information evolves over time through the variance function GB- A simple example of the variance function is Gs(X,t) exp(BX 1)), where positive, zero, and negative values of B X correspond to increasing, constant, and decreasing variance over time.
As will demonstrate, this relatively simple structure using different variance functions often works well in practice. Figure 6] Appendix B show that even with only two scalar parameters under this specification, GLIM is able model wide variety of probability path structures. However, other parameterization are also possible. For instance, deep kernel learning [Wilson et al. 2016 could be applied to facilitate more complex covariance matrix structure.
4.1 Experiment setup
We now turn to two more complex real-world prediction problems with evolving probability paths to assess the GLIM's performance: (1) the minute-by-minn teams, updated as the game unfolds; and (2) forecasts of rainfall in Australian cities, updated daily as new information becomes available. In both prediction tasks, we observe the initial point prediction Yo given by pre-trained random forest predictor about the final outcome Yr. The goal here is to accurately estimate the posterior probability path distribution, approximated by sample of simulated posterior probability paths generated by the models to be evaluated. Figure Salin Appendix D verifies that for both basketball and weather datasets, the probability paths themselves are generally well calibrated.
win probability of professional basketball
We compare GLIM against three representative baselines from each of the three aforementioned classes of models in Section 2 (1) MMFE: martingale method of forecast evolution (Heath and Jackson 1994 Zhao al. 2013): (2) LR: set of linear regression models {m²) } that predict the future estimated probability at time / [Brockwell and Davis 2016); and (3) MQLSTM: Bayesian
multi-horizon quantile LSTM model [Wen et al. 2017 Eisenach al. 2020).
Assessing the quality of an estimated distribution is tricky: For each probability path in the dataset, we only observe one realized trajectory of the path, rendering metrics comparing two distributions statistic or cross-entrowe cross-entropy) inappropriate Luckily. by taking advantage
(e.g., Kolmogorov-Sminng of the fact that we are modeling probability paths, know there are a few key properties the true probability path distribution must satisfy and have accordingly devised three evaluation metrics to benchmark GL.M against the baseline models.
Although it is possible to treat the observed path as coarse approximation of ground truth distribution (i.e., one one-point distribution), doing will likely introduce unacceptable level of noise in evaluation.
7

---

MNFE
MOLSTM
100%
Probability
75%
25%
Minutes
Figure 3: A comparison of real and predicted probability paths for a single basketball game, for GLIM and for our three baseline models: MMFE, LR, and MQLSTM. The blue lines are the observed ground-truth paths, the grey lines show 100 samples drawn from the distribution of paths predicted by each model, and the red lines show the mean predictions over time.
The first metric is the posterior mean calibration mean squared error (mean calibration MSE). Since the probability paths are expected to be martingales, posterior means of simulated paths should be around the initial prediction (i.e., yo) for i 1,..., and we can empirically examine the generated paths have a sample mean of yo-
The second metric check is the expected volatility mean squared ervor (volatility MSE). As proven by Augenblick and Rabin (202T] and Taleb) (2018), for an evolving belief stream (i.e., a probability path), given initial probabilistie prediction yo: the accumulated squared increments, or volatility, defined as Qr E (Y; - 1 should be equal yo - 3/) expectation. i.e. [[Q2]= = Vo( - yo). In practice, E[Qr] can be estimated through averaging over the squared increment sum of collection of sample paths.
Finally, if the realized paths are from the estimated probability path distribution, then for any 0 < i T, an empirical credible interval at significance level is expected cover exactly proportion of observed paths. Specifically, we examine the credible interval coverage ervor (CI coverage error) significance level E {50%, 80%, 90%, 95%) } at the beginning, middle, and the end of probability paths for both the basketball and weather datasets.
In the experiments, for each model, all metrics are calculated using 100 simulated samples per probability path. All paths are simulated up until time T 1, the the final end points are generated by drawing from Bernouli(p = Yr-1) to ensure they converge to or at time T,
4.2 Basketball game outcome predictions
Suppose the second half of a basketball game is about start. Based on all the available information, the probability that the home team wins is 60%. How is the game likely to evolve? Do we expect game-deciding event to occur in the next few minutes? Can we safely run the bathroom and not miss the action? More specifically, do we expect the 60% prediction to remain stable until the final minutes of the game, or, alternatively, do we expect to quickly veer toward or 100%?
Here we analyze regular-season NBA basketball games for the 2017-2018 and 2018-2019 seasons, training our model on the first season, and evaluating our predictions on the second To do so, we first created minute- by-intte probability paths for every 48-minute game. For every minute, we trained separate random forest model predict the final game outcome based on information available at that point, including: the current home-team and away-team score, the maximum score margin up until that point in the game, and the win rates for the home and away teams for the previous season. On the resulting probability paths, we fit our GL.M model -as well as our three baseline methods-to model predictions for the second half of the game?) All four models have access to the same information available at half-time, including the score, each team's performance in the previous season, and summary statistics for the probability path in the first half of the game (e.g., the minimum, maximum, average, and standard deviation of predictions). We use a regularized linear function g3 (X,()) () for GLIM, which we describe detail in Appendix
'Our basketball data collected from .ba. with third-party API client (Patel 2018 To avoid the problem of varying horizon lengths because of overtime periods, for any particular game, if home team's score is greater or equal to the visiting team's, we mark the home team as the winner.
For modeling purposes, let t represent the 24th minute of the basketball game.

---

16-01
36:02
Expected Val, MSE
1w-00-
MMFE
LR
10:02
Calibration MSE
Callbration MSE
GUM
1e-03
MIFE
LR
MOLSTM
GLIM
1e-03
(a) Basketball
Expected Vol. MSE
3e-00
1e:00
24
02
40
Minutes
Days
MMFE
LR
MGLSTM
GLM
(a) Basketball
(b) Weather
(b) Weather
Figure 4: Mean calibration MSE, plotted log scale. The esti- mated calibration MSE for GLIM is not identically zero because estimate is based finite number of sample paths.
Figure 5: Volatility MSE, plotted on log scale.
In Figure 3
we show an illustrative example of a single game, modeled by all four methods. The identical blue curve in all four panels the ground-truth probability path of the game. Starting the 25th minute (i.e., at the start of the second half), each panel displays the distribution of probability paths predicted by each method, with the dashed red line indicating the mean prediction under each model over time. In this example, MQLSTM produces probability paths that have far too much movement showing significant oscillation between adjacent time steps. In contrast, LR produces paths that appear to appropriately fan out over time. Those paths, however, skew towards the positive outcome just like MQLSTM paths, failing satisfy the martingale property, as indicated by the red line. MMFE is the only baseline model that is able to maintain the posterior mean being centered around yo- Unfortunately.
paths are generally concentrated around Yo until the end and have assigned pretty small density around the ground-truth blue path. Finally, GLIM generates predicted paths that are both calibrated (as theoretically guaranteed) and that appear to appropriately capture the path volatility. One interpretation for GLIM's simulated paths is that given all information have seen at the 24th minute and the point prediction yo being around 30%, if the home team is going to lose the game, we can expect to have confident prediction at around the 36th minute; but if the home team going to win, we might not be so sure about until the very end of the game.
Min.
MMFE
LR
MQLSTM
GLIM
Day
MMFE
LR
MQLSTM
GLIM
50%
0.13
-0.32
-0,08
0.07
50%
0.06
-0.14
-0.01
0.01
80%
0.04
-0.46
-0.10
0.03
80%
0.03
-0.19
-0.02
-0.01
25
90%
0.00
-0.46
-0.10
0.01
90%
-0.19
-0.03
-0.03
95%
-0.02
-0.45
-0.10
-0.01
95%
0.00
-0.17
-0.03
-0.03
50%
0.03
-0.06
-0.09
0.07
50%
-0.09
-0.02
0.03
80%
-0.03
-0.08
-0.12
0.05
80%
0.00
-0.13
-0.03
0.01
90%
-0.05
-0.08
-0.12
0.02
90%
-0.01
-0.12
-0.04
0.00
95%
-0.07
-0.12
0.01
95%
-0.02
-0.11
-0.05
-0.02
50%
-0.01
0.02
-0.14
0.01
50%
0.15
0.15
80%
-0.06
0.01
-0.22
0.02
80%
0.11
-0.01
47
90%
-0.06
-0.01
-0.24
0.03
90%
0.05
0.07
0.06
-0.01
95%
-0.05
-0,02
-0.24
0.02
95%
0.02
0.03
0.03
-0.02
(a) Basketball o-level CI coverage
(b) Weather a-level CI coverage
Table 1: Credible interval coverage at significance level a at different time points. Positive values
indicate overly wide credible intervals, and negative values suggest overly narrow credible intervals. Standard errors are generally at or below 0.01 level and are omitted from this table. The lowest coverage errors (or within +0002 from the lowest absolute error) are in bold. GLIM's performance is consistently the best or indistinguishable from the best in nearly all (time, a:) combinations.

---

Moving beyond this single example, Figure 4a Figure Sa and Table la show the mean calibration MSE, expected volatility MSE, and credible ineeval coverage respectively. As displayed in the plot, GL.M outperforms all three baselines across the board. Particularly, Figure 4a are Figure [a are plotted in log scale, suggesting GLJM outperforming baselines by a few orders of magnitudes on those metrics.
4.3 Weather predictions
Finally, we consider the problem of modeling dynamic forecasts of precipitation. Whereas basketball games may ostensibly be well-modeled as a biased random walk, weather outcomes arguably have much more complicated temporal dynamics [Palmer, 1997]. Specifically, we use dataset of Australian rainfall observations [Williams] 201 Young and Young 2018], and construct daily predictions starting seven days in advance of the target date. Here, starting day zero, we aim to model the evolution of our predictions on whether it will rain on the target date. In this case, our predictions were based on a variety of meteorological features, including air pressure, wind speed, location, and month of year. We randomly sampled 10,000 target dates in the dataset prior to 2014 for training our models, and randomly sampled 10,000 target dates or after 2014 for testing. GLIM and our baseline models were then fit to the training paths, using the same features above. We set = 0 in this case and use a regularized quadratic function Ga(,, t) for GLIM, which we describe in detail in Appendix C.2
GLIM and three baseline models' mean calibration MSE, expected volatility MSE, and credible interval coverage for the weather dataset are displayed in Figure 4b Figure 5b and Table Tb respectively. Similar to what we observe in the basketball dataset, GLIM performs better than all three baseline models. In summary, the results from both the basketball game outcome predictions and weather predictions datasets suggest that GLIM is a desirable framework for modeling probability paths both in theory and in practice.
5 Conclusion
In this paper, we formally investigated the structure of probabilistic predictions that evolve over time. In doing so, introduced the Gaussian latent information martingale (GLIM), Bayesian framework to model these probability paths. contrast to previous work this area, GLIM can naturally incorporate covariates, is able to produce multi-step predictions without explicitly modeling changes in covariates over time, and, perhaps most importantly, preserves important properties of probability paths such the martingale structure and amount of movements. We investigated GLIM's performance on two real-world datasets in addition to a synthetic dataset (Appendix B). helping us understand the dynamic structure of predictions over time.
Aside from being an intriguing problem in and itself, understanding the structure of evolving probability paths can aid time-sensitive decision makers choose appropriate action [Ferguson 2004 Tsitsiklis and Van Roy 1999 Chow and Robbins, 1963]. Examples range from helping ordinary individuals plan weather-continger events, to physicians treating patients with rapidly changing symptoms [Wassenaar et al. 2015), to prosecutors making judgements based on updating information [Lin et al. 2019].
With the increasing availability of real-time data across domains, rapidly evolving forecasts are also likely to become more common. GLIM offers promising route for explicitly modeling- in turn, predicting- trajectory of forecasts. Looking forward, we hope our work sparks further interest in uncovering and leveraging the subtle structure of such dynamic predictions.
Acknowledgements
We thank Johann Gaebler, Daniel Jiang, Zhen Liu, Sabina Tomkins, and Yuchen Xie for helpful conversations throughout the project, and Nicolas Lambert and Jessica Su for in in-depth discussions on a preliminary version of this work. Code to replicate our experiments is available online at: https://gitthu... htttbb/......ttttaaaaarraaaaaai.
Funding support: The authors acknowledge that they received no funding in support of this research. Competing interests: The authors declare that they have competing interests.
10

---

References
Erinc Albey, Amirhosein Norouzi, Karl G Kempf, and Reha Uzsoy. Demand modeling with fore- cast evolution: application to production planning. IEEE Transactions Semiconductor Manufacturing, 28(3):374-388 384, 2015.
Ned Augenblick and Matthew Rabin. Belief movement, uncertainty reduction, and rational updating. Q. Econ., February 2021.
Anil K Bera, Matthew L Higgins, and Sangkyu Lec. Interaction between autocorrelation and conditional heteroscedassicity: A random-coefficien approach. Journal of Business Economic Statistics, 10(2):133-142, 1992.
Charles Blundell, Julien Corneiise, Koray Kavukcuoglu, and Daan Wierstra. Weight uncertainty in neural networks. arXiv preprint 1rX1v:1505.05424, 2015.
Tim Bollerslev. Generalized autoregressive conditional heteroskedasticity. Journal of econometrics, 313))307-327, 1986.
Youssef Boulaksil. Safety stock placement in supply chains with demand forecast updates. Operations Research Perspectives, 3:27-31, 2016.
Peter Brockwell and Richard Davis. Introduction to time series and forecasting. Springer, 2016.
Bob Carpenter, Andrew Gelman, Matthew D Hoffmann Daniel Lee, Ben Goodrich, Michael Be- tancourt, Marcus Brubaker, Jiqiang Guo, Peter Li, and Allen Riddell. Stan: A probabilistic programming language. Journal of statistical software, 76(1), 2017.
Alexandra Chouldechova. Fair prediction with disparate impact: A study of bias in recidivism prediction instruments. Big data, 5(2):153-163, 2017.
Yuan Shih Chow and Herbert Robbins. On optimal stopping rules. Zeitschrift für Wahrscheinlichkeit. stheorie und verwandte Gebiete, 2(1):33 49, 1963.
Carson Fisenach, Yagna Patel, and Dhruv Madeka. Mqtransformer: Multi-horizon forecasts with context dependent and feedback-aware attention. arXiv preprint arXiv:2009. 147999 2020.
Jeffrey Ely, Alexander Frankel, and Emir Kamenica. Suspense and surprise. Polit. Econ., 123(1): 215-260, 2015.
Robert Engle. Autoregressive conditional heteroscedasticity with estimates of the variance of United Kingdom inflation. Econometrica: Journal of the Econometric Society, pages 987-1007, 1982.
Robert F Engle and Andrew Patton. What good is a volatility model? In Forecasting volatility in the financial markets, pages 47-63. Elsevier, 2007.
Robert S Erikson and Christopher Wlezien. Markets vs. polls as election predictors: An historical assessment. Electoral Studies, 31 31():5332-539 2012.
João Trevizoli Esteves, Glauco de Souza Rolim, and Antonio Sergio Ferraudo. Rainfall prediction methodology with binary multilayer perceptron neural networks. Climate Dynamics, 52(3-4): 2319-2331, 2019.
Ferguson. Optimal stopping and applications. mathematics department ucla, 2004.
Dean P Foster and Robert A Stine. Threshold martingales and the evolution of forecasts. arXiv preprint arXiv:2 105.06834, 2021.
Christian Francq and Le Quyen Thieu. Qml inference for volatility models with covariates. Econo- metric Theory, 35(1)337-72, 2019. doi: 10.1017/502644666170000
Yarin Gal and Zoubin Ghahramani. Bayesian convolutional neural networks with bernoulli approxi- 2015.
mate variational inference. arXiv preprint

---

Andrew Gelman, Jessica Hullman, Christopher Wlezien, and George Elliott Morris. Information, incentives, and goals in election forecasts. Judgment and Decision Making, 15(5):863, 2020.
Ian Goodfellow, Yoshua Bengio, Aaron Courville, and Yoshua Bengio. Deep learning, volume 1. MIT press Cambridge, 2016.
Stephen Graves, Harlan Meal, Sriram Dasu, and Yuping Qui. Two-stage production planning in dynamic environment. In stage production planning and inventory control, pages 9-43. Springer, 1986.
David C Heath and Peter Jackson. Modeling the evolution of demand forecasts ith application to safety stock analysis in production/ddittriution systems. IIE transactions, 26(3):17-30, 1994.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8): 1785, 1997.
Durk P Kingma, Tim Salimans, and Max Welling. Variational dropout and the local reparametcrizzation trick. In Advances in neural information processing systems, pages 2575-2583, 2015.
Zhiyuan Lin, Alex Chohlas-Wore and Sharad Goel. Guiding prosecutorial decisions with an interpretable statistical model. Proceedings of the 2019 AAAI/ACM Conference on Al, Ethics, and Society, pages 469-476, 2019.
Zhiyuan Jenry Lin, Jongbin Jung, Sharad Goel, and Jennifer Skeem. The limits of human predictions of recidivism. Science advances, 6(7):eaa20652, 2020.
Klas Malmberg, Lars Rydén, Anders Hamsten, Johan Herlitz, Anders Waldenström, and Hans Wedel. Mortality prediction diabetic patients with myocardial infarction: experiences from the digami study. Cardiovascular research, 34(1):248-22) 253, 1997.
J Martinez-Alario, ID Tuesta, E Plasencia, M Santana, and ML Mora. Mortality prediction in cardiac surgery patients: comparative performance of parsonnet and general severity systems. Circulation, 99(18):2378-233) 1999.
Jeffrey W Miller. A detailed treatment of Doob's theorem. arXiv preprint arXiv: 1801:03122.22 2018.
Tim Palmer. A weather eye on unpredictability. Exploring Chaos, A Guide to the New Science of Disorder, pages 69-81, 1991.
Swar Patel. An api client package access the apis for nba.com. https://github. cow.suuu..cmm/ara.... nba_api 2018. Accessed: 2020-10-1.
Syama Sundar Rangapuram, Matthias W Seeger, Jan Gasthaus, Lorenzo Stella, Yuyang Wang, and Tim Januschowski. Deep state space models for time series forecasting. Advances neural information processing systems, 31:7785-7794, 2018.
David Salinas, Valentin Flunkert, Jan Gasthaus, and Tim Januschowski. Deepar: Probabilistic forecasting with autoregressive recurrent networks. International Journal of Forecasting, 2019.
Houshmand Shirani-Mehr, David Rothschild, Sharad Goel, and Andrew Gelman. Disentangling bias and variance election polls. Journal of the American Statistical Asxociation, 1136522)6007-614 2018.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, llya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, 15(1):1929-199 1958, 2014.
Genaro Sucarrat. garchx: Flexible and robust garch-x modelling, 2020.
Nassim Nicholas Taleh. Election predictions as martingales: an arbitrage approach. Quant. Finance, 18(1):1-5, January 2018.
Nassim Nicholas Taleb, Dhruv Madeka, et al. All roads lead to quantitative finance. Quantitative Finance, 19(11):175-11776 2019.
12

---

Daniel Teres, Stanley Lemeshow, JILL SPITZ Avrunin, and HARRIS Pastides. Validation of the mortality prediction model for patients. Critical care medicine, 15(3):208-213 1987.
John N Tsitsiklis and Benjamin Van Roy. Optimal stopping Markov processes: Hilbert space theory, approximation algorithms, and application to pricing high-dimensional financial derivatives. IEEE Transactions on Automatic Control, 44410):1840-1851 1999.
Wassenaar, MHWA van den Boogaard, Theo van Achterberg, AJC Slooter, MA Kuiper, ME Hoogendoom, KS Simons, E Maseda, N Pinto, C Jones, al. Multinational development and validation of an early prediction model for delirium in icu patients. Intensive care medicine, 41(6): 1048-1056, 2015.
Ruofeng Wen, Kari Torkkola, Balakrishnan Narayanaswamy, and Dhruv Madeka. multi-horizon quantile recurrent forecaster. arXiv preprint arXiv:1711.11055 2017.
Graham Williams. Data mining with Rattle and R: The art of excavating data for knowledge discovery. Springer Science Business Media, 2011.
Andrew G Wilson and Zoubin Ghahramani. Copula processes. In Advances in Neural Information Processing Systems, pages 2460-2468, 2010.
Andrew Gordon Wilson, Zhiting Hu, Ruslan Salakhutdinov, and Eric P Xing. Deep kernel learning. In Artificial intelligence and statistics, pages 370-378, 2016.
Bin Xu, Feilin Zhu, Ping-an Zhong, Juan Chen, Weifeng Liu, Yufei Ma, Le Guo, and Xiaoliang Deng. Identifying long-term effects of using hydropower complement wind power uncertainty through stochastic programming. Applied Energy, 253:1 13535, 2019.
Li Yan, Hai-Tao Zhang, Jorge Goncalves, Yang Xiao, Maolin Wang. Yuqi Guo, Chuan Sun, Xiuchuan Tang, Liang Jing, Mingyang Zhang, et al. An interpretable mortality prediction model for covid-19 patients. Nature Machine Intelligence, pages 1-6, 2020.
Joe Young and Adam Young. Kaggle: Rain in australia, 2018. https://www.kett kaggle. com/ Saaal///aaaaa--dtttttt----a--e-
Tongtiegang Zhao, Ximing Cai, and Dawen Yang. Effect of streamflow forecast uncertainty on real-time reservoir operation. Advances in water resources, 34(4):495-504, 2011.
Tongtiegang Zhao, Jianshi Zhao, Dawen Yang, and Hao Wang. Generalized martingale model of the uncertainty evolution of streamflow forecasts. Advances in water resources, 57:41-51, 2013.
13


Reviews:
[{"review_id": "vHzhJLvmtj", "reviewer": "Reviewer_j4Fr", "summary": "The main contributions of this paper are proposing a Bayesian model that satisfies the martingale property point forecasts should have:\n\n- This paper provides a nice cool idea to make Bayesian models satisfy the martingale property. Eqn (1) is a cool idea -\n- Its a pretty nice comparison across datasets - its nice to actually track this\n- The metrics actually track something useful (like Talebs/AugRab's martingale trade). This converges to the volatility of a diffusion (as shown by Taleb later - https://d1wqtxts1xzle7.cloudfront.net/60195149/clayton3-with-cover-page-v2.pdf?Expires=1626415145&Signature=d~fkCG~WwZ14pZopiG5MwXC~1Dbt4y9C2w-X1GQeKvFJhwpPoiKg0wV9DIN5XrdWEVbY8QGTLTzxcFRVJ3URwCSFeMQjCrapJ3BKf5m7fx7g0Q4X2FpLhfFfqTA6rse3oW8AOADxHDm7yK0-9bE6l3cyDZ2xjI43FMEcoZzWqXoBGN6RS1KnqmbVwXf4SiT6~zItizMdMGo0wYOUIKtei6-zSZ8cM8BP0eOtTAaqy~pnRo11yqXR0vjQWN0uZERfyFmHnBGIXo4vGnihTPbOs5SOnyZabMlbiqcTT8rc7sBV4N3GnwYYF5zeX4w8KW9FELrcPTvqFKVzOm5axR2aTA__&Key-Pair-Id=APKAJLOHF5GGSLRBV4ZA)\n\n", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Though the paper tracks really interesting metrics. I think it still misses the point of what the martingale property really means. In Foster and Stine, the core idea seems to be that the martingale filter allows you to have a *more* accurate forecast. That means any other metric but accuracy (so straight MSE) doesn't really matter - as the martingale filter illustrates. In that context, Im not sure what this paper adds. That seems to be the crux of Eisenach et al too - the decoder self allows the MQNet to be more accurate.\n\nA few clarifications - \n\n1) Do you assume your forecasts are already calibrated? If so, do you ensure this for all the models (specially MMFE before you compare)\n2) Is MQLSTM a Bayesian model?\n3) MQLSTM seems the wrong baseline - MQ_CNN_Wave within the paper is more accurate (seems to just be a wavenet encoder instead of an LSTM encoder) but even then Eisenach et al (MQTransformer) actually tries to make the MQLSTM a martingale through the decoder self attention", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "1) I think overall the model fails to address the primary ideas of the martingale property elucidated by Foster and Stine and Taleb et al. The martingale property is fundamentally about a more accurate forecast\n2) To that end comparing against fair baselines (MQTransformer) and other models to which the Foster and Stine Martingale Filter have been applied are more relevant\n3) This claim about Foster and Stine (\"but they usually focus on developing diagnostic metrics instead of building predictive frameworks that capture probability path heterogeneity like GLIM\") I think is wrong. Foster and Stine seem to formalize the connection to why we care about the martingale property at all (it like calibration guarantees a >= in accuracy forecast). So I do think that might be the connection the authors are missing as the core idea of the paper - that MSE is everything\n\nMore deeply, I am not sure any decision making algorithm would actually care about the diffusion coefficient of an SDE", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "_Atigs4J11L", "reviewer": "Reviewer_oVRK", "summary": "The paper presents a latent Gaussian model for the progression of probability forecasts. Under this model, forecasts are a martingale and the paper describes a sampling strategy to predict the trajectory of the remaining (T - t) forecasts based on the first t observed forecasts. The paper applies the method on real world data and the results are promising.\n\n-----\nUpdate after author feedback period:\nI appreciate the authors' responses to other referee's comments. After going through the responses and subsequent discussion, I unfortunately must lower my score. While the model is interesting, I do not believe the relevant empirical comparisons have been done. Namely, the methods of Foster & Stine can serve as a way to \"post-process\" a probability path to improve the forecast accuracy (measured, e.g., by MSE). I think a comparison to post-processed procedures is vitally necessary here and, in particular, using MSE. At present it is not clear to me that the proposed model will outperform these straw-men. ", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: The latent Gaussian model seems clever (though I have some clarifying questions, see below) and the idea of modeling trajectories of forecasts is a nice one.\n\nQuality: The model and inferential approach seems sound. However, the specific form of the model seems a bit ad hoc. Can you provide more motivation / intuition for the additive term $\\sum_{t = 1}^{T}{Z_{t}}$ in Equation (1)? One might reasonably consider an alternative model of the form $Y_{t} = P(\\gamma + Z_{t})$ where the $Z_{t}$'s are dependent (e.g. the covariance matrix of $Z$ has a first-order autoregressive structure). As presented the model seems a bit ad hoc / under-motivated and I think the paper would be considerably strengthened by arguing why some \"straw man\" alternatives are inadequate (e.g. do not satisfy the martingale property or have some other disqualifying characteristic).\n\nSemi-relatedly, how sensitive are your results to the choice of variance function? The elicitation in Appendix D for the basketball dataset is a good start but it would be useful to, at a minimum, comment in the main text on the sensitivity to the choice of variance function and give interested practitioners some guidance for choosing this function.\n\nClarity: On the whole, the paper is very well-written and easy to follow.\n\nSignificance: I think the basic model presented here is quite interested and will stimulate new research interests. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Not addressed", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "XeMFwsEfn6L", "reviewer": "Reviewer_DdpC", "summary": "A new model for the evolution of predictions with strong empirical results.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper considers the problem of estimating the probability of a future binary outcome. The authors look at the question of how a prediction evolves over time as new information is made available. They develop a Bayesian time series model to look at the structure of probabilistic forecasts over time. A salient feature of such forecasts is a martingale property, viz., the expected value of future predictions along the forecast path is equal to the expected value of the current prediction. This effectively constrains the forecasts and puts the focus on the study of conditional distributions of paths given the information at the forecast time. The authors perform a good review of work in modelling time series and point out the special structure in probability paths that needs to be captured and is not performed by some existing model classes.\n\nThe paper then moves on to presenting the main model, the GLIM. This is a kind of Gaussian latent variable model, which models the evolution of the probability of the binary outcome by adding `positive' or `negative' information incrementally over time. By conditioning on information previously added, the certainty of the prediction increases as the event of interest is approached. A simple and clear inference process is described for the model allowing computing the likelihood by first applying a recursion to update the means and variances of the Gaussians in the latent space and then computing an overall likelihood of the probability path. Since this is a novel model, I would have appreciated some more intuition on the interpretations of these recursions, ie,. giving a bit more meaning to the parameters that are being computed. \n\nFinally, two well-chosen data sets (basketball and weather) are used to test the model. While the description of the model and inference process is very clear, the experiments section is harder to follow. In part, I think this is due to the novelty of the model. You should clearly outline the experiment you are doing and use the appropriate notation. Thus, for example, how exactly do you fit your model to the basketball data?\n\nI understood that you take a set of basketball games, say 1 , ... , N. You compute a probability path for each game, {y_1^1 , ... , y_T^1} , ... , {y_1^N , ... , y_T^N}. Then you take these paths and plug them into the procedure described by Theorem 1. This gives you an estimate of the covariance parameters for the structure you described in lines 227-228. You allude to a prior on these parameters that is `weakly informative' but do not specify it anywhere. You then test the model on a single game given (1) a prediction at the 24th minute, presumably from the same sort of random forest model, and (2) the parameters inferred on the entire set of data available. Is this single game part of a test set? Finally, a general comment is why do you do your inference by drawing a single value from the posterior, wouldn't it make more sense to use the posterior mean or even the maximum likelihood estimate for that matter? Can you say something about the posterior of theta?\n\nOverall, I think this paper is certainly an accept, the model is novel, promising, and supported by good empirical results.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"ethical_concerns": "None.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "ASGJUi4E3mA", "reviewer": "Reviewer_tNwH", "summary": "The paper titled \"Probability Paths and the Structure of Predictions over Time\" proposes and analyses a probabilistic model called Gaussian Latent Information Martingale (GLIM) to model probability paths, a finite sequence $\\\\{Y_t\\\\}_{t \\in \\\\{0,1,\\ldots,T\\\\}}$ of probabilities for an event happening at time $T$ with $Y_t$ being the best guess of the probability of the event at time $t$ conditional on information available till that time. After proposing the generative model which satisfies certain desirable properties like martingale property and $Y_T$ being either $0$ or $1$, the authors discuss an algorithm to compute the likelihood and thereby allowing inference. This is followed by experiments which range from doing inference on a synthetic data set for sanity check to comparing GLIM against three other models on basketball game outcome and rain prediction data.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This is a well-written paper discussing a very interesting problem complete with motivation, technical analysis and experiments. The problem of modeling probability paths seems so obviously important in retrospect, and yet, I am embarrassed to admit, I had never thought about it or seen it discussed before. The latent variable model, GLIM, is a very simple and elegant yet powerful model that has been proposed by the authors to solve this very problem. This model can be easily extended to a more complicated one by adding various bells and whistles, as also acknowledged by the authors.\n\nOne shortcoming is that the paper fails to discuss why those specific $\\mathcal{G}_\\beta(X,t)$ functions were used for the experiments. They seem very ad-hoc and could potentially decide whether the model works well or not.\n\nEDIT: After extensive reviewer discussions I am decreasing my rating of the paper from a clear accept to a weak reject. My earlier positive review was based on incomplete understanding of the subject of forecasting. The main point because of which I cannot accept this paper is that martingality in and of itself is not the goal of a forecasting model. The three metrics used in the paper fail to be relevant for forecast accuracy, something like Brier score would have been better. The model proposed, though elegant, may be sacrificing accuracy and without proper metrics we wouldn't know. In light of this it isn't clear what the goal of the paper is anymore. This fatal flaw along with the fact that the authors failed to engage in discussions expect with one reviewer (j4Fr) brings the paper into the territory of reject.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Not relevant.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6 hours"}, "extra_text": {}}]

Author rebuttal:

Meta-review: After extensive discussions that led to reviewers rethinking their opinions significantly, there was a consensus that this paper, though well-written and proposing an interesting and novel model, is not quite ready for publication. The core concern is that martingality in and of itself is not really an end goal -- and to the extent that it is, the Foster & Stine approach can be used to post-process martingality onto a model. Thus, to really demonstrate the value of the proposed approach, it seems necessary to at least understand its accuracy (say, MSE) in comparison to that simple baseline. Please see the updated reviews for additional details.
Decision: Accept (Poster)