Probability Paths and the Structure of Predictions over Time
Authors: Zhiyuan Jerry Lin
Hao Sheng
Sharad Goel
Venue: NeurIPS 2021 Poster (2021)
Keywords: Probability Path
Bayesian Time Series Modeling
Evolving Prediction
Martingale

Abstract: In settings ranging from weather forecasts to political prognostications to financial projections, probability estimates of future binary outcomes often evolve over time. For example, the estimated likelihood of rain on a specific day changes by the hour as new information becomes available. Given a collection of such probability paths, we introduce a Bayesian framework -- which we call the Gaussian latent information martingale, or GLIM -- for modeling the structure of dynamic predictions over time. Suppose, for example, that the likelihood of rain in a week is 50%, and consider two hypothetical scenarios. In the first, one expects the forecast to be equally likely to become either 25% or 75% tomorrow; in the second, one expects the forecast to stay constant for the next several days. A time-sensitive decision-maker might select a course of action immediately in the latter scenario, but may postpone their decision in the former, knowing that new information is imminent. We model these trajectories by assuming predictions update according to a latent process of information flow, which is inferred from historical data. In contrast to general methods for time series analysis, this approach preserves important properties of probability paths such as the martingale structure and appropriate amount of volatility and better quantifies future uncertainties around probability paths. We show that GLIM outperforms three popular baseline methods, producing better estimated posterior probability path distributions measured by three different metrics. By elucidating the dynamic structure of predictions over time, we hope to help individuals make more informed choices.

Probability Paths
and the Structure of Predictions over Time
Zhiyuan Jerry Lin"
Hao Sheng
Sharad Goel Harvard University sgoel0hks harvard. edu
Facebook zylin@fb. com
Stanford University
haoshengOcs. stanford.
Abstract
In settings ranging from weather forecasts political prognostications to financial projections, probability estimates of future binary outcomes often evolve over time. For example, the estimated likelihood of rain on specific day changes by the hour as new information becomes available. Given collection of such probability paths, we introduce Bayesian whaneworrkwhich we call the Gaussian latent information martingale, or GLIM-for modeling the structure of dynamic predictions over time. Suppose, example, that the likelihood of rain in a week is 50%, and consider two hypothetical scenarios. In the first, one expects the forecast be equally likely become either 25% or 75% tomorrow; in the second, one expects the forecast to stay constant for the next several days. A time-sensitive decision-maker might select a course of action immediately in latter scenario, but may postpone their decision in the former, knowing that new information is imminent. We model these trajectories by assuming predictions update according latent process of information flow, which is inferred from historical data. In contrast to general methods for time series analysis, this approach preserves important properties of probability paths such the martingale structure and appropriate amount of volatility and better quantifies future uncertainties around probability paths. We show that GLIM outperforms three popular baseline methods, producing better estimated posterior probability path distributions measured by three different metrics. By elucidating the dynamic structure of predictions over time, we hope to help individuals make more informed choices.
Introduction
Probabilistic predictions of future binary outcomes are ubiquitous in real-world statistical and machine
learning problems, ranging from election modeling [Erikson and Wlezien 2012 Shirani- et al.)
2018 Gelman et al.] 2020 to weather forecasting |Esteves et al.)
2019 to assessing mortality
risk Teres et al.) 1987 Malmberg et al. 1997 Mlrtinez-Allrio al. 1999 Yan al. 2020 risk
assessment tools in criminal justice applications [Chouldechova] 2017 Lin et al. 2020 Offen such probabilistic predictions are not static, rather evolve over time as more information becomes available. For instance, the estimated likelihood candidate wins election updates with every new poll that is conducted, and the estimated chance rain on a given future day changes by the hour as new metcorological data become available.
In many of these domains, a large body of work aims to provide accurate, real-time forecasts that account for the very latest information. However, significantly less attention has been paid to understanding how the predictions themselves evolve over time. If there's 's 35% chance of rain one week from now, what can say about tomorrow's prediction of rain on that same day? In expectation, tomorrow's 's prediction must be the same as today's- were not, we would be better
"Research conducted when the author Stanford University.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Summer
Winter
100%
Probability
75%
50%
25%
0%
0
3
4
5
2
3
6
7
Days
Figure I: Dynamic forecasts of rainfall across several Australian cities during the summer and winter, starting one week before the target date. All forecasts initially indicate approximately 35% chance of rain. But the forecast trajectories- which we call probability paths-have higher volatility the summer than in the winter.
off updating today's prediction to match our expectation tomorrow's forecast. In other words, such
sequence of probabilistic predictions should be martingale [Augenblick and Rabin 2021 Foster and Stine 2021, Gelman al.) 2020 Taleb et al., [2019 Taleb] 2018 Ely et al. 20155. But aside from satisfying this martingale property, the forecast trajectories vary widely from setting the next.
Consider, for example, the collection of dynamic weather forecasts presented in Figure The plot shows the evolution of 7-day precipitation predictions for a of Australian cities [Williams] 2011 Young and Young 2018), where we have disaggregated the predictions by season (summer or winter) and restricted those instances for which the 7-day forecast starts at approximately 35%.
visually apparent, the mean prediction on any later day is still approximately 35%, expected. But the summer forecasts are considerable more volatile than those the winter. the winter months (June-August), the prediction is unlikely to change substantially until immediately before the target date. In contrast, the summer months (December-Februage the prediction is likely to oscillate considerably-eittee up down-with each passing day.
These patterns have immediate consequences for seneisensi decision makers [Ferguson] 2004 Tsitsiklis and Van Roy 1999 Chow and Robbins 1963]. As a simple example, suppose one is planning weekend picnic, with preference both sunshine and for sending out invitations as soon possible. If one knows that a weather forecast is unlikely to change in the immediate future, one might opt to simply send out the invitations and hope for the best. Alternatively, knowing that more information is imminent- as reflected by forecasts that are likely to soon change one might choose wait another day before deciding to hold the party.
In this paper, we develop a Bayesian framework to investigate- and predict-the structure of dynamic probabilistic forecasts: the Gaussian latent information martingale, GLIM. In contrast existing techniques for time-series modeling, our approach is tailored to the specific properties of evolving probabilistic forecasts, which we call probability path- Most importantly, our approach satisfies the martingale property described above, with the expected value of future predictions along the path equal to the current prediction. Our model is trained directly on historical paths, and learns the heterogeneous, covariate-dependen structure of forecasts. As result, it is rich enough to predict the type of structure illustrated Figure We show that this approach often outperforms existing methods for quantifying uncertainty for evolving time series forecast, ostensibly because past general-purpose methods do not leverage the idiosyncratic properties of probability paths. As real-time forecasting becomes increasingly popular, we hope this work helps researchers and decision makers investigate and act on these dynamic predictions.
Also known stream of beliefs, streamflow forecast, evolution of forecasts under different contexts.

---

2
Background
In this section, we formally define the problem we aim to address and introduce related literature pertaining to this topic.
2.1 Problem Setup
Suppose we are interested in a binary outcome YT E {0, 1} that will either occur or not at some fixed future time T and the current time is f 0. For example, may be interested in whether it will rain T= 7 days from today. At time O, we observe an initial probabilistic point estimate Yo- = yo about the final outcome (e.g.. from static weather forecast model), as well collection of covariates X (e.g., current and recent metcorological measurements). Our goal, then, is to accurately estimate how the predictions [=] of the event in question will evolve over time, described by posterior distribution of {Y) given all the information we have observed at 0.
While this problem might seem to be straightforward time series modeling application at first
sight, as shown by statisticians and economists AAgenblick and Rabin 2021 Foster and Stine 2021
Gelman et al. 2020 Taleb al. 2019 Taleb 2018 Ely al., 2015], J. such evolving probabilistic predictions need to have an expectation equal to the initial prediction (i.e., being a martingale
starting from yo) and shall not display excessive or insufficient movement (i.e., appropriate amount
of volatility as we will explain later in Section 4. putting additional constraints on the shape of
predicted probability path distributions. Indeed, without any additional information, the only sensible posterior distributions of future predictions (Y]== are ones with the mean equal to yo [Gelman
al. 2020
Taleb 2018 the current observed probabilistic point estimation.
is worth noting that in this problem, unlike in many common time series modeling tasks, we are not interested in producing a sequence of point estimates for the future probability path {Y) as model that always outputs the current best probabilistic prediction yo for all future (Y) is arguably the best point estimates we could possibly give without observing additional information. Instead, here we wish to provide an accurate estimate of the entire posterior probability path distribution conditioning on having observed all information available
O. As we will explain in detail in Section 4 the quality of the estimated distribution can be assessed via examining some of the properties probability path distribution should possess such as its posterior mean calibration, expected volatility, and credible interval coverage at different significance levels.
2.2 Related Work
In this subsection, we briefly review three classes of methods relevant to the problem of estimating the posterior distribution of probability paths (Y)T
Martingale model of forecast evolution. The model most relevant to our setting is probably the martingale model of forecast evolution, or MMFE, originally developed by Graves et al. [1986] and Heath and Jackson [1994) estimate the evolution of production forecasts in supply chain optimization. In order to estimate how the forecasts of the production of a commodity at a future time T evolve for = 0 to T, learning from historical data, MMFE directly estimates the covariance matrix of increments between consecutive time points. With the learned covariance matrix, at t = 0 we are able to simulate arbitrary number of possible scenarios (hence the posterior distribution of forecast evolution) of how the production predictions for ( = T will be at time / = o,.... T. Although MMFE was originally invented few decades ago, there are many recent successful applications and improvements of MMFE in wide range of fields such as energy management [Xu et al. 2019).
reservoir operation [Zhao et al. 2013 [011]]. stock replacement [Boulaksil] 20166), and production planning (Aley et al.) 2015]. Finally, recent work such as Augenblick and Rabin [202] and Foster and Stine [021] are studying martingale structures of probability paths, but they usually focus on developing diagnostic metries for existing predictions instead of constructing predictive frameworks that capture probability path heterogeneity like GLIM.
Probabilistic time series models. When it comes to modeling time series data, probability path or not, arguably the most widely known class of models probabilistic time series models. Famous examples include the autoregressive (AR), autoregressive moving average (ARMA), and autoregres- sive integrated moving average (ARIMA) models [Brockwell and Davis 2016). The autoregressive
structure and the estimated Gaussian noise term allow us to simulate future time series evolution
3

---

through recursive sampling. If we additionally model the noise terms themselves as a function of noise terms from previous time steps, we have a stochastic volatility model. For example, when we assume the variance of the error terms or innovation-follon
AR structure, we have a ARCH [Engle model. Similarly, assuming ARMA structure of error terms give the GARCH (Bollersiev
1982
1986 model. More recent development on stochastic volatility models include GARCHX [Sucarrat
2020 Francq and Thieu 2019 Engle and Patton 2007]. Augmented ARCH [Bera et al. 1992]. and
the Gaussian Copula Process Volatility (GCPV) model [Wilson and Ghahramani 2010). which all
allow for more complex structure of estimated variance over time and some of them grant the use of covariate information. Despite their vast success in numerous applications, most of these classical probabilistic time series models are primarily designed to model a single time series, rather than to model the multiple, independent time series in our motivating examples. Further, while some of these approaches may still be applied to model probability paths with some modifications, they are not designed to fully leverage the distinctive characteristics of probability paths, such as their martingale property, volatility level, or their finite time horizon (meaning the predictions must converge to O or 1 at fixed time T).
Bayesian deep sequence models. Rapid development of deep learning models over the past decade has offered new avenue of modeling time series data. Specifically, modern deep-learning based recurrent neural networks such as LSTM [Hochreiter and Schmidhuber 1997, Goodfellow et al. 2016) and DeepAR (Salinas al. 2019] can not only model and predict time series data, but also naturally incorporate the information of covariates into their predictions. There is a stream of recent work (Wen et al.) 2017 Eisenach et al. 2020) Rangapuram et al. 2018) particularly focusing providing better calibrated future time series predictions. Moreover, recent advancement in Bayesian deep learning has made uncertainty estimation possible in addition point predictions through random regularization [Srivastava et al.) 2014 Gal and Ghahramani] 2015 Kingma et al.] 2013) or
weights distribution [Blundell et al. 2015). J. In spite of their significant model capacity, typical deep sequence models suffer from the same problem as classical probablistic time series models: they not constrained by the martingale property in this bounded probability path setting. As we will show in Section 4 deep models tend to become overly conservative when it comes to uncertainty estimation and do not conform with the martingale requirement.
In summary, all of these techniques are powerful methods for modeling general time-series data, but none are tailored the idiosyneratic properties of probability paths. In Section 4 we will evaluate GLIM against representative baseline models from each of the three classes of models and illustrate how GLIM produce posterior future probability path samples with more accurate uncertainty estimation and better mean calibration.
3 Modeling Probability Paths
3.1 Gaussian latent information martingale
Now, we formally introduce the Gaussian latent information martingale, or GLIM, as a framework to model and predict probability paths. To model a probability path (Y)To we introduce corre- sponding sequence of latent variables { where the scalar Z, intuitively represents the amount of information received between time and t. Receiving positive information (i.e., positive Z,) makes the ultimate outcome Yr more likely, while receiving negative information makes the ultimate outcome less likely.
Now, we assume the latent variables {2)T- follow a multivariate Gaussian distribution with mean zero and T T covariance matrix Σ- Σ(X, f). where the covariance matrix can depend on both the observed covariates and a parameter 0. Finally, we model (Y)To by assuming predictions along the probability path can be expressed in terms of the following conditional probabilities:
Y, = Pr 7
Zi
0
Z1
Z
where y is a fixed constant.
Eq. (T) is closely related to the latent variable formulation of logistic regression models, but we additionally condition on the latent variables. This expression has several attractive properties for modeling probability paths. First, Y, is naturally constrained to lie between 0 and 1. Second, Yr is

---

guaranteed to be either 0 or 1, since once we condition on Zi..... Zr, there is no randomness left in the expression. Finally, and most importantly, Y satisfies the martingale property, as shown below. Proposition 1. For (Y) satisfying Eq.
① we have B| Yo. Yol = Y.
SK<T.
The proof is given by Appendix A.3 in the supplement material. Given collection of probability paths, our goal is to infer the parameter 6, as we describe next.
3.2 Model inference
We take Bayesian approach to fit GLIM defined above. The main difficulty is deriving tractable expression for the corresponding likelihood function. so, we first introduce some notation.
Given any 0 < t < T, we divide the covariance matrix Σ into four quadrants as follows:
T
}
Σ
Σ1
22
}
T-t
For t O, we further define Σg E. For arbitrary matrix M, use the notation M to denote the (i,j) entry in the matrix. Similarly, for an arbitrary vector we use the notation V(t) to denote the i-th entry in the vector.
Now, using this notation, Theorem I presents recursive formula for computing the (log) probability density function for our Gaussian latent information martingale with parameter 0. That expression can in turn be used to infer the parameters of the model from the observed data via maximum likelihood estimation or. alternatively. fully Bayesian inference.
Theorem 1. Consider probability path yuu.... Vr, with associated covariate vector X. For parameter and covariance matrix {X, f), suppose Y, is given by the GLIM model, defined in Eq. TT. Then there is unique y such that Yo yo:
7 = + (vo)
(1)))
(2)
3
where & is the CDF of the standard normal distribution. Further, the log probability density, function the path ,,, BT under the model is given by
① (vt) fu,)
(Φ ( (yx))
log [() - E,);
Σ log ā.
27?
2
yr log(yr-)) + (1 yr) log(1 1 yr vr-1),
where jt and ō; are computed according to the following four-step procedure:
1. For < < T, calculate E and a' according to the following expressions (and set
E E):
E'= E - - (Ei)
a' = 1=11 IT -1
2. For 1st< T, iteratively compute 24:
Ens Elann (vr) E1=1 a )2 -Y
24
1 1+a
3. For 1 St T. calculate μt (and set pd 0):
5

---

x
Z
NO 2)
Σ
1
T
Figure 2: The directed graphical model under consideration of GLIM. Solid lines denote the generative model. Dashed lines denote the identification of ス from realized values of Y; this induction also requires knowledge of Σ and :1, 1t but those arrows omitted in the diagram for simplicity. The parameters 0 of the covariance matrix E are learned from the data.
4. Finally, for 1 1st< T, calculate jt and oz
P + - (1 + a (0) 2 + (1 (0)
( + aQr))
Viid (1))
Dis E(a.s
Figure _ shows graphical representation of the model and inference process. proof of Theorem is given in Appendix of the supplementary material. The key observation is that given realized values of .i..... Y,, together with the parameters of the underlying data- gettraaeneatingg process, one can compute the implied values of the latent variables Z1, Zr. Using change of variables, one can then compute the probability density function itself.
Much of the notational complexity in Theorem stems from the fact that the latent variables can have non-trivial correlation structure. This flexibility allows our model better capture the patterns of real-world data. However, in some cases, it is sufficient to assume the latent variables are independent, which in turn considerably simplities our expression of the density.
Corollary 1. Suppose the latent variables {ZYT [1=1 independent, with diagonal covariance matrix E having diagonal entries of oz,... ... .7. Then the expressions for jar and ō, in Theorem have the following simplified form:
Lizt o?
((V-1)))
ō:
2i=r+1 o?
List
A proof of Corollary is given in Appendix A.2 of the supplementary material.
Given the expression for the likelihood derived in Theorem [ there multiple ways carry out model inference. For example, one could sweep over the parameter space to maximize the likelihood of the observed probability paths under the model. Here instead apply Bayesian approach, putting weakly informative prior on 0, and then approximating its posterior distribution via Hamiltonian Monte Carlo (HMC), implemented Stan [Carpenter et al. (017).
Without further constraints, is not fully identified by the data, since multiplying all of the latent variables in Eq. 4 by positive constant does not affect the sign of the relevant expression. Thus, in our applications below, we constrain the scale of the latent variables by requiring Var = of 1.
From fitted GLIM model, it is straightforward to draw probability path from the posterior distribution over paths. We do in three steps. First, we draw value of from the inferred posterior.

---

Next, we draw the vector of latent variables (,1....,2r) from the multivariate normal distribution (O E(X, 0)). Finally, compute values of ........8 yr according to Eq. (T). As shown in the supplementary material, the value of Eq. ① can be computed analytically, yielding:
7
+ 3; M()
Vt = +
(3)
E (1,))
for 1 where 7, p², and E are defined as Theorem only y -' 2 2220 0.
For = we have Yr if and
Experiments
We now explore the efficacy of GLIM through series of experiments on two real-world datasets. We have additionally included simulation study on synthetic dataset in Appendix Bo demonstrate GILM's empirical finite sample behavior. E(X,0) 0) with autoregressive structure and heteroskedastic variance. Specifically, we set
our experiments, we use a covariance matrix
E(e)) 0,0 o:, where the variance time is of ts(
In this setup, the covariance matrix E is parameterized by 9 = (). The first parameter, P. controls the correlation between latent variables, with p = O corresponding to independence. The second parameter, 3, controls how the latent information evolves over time through the variance function GB- A simple example of the variance function is Gs(X,t) exp(BX 1)), where positive, zero, and negative values of B X correspond to increasing, constant, and decreasing variance over time.
As will demonstrate, this relatively simple structure using different variance functions often works well in practice. Figure 6] Appendix B show that even with only two scalar parameters under this specification, GLIM is able model wide variety of probability path structures. However, other parameterization are also possible. For instance, deep kernel learning [Wilson et al. 2016 could be applied to facilitate more complex covariance matrix structure.
4.1 Experiment setup
We now turn to two more complex real-world prediction problems with evolving probability paths to assess the GLIM's performance: (1) the minute-by-minn teams, updated as the game unfolds; and (2) forecasts of rainfall in Australian cities, updated daily as new information becomes available. In both prediction tasks, we observe the initial point prediction Yo given by pre-trained random forest predictor about the final outcome Yr. The goal here is to accurately estimate the posterior probability path distribution, approximated by sample of simulated posterior probability paths generated by the models to be evaluated. Figure Salin Appendix D verifies that for both basketball and weather datasets, the probability paths themselves are generally well calibrated.
win probability of professional basketball
We compare GLIM against three representative baselines from each of the three aforementioned classes of models in Section 2 (1) MMFE: martingale method of forecast evolution (Heath and Jackson 1994 Zhao al. 2013): (2) LR: set of linear regression models {m²) } that predict the future estimated probability at time / [Brockwell and Davis 2016); and (3) MQLSTM: Bayesian
multi-horizon quantile LSTM model [Wen et al. 2017 Eisenach al. 2020).
Assessing the quality of an estimated distribution is tricky: For each probability path in the dataset, we only observe one realized trajectory of the path, rendering metrics comparing two distributions statistic or cross-entrowe cross-entropy) inappropriate Luckily. by taking advantage
(e.g., Kolmogorov-Sminng of the fact that we are modeling probability paths, know there are a few key properties the true probability path distribution must satisfy and have accordingly devised three evaluation metrics to benchmark GL.M against the baseline models.
Although it is possible to treat the observed path as coarse approximation of ground truth distribution (i.e., one one-point distribution), doing will likely introduce unacceptable level of noise in evaluation.
7

---

MNFE
MOLSTM
100%
Probability
75%
25%
Minutes
Figure 3: A comparison of real and predicted probability paths for a single basketball game, for GLIM and for our three baseline models: MMFE, LR, and MQLSTM. The blue lines are the observed ground-truth paths, the grey lines show 100 samples drawn from the distribution of paths predicted by each model, and the red lines show the mean predictions over time.
The first metric is the posterior mean calibration mean squared error (mean calibration MSE). Since the probability paths are expected to be martingales, posterior means of simulated paths should be around the initial prediction (i.e., yo) for i 1,..., and we can empirically examine the generated paths have a sample mean of yo-
The second metric check is the expected volatility mean squared ervor (volatility MSE). As proven by Augenblick and Rabin (202T] and Taleb) (2018), for an evolving belief stream (i.e., a probability path), given initial probabilistie prediction yo: the accumulated squared increments, or volatility, defined as Qr E (Y; - y should be equal yo - 3/) expectation. i.e. [[Q2]= = Vo( - yo). In practice, E[Qr] can be estimated through averaging over the squared increment sum of collection of sample paths.
Finally, if the realized paths are from the estimated probability path distribution, then for any 0 < i T, an empirical credible interval at significance level is expected cover exactly proportion of observed paths. Specifically, we examine the credible interval coverage ervor (CI coverage error) significance level E {50%, 80%, 90%, 95%) } at the beginning, middle, and the end of probability paths for both the basketball and weather datasets.
In the experiments, for each model, all metrics are calculated using 100 simulated samples per probability path. All paths are simulated up until time T 1, the the final end points are generated by drawing from Bernouli(p = Yr-1) to ensure they converge to or at time T,
4.2 Basketball game outcome predictions
Suppose the second half of a basketball game is about start. Based on all the available information, the probability that the home team wins is 60%. How is the game likely to evolve? Do we expect game-deciding event to occur in the next few minutes? Can we safely run the bathroom and not miss the action? More specifically, do we expect the 60% prediction to remain stable until the final minutes of the game, or, alternatively, do we expect to quickly veer toward or 100%?
Here we analyze regular-season NBA basketball games for the 2017-2018 and 2018-2019 seasons, training our model on the first season, and evaluating our predictions on the second To do so, we first created minute- by-intte probability paths for every 48-minute game. For every minute, we trained separate random forest model predict the final game outcome based on information available at that point, including: the current home-team and away-team score, the maximum score margin up until that point in the game, and the win rates for the home and away teams for the previous season. On the resulting probability paths, we fit our GL.M model -as well as our three baseline methods-to model predictions for the second half of the game?) All four models have access to the same information available at half-time, including the score, each team's performance in the previous season, and summary statistics for the probability path in the first half of the game (e.g., the minimum, maximum, average, and standard deviation of predictions). We use a regularized linear function g3 (X,()) () for GLIM, which we describe detail in Appendix
'Our basketball data collected from .ba. with third-party API client (Patel 2018 To avoid the problem of varying horizon lengths because of overtime periods, for any particular game, if home team's score is greater or equal to the visiting team's, we mark the home team as the winner.
For modeling purposes, let t represent the 24th minute of the basketball game.

---

16-01
36:02
Expected Val, MSE
1e-00-
MMFE
LR
10:02
Calibration MSE
Callbration MSE
GUM
1e-03
MNFE
LR
MOLSTM
GLIM
1e-03
(a) Basketball
Expected Vol. MSE
3e-00
te:00
24
02
40
Minutes
Days
MMFE
LR
MGLSTM
GLM
(a) Basketball
(b) Weather
(b) Weather
Figure 4: Mean calibration MSE, plotted log scale. The esti- mated calibration MSE for GLIM is not identically zero because estimate is based finite number of sample paths.
Figure 5: Volatility MSE, plotted on log scale.
In Figure 3
we show an illustrative example of a single game, modeled by all four methods. The identical blue curve in all four panels the ground-truth probability path of the game. Starting the 25th minute (i.e., at the start of the second half), each panel displays the distribution of probability paths predicted by each method, with the dashed red line indicating the mean prediction under each model over time. In this example, MQLSTM produces probability paths that have far too much movement showing significant oscillation between adjacent time steps. In contrast, LR produces paths that appear to appropriately fan out over time. Those paths, however, skew towards the positive outcome just like MQLSTM paths, failing satisfy the martingale property, as indicated by the red line. MMFE is the only baseline model that is able to maintain the posterior mean being centered around yo- Unfortunately.
paths are generally concentrated around Yo until the end and have assigned pretty small density around the ground-truth blue path. Finally, GLIM generates predicted paths that are both calibrated (as theoretically guaranteed) and that appear to appropriately capture the path volatility. One interpretation for GLIM's simulated paths is that given all information have seen at the 24th minute and the point prediction yo being around 30%, if the home team is going to lose the game, we can expect to have confident prediction at around the 36th minute; but if the home team going to win, we might not be so sure about until the very end of the game.
Min.
MMFE
LR
MQLSTM
GLIM
Day
MMFE
LR
MQLSTM
GLIM
50%
0.13
-0.32
-0,08
0.07
50%
0.06
-0.14
-0.01
0.01
80%
0.04
-0.46
-0.10
0.03
80%
0.03
-0.19
-0.02
-0.01
25
90%
0.00
-0.46
-0.10
0.01
90%
-0.19
-0.03
-0.03
95%
-0.02
-0.45
-0.10
-0.01
95%
0.00
-0.17
-0.03
-0.03
50%
0.03
-0.06
-0.09
0.07
50%
-0.09
-0.02
0.03
80%
-0.03
-0.08
-0.12
0.05
80%
0.00
-0.13
-0.03
0.01
90%
-0.05
-0.08
-0.12
0.02
90%
-0.01
-0.12
-0.04
0.00
95%
-0.07
-0.12
0.01
95%
-0.02
-0.11
-0.05
-0.02
50%
-0.01
0.02
-0.14
0.01
50%
0.15
0.15
80%
-0.06
0.01
-0.22
0.02
80%
0.11
-0.01
47
90%
-0.06
-0.01
-0.24
0.03
90%
0.05
0.07
0,06
-0.01
95%
-0.05
-0,02
-0.24
0.02
95%
0.02
0.03
0.03
-0.02
(a) Basketball o-level CI coverage
(b) Weather a-level CI coverage
Table 1: Credible interval coverage at significance level a at different time points. Positive values
indicate overly wide credible intervals, and negative values suggest overly narrow credible intervals. Standard errors are generally at or below 0.01 level and are omitted from this table. The lowest coverage errors (or within +0002 from the lowest absolute error) are in bold. GLIM's performance is consistently the best or indistinguishable from the best in nearly all (time, a:) combinations.

---

Moving beyond this single example, Figure 4a Figure Sa and Table la show the mean calibration MSE, expected volatility MSE, and credible ineeval coverage respectively. As displayed in the plot, GL.M outperforms all three baselines across the board. Particularly, Figure 4a are Figure [a are plotted in log scale, suggesting GLJM outperforming baselines by a few orders of magnitudes on those metrics.
4.3 Weather predictions
Finally, we consider the problem of modeling dynamic forecasts of precipitation. Whereas basketball games may ostensibly be well-modeled as a biased random walk, weather outcomes arguably have much more complicated temporal dynamics [Palmer, 1997]. Specifically, we use dataset of Australian rainfall observations [Williams] 201 Young and Young 2018], and construct daily predictions starting seven days in advance of the target date. Here, starting day zero, we aim to model the evolution of our predictions on whether it will rain on the target date. In this case, our predictions were based on a variety of meteorological features, including air pressure, wind speed, location, and month of year. We randomly sampled 10,000 target dates in the dataset prior to 2014 for training our models, and randomly sampled 10,000 target dates or after 2014 for testing. GLIM and our baseline models were then fit to the training paths, using the same features above. We set = 0 in this case and use a regularized quadratic function Ga(,, t) for GLIM, which we describe in detail in Appendix C.2
GLIM and three baseline models' mean calibration MSE, expected volatility MSE, and credible interval coverage for the weather dataset are displayed in Figure 4b Figure 5b and Table Tb respectively. Similar to what we observe in the basketball dataset, GLIM performs better than all three baseline models. In summary, the results from both the basketball game outcome predictions and weather predictions datasets suggest that GLIM is a desirable framework for modeling probability paths both in theory and in practice.
5 Conclusion
In this paper, we formally investigated the structure of probabilistic predictions that evolve over time. In doing so, introduced the Gaussian latent information martingale (GLIM), Bayesian framework to model these probability paths. contrast to previous work this area, GLIM can naturally incorporate covariates, is able to produce multi-step predictions without explicitly modeling changes in covariates over time, and, perhaps most importantly, preserves important properties of probability paths such the martingale structure and amount of movements. We investigated GLIM's performance on two real-world datasets in addition to a synthetic dataset (Appendix B). helping us understand the dynamic structure of predictions over time.
Aside from being an intriguing problem in and itself, understanding the structure of evolving probability paths can aid time-sensitive decision makers choose appropriate action [Ferguson 2004 Tsitsiklis and Van Roy 1999 Chow and Robbins, 1963]. Examples range from helping ordinary individuals plan weather-continger events, to physicians treating patients with rapidly changing symptoms [Wassenaar et al. 2015), to prosecutors making judgements based on updating information [Lin et al. 2019].
With the increasing availability of real-time data across domains, rapidly evolving forecasts are also likely to become more common. GLIM offers promising route for explicitly modeling- in turn, predicting- trajectory of forecasts. Looking forward, we hope our work sparks further interest in uncovering and leveraging the subtle structure of such dynamic predictions.
Acknowledgements
We thank Johann Gaebler, Daniel Jiang, Zhen Liu, Sabina Tomkins, and Yuchen Xie for helpful conversations throughout the project, and Nicolas Lambert and Jessica Su for in in-depth discussions on a preliminary version of this work. Code to replicate our experiments is available online at: https://gitthu... htttbb/...../ttttaaaaaaraaaaaai.
Funding support: The authors acknowledge that they received no funding in support of this research. Competing interests: The authors declare that they have competing interests.
10

---

References
Erinc Albey, Amirhosein Norouzi, Karl G Kempf, and Reha Uzsoy. Demand modeling with fore- cast evolution: application to production planning. IEEE Transactions Semiconductor Manufacturing, 28(3):374-388 384, 2015.
Ned Augenblick and Matthew Rabin. Belief movement, uncertainty reduction, and rational updating. Q. Econ., February 2021.
Anil K Bera, Matthew L Higgins, and Sangkyu Lec. Interaction between autocorrelation and conditional heteroscedassicity: A random-coefficien approach. Journal of Business Economic Statistics, 10(2):133-142, 1992.
Charles Blundell, Julien Corneiise, Koray Kavukcuoglu, and Daan Wierstra. Weight uncertainty in neural networks. arXiv preprint 1rX1v:1505.05424, 2015.
Tim Bollerslev. Generalized autoregressive conditional heteroskedasticity. Journal of econometrics, 313))307-327, 1986.
Youssef Boulaksil. Safety stock placement in supply chains with demand forecast updates. Operations Research Perspectives, 3:27-31, 2016.
Peter Brockwell and Richard Davis. Introduction to time series and forecasting. Springer, 2016.
Bob Carpenter, Andrew Gelman, Matthew D Hoffmann Daniel Lee, Ben Goodrich, Michael Be- tancourt, Marcus Brubaker, Jiqiang Guo, Peter Li, and Allen Riddell. Stan: A probabilistic programming language. Journal of statistical software, 76(1), 2017.
Alexandra Chouldechova. Fair prediction with disparate impact: A study of bias in recidivism prediction instruments. Big data, 5(2):153-163, 2017.
Yuan Shih Chow and Herbert Robbins. On optimal stopping rules. Zeitschrift für Wahrscheinlichkeit. stheorie und verwandte Gebiete, 2(1):33 49, 1963.
Carson Fisenach, Yagna Patel, and Dhruv Madeka. Mqtransformer: Multi-horizon forecasts with context dependent and feedback-aware attention. arXiv preprint arXiv:2009. 147999 2020.
Jeffrey Ely, Alexander Frankel, and Emir Kamenica. Suspense and surprise. Polit. Econ., 123(1): 215-260, 2015.
Robert Engle. Autoregressive conditional heteroscedasticity with estimates of the variance of United Kingdom inflation. Econometrica: Journal of the Econometric Society, pages 987-1007, 1982.
Robert F Engle and Andrew Patton. What good is a volatility model? In Forecasting volatility in the financial markets, pages 47-63. Elsevier, 2007.
Robert S Erikson and Christopher Wlezien. Markets vs. polls as election predictors: An historical assessment. Electoral Studies, 31 31():5332-539 2012.
João Trevizoli Esteves, Glauco de Souza Rolim, and Antonio Sergio Ferraudo. Rainfall prediction methodology with binary multilayer perceptron neural networks. Climate Dynamics, 52(3-4): 2319-2331, 2019.
Ferguson. Optimal stopping and applications. mathematics department ucla, 2004.
Dean P Foster and Robert A Stine. Threshold martingales and the evolution of forecasts. arXiv preprint arXiv:2 105.06834, 2021.
Christian Francq and Le Quyen Thieu. Qml inference for volatility models with covariates. Econo- metric Theory, 35(1)337-72, 2019. doi: 10.1017/502644666170000
Yarin Gal and Zoubin Ghahramani. Bayesian convolutional neural networks with bernoulli approxi- mate variational inference. arXiv preprint arXiv: 150.:0058800 2015.

---

Andrew Gelman, Jessica Hullman, Christopher Wlezien, and George Elliott Morris. Information, incentives, and goals in election forecasts. Judgment and Decision Making, 15(5):863, 2020.
Ian Goodfellow, Yoshua Bengio, Aaron Courville, and Yoshua Bengio. Deep learning, volume 1. MIT press Cambridge, 2016.
Stephen Graves, Harlan Meal, Sriram Dasu, and Yuping Qui. Two-stage production planning in dynamic environment. In stage production planning and inventory control, pages 9-43. Springer, 1986.
David C Heath and Peter Jackson. Modeling the evolution of demand forecasts ith application to safety stock analysis in production/ddittriution systems. IIE transactions, 26(3):17-30, 1994.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8): 1785, 1997.
Durk P Kingma, Tim Salimans, and Max Welling. Variational dropout and the local reparametcrizzation trick. In Advances in neural information processing systems, pages 2575-2583, 2015.
Zhiyuan Lin, Alex Chohlas-Wore and Sharad Goel. Guiding prosecutorial decisions with an interpretable statistical model. Proceedings of the 2019 AAAI/ACM Conference on Al, Ethics, and Society, pages 469-476, 2019.
Zhiyuan Jenry Lin, Jongbin Jung, Sharad Goel, and Jennifer Skeem. The limits of human predictions of recidivism. Science advances, 6(7)xeaa20652, 2020.
Klas Malmberg, Lars Rydén, Anders Hamsten, Johan Herlitz, Anders Waldenström, and Hans Wedel. Mortality prediction diabetic patients with myocardial infarction: experiences from the digami study. Cardiovascular research, 34(1):248-22) 253, 1997.
J Martinez-Alario, ID Tuesta, E Plasencia, M Santana, and ML Mora. Mortality prediction in cardiac surgery patients: comparative performance of parsonnet and general severity systems. Circulation, 99(18):2378-233 1999.
Jeffrey W Miller. A detailed treatment of Doob's theorem. arXiv preprint arXiv: 1801:03122.22 2018.
Tim Palmer. A weather eye on unpredictability. Exploring Chaos, A Guide to the New Science of Disorder, pages 69-81, 1991.
Swar Patel. An api client package access the apis for nba.com. https://github. cow.suuu..cmm/ara.... nba_api 2018. Accessed: 2020-10-1.
Syama Sundar Rangapuram, Matthias W Seeger, Jan Gasthaus, Lorenzo Stella, Yuyang Wang, and Tim Januschowski. Deep state space models for time series forecasting. Advances neural information processing systems, 31:7785-7794, 2018.
David Salinas, Valentin Flunkert, Jan Gasthaus, and Tim Januschowski. Deepar: Probabilistic forecasting with autoregressive recurrent networks. International Journal of Forecasting, 2019.
Houshmand Shirani-Mehr, David Rothschild, Sharad Goel, and Andrew Gelman. Disentangling bias and variance election polls. Journal of the American Statistical Asxociation, 1136522)6007-614 2018.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, llya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, 15(1):1929-199 1958, 2014.
Genaro Sucarrat. garchx: Flexible and robust garch-x modelling, 2020.
Nassim Nicholas Taleh. Election predictions as martingales: an arbitrage approach. Quant. Finance, 18(1):1-5, January 2018.
Nassim Nicholas Taleb, Dhruv Madeka, et al. All roads lead to quantitative finance. Quantitative Finance, 19(11):175-11776 2019.
12

---

Daniel Teres, Stanley Lemeshow, JILL SPITZ Avrunin, and HARRIS Pastides. Validation of the mortality prediction model for patients. Critical care medicine, 15(3):208-213 1987.
John N Tsitsiklis and Benjamin Van Roy. Optimal stopping Markov processes: Hilbert space theory, approximation algorithms, and application to pricing high-dimensional financial derivatives. IEEE Transactions on Automatic Control, 44410):1840-1851 1999.
Wassenaar, MHWA van den Boogaard, Theo van Achterberg, AJC Slooter, MA Kuiper, ME Hoogendoom, KS Simons, E Maseda, N Pinto, C Jones, al. Multinational development and validation of an early prediction model for delirium in icu patients. Intensive care medicine, 41(6): 1048-1056, 2015.
Ruofeng Wen, Kari Torkkola, Balakrishnan Narayanaswamy, and Dhruv Madeka. multi-horizon quantile recurrent forecaster. arXiv preprint arXiv:1711.11055 2017.
Graham Williams. Data mining with Rattle and R: The art of excavating data for knowledge discovery. Springer Science Business Media, 2011.
Andrew G Wilson and Zoubin Ghahramani. Copula processes. In Advances in Neural Information Processing Systems, pages 2460-2468, 2010.
Andrew Gordon Wilson, Zhiting Hu, Ruslan Salakhutdinov, and Eric P Xing. Deep kernel learning. In Artificial intelligence and statistics, pages 370-378, 2016.
Bin Xu, Feilin Zhu, Ping-an Zhong, Juan Chen, Weifeng Liu, Yufei Ma, Le Guo, and Xiaoliang Deng. Identifying long-term effects of using hydropower complement wind power uncertainty through stochastic programming. Applied Energy, 253:1 13535, 2019.
Li Yan, Hai-Tao Zhang, Jorge Goncalves, Yang Xiao, Maolin Wang. Yuqi Guo, Chuan Sun, Xiuchuan Tang, Liang Jing, Mingyang Zhang, et al. An interpretable mortality prediction model for covid-19 patients. Nature Machine Intelligence, pages 1-6, 2020.
Joe Young and Adam Young. Kaggle: Rain in australia, 2018. https://www.kett kaggle. com/ Saaalr//aaaaa--dtttttt----a--e-
Tongtiegang Zhao, Ximing Cai, and Dawen Yang. Effect of streamflow forecast uncertainty on real-time reservoir operation. Advances in water resources, 34(4):495-504, 2011.
Tongtiegang Zhao, Jianshi Zhao, Dawen Yang, and Hao Wang. Generalized martingale model of the uncertainty evolution of streamflow forecasts. Advances in water resources, 57:41-51, 2013.
13


Reviews:
[{"review_id": "kEok90W-aRf", "reviewer": "Reviewer_KTCw", "summary": "This paper proposes a Bayesian time-series model to predict the structure of dynamic probabilistic forecasts. The main idea is to model prediction trajectories by assuming predictions update according to a latent process of information flow inferred from historical data. ", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "+ The reviewer finds this paper is joyful to read as it is clear and well organized.\n\n+ Modeling evolving predictions is well-motivated although it is not the first work proposed in this field.\n\n+ Theory 1 is well presented with clear explanation for easy understanding.\n\n+ The proposed Bayesian time-series model is technically sound and its performance is investigated through designed experiments.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "- Given the relatively large section of Theory 1,  it is unclear how the proposed four-step procedure is developed from existing works. The author is suggested to highlight the contribution in a more detailed manner.\n\n- As the problem of evolving predictions is highly relevant to Bayesian learning-based uncertainty quantification, it would be very interesting to see more discussion on this point.\n\n- Computational complexity analysis is missing, which is important when considering the time-sensitive predictive modeling.", "review_text": "", "extra_scores": {"ethical_concerns": "No ethical concerns", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1"}, "extra_text": {}}, {"review_id": "KppSOYGWfUa", "reviewer": "Reviewer_mTu3", "summary": "If 2 months before election day, Nate Silver forecasts that Democrats have an 80% chance of taking the Senate, what is the distribution over Nate Silver's forecast 1 month before election day? (Or 2 weeks, 2 days, etc.?) That is the motivation of this paper---it seeks to develop a model for _probability paths_ (i.e., sequences of probabilistic forecasts of an event's occurrence, up until the event) to understand the volatility of forecasts in different domains (e.g., elections, basketball games, weather). \n\nThe paper cites recent work in stats/econ to assert that such a model should be a martingale (i.e., $E[Y_{t+1} | Y_t] = Y_t$ where $Y_t \\in [0,1]$ is the forecasted probability of occurrence). The paper then develops a time-series model that defines the forecast $Y_t$ to be\n\n$$Y_t = Pr\\left(\\gamma + \\sum_{t=1}^T Z_t \\geq 0 \\,|\\,Z_1,\\dots,Z_t\\right)$$\n\nwhich is essentially the expectation of the normal CDF applied to a discretized Brownian motion.\n\nThe paper shows that this model satisfies the martingale property and further derives a forward-filtering algorithm to analytically compute the likelihood (with the Gaussian innovations $Z_t$ marginalized out). The paper then proposes to use HMC (with vague priors on the parameters) to fit the model.\n\nIn experiments, the paper compares the proposed model to three baselines---1) an alternative method that satisfies the martingale property (MMFE), and two others that do not, including 2) a naive linear model (LR) that also does not respect the bounded support of $Y_t \\in [0,1]$, and 3) and deep sequence model (MQLSTM). The paper compares all four methods on (real) basketball prediction data and weather forecast data. Each model is fit to historical sequences of probabilistic forecasts and used to generate posterior predictive paths on a test set. The posterior predictive paths are evaluated using three metrics---the first two metrics assess how well the posterior paths collectively satisfy the desirable martingale property and the third metric assesses the posterior's confidence interval coverage of the true path.\n\nThe proposed model obtains better performance on the metrics that evaluate whether it satisfies the martingale property (including better performance over the other martingale baseline). The proposed model's posterior paths also obtain better or similar CI coverage than the baselines most of the time. ", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is well-written and clear. The technical bits are presented precisely and correctly (including the proofs in the appendix, which look correct to me).\n\nThe paper's high-level motivation is interesting but fairly niche/narrow. The paper is explicit about not seeking to _improve_ forecasts but simply to model observed sequences of forecasts in order to understand the volatility of such forecasts in different domains. It's true that popular forecasters (e.g., Nate Silver) release sequences of forecasts that seem to themselves impact the real world and thus may be worthy of being modeled themselves. \n\nThe technical meat of the paper is mainly motivated by developing a martingale model (with \"zero-centered expectation\") which it defends by saying \"as shown by statisticians and economists...predictions need to have a zero-centered expectation...\". Since all of the methodological development in the paper, as well as the evaluation metrics in the results section, are then based around this premise, the paper would be stronger if it included a (brief) summary/description of why statistician/economists say we \"need\" zero-centered expectations. To be clear, it seems like a natural desideratum to me, but the paper should do more to justify its main motivation to readers, such as myself, who are unfamiliar with the econ literature on probability paths.\n\nThe main difficulty here is in building and fitting a martingale model for _bounded-support_ sequences $Y_t \\in [0,1]$. The paper's solution is appealing and elegant. By defining the observed data as a probability $Y_t = P(\\cdot\\,|\\,Z_1,\\dots,Z_t)$ in terms of Gaussian innovations $Z_t$ the paper is able to combine familiar to expressions and identities (mostly involving iterated expectations and normal CDFs) to obtain a martingale model with analytic marginal likelihood.\n\nNote that the proof in A.3 for the martingale property of the proposed model can be easily generalized. For any filtration process $Y_t | H_t$, where the history $H_t = \\{Z_t, H_{t-1}\\}$ is the union of the history up to $t-1$ and some innovation $Z_t$, then if the observation is defined to be a probability $Y_t = P(\\,\\cdot\\, \\,|\\, H_t)$ then:\n\n$$E[Y_{t+1} \\,|\\, H_t] = E[P(\\,\\cdot\\, \\,|\\, H_{t+1}) \\,|\\,H_t] = P(\\cdot\\,|\\,H_t) = Y_t$$\n\nThe generality of this proof makes me wonder whether the paper has missed some related work in stochastic processes, particularly in transformations of Brownian motion to the unit interval. More generally, I would have wanted to see more coverage of existing time-series work on bounded-support data (e.g., [1]). \n\nIn terms of the experiments, I wonder how much of the variation in the results across models is being driven by the number of sampled paths being set to 100. The reason I wonder this is because, in theory, there should be no difference between GLIM and MMFE on the metrics that asssess whether the model satisfies the martingale property, yet the proposed GLIM appears to obtain orders of magnitude better performance than MMFE. One then wonders whether the difference between the confidence interval results are also being driven by this small -(posterior)sample variation or if there is something fundamentally better about the proposed model.\n\nI also wonder why the LR model does not satisfy the martingale property. It should be trivial to write down a linear model, that does not respect the bounded-support of $Y_t$ but still has zero-centered expectations. \n\nOverall, I think the paper has superb _clarity_ (very well written) and high _quality_ (the technical bits are correct). Its _significance_ and _originality_ depend on whether you buy the motivation that one needs a martingale model of observed forecasts and that existing approaches are insignificant; the paper does not do a good job of defending either point (I reserve judgement on this until I hear from other reviewers and the authors).\n\n[1] Jara, Nieto-Barajas, and Quintana. A time series model for responses on the unit interval. (2013).\n\n#### Questions for the authors\n* Why wasn't a martingale model selected for the naive linear baseline? \n\n* Why does GLIM perform an order of magnitude better than MMFE on mean calibration? Both are martingales and MMFE's samples are tightly clustered around y0---I would have expected both to perform the same given infinite posterior samples (i.e., have zero calibration MSE) and for MMFE to perform better (or the same) given finite samples.\n\n* The reported results suggest that MMFE obtains poor confidence interval coverage in certain settings while GLIM (the proposed model) does not. Is it understood why MMFE performs poorly here? More generally, the paper doesn't mention why development of the proposed model is motivated given that an existing martingale approach exists (MMFE)---what are the intrinsic/theoretical aspects of MMFE that are lacking or insufficient and would prompt development of an alternative model? \n\n\n#### Typos\n* Line 104: \"evolvement\" -> \"evolution\"\n* Line 118: \"assumes\" -> \"assume\"\n* Line 299: missing period", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"ethical_concerns": "None.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "GVU7O_i8hgr", "reviewer": "Reviewer_1PPk", "summary": "This paper proposes the Gaussian latent information martingale (GLIM), a Bayesian methodology to model probability paths.\nThis paper introduces the notion of a probability path and shows how Bayesian methods can be used to carry out inference on such an object. Two applications showcase the potential usefulness of the methodology.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The idea of the paper is quite original. The methodology is nicely illustrated and in my opinion it achieves its objectives. Moreover, the empirical illustration nicely showcases the idea of this work and the gains of the procedure. This is not the best paper I reviewed but this could be a nice \"fresh\" contribution to the program.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "This paper does not raise any concerns", "review_text": "", "extra_scores": {"ethical_concerns": "This paper does not raise any ethical concerns.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "AlHMBXK7uon", "reviewer": "Reviewer_BQ9v", "summary": "The paper presents a novel model called Gaussian Latent Information Martingale for capturing the structure of dynamically evolving binary probability forecasts, called probability paths, over the interval [0,T], for a binary (yes/no) event that is considered at some fixed point in time, T, in the future. The goal is to capture the evolution of the binary forecast over [0,T], while sitting at t=0 (or any t < T), and this needs to satisfy a Martingale property at each time t < T, since there is no special information that is specifically available about the future (i.e. beyond time t). The proposed model is shown to satisfy this property through conditioning on a sequence of latent variables that are assumed to enjoy a multivariate normal distribution structure. The authors also exploit the analytical properties of a multivariate normal distribution along with the graphical model structure of the proposed latent model, to derive the probability density function for 'probability path' data according to the proposed model, and this is their main result presented as Theorem 1. This allows them to carry out computations for estimating the model parameters using maximum log likelihood or Bayesian inference. Experimental results are presented for basketball game data and weather data. The model admits specification of covariates at t=0, and has parameters for capturing the covariate-dependent covariance matrix for the latent multivariate normal distribution. ", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper addresses a tricky problem of how to capture dynamically evolving probability forecasts for binary outcomes at some fixed point in time in the (near) future. The motivation is that decision-makers could benefit from knowing the volatility of how a starting point-in-time forecast would evolve in time as one would get closer in time to the binary outcome of interest. \nThe proposed model, GLIM appears original and novel. It assumes a multivariate normal distribution for an evolving sequence of latent information variables that produce the future looking estimates/forecasts for the dynamically evolving binary probability forecast of interest, in the form of a conditional probability. The conditional probability also allows preservation of the martingale property that is required for forward looking estimates of the dynamically evolving forecast with no further specific information about the future.\nThe writing is clear.\nEquation 1) could benefit from explicitly clarifying that Y_t is by definition the conditional expectation of the binary variable Y_T, conditional on the latent sequence Z_1, Z_2,...,Z_t.\nThe paper could also benefit from providing more details about the Bayesian HMC based inference computations that they use in experiments. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "No negative societal impact.", "review_text": "", "extra_scores": {"ethical_concerns": "No ethical concerns.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors formulate a new problem of predicting the probability paths of a forecaster over time as it obtains new evidence, and a new algorithm for doing so, based on the fact that forecasts are expected to satisfy a martingale property. Reviewers agreed the problem setting and method were novel, interesting, and technically well executed, and gave scores of 5, 6, 6, 6. There were no major critiques raised, though Reviewer mTu3 wrote a long and insightful review that raised some questions. The meta-reviewer is unsure about the eventual scope of application for this problem and method, but overall finds that the ideas are fresh, interesting, and well executed, so should be of interest to the NeurIPS community. 
Decision: Accept (Poster)