Probability Paths and the Structure of Predictions over Time

Abstract: In settings ranging from weather forecasts to political prognostications to financial projections, probability estimates of future binary outcomes often evolve over time. For example, the estimated likelihood of rain on a specific day changes by the hour as new information becomes available. Given a collection of such probability paths, we introduce a Bayesian framework -- which we call the Gaussian latent information martingale, or GLIM -- for modeling the structure of dynamic predictions over time. Suppose, for example, that the likelihood of rain in a week is 50%, and consider two hypothetical scenarios. In the first, one expects the forecast to be equally likely to become either 25% or 75% tomorrow; in the second, one expects the forecast to stay constant for the next several days. A time-sensitive decision-maker might select a course of action immediately in the latter scenario, but may postpone their decision in the former, knowing that new information is imminent. We model these trajectories by assuming predictions update according to a latent process of information flow, which is inferred from historical data. In contrast to general methods for time series analysis, this approach preserves important properties of probability paths such as the martingale structure and appropriate amount of volatility and better quantifies future uncertainties around probability paths. We show that GLIM outperforms three popular baseline methods, producing better estimated posterior probability path distributions measured by three different metrics. By elucidating the dynamic structure of predictions over time, we hope to help individuals make more informed choices.


Reviews and author responses:

 We thank the reviewer for the detailed review and thoughtful suggestions. In response to the specific questions and concerns raised:

1. MMFE indeed guarantees that the martingale property is satisfied in most cases where the Y_t are not bounded. However, many sampled paths generated by MMFE go beyond the [0, 1] bound (and thus are not valid probability paths). We therefore truncated the paths to adhere to this constraint, but that in turn results in the t MMFE losing its martingale property. Similarly, a simple LR model that does not respect the bounded-support of Y_t may suffer from the same problem.

2. Regarding the confidence interval coverage performance gap between MMFE and GLIM, one explanation is that MMFE does not account for the idiosyncratic structure of each individual probability path in the test set. On the other hand, GLIM’s formulation allows it to encode such heterogeneity of each probability path via the variance function $\mathcal{G}_{\beta}(X, t)$ and the parameterized covariance matrix in general. Together with the point above, GLIM extends MMFE by incorporating information from covariates X for each individual path in a bounded (i.e., [0, 1]) space.

3. On the motivation of this work, as the reviewer has pointed out, the martingale property is a natural desideratum to us, as it ensures the internal consistency of the underlying prediction mechanism. In other words, probabilistic predictions should not only be calibrated at the time of prediction (i.e., when t = 0), but also across all time horizons (i.e., when t = 1, 2, … T), hence the martingale property. Violation of the martingale property makes it difficult to interpret the inferred paths as probabilistic forecasts. While it is still possible to optimize for some utility functions without probabilistic forecasts, unfaithful probabilistic forecasts are often hard to reason about by human decision-makers. Further, the martingale property offers additional protection in certain situations such as arbitrage (Taleb, 2017) in financial applications and can generally lead to better predictions (Foster and Stine, 2021).

This paper proposes the Gaussian latent information martingale (GLIM), a Bayesian methodology to model probability paths.
This paper introduces the notion of a probability path and shows how Bayesian methods can be used to carry out inference on such an object. Two applications showcase the potential usefulness of the methodology.  The idea of the paper is quite original. The methodology is nicely illustrated and in my opinion it achieves its objectives. Moreover, the empirical illustration nicely showcases the idea of this work and the gains of the procedure. This is not the best paper I reviewed but this could be a nice "fresh" contribution to the program. This paper does not raise any concerns

This paper proposes a Bayesian time-series model to predict the structure of dynamic probabilistic forecasts. The main idea is to model prediction trajectories by assuming predictions update according to a latent process of information flow inferred from historical data.   + The reviewer finds this paper is joyful to read as it is clear and well organized.

+ Modeling evolving predictions is well-motivated although it is not the first work proposed in this field.

+ Theory 1 is well presented with clear explanation for easy understanding.

+ The proposed Bayesian time-series model is technically sound and its performance is investigated through designed experiments. - Given the relatively large section of Theory 1,  it is unclear how the proposed four-step procedure is developed from existing works. The author is suggested to highlight the contribution in a more detailed manner.

- As the problem of evolving predictions is highly relevant to Bayesian learning-based uncertainty quantification, it would be very interesting to see more discussion on this point.

- Computational complexity analysis is missing, which is important when considering the time-sensitive predictive modeling.

If 2 months before election day, Nate Silver forecasts that Democrats have an 80% chance of taking the Senate, what is the distribution over Nate Silver's forecast 1 month before election day? (Or 2 weeks, 2 days, etc.?) That is the motivation of this paper---it seeks to develop a model for _probability paths_ (i.e., sequences of probabilistic forecasts of an event's occurrence, up until the event) to understand the volatility of forecasts in different domains (e.g., elections, basketball games, weather). 

The paper cites recent work in stats/econ to assert that such a model should be a martingale (i.e., $E[Y_{t+1} | Y_t] = Y_t$ where $Y_t \in [0,1]$ is the forecasted probability of occurrence). The paper then develops a time-series model that defines the forecast $Y_t$ to be

$$Y_t = Pr\left(\gamma + \sum_{t=1}^T Z_t \geq 0 \,|\,Z_1,\dots,Z_t\right)$$

which is essentially the expectation of the normal CDF applied to a discretized Brownian motion.

The paper shows that this model satisfies the martingale property and further derives a forward-filtering algorithm to analytically compute the likelihood (with the Gaussian innovations $Z_t$ marginalized out). The paper then proposes to use HMC (with vague priors on the parameters) to fit the model.

In experiments, the paper compares the proposed model to three baselines---1) an alternative method that satisfies the martingale property (MMFE), and two others that do not, including 2) a naive linear model (LR) that also does not respect the bounded support of $Y_t \in [0,1]$, and 3) and deep sequence model (MQLSTM). The paper compares all four methods on (real) basketball prediction data and weather forecast data. Each model is fit to historical sequences of probabilistic forecasts and used to generate posterior predictive paths on a test set. The posterior predictive paths are evaluated using three metrics---the first two metrics assess how well the posterior paths collectively satisfy the desirable martingale property and the third metric assesses the posterior's confidence interval coverage of the true path.

The proposed model obtains better performance on the metrics that evaluate whether it satisfies the martingale property (including better performance over the other martingale baseline). The proposed model's posterior paths also obtain better or similar CI coverage than the baselines most of the time.   The paper is well-written and clear. The technical bits are presented precisely and correctly (including the proofs in the appendix, which look correct to me).

The paper's high-level motivation is interesting but fairly niche/narrow. The paper is explicit about not seeking to _improve_ forecasts but simply to model observed sequences of forecasts in order to understand the volatility of such forecasts in different domains. It's true that popular forecasters (e.g., Nate Silver) release sequences of forecasts that seem to themselves impact the real world and thus may be worthy of being modeled themselves. 

The technical meat of the paper is mainly motivated by developing a martingale model (with "zero-centered expectation") which it defends by saying "as shown by statisticians and economists...predictions need to have a zero-centered expectation...". Since all of the methodological development in the paper, as well as the evaluation metrics in the results section, are then based around this premise, the paper would be stronger if it included a (brief) summary/description of why statistician/economists say we "need" zero-centered expectations. To be clear, it seems like a natural desideratum to me, but the paper should do more to justify its main motivation to readers, such as myself, who are unfamiliar with the econ literature on probability paths.

The main difficulty here is in building and fitting a martingale model for _bounded-support_ sequences $Y_t \in [0,1]$. The paper's solution is appealing and elegant. By defining the observed data as a probability $Y_t = P(\cdot\,|\,Z_1,\dots,Z_t)$ in terms of Gaussian innovations $Z_t$ the paper is able to combine familiar to expressions and identities (mostly involving iterated expectations and normal CDFs) to obtain a martingale model with analytic marginal likelihood.

Note that the proof in A.3 for the martingale property of the proposed model can be easily generalized. For any filtration process $Y_t | H_t$, where the history $H_t = \{Z_t, H_{t-1}\}$ is the union of the history up to $t-1$ and some innovation $Z_t$, then if the observation is defined to be a probability $Y_t = P(\,\cdot\, \,|\, H_t)$ then:

$$E[Y_{t+1} \,|\, H_t] = E[P(\,\cdot\, \,|\, H_{t+1}) \,|\,H_t] = P(\cdot\,|\,H_t) = Y_t$$

The generality of this proof makes me wonder whether the paper has missed some related work in stochastic processes, particularly in transformations of Brownian motion to the unit interval. More generally, I would have wanted to see more coverage of existing time-series work on bounded-support data (e.g., [1]). 

In terms of the experiments, I wonder how much of the variation in the results across models is being driven by the number of sampled paths being set to 100. The reason I wonder this is because, in theory, there should be no difference between GLIM and MMFE on the metrics that asssess whether the model satisfies the martingale property, yet the proposed GLIM appears to obtain orders of magnitude better performance than MMFE. One then wonders whether the difference between the confidence interval results are also being driven by this small -(posterior)sample variation or if there is something fundamentally better about the proposed model.

I also wonder why the LR model does not satisfy the martingale property. It should be trivial to write down a linear model, that does not respect the bounded-support of $Y_t$ but still has zero-centered expectations. 

Overall, I think the paper has superb _clarity_ (very well written) and high _quality_ (the technical bits are correct). Its _significance_ and _originality_ depend on whether you buy the motivation that one needs a martingale model of observed forecasts and that existing approaches are insignificant; the paper does not do a good job of defending either point (I reserve judgement on this until I hear from other reviewers and the authors).

[1] Jara, Nieto-Barajas, and Quintana. A time series model for responses on the unit interval. (2013).

#### Questions for the authors
* Why wasn't a martingale model selected for the naive linear baseline? 

* Why does GLIM perform an order of magnitude better than MMFE on mean calibration? Both are martingales and MMFE's samples are tightly clustered around y0---I would have expected both to perform the same given infinite posterior samples (i.e., have zero calibration MSE) and for MMFE to perform better (or the same) given finite samples.

* The reported results suggest that MMFE obtains poor confidence interval coverage in certain settings while GLIM (the proposed model) does not. Is it understood why MMFE performs poorly here? More generally, the paper doesn't mention why development of the proposed model is motivated given that an existing martingale approach exists (MMFE)---what are the intrinsic/theoretical aspects of MMFE that are lacking or insufficient and would prompt development of an alternative model? 


#### Typos
* Line 104: "evolvement" -> "evolution"
* Line 118: "assumes" -> "assume"
* Line 299: missing period Yes.

The paper presents a novel model called Gaussian Latent Information Martingale for capturing the structure of dynamically evolving binary probability forecasts, called probability paths, over the interval [0,T], for a binary (yes/no) event that is considered at some fixed point in time, T, in the future. The goal is to capture the evolution of the binary forecast over [0,T], while sitting at t=0 (or any t < T), and this needs to satisfy a Martingale property at each time t < T, since there is no special information that is specifically available about the future (i.e. beyond time t). The proposed model is shown to satisfy this property through conditioning on a sequence of latent variables that are assumed to enjoy a multivariate normal distribution structure. The authors also exploit the analytical properties of a multivariate normal distribution along with the graphical model structure of the proposed latent model, to derive the probability density function for 'probability path' data according to the proposed model, and this is their main result presented as Theorem 1. This allows them to carry out computations for estimating the model parameters using maximum log likelihood or Bayesian inference. Experimental results are presented for basketball game data and weather data. The model admits specification of covariates at t=0, and has parameters for capturing the covariate-dependent covariance matrix for the latent multivariate normal distribution.   The paper addresses a tricky problem of how to capture dynamically evolving probability forecasts for binary outcomes at some fixed point in time in the (near) future. The motivation is that decision-makers could benefit from knowing the volatility of how a starting point-in-time forecast would evolve in time as one would get closer in time to the binary outcome of interest. 
The proposed model, GLIM appears original and novel. It assumes a multivariate normal distribution for an evolving sequence of latent information variables that produce the future looking estimates/forecasts for the dynamically evolving binary probability forecast of interest, in the form of a conditional probability. The conditional probability also allows preservation of the martingale property that is required for forward looking estimates of the dynamically evolving forecast with no further specific information about the future.
The writing is clear.
Equation 1) could benefit from explicitly clarifying that Y_t is by definition the conditional expectation of the binary variable Y_T, conditional on the latent sequence Z_1, Z_2,...,Z_t.
The paper could also benefit from providing more details about the Bayesian HMC based inference computations that they use in experiments.  No negative societal impact.

Meta-review: The authors formulate a new problem of predicting the probability paths of a forecaster over time as it obtains new evidence, and a new algorithm for doing so, based on the fact that forecasts are expected to satisfy a martingale property. Reviewers agreed the problem setting and method were novel, interesting, and technically well executed, and gave scores of 5, 6, 6, 6. There were no major critiques raised, though Reviewer mTu3 wrote a long and insightful review that raised some questions. The meta-reviewer is unsure about the eventual scope of application for this problem and method, but overall finds that the ideas are fresh, interesting, and well executed, so should be of interest to the NeurIPS community. 

Decision: accept