SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Eric Neyman
eric.neyman@columbia.edu
Tim Roughgarden
tim.roughgarden@gmail.com
For each of  $T$  time steps,  $m$  experts report probability distributions over  $n$  outcomes; we wish to learn to aggregate these forecasts in a way that attains a no-regret guarantee. We focus on the fundamental and practical aggregation method known as logarithmic pooling — a weighted average of log odds — which is in a certain sense the optimal choice of pooling method if one is interested in minimizing log loss (as we take to be our loss function). We consider the problem of learning the best set of parameters (i.e. expert weights) in an online adversarial setting. We assume (by necessity) that the adversarial choices of outcomes and forecasts are consistent, in the sense that experts report calibrated forecasts. Imposing this constraint creates a (to our knowledge) novel semi-adversarial setting in which the adversary retains a large amount of flexibility. In this setting, we present an algorithm based on online mirror descent that learns expert weights in a way that attains  $O(\sqrt{T}\log T)$  expected regret as compared with the best weights in hindsight.
1.1 Logarithmic pooling
Suppose that  $m$  experts report probability distributions  $\mathbf{p}^1, \ldots, \mathbf{p}^m \in \Delta^n$  over  $n$  disjoint, exhaustive outcomes. We are interested in aggregating these distributions into a single distribution  $\mathbf{p}^*$ , a task known as probabilistic opinion pooling. Perhaps the most straightforward way to do this is to take the arithmetic mean, also called the linear pool:  $\mathbf{p}^* = \frac{1}{m} \sum_{i=1}^{m} \mathbf{p}^i$ .
While commonly studied and frequently used, the linear pool is by no means the definitive opinion pooling method. The choice of pooling method ought to depend on context, and in particular on the loss function with respect to which forecasts are assessed. Neyman and Roughgarden (2023) showed a correspondence between proper loss functions and opinion pooling methods, which they termed quasi-arithmetic (QA) pooling with respect to a loss function. Specifically, the QA pool with respect to a loss is the forecast that guarantees the largest possible overperformance (as judged by the loss function) compared with the strategy of choosing a random expert to trust.<sup>1</sup>
In this work we will be using log loss function. The QA pooling technique with respect to the log loss is known as logarithmic pooling. Instead of averaging the experts' forecasts, logarithmic pooling averages the experts' log odds (i.e. logits). Put otherwise, the logarithmic pool is defined by
$$
p _ {j} ^ {*} = c \prod_ {i = 1} ^ {m} \left(p _ {j} ^ {i}\right) ^ {1 / m}
$$
1More formally, the QA pool maximizes the minimum (over possible outcomes) improvement in the loss.
for all events  $j \in [n]$ , where  $c$  is a normalizing constant to ensure that the probabilities add to 1. While the linear pool is an arithmetic mean, the logarithmic pool behaves much more like a geometric mean. For instance, the logarithmic pool of (0.001, 0.999) and (0.5, 0.5) with equal weights is approximately (0.03, 0.97).
The logarithmic pool has been studied extensively because of its naturalness (Genest, 1984; Genest and Zidek, 1986; Givens and Roback, 1999; Poole and Raftery, 2000; Kascha and Ravazzolo, 2008; Rufo et al., 2012; Allard et al., 2012). Logarithmic pooling can be interpreted as averaging experts' Bayesian evidence (Neyman and Roughgarden, 2023, §A). It is also the most natural pooling method that satisfies external Bayesianity, meaning roughly that it does not matter whether a Bayesian update is applied to each expert's forecast before aggregation, or if instead the Bayesian update is applied to the pooled forecast (Genest, 1984). Logarithmic pooling can also be characterized as the pooling method that minimizes the average KL divergence to the experts' reports (Abbas, 2009). Finally, empirical work has found logarithmic pooling performs very well on real-world data (Satopää et al., 2014; Sevilla, 2021).
1.2 Logarithmic pooling with weighted experts
Forecast aggregators often assign different weights to different experts, e.g. based on each expert's level of knowledge or track record (Tetlock and Gardner, 2015). There is a principled way to include weights  $w_{1}, \ldots, w_{i}$  (summing to 1) in the logarithmic pool, namely:
$$
p _ {j} ^ {*} (\mathbf {w}) = c (\mathbf {w}) \prod_ {i = 1} ^ {m} \left(p _ {j} ^ {i}\right) ^ {w _ {i}},
$$
where  $c(\mathbf{w})$  is again a normalizing constant that now depends on the weights. This more general notion continues to have all of the aforementioned natural properties.
The obvious question is: how does one know what these weights should be? Perhaps the most natural answer is to weight experts according to their past performance. Finding appropriate weights for experts is thus an online learning problem. This learning problem is the focus of our work.
1.3 Choosing the right benchmark
Our goal is to develop an algorithm for learning weights for logarithmic pooling in a way that achieves vanishing regret as judged by the log loss function (i.e. the loss function most closely associated with this pooling method). Within the field of online prediction with expert advice, this is a particularly challenging setting. In part, this is because the losses are potentially unbounded. However, that is not the whole story: finding weights for linear pooling, by contrast, is a well-studied problem that has been solved even in the context of log loss. On the other hand, because logarithmic pooling behaves more as a geometric than an arithmetic mean, if some expert assigns a very low probability to the eventual outcome (and the other experts do not) then the logarithmic pool will also assign a low probability, incurring a large loss. This makes the combination of logarithmic pooling with log loss particularly difficult.
We require that our algorithm not have access to the experts' forecasts when choosing weights: an algorithm that chooses weights in a way that depends on forecasts can output an essentially arbitrary function of the forecasts, and thus may do something other than learn optimal weights for logarithmic pooling. For example, suppose that  $m = n = 2$  and an aggregator wishes to subvert our intentions and take an equally weighted linear pool of the experts' forecasts. Without knowing the experts' forecasts, this is impossible; on the other hand, if the aggregator knew that e.g.  $\mathbf{p}_1 = (90\%, 10\%)$  and  $\mathbf{p}_2 = (50\%, 50\%)$ , they could assign weights for logarithmic pooling so as to produce the post-hoc desired result, i.e.  $(70\%, 30\%)$ . We wish to disallow this.
One might suggest the following setup: at each time step, the algorithm selects weights for each expert. Subsequently, an adversary chooses each expert's forecast and the outcome, after which the algorithm and each expert incur a log loss. Unfortunately — due to the unboundedness of log loss and the behavior of logarithmic pooling — vanishing regret guarantees in this setting are impossible.
2Were it not for the normalizing constant, the log loss incurred by the logarithmic pool  $p_j^* (\mathbf{w})$  would be a linear function of the weights. However, the  $c(\mathbf{w})$  term significantly complicates this picture.
Example 1.1. Consider the case of  $m = n = 2$ . Without loss of generality, suppose that the algorithm assigns Expert 1 a weight  $w \geq 0.5$  in the first time step. The adversary chooses reports  $(e^{-T}, 1 - e^{-T})$  for Expert 1 and  $\left(\frac{1}{2}, \frac{1}{2}\right)$  for Expert 2, and for Outcome 1 to happen. The logarithmic pool of the forecasts turns out to be approximately  $(e^{-wT}, 1 - e^{-wT})$ , so the algorithm incurs a log loss of approximately  $wT \geq 0.5T$ , compared to  $O(1)$  loss for Expert 2. On subsequent time steps, Expert 2 is perfect (assigns probability 1 to the correct outcome), so the algorithm cannot catch up.
What goes wrong in Example 1.1 is that the adversary has full control over experts' forecast and the realized outcome, and is not required to couple the two in any way. This unreasonable amount of adversarial power motivates assuming that the experts are calibrated: for example, if an expert assigns a  $10\%$  chance to an outcome, there really is a  $10\%$  chance of that outcome (conditional on the expert's information).
We propose the following setting: an adversary chooses a joint probability distribution over the experts' beliefs and the outcome — subject to the constraint that each expert is calibrated. The adversary retains full control over correlations between forecasts and outcomes, subject to the calibration property. Subsequently, nature randomly samples each expert's belief and the eventual outcome from the distribution. In this setting, we seek to prove upper bounds on the expected value of our algorithm's regret.
Why impose this constraint, instead of a different one? Our reasons are twofold: theoretical and empirical. From a theoretical standpoint, the assumption that experts are calibrated is natural because experts who form Bayesian rational beliefs based on evidence will be calibrated, regardless of how much or how little evidence they have. The assumption is also motivated if we model experts as learners rather than Bayesian agents: even if a forecaster starts out completely uninformed, they can quickly become calibrated in a domain simply by observing the frequency of events (Foster and Vohra, 1997).
Second, recent work has shown that modern deep neural networks are calibrated when trained on a proper loss function such as log loss. This is true for a variety of tasks, including image classification (Minderer et al., 2021; Hendrycks et al., 2020) and language modeling (Kadavath et al., 2022; Desai and Durrett, 2020; OpenAI, 2023); see (Blasiok et al., 2023) for a review of the literature. We may wish to use an ensemble of off-the-shelf neural networks for some prediction or classification task. If we trust these networks to be calibrated (as suggested by recent work), then we may wish to learn to ensemble these experts (models) in a way that has strong worst-case theoretical guarantees under the calibration assumption.
Logarithmic pooling is particularly sensible in the context of calibrated experts because it takes confident forecasts "more seriously" as compared with linear pooling (simple averaging). If Expert 1 reports probability distribution  $(0.1\%, 99.9\%)$  over two outcomes and Expert 2 reports  $(50\%, 50\%)$ , then the logarithmic pool (with equal weights) is approximately  $(3\%, 97\%)$ , as compared with a linear pool of roughly  $(25\%, 75\%)$ . If Expert 1 is calibrated (as we are assuming), then the  $(0.1\%, 99.9\%)$  forecast entails very strong evidence in favor of Outcome 2 over Outcome 1. Meanwhile, Expert 2's forecast gives no evidence either way. Thus, it is sensible for the aggregate to point to Outcome 2 over Outcome 1 with a fair amount of confidence.
As another example, suppose that Expert 1 reports  $(0.04\%, 49.98\%, 49.98\%)$  and Expert 2 reports  $(49.98\%, 0.04\%, 49.98\%)$  (a natural interpretation: Expert 1 found strong evidence against Outcome 1 and Expert 2 found strong evidence against Outcome 2). If both experts are calibrated, a sensible aggregate should arguably assign nearly all probability to Outcome 3. Logarithmic pooling returns roughly  $(2.7\%, 2.7\%, 94.6\%)$ , which (unlike linear pooling) accomplishes this.
Since we are allowing our algorithm to learn the optimal logarithmic pool, perhaps there is hope to compete not just with the best expert in hindsight, but the optimally weighted logarithmic pool of experts in hindsight. We will aim to compete with this stronger benchmark.
This paper demonstrates that the "calibrated experts" condition allows us to prove regret bounds when no such bounds are possible for an unrestricted adversary. While that is our primary motivation, the relaxation may also be of independent interest. For example, even in settings where vanishing regret is attainable in the presence of an unrestricted adversary, even stronger regret bounds might be achievable if calibration is assumed.
1.4 Our main result
Is vanishing regret possible in our setting? Our main result is that the answer is yes. We exhibit an algorithm that attains expected regret that scales as  $O(\sqrt{T}\log T)$  with the number of time steps  $T$ . Our algorithm uses online mirror descent (OMD) with the Tsallis entropy regularizer  $R(\mathbf{w}) = \frac{-1}{\alpha} (w_1^\alpha +\dots +w_m^\alpha)$  and step size  $\eta \approx \frac{1}{\sqrt{T}\ln T}$ , where any choice of  $\alpha \in (0,1 / 2)$  attains the regret bound.
Our proof has two key ideas. One is to use the calibration property to show that the gradient of loss with respect to the weight vector is likely to be small (Section 4.4). This is how we leverage the calibration property to turn an intractable setting into one where — despite the unboundedness of log loss and the behavior of logarithmic pooling — there is hope for vanishing regret.
The other key idea (Section 4.3) involves keeping track of a function that, roughly speaking, reflects how much "regret potential" the algorithm has. We show that if the aforementioned gradient updates are indeed small, then this potential function decreases in value at each time step. This allows us to upper bound the algorithm's regret by the initial value of the potential function.
This potential argument is an important component of the proof. A naive analysis might seek to use our bounds on the gradient steps to myopically bound the contribution to regret at each time step. Such an analysis, however, does not achieve our  $O(\sqrt{T}\log T)$  regret bound. In particular, an adversary can force a large accumulation of regret if some experts' weights are very small (specifically by making the experts with small weights more informed than those with large weights) — but by doing so, the small weights increase and the adversary "spends down" its potential. Tracking this potential allows us to take this nuance into consideration, improving our bound.
We extend our main result by showing that the result holds even if experts are only approximately calibrated: so long as no expert understates the probability of an outcome by more than a constant factor, we still attain the same regret bound (see Corollary A.8). We also show in Appendix B that no OMD algorithm with a constant step size can attain expected regret better than  $\Omega (\sqrt{T})$ .
2.1 Probabilistic opinion pooling
There has been substantial mathematical work on probabilistic opinion pooling (i.e. forecast aggregation) since the 1980s. One line of work is axiomatic in nature: motivating opinion pooling methods by describing axioms that they satisfy. For example, logarithmic pooling satisfies unanimity preservation and external Bayesianity (Genest, 1984). There has also been work on Bayesian approaches to pooling, e.g. under the assumption that experts' signals are drawn from some parameterized class of distributions (Winkler, 1981; Lichtendahl et al., 2017).
Neyman and Roughgarden (2023) show that every proper loss function has an associated pooling method (the QA pool with respect to the loss function), which is the forecast that guarantees the largest possible overperformance (as judged by the loss function) compared with the strategy of choosing a random expert to trust. This mapping is a bijection: a pooling method is QA pooling with respect to some proper loss function if and only if it satisfies certain natural axioms.
2.2 Online prediction with expert advice
In the subfield of prediction with expert advice, for  $T$  time steps, experts report "predictions" from a decision space  $\mathcal{D}$  (often, as in our case, the space of probability distributions over a set of outcomes). A forecaster must then output their own prediction from  $\mathcal{D}$ . Then, predictions are assessed according to a loss function. See Cesa-Bianchi and Lugosi (2006) for an survey of this field.
We are particularly interested in mixture forecasters: forecasters who, instead of choosing an expert to trust at each time step, aggregate the expert' reports. Linear mixtures, i.e. convex combinations of predictions, have been especially well-studied, generally with the goal of learning weights for the convex combination to compete with the best weights in hindsight. Standard convex optimization algorithms achieve  $O(\sqrt{T})$  regret for bounded, convex losses, but it is sometimes possible to do
better. For example, if the loss function is bounded and exp-concave, then logarithmic regret in  $T$  is attainable (Cesa-Bianchi and Lugosi, 2006, §3.3).
Portfolio theory studies optimal stock selection for maximizing return on investment, often in a no-regret setting. Cover (1991) introduced the "universal portfolio" algorithm, which, for each of  $T$  time steps, selects a portfolio (convex combination of stocks). Our setting translates naturally to Cover's: experts play the role of stocks, and the return of a stock corresponds to the probability that and expert assigns to the eventual outcome. The universal portfolio algorithm achieves logarithmic regret compared with the best portfolio in hindsight (Cover and Ordentlich, 1996); in our terms, this means that logarithmic regret (for log loss) is attainable for the linear pooling of experts. We refer the reader to (Li and Hoi, 2014) for a survey of this area.
To our knowledge, learning weights for logarithmic pooling has not been previously studied. As shown in Example 1.1, it is not possible to achieve vanishing regret if the setting is fully adversarial. We relax our setting by insisting that the experts be calibrated (see Section 3.1). To our knowledge, online prediction with expert advice has also not previously been studied under this condition.
3.1 Calibration property
We define calibration as follows. Note that the definition is in the context of our setting, i.e.  $m$  experts reporting probability distributions  $\mathbf{p}^1,\dots ,\mathbf{p}^m$  over  $n$  outcomes. We will use  $J$  to denote the random variable corresponding to the outcome, i.e.  $J$  takes values in  $[n]$ .
Definition 3.1. Consider a joint probability distribution  $\mathbb{P}$  over experts' reports and the outcome.<sup>3</sup> We say that expert  $i$  is calibrated if for all  $\mathbf{p} \in \Delta^n$  and  $j \in [n]$ , we have that
$$
\mathbb {P} \left[ J = j \mid \mathbf {p} ^ {i} = \mathbf {p} \right] = p _ {j}.
$$
That is, expert  $i$  is calibrated if the probability distribution of  $J$  conditional on their report  $\mathbf{p}^i$  is precisely  $\mathbf{p}^i$ . We say that  $\mathbb{P}$  satisfies the calibration property if every expert is calibrated.
The key intuition behind the usefulness of calibration is that if an expert claims that an outcome is very unlikely, this is strong evidence that the outcome is in fact unlikely. In Section 4.4 we will use the calibration property to show that the gradient of the loss with respect to the weight vector is likely to be relatively small at each time step.
3.2 Our online learning setting
The setting for our online learning problem is as follows. For each time step  $t \in [T]$ :
(1) Our algorithm reports a weight vector  $\mathbf{w}^t\in \Delta^m$
(2) An adversary (with knowledge of  $\mathbf{w}^t$ ) constructs a probability distribution  $\mathbb{P}$ , over reports and the outcome, that satisfies the calibration property.
(3) Reports  $\mathbf{p}^{t,1},\dots ,\mathbf{p}^{t,m}$  and an outcome  $j$  are sampled from  $\mathbb{P}$
(4) The loss of a weight vector  $\mathbf{w}$  is defined as  $L^{t}(\mathbf{w})\coloneqq -\ln (p_{j}^{*}(\mathbf{w}))$ , the log loss of the logarithmic pool of  $\mathbf{p}^{t,1},\ldots ,\mathbf{p}^{t,m}$  with weights  $\mathbf{w}$ . Our algorithm incurs loss  $L^t (\mathbf{w}^t)$ .
We define the regret of our algorithm as
$$
\operatorname {R e g r e t} = \sum_ {t = 1} ^ {T} L ^ {t} (\mathbf {w} ^ {t}) - \min  _ {\mathbf {w} \in \Delta^ {m}} \sum_ {t = 1} ^ {T} L ^ {t} (\mathbf {w}).
$$
That is, the benchmark for regret is the best weight vector in hindsight. Since our setting involves randomness, our goal is to provide an algorithm with vanishing expected regret against any adversarial strategy, where the expectation is taken over the sampling in Step (3).
<sup>3</sup>An equivalent, more mechanistic formulation views  $\mathbb{P}$  instead as a joint probability distribution over signals received by each expert (i.e. the expert's private knowledge) and the outcome. Such a probability distribution is known as an information structure, see e.g. (Bergemann and Morris, 2019). Each expert computes their belief from their signal.
Even subject to the calibration property, the adversary has a large amount of flexibility, because the adversary retains control over the correlation between different experts' forecasts. An unrestricted adversary has exponentially many degrees of freedom (as a function of the number of experts), whereas the calibration property imposes a mere linear number of constraints.<sup>4</sup>
3.3 Our algorithm
We use Algorithm 1 to accomplish this goal. The algorithm is online mirror descent (OMD) on the weight vector. Fix any  $\alpha \in (0,1 / 2)$ . We use the regularizer
$$
R (\mathbf {w}) := \frac {- 1}{\alpha} (w _ {1} ^ {\alpha} + \dots + w _ {m} ^ {\alpha}).
$$
This is known as the Tsallis entropy regularizer; see e.g. (Zimmert and Seldin, 2021) for previous use in the online learning literature. We obtain the same result (up to a multiplicative factor that depends on  $\alpha$ ) regardless of the choice of  $\alpha$ . Because no choice of  $\alpha$  stands out, we prove our result for all  $\alpha \in (0,1/2)$  simultaneously.
We will generally use a step size  $\eta = \frac{1}{\sqrt{T}\ln T} \cdot \frac{1}{12m^{(1 + \alpha) / 2}n}$ . However, in the (unlikely, as we show) event that some expert's weight becomes unusually small, we will reduce the step size.
ALGORITHM 1: OMD algorithm for learning weights for logarithmic pooling
$R(\mathbf{w}):= \frac{-1}{\alpha} (w_1^\alpha +\dots +w_m^\alpha);$ $\eta \leftarrow \frac{1}{\sqrt{T}\ln T}\cdot \frac{1}{12m^{(1 + \alpha) / 2n}};$ $\mathbf{w}^1\gets (1 / m,\ldots ,1 / m)$    
for  $t = 1$  to  $T$  do if  $\eta \leq \min_i((w_i^t)^\alpha)$  then  $\eta_t\gets \min (\eta_{t - 1},\eta);$  else  $\eta_t\gets \min (\eta_{t - 1},\min_i w_i^t);$  /\* Edge case; happens with low probability \*/ end Observe loss function  $L^t$ $\begin{array}{r}{/\ast L^{t}\mathrm{is~chown~as~described~in~Section~3.2~*}} \end{array}$  Define  $\mathbf{w}^{t + 1}$  such that  $\nabla R(\mathbf{w}^{t + 1}) = \nabla R(\mathbf{w}^t) - \eta_t\nabla L^t (\mathbf{w}^t)$    
end
In Appendix A, we prove that Algorithm 1 is efficient, taking  $O(mn)$  time per time step.
Theorem 3.2 formally states our no-regret guarantee.
Theorem 3.2. For any adversarial strategy, the expected regret<sup>5</sup> of Algorithm 1 is at most
$$
O \left(m ^ {(3 - \alpha) / 2} n \sqrt {T} \log T\right).
$$
4 Proof of no-regret guarantee
In this section, we prove Theorem 3.2.
4.1 Outline of proof
We use the following fact, which follows from a more general statement about how losses relate to their associated quasi-arithmetic pools.
This follows from the perspective of the adversary choosing an information structure from which experts' signals are drawn, as also alluded to in the previous footnote. The information structure specifies the probability of every possible combination of signals received by the experts, and thus has dimension that is exponential in the number of experts. The calibration property imposes linearly many constraints on this space.
The given asymptotics assume that  $T \gg m, n$ , i.e. ignore terms that are lower-order in  $T$ .