Title: No-Regret Learning with Unbounded Losses: The Case of Logarithmic Pooling
Venue: NeurIPS 2023
Accepted: yes
Keywords: Logarithmic pooling

online learning

no-regret learning

calibrated experts

online mirror descent

prediction with expert advice
Abstract: For each of $T$ time steps, $m$ experts report probability distributions over $n$ outcomes; we wish to learn to aggregate these forecasts in a way that attains a no-regret guarantee. We focus on the fundamental and practical aggregation method known as *logarithmic pooling* -- a weighted average of log odds -- which is in a certain sense the optimal choice of pooling method if one is interested in minimizing log loss (as we take to be our loss function). We consider the problem of learning the best set of parameters (i.e. expert weights) in an online adversarial setting.  We assume (by necessity) that the adversarial choices of outcomes and forecasts are consistent, in the sense that experts report calibrated forecasts. Imposing this constraint creates a (to our knowledge) novel semi-adversarial setting in which the adversary retains a large amount of flexibility. In this setting, we present an algorithm based on online mirror descent that learns expert weights in a way that attains $O(\sqrt{T} \log T)$ expected regret as compared with the best weights in hindsight.

No-Regret Learning with Unbounded Losses: The Case of Logarithmic Pooling

Abstract

For each of T time steps, m experts report probability distributions over n outcomes; we wish to learn to aggregate these forecasts in a way that attains a no-regret guarantee. We focus on the fundamental and practical aggregation method known as logarithmic pooling -aweighted average of log odds - which is in a certain sense the optimal choice of pooling method if one is interested in minimizing log loss (as we take to be our loss function). We consider the problem of learning the best set of parameters (i.e. expert weights) in an online adversarial setting. We assume (by necessity) that the adversarial choices of outcomes and forecasts are consistent, in the sense that experts report calibrated forecasts. Imposing this constraint creates a (to our knowledge) novel semi-adversarial setting in which the adversary retains a large amount of flexibility. In this setting, we present an algorithm based on online mirror descent that learns expert weights in a way that attains O ( √ T log T ) expected regret as compared with the best weights in hindsight.

1 Introduction

1.1 Logarithmic pooling

Suppose that m experts report probability distributions p 1 , . . . , p m ∈ ∆ n over n disjoint, exhaustive outcomes. We are interested in aggregating these distributions into a single distribution p ∗ , a task known as probabilistic opinion pooling . Perhaps the most straightforward way to do this is to take the arithmetic mean, also called the linear pool: p ∗ = 1 m ∑ m i =1 p i .

While commonly studied and frequently used, the linear pool is by no means the definitive opinion pooling method. The choice of pooling method ought to depend on context, and in particular on the loss function with respect to which forecasts are assessed. Neyman and Roughgarden (2023) showed a correspondence between proper loss functions and opinion pooling methods, which they termed quasi-arithmetic (QA) pooling with respect to a loss function. Specifically, the QA pool with respect to a loss is the forecast that guarantees the largest possible overperformance (as judged by the loss function) compared with the strategy of choosing a random expert to trust. 1

In this work we will be using log loss function. The QA pooling technique with respect to the log loss is known as logarithmic pooling . Instead of averaging the experts' forecasts, logarithmic pooling averages the experts' log odds (i.e. logits). Put otherwise, the logarithmic pool is defined by

for all events j ∈ [ n ] , where c is a normalizing constant to ensure that the probabilities add to 1 . While the linear pool is an arithmetic mean, the logarithmic pool behaves much more like a geometric mean. For instance, the logarithmic pool of (0 . 001 , 0 . 999) and (0 . 5 , 0 . 5) with equal weights is approximately (0 . 03 , 0 . 97) .

The logarithmic pool has been studied extensively because of its naturalness (Genest, 1984; Genest and Zidek, 1986; Givens and Roback, 1999; Poole and Raftery, 2000; Kascha and Ravazzolo, 2008; Rufo et al., 2012; Allard et al., 2012). Logarithmic pooling can be interpreted as averaging experts' Bayesian evidence (Neyman and Roughgarden, 2023, §A). It is also the most natural pooling method that satisfies external Bayesianality , meaning roughly that it does not matter whether a Bayesian update is applied to each expert's forecast before aggregation, or if instead the Bayesian update is applied to the pooled forecast (Genest, 1984). Logarithmic pooling can also be characterized as the pooling method that minimizes the average KL divergence to the experts' reports (Abbas, 2009). Finally, empirical work has found logarithmic pooling performs very well on real-world data (Satopää et al., 2014; Sevilla, 2021).

1.2 Logarithmic pooling with weighted experts

Forecast aggregators often assign different weights to different experts, e.g. based on each expert's level of knowledge or track record (Tetlock and Gardner, 2015). There is a principled way to include weights w 1 , . . . , w i (summing to 1 ) in the logarithmic pool, namely:

where c ( w ) is again a normalizing constant that now depends on the weights. 2 This more general notion continues to have all of the aforementioned natural properties.

The obvious question is: how does one know what these weights should be? Perhaps the most natural answer is to weight experts according to their past performance. Finding appropriate weights for experts is thus an online learning problem. This learning problem is the focus of our work.

1.3 Choosing the right benchmark

Our goal is to develop an algorithm for learning weights for logarithmic pooling in a way that achieves vanishing regret as judged by the log loss function (i.e. the loss function most closely associated with this pooling method). Within the field of online prediction with expert advice, this is a particularly challenging setting. In part, this is because the losses are potentially unbounded. However, that is not the whole story: finding weights for linear pooling, by contrast, is a well-studied problem that has been solved even in the context of log loss. On the other hand, because logarithmic pooling behaves more as a geometric than an arithmetic mean, if some expert assigns a very low probability to the eventual outcome (and the other experts do not) then the logarithmic pool will also assign a low probability, incurring a large loss. This makes the combination of logarithmic pooling with log loss particularly difficult.

We require that our algorithm not have access to the experts' forecasts when choosing weights: an algorithm that chooses weights in a way that depends on forecasts can output an essentially arbitrary function of the forecasts, and thus may do something other than learn optimal weights for logarithmic pooling. For example, suppose that m = n = 2 and an aggregator wishes to subvert our intentions and take an equally weighted linear pool of the experts' forecasts. Without knowing the experts' forecasts, this is impossible; on the other hand, if the aggregator knew that e.g. p 1 = (90% , 10%) and p 2 = (50% , 50%) , they could assign weights for logarithmic pooling so as to produce the post-hoc desired result, i.e. (70% , 30%) . We wish to disallow this.

One might suggest the following setup: at each time step, the algorithm selects weights for each expert. Subsequently, an adversary chooses each expert's forecast and the outcome, after which the algorithm and each expert incur a log loss. Unfortunately - due to the unboundedness of log loss and the behavior of logarithmic pooling - vanishing regret guarantees in this setting are impossible.

Example 1.1. Consider the case of m = n = 2 . Without loss of generality, suppose that the algorithm assigns Expert 1 a weight w ≥ 0 . 5 in the first time step. The adversary chooses reports ( e -T , 1 -e -T ) for Expert 1 and ( 1 2 , 1 2 ) for Expert 2, and for Outcome 1 to happen. The logarithmic pool of the forecasts turns out to be approximately ( e -wT , 1 -e -wT ) , so the algorithm incurs a log loss of approximately wT ≥ 0 . 5 T , compared to O (1) loss for Expert 2. On subsequent time steps, Expert 2 is perfect (assigns probability 1 to the correct outcome), so the algorithm cannot catch up.

What goes wrong in Example 1.1 is that the adversary has full control over experts' forecast and the realized outcome, and is not required to couple the two in any way. This unreasonable amount of adversarial power motivates assuming that the experts are calibrated : for example, if an expert assigns a 10% chance to an outcome, there really is a 10% chance of that outcome (conditional on the expert's information).

We propose the following setting: an adversary chooses a joint probability distribution over the experts' beliefs and the outcome - subject to the constraint that each expert is calibrated. The adversary retains full control over correlations between forecasts and outcomes, subject to the calibration property. Subsequently, nature randomly samples each expert's belief and the eventual outcome from the distribution. In this setting, we seek to prove upper bounds on the expected value of our algorithm's regret.

Why impose this constraint, instead of a different one? Our reasons are twofold: theoretical and empirical. From a theoretical standpoint, the assumption that experts are calibrated is natural because experts who form Bayesian rational beliefs based on evidence will be calibrated, regardless of how much or how little evidence they have. The assumption is also motivated if we model experts as learners rather than Bayesian agents: even if a forecaster starts out completely uninformed, they can quickly become calibrated in a domain simply by observing the frequency of events (Foster and Vohra, 1997).

Second, recent work has shown that modern deep neural networks are calibrated when trained on a proper loss function such as log loss. This is true for a variety of tasks, including image classification (Minderer et al., 2021; Hendrycks et al., 2020) and language modeling (Kadavath et al., 2022; Desai and Durrett, 2020; OpenAI, 2023); see (Blasiok et al., 2023) for a review of the literature. We may wish to use an ensemble of off-the-shelf neural networks for some prediction or classification task. If we trust these networks to be calibrated (as suggested by recent work), then we may wish to learn to ensemble these experts (models) in a way that has strong worst-case theoretical guarantees under the calibration assumption.

Logarithmic pooling is particularly sensible in the context of calibrated experts because it takes confident forecasts 'more seriously" as compared with linear pooling (simple averaging). If Expert 1 reports probability distribution (0 . 1% , 99 . 9%) over two outcomes and Expert 2 reports (50% , 50%) , then the logarithmic pool (with equal weights) is approximately (3% , 97%) , as compared with a linear pool of roughly (25% , 75%) . If Expert 1 is calibrated (as we are assuming), then the (0 . 1% , 99 . 9%) forecast entails very strong evidence in favor of Outcome 2 over Outcome 1. Meanwhile, Expert 2's forecast gives no evidence either way. Thus, it is sensible for the aggregate to point to Outcome 2 over Outcome 1 with a fair amount of confidence.

As another example, suppose that Expert 1 reports (0 . 04% , 49 . 98% , 49 . 98%) and Expert 2 reports (49 . 98% , 0 . 04% , 49 . 98%) (a natural interpretation: Expert 1 found strong evidence against Outcome 1 and Expert 2 found strong evidence against Outcome 2). If both experts are calibrated, a sensible aggregate should arguably assign nearly all probability to Outcome 3. Logarithmic pooling returns roughly (2 . 7% , 2 . 7% , 94 . 6%) , which (unlike linear pooling) accomplishes this.

Since we are allowing our algorithm to learn the optimal logarithmic pool, perhaps there is hope to compete not just with the best expert in hindsight, but the optimally weighted logarithmic pool of experts in hindsight. We will aim to compete with this stronger benchmark.

This paper demonstrates that the 'calibrated experts" condition allows us to prove regret bounds when no such bounds are possible for an unrestricted adversary. While that is our primary motivation, the relaxation may also be of independent interest. For example, even in settings where vanishing regret is attainable in the presence of an unrestricted adversary, even stronger regret bounds might be achievable if calibration is assumed.

1.4 Our main result

Is vanishing regret possible in our setting? Our main result is that the answer is yes. We exhibit an algorithm that attains expected regret that scales as O ( √ T log T ) with the number of time steps T . Our algorithm uses online mirror descent (OMD) with the Tsallis entropy regularizer R ( w ) = -1 α ( w α 1 + · · · + w α m ) and step size η ≈ 1 √ T ln T , where any choice of α ∈ (0 , 1 / 2) attains the regret bound.

Our proof has two key ideas. One is to use the calibration property to show that the gradient of loss with respect to the weight vector is likely to be small (Section 4.4). This is how we leverage the calibration property to turn an intractable setting into one where - despite the unboundedness of log loss and the behavior of logarithmic pooling - there is hope for vanishing regret.

The other key idea (Section 4.3) involves keeping track of a function that, roughly speaking, reflects how much 'regret potential" the algorithm has. We show that if the aforementioned gradient updates are indeed small, then this potential function decreases in value at each time step. This allows us to upper bound the algorithm's regret by the initial value of the potential function.

This potential argument is an important component of the proof. A naïve analysis might seek to use our bounds on the gradient steps to myopically bound the contribution to regret at each time step. Such an analysis, however, does not achieve our O ( √ T log T ) regret bound. In particular, an adversary can force a large accumulation of regret if some experts' weights are very small (specifically by making the experts with small weights more informed than those with large weights) - but by doing so, the small weights increase and the adversary 'spends down" its potential. Tracking this potential allows us to take this nuance into consideration, improving our bound.

We extend our main result by showing that the result holds even if experts are only approximately calibrated: so long as no expert understates the probability of an outcome by more than a constant factor, we still attain the same regret bound (see Corollary A.8). We also show in Appendix B that no OMDalgorithm with a constant step size can attain expected regret better than Ω( √ T ) .

2 Related work

2.1 Probabilistic opinion pooling

There has been substantial mathematical work on probabilistic opinion pooling (i.e. forecast aggregation) since the 1980s. One line of work is axiomatic in nature: motivating opinion pooling methods by describing axioms that they satisfy. For example, logarithmic pooling satisfies unanimity preservation and external Bayesianality (Genest, 1984). There has also been work on Bayesian approaches to pooling, e.g. under the assumption that experts' signals are drawn from some parameterized class of distributions (Winkler, 1981; Lichtendahl et al., 2017).

Neyman and Roughgarden (2023) show that every proper loss function has an associated pooling method (the QA pool with respect to the loss function), which is the forecast that guarantees the largest possible overperformance (as judged by the loss function) compared with the strategy of choosing a random expert to trust. This mapping is a bijection: a pooling method is QA pooling with respect to some proper loss function if and only if it satisfies certain natural axioms.

2.2 Online prediction with expert advice

In the subfield of prediction with expert advice , for T time steps, experts report 'predictions" from a decision space D (often, as in our case, the space of probability distributions over a set of outcomes). A forecaster must then output their own prediction from D . Then, predictions are assessed according to a loss function. See Cesa-Bianchi and Lugosi (2006) for an survey of this field.

We are particularly interested in mixture forecasters : forecasters who, instead of choosing an expert to trust at each time step, aggregate the expert' reports. Linear mixtures, i.e. convex combinations of predictions, have been especially well-studied, generally with the goal of learning weights for the convex combination to compete with the best weights in hindsight. Standard convex optimization algorithms achieve O ( √ T ) regret for bounded, convex losses, but it is sometimes possible to do

better. For example, if the loss function is bounded and exp-concave, then logarithmic regret in T is attainable (Cesa-Bianchi and Lugosi, 2006, §3.3).

Portfolio theory studies optimal stock selection for maximizing return on investment, often in a no-regret setting. Cover (1991) introduced the 'universal portfolio" algorithm, which, for each of T time steps, selects a portfolio (convex combination of stocks). Our setting translates naturally to Cover's: experts play the role of stocks, and the return of a stock corresponds to the probability that and expert assigns to the eventual outcome. The universal portfolio algorithm achieves logarithmic regret compared with the best portfolio in hindsight (Cover and Ordentlich, 1996); in our terms, this means that logarithmic regret (for log loss) is attainable for the linear pooling of experts. We refer the reader to (Li and Hoi, 2014) for a survey of this area.

To our knowledge, learning weights for logarithmic pooling has not been previously studied. As shown in Example 1.1, it is not possible to achieve vanishing regret if the setting is fully adversarial. We relax our setting by insisting that the experts be calibrated (see Section 3.1). To our knowledge, online prediction with expert advice has also not previously been studied under this condition.

3 Preliminaries

3.1 Calibration property

We define calibration as follows. Note that the definition is in the context of our setting, i.e. m experts reporting probability distributions p 1 , . . . , p m over n outcomes. We will use J to denote the random variable corresponding to the outcome, i.e. J takes values in [ n ] .

Definition 3.1. Consider a joint probability distribution P over experts' reports and the outcome. 3 We say that expert i is calibrated if for all p ∈ ∆ n and j ∈ [ n ] , we have that

That is, expert i is calibrated if the probability distribution of J conditional on their report p i is precisely p i . We say that P satisfies the calibration property if every expert is calibrated.

The key intuition behind the usefulness of calibration is that if an expert claims that an outcome is very unlikely, this is strong evidence that the outcome is in fact unlikely. In Section 4.4 we will use the calibration property to show that the gradient of the loss with respect to the weight vector is likely to be relatively small at each time step.

3.2 Our online learning setting

The setting for our online learning problem is as follows. For each time step t ∈ [ T ] :

Our algorithm reports a weight vector w t ∈ ∆ m .

An adversary (with knowledge of w t ) constructs a probability distribution P , over reports and the outcome, that satisfies the calibration property.

Reports p t, 1 , . . . , p t,m and an outcome j are sampled from P .

The loss of a weight vector w is defined as L t ( w ) := -ln( p ∗ j ( w )) , the log loss of the logarithmic pool of p t, 1 , . . . , p t,m with weights w . Our algorithm incurs loss L t ( w t ) .

We define the regret of our algorithm as

That is, the benchmark for regret is the best weight vector in hindsight. Since our setting involves randomness, our goal is to provide an algorithm with vanishing expected regret against any adversarial strategy, where the expectation is taken over the sampling in Step (3).

Even subject to the calibration property, the adversary has a large amount of flexibility, because the adversary retains control over the correlation between different experts' forecasts. An unrestricted adversary has exponentially many degrees of freedom (as a function of the number of experts), whereas the calibration property imposes a mere linear number of constraints. 4

3.3 Our algorithm

We use Algorithm 1 to accomplish this goal. The algorithm is online mirror descent (OMD) on the weight vector. Fix any α ∈ (0 , 1 / 2) . We use the regularizer

This is known as the Tsallis entropy regularizer; see e.g. (Zimmert and Seldin, 2021) for previous use in the online learning literature. We obtain the same result (up to a multiplicative factor that depends on α ) regardless of the choice of α . Because no choice of α stands out, we prove our result for all α ∈ (0 , 1 / 2) simultaneously.

We will generally use a step size η = 1 √ T ln T · 1 12 m (1+ α ) / 2 n . However, in the (unlikely, as we show) event that some expert's weight becomes unusually small, we will reduce the step size.

ALGORITHM 1: OMDalgorithm for learning weights for logarithmic pooling

In Appendix A, we prove that Algorithm 1 is efficient, taking O ( mn ) time per time step.

Theorem 3.2 formally states our no-regret guarantee.

Theorem 3.2. For any adversarial strategy, the expected regret 5 of Algorithm 1 is at most

4 Proof of no-regret guarantee

In this section, we prove Theorem 3.2.

4.1 Outline of proof

We use the following fact, which follows from a more general statement about how losses relate to their associated quasi-arithmetic pools.

Proposition 4.1 (Follows from (Neyman and Roughgarden, 2023, Theorem 5.1)) . Let p 1 , . . . , p m be forecasts over n outcomes, j ∈ [ n ] be an outcome, and w ∈ ∆ m be a weight vector. Let p ∗ ( w ) be the logarithmic pool of the forecasts with weight vector w and let L ( w ) := -ln( p ∗ j ( w )) be the log loss of p ∗ ( w ) if Outcome j is realized. Then L is a convex function.

In particular, all of our loss functions L t are convex, which means that standard regret bounds apply. In particular, to bound the expected regret of Algorithm 1, we will use a well-known regret bound for follow the regularized leader (FTRL) with linearized losses (Hazan, 2021, Lemma 5.3), which in our case is equivalent to OMD. 6

Lemma 4.2 (Follows from (Hazan, 2021, Lemma 5.3)) . If η t = η for all t , the regret of Algorithm 1 is at most

Informally, this bound means that if the vectors ∇ L t ( w t ) are small in magnitude, our regret is also small. Conversely, if some ∇ L t ( w t ) is large, this may be bad for our regret bound. We expect the gradient of the loss to be large if some expert is very wrong (assigns a very low probability to the correct outcome), since the loss would then be steeply increasing as a function of that expert's weight. Fortunately, the calibration property guarantees this to be unlikely. Specifically, we define the small gradient assumption as follows.

Definition 4.3. Define γ := 12 n ln T . The small gradient assumption holds for a particular run of Algorithm 1 if for every t ∈ [ T ] and i ∈ [ m ] , we have

where ∂ i denotes the partial derivative with respect to the i -th weight. 7

In Section 4.4, we prove that the small gradient assumption is very likely to hold. This is a key conceptual step in our proof, as it is where we leverage the calibration property to prove bounds that ultimately let us bound our algorithm's regret. We then use the low likelihood of the small gradient assumption failing in order to bound the contribution to the expected regret from the case where the assumption fails to hold.

In Sections 4.2 and 4.3, we bound regret under the condition that the small gradient assumption holds. We show that under the assumption, for all i, t we have ( w t i ) α ≥ η . Consequently, η = 1 √ T ln T · 1 12 m (1+ α ) / 2 n at all time steps, so we can apply Lemma 4.2. The first term in the bound is O (1 /η ) = O ( √ T log T ) . As for the summation term, we upper bound it by keeping track of the following quantity:

The first term is exactly the summation in Lemma 4.2 up through step t . The 19 m 2 γ 2 η is something akin to an upper bound on the value of L t ( w t ) · ( w t -w t +1 ) at a given time step (times T -t remaining time steps). This upper bound is not strict: in particular, large summands are possible when some weights are small (because of the fact that the lower bound in the small gradient assumption is inversely proportional to w t i ). However, attaining a large summand requires these small weights to increase, thus 'spending potential" for future large summands. The last term keeps track of this potential.

We show that under the small gradient assumption, φ ( t ) necessarily decreases with t . This argument, which we give in Section 4.3, is another key conceptual step, and is arguably the heart of the proof. Since φ ( T ) is equal to the summation term in Lemma 4.2 (plus a positive number), and φ (0) ≥ φ ( T ) , the summation term is less than or equal to φ (0) , which is at most O ( m (3 -α ) / 2 √ T log T ) . This completes the proof.

4.2 Bounds on w t under the small gradient assumption

In this section, we state bounds on expert weights and how quickly they change from one time step to the next, conditional on the small gradient assumption. We use the following lemma, whose proof we defer to Appendix A.

Lemma 4.4. Consider a particular run of Algorithm 1. Let ζ be a constant such that -ζ w t i ≤ ∂ i L t ( w t ) ≤ ζ for all i, t . Then for every i, t , we have

Furthermore, if η t ζ ≤ (1 -α ) 2 ( w t i ) α for all i , then for every i we have

Intuitively, this result states that when the gradient update is small, w t +1 i is not too different from w t i . Note that the lower bound -ζ w t i that we place on the gradient is not a simple Lipschitz bound but instead depends on w t i ; this makes the bounds in Lemma 4.4 less straightforward to prove. In particular, we bound each component w t i individually, using bounds on the gradient of the loss for all other components and convexity arguments.

Lemma 4.4 can be translated into bounds on each w t i and on the change between w t i and w t +1 i :

Corollary 4.5. Under the small gradient assumption, for sufficiently large T we have for all i ∈ [ m ] , t ∈ [ T ] that:

We defer the proof of Corollary 4.5 to Appendix A. The key idea for (#1) is to proceed by induction on t on the two sub-statements in parallel: so long as ( w t i ) α ≥ 4 ηγ , we may use the second part of Lemma 4.4 with ζ = γ to bound ( w t +1 i ) α -1 in terms of ( w t i ) α -1 , which we can leverage to prove both sub-statements for t +1 . (#2) then follows from (#1) by routine (though nontrivial) algebra.

Armed with the bounds of Corollary 4.5, we are now able to show that under the small gradient assumption, Algorithm 1 attains vanishing regret.

4.3 Bounding regret under the small gradient assumption

Assume the small gradient assumption. Note that since 4 γ ≥ 1 , by Corollary 4.5 (#1) we have that η t = η for all t . This means that we may apply the bound in Lemma 4.2, and in particular we have

It remains to bound the summation component of the regret bound in Lemma 4.2. To do so, we prove the following lemma, which we alluded to in Section 4.1 as the heart of the proof of Theorem 3.2.

Lemma 4.6. For t ∈ { 0 , 1 , . . . , T } , let

Under the small gradient assumption, for sufficiently large T , φ ( t ) is a decreasing function of t .

To prove this claim, consider a particular t ∈ [ T ] . We may write

and we wish to show that this quantity is negative. In fact, we show that the contribution from every i ∈ [ m ] is negative. The key idea is to consider two cases: w t +1 i ≤ w t i and w t +1 i ≥ w t i . In each case, Corollary 4.5 provides an upper bound on the magnitude of the difference between w t i and w t +1 i . If w t +1 i ≤ w t i then the first and third terms in the summation are positive but small, and are dominated by the middle term. If w t +1 i ≥ w t i then the first term may be quite large, because of the asymmetric bound in the small gradient assumption (and the consequently asymmetric bound in Corollary 4.5). However, in this case the contribution of the third term is very negative, enough to make the overall expression negative. In this sense, the third term keeps track of unspent potential for future regret, which gets 'spent down" whenever a large amount of regret is realized (as measured by the first term).

We now prove formally that each term of the summation in Equation 1 is negative.

Proof. First assume that w t i -w t +1 i ≤ 0 . Note that by combining (#1) and (#2) of Corollary 4.5, we have

By the small gradient assumption we have that

On the other hand, we have

for T large enough. (Here we use that w t +1 i -w t i ≤ 8 w t i and that ln(1 + x ) ≥ x 4 for x ≤ 8 .) Thus, the first and third terms in Equation 1 are net negative; meanwhile, the second term is also negative, so the expression is negative.

Now assume that w t i -w t +1 i ≥ 0 . Again by the small gradient assumption, we have that

and

for T sufficiently large, where in the last step we use that ln(1 -x ) ≥ -4 3 x for x > 0 sufficiently small (and we have lim T →∞ 3 mηγ = 0 ). Since 16 + 3 ≤ 19 , the right-hand side of Equation 1 is negative. This concludes the proof.

Corollary 4.7. For sufficiently large T , under the small gradient assumption, the regret of Algorithm 1 is at most ( 240 + 12 α ) m (3 -α ) / 2 n √ T ln T .

Proof. We have already bounded the first term in the regret bound in Lemma 4.2. It remains only to bound the second term. This term is exactly equal to φ ( T ) + 4 γ ∑ m i =1 ln w T +1 i ≤ φ ( T ) , and φ ( T ) ≤ φ (0) , by Lemma 4.6. We have

for sufficiently large T . Plugging in γ = 12 n ln T and η = 1 √ T ln T · 1 12 m (1+ α ) / 2 n concludes the proof.

4.4 The case where the small gradient assumption fails

It remains to consider the case in which the small gradient assumption does not hold. This part of the proof consists primarily of technical lemmas, which we defer to Appendix A. The key lemma is a bound on the probability that the small gradient assumption fails by a given margin:

Lemma 4.8. For any weight vector w , i ∈ [ m ] , and ζ ≥ 0 , we have that

and

Note that plugging in ζ = γ yields a bound of mT ( ne -γ + mn 2 e -γ/n ) on the probability that the small gradient assumption fails to hold. (Since γ = 12 n ln T , this quantity is on the order of T -11 .)

The proof of Lemma 4.8 is the only part of the proof of Theorem 3.2 that uses the calibration property. While we defer the full proof to Appendix A, we highlight how the calibration property is used to prove Equation 2. In brief, it is straightforward to show that ∂ i L ( w ) ≤ -ln p i j , where J is the random variable corresponding to the realized outcome. 8 Therefore, we have

where the last step follows by the calibration property, thus proving Equation 2.

Combining Lemma 4.8 with an analysis of our algorithm using the standard regret bound for online mirror descent (Orabona, 2021, Theorem 6.8) gives us the following result as a corollary.

Corollary 4.9. The expected total regret of our algorithm conditional on the small gradient assumption not holding, times the probability of this event, is at most ˜ O ( T (5 -α ) / (1 -α ) -10 ) .

It follows that the contribution to expected regret from the case that the small gradient assumption does not hold is ˜ O ( T -1 ) , which is negligible. Together with Corollary 4.7 (which bounds regret under the small gradient assumption), this proves Theorem 3.2. As a matter of fact, Theorem 3.2 holds even if experts are only approximately calibrated. As with other details of this section, we refer the reader to Appendix A.

5 Conclusion

In this work we have considered the problem of learning optimal weights for the logarithmic pooling of expert forecasts. It quickly became apparent that under the usual fully adversarial setup, attaining vanishing regret is impossible (Example 1.1). We chose to relax the environment by imposing the constraint on the adversary that experts must be calibrated. Put otherwise, the adversary is allowed to choose a joint probability distribution over the experts' reports and the outcome however it wants to, so long as the experts' reports are calibrated, after which the realized reports and outcome are selected at random from this distribution. To our knowledge, this setting is a novel contribution to the literature on prediction with expert advice. The setting may be of independent interest: we have demonstrated that no-regret bounds are possible in this setting when they are otherwise impossible, and it seems plausible that even in settings where no-regret bounds are attainable in a fully adversarial setting, the calibration property allows for stronger results.

Another important direction for future work is learning weights for other pooling methods. In particular, because of the close connection between a proper loss function and its associated quasiarithmetic pool, it is natural to ask for which proper loss functions it is possible to achieve vanishing regret when learning weights for quasi-arithmetic pooling with respect to the loss function. (Neyman and Roughgarden, 2023, §5) showed that the loss of a quasi-arithmetic pool is convex in the experts' weights, and therefore the usual no-regret algorithms (e.g. online gradient descent) guarantee O ( √ T ) regret - so long as the loss function is bounded. In this work, we extended their result to the log loss (with the associated QA pooling method, i.e. logarithmic pooling.) Extending our techniques to other unbounded loss functions is a promising avenue for future exploration.

Acknowledgments and Disclosure of Funding

Funding in direct support of this work: NSF grant DGE-2036197; NSF grant CCF-2006737; and ARO grant W911NF1910294.

We would like to thank Christian Kroer, Rafael Frongillo, and Bo Waggoner for discussion.

References

Ali E. Abbas. 2009. A Kullback-Leibler View of Linear and Log-Linear Pools. Decision Analysis 6, 1 (3 2009), 25-37. ❤/a116/a116♣/a115✿✴✴❞♦✐✳♦/a114❣✴✶✵✳✶✷✽✼✴❞❡❝❛✳✶✵✽✵✳✵✶✸✸

D Allard, A Comunian, and Philippe Renard. 2012. Probability aggregation methods in geoscience. Mathematical Geosciences 44, 5 (2012), 545-581.

Dirk Bergemann and Stephen Morris. 2019. Information Design: A Unified Perspective. Journal of Economic Literature 57, 1 (3 2019), 44-95. ❤/a116/a116♣/a115✿✴✴❞♦✐✳♦/a114❣✴✶✵✳✶✷✺✼✴❥❡❧✳✷✵✶✽✶✹✽✾

Jaroslaw Blasiok, Parikshit Gopalan, Lunjia Hu, and Preetum Nakkiran. 2023. When Does Optimizing a Proper Loss Yield Calibration? CoRR abs/2305.18764 (2023). ❤/a116/a116♣/a115✿✴✴❞♦✐✳♦/a114❣✴✶✵✳✹✽✺✺✵✴ ❛/a114❳✐✈✳✷✸✵✺✳✶✽✼✻✹ arXiv:2305.18764

Nicolò Cesa-Bianchi and Gábor Lugosi. 2006. Prediction, Learning, and Games . ❤/a116/a116♣/a115✿✴✴❞♦✐✳ ♦/a114❣✴✶✵✳✶✵✶✼✴❈❇❖✾✼✽✵✺✶✶✺✹✻✾✷✶

Thomas M. Cover. 1991. Universal Portfolios. Mathematical Finance 1, 1 (1991), 1-29. ❤/a116/a116♣/a115✿✴✴❞♦✐✳♦/a114❣✴✶✵✳✶✶✶✶✴❥✳✶✹✻✼✲✾✾✻✺✳✶✾✾✶✳/a116❜✵✵✵✵✷✳① arXiv:https://onlinelibrary.wiley.com/doi/pdf/10.1111/j.1467-9965.1991.tb00002.x

Thomas M. Cover and Erik Ordentlich. 1996. Universal portfolios with side information. IEEE Trans. Inf. Theory 42, 2 (1996), 348-363. ❤/a116/a116♣/a115✿✴✴❞♦✐✳♦/a114❣✴✶✵✳✶✶✵✾✴✶✽✳✹✽✺✼✵✽

Shrey Desai and Greg Durrett. 2020. Calibration of Pre-trained Transformers. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020 , Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (Eds.). Association for Computational Linguistics, 295-302. ❤/a116/a116♣/a115✿✴✴❞♦✐✳♦/a114❣✴✶✵✳✶✽✻✺✸✴✈✶✴✷✵✷✵✳ ❡♠♥❧♣✲♠❛✐♥✳✷✶

Dean P. Foster and Rakesh V. Vohra. 1997. Calibrated Learning and Correlated Equilibrium. Games and Economic Behavior 21, 1 (1997), 40-55. ❤/a116/a116♣/a115✿✴✴❞♦✐✳♦/a114❣✴✶✵✳✶✵✵✻✴❣❛♠❡✳✶✾✾✼✳✵✺✾✺

Christian Genest. 1984. A Characterization Theorem for Externally Bayesian Groups. Ann. Statist. 12, 3 (09 1984), 1100-1105. ❤/a116/a116♣/a115✿✴✴❞♦✐✳♦/a114❣✴✶✵✳✶✷✶✹✴❛♦/a115✴✶✶✼✻✸✹✻✼✷✻

Christian Genest and James V. Zidek. 1986. Combining Probability Distributions: A Critique and an Annotated Bibliography. Statist. Sci. 1, 1 (1986), 114-135. ❤/a116/a116♣✿✴✴✇✇✇✳❥/a115/a116♦/a114✳♦/a114❣✴/a115/a116❛❜❧❡✴ ✷✷✹✺✺✶✵

Geof H. Givens and Paul J. Roback. 1999. Logarithmic Pooling of Priors Linked by a Deterministic Simulation Model. Journal of Computational and Graphical Statistics 8, 3 (1999), 452-478. ❤/a116/a116♣✿✴✴✇✇✇✳❥/a115/a116♦/a114✳♦/a114❣✴/a115/a116❛❜❧❡✴✶✸✾✵✽✻✾

Elad Hazan. 2021. Introduction to Online Convex Optimization. CoRR abs/1909.05207 (2021). arXiv:1909.05207 ❤/a116/a116♣✿✴✴❛/a114①✐✈✳♦/a114❣✴❛❜/a115✴✶✾✵✾✳✵✺✷✵✼

Dan Hendrycks, Norman Mu, Ekin Dogus Cubuk, Barret Zoph, Justin Gilmer, and Balaji Lakshminarayanan. 2020. AugMix: A Simple Data Processing Method to Improve Robustness and Uncertainty. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020 . OpenReview.net. ❤/a116/a116♣/a115✿✴✴♦♣❡♥/a114❡✈✐❡✇✳♥❡/a116✴❢♦/a114✉♠❄✐❞❂ ❙✶❣♠/a114①❍❋✈❇

Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, Scott Johnston, Sheer El Showk, Andy Jones, Nelson Elhage, Tristan Hume, Anna Chen, Yuntao Bai, Sam Bowman, Stanislav Fort, Deep Ganguli, Danny Hernandez, Josh Jacobson, Jackson Kernion, Shauna Kravec, Liane Lovitt, Kamal Ndousse, Catherine Olsson, Sam Ringer, Dario Amodei, Tom Brown, Jack Clark, Nicholas Joseph, Ben Mann, Sam McCandlish, Chris Olah, and Jared Kaplan. 2022. Language Models (Mostly) Know What They Know. CoRR abs/2207.05221 (2022). ❤/a116/a116♣/a115✿✴✴❞♦✐✳♦/a114❣✴ ✶✵✳✹✽✺✺✵✴❛/a114❳✐✈✳✷✷✵✼✳✵✺✷✷✶ arXiv:2207.05221

Christian Kascha and Francesco Ravazzolo. 2008. Combining inflation density forecasts . Working Paper 2008/22. Norges Bank. ❤/a116/a116♣/a115✿✴✴✐❞❡❛/a115✳/a114❡♣❡❝✳♦/a114❣✴♣✴❜♥♦✴✇♦/a114♣❛♣✴✷✵✵✽❴✷✷✳❤/a116♠❧

Bin Li and Steven C. H. Hoi. 2014. Online portfolio selection: A survey. ACM Comput. Surv. 46, 3 (2014), 35:1-35:36. ❤/a116/a116♣/a115✿✴✴❞♦✐✳♦/a114❣✴✶✵✳✶✶✹✺✴✷✺✶✷✾✻✷

Kenneth C. Lichtendahl, Yael Grushka-Cockayne, Victor Richmond R. Jose, and Robert L. Winkler. 2017. Extremizing and Anti-Extremizing in Bayesian Ensembles of Binary-Event Forecasts. ERN: Forecasting Techniques (Topic) (2017).

Matthias Minderer, Josip Djolonga, Rob Romijnders, Frances Hubis, Xiaohua Zhai, Neil Houlsby, Dustin Tran, and Mario Lucic. 2021. Revisiting the Calibration of Modern Neural Networks. In Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual , Marc'Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan (Eds.). 15682-15694. ❤/a116/a116♣/a115✿✴✴♣/a114♦❝❡❡❞✐♥❣/a115✳♥❡✉/a114✐♣/a115✳❝❝✴♣❛♣❡/a114✴✷✵✷✶✴❤❛/a115❤✴ ✽✹✷✵❞✸✺✾✹✵✹✵✷✹✺✻✼❜✺❛❡❢❞❛✶✷✸✶❛❢✷✹✲❆❜/a115/a116/a114❛❝/a116✳❤/a116♠❧

Eric Neyman and Tim Roughgarden. 2023. From Proper Scoring Rules to Max-Min Optimal Forecast Aggregation. Operations Research (Jan. 2023). ❤/a116/a116♣/a115✿✴✴❞♦✐✳♦/a114❣✴✶✵✳✶✷✽✼✴♦♣/a114❡✳✷✵✷✷✳✷✹✶✹

OpenAI. 2023. GPT-4 Technical Report. CoRR abs/2303.08774 (2023). ❤/a116/a116♣/a115✿✴✴❞♦✐✳♦/a114❣✴✶✵✳ ✹✽✺✺✵✴❛/a114❳✐✈✳✷✸✵✸✳✵✽✼✼✹ arXiv:2303.08774

Francesco Orabona. 2021. A Modern Introduction to Online Learning. CoRR abs/1912.13213 (2021). arXiv:1912.13213 ❤/a116/a116♣✿✴✴❛/a114①✐✈✳♦/a114❣✴❛❜/a115✴✶✾✶✷✳✶✸✷✶✸

David Poole and Adrian E. Raftery. 2000. Inference for Deterministic Simulation Models: The Bayesian Melding Approach. J. Amer. Statist. Assoc. 95, 452 (2000), 1244-1255. ❤/a116/a116♣/a115✿ ✴✴❞♦✐✳♦/a114❣✴✶✵✳✶✵✽✵✴✵✶✻✷✶✹✺✾✳✷✵✵✵✳✶✵✹✼✹✸✷✹

María Rufo, Jacinto Martín, and C. Pérez. 2012. Log-Linear Pool to Combine Prior Distributions: A Suggestion for a Calibration-Based Approach. Bayesian Analysis 7 (06 2012), 1-28. ❤/a116/a116♣/a115✿ ✴✴❞♦✐✳♦/a114❣✴✶✵✳✶✷✶✹✴✶✷✲❇❆✼✶✹

Ville Satopää, Jonathan Baron, Dean Foster, Barbara Mellers, Philip Tetlock, and Lyle Ungar. 2014. Combining multiple probability predictions using a simple logit model. International Journal of Forecasting 30 (04 2014), 344-356. ❤/a116/a116♣/a115✿✴✴❞♦✐✳♦/a114❣✴✶✵✳✶✵✶✻✴❥✳✐❥❢♦/a114❡❝❛/a115/a116✳✷✵✶✸✳✵✾✳ ✵✵✾

Jaime Sevilla. 2021. When pooling forecasts, use the geometric mean of odds. ❤/a116/a116♣/a115✿✴✴❢♦/a114✉♠✳❡❢❢❡❝/a116✐✈❡❛❧/a116/a114✉✐/a115♠✳♦/a114❣✴♣♦/a115/a116/a115✴/a115▼❥❝❥♥♥♣♦❆◗❈❝❡❞▲✷✴ ✇❤❡♥✲♣♦♦❧✐♥❣✲❢♦/a114❡❝❛/a115/a116/a115✲✉/a115❡✲/a116❤❡✲❣❡♦♠❡/a116/a114✐❝✲♠❡❛♥✲♦❢✲♦❞❞/a115

Shai Shalev-Shwartz. 2007. Online Learning: Theory, Algorithms, and Applications . Ph. D. Dissertation.

Tetlock and D. Gardner. 2015. Superforecasting: The Art and Science of Prediction . Random House. ❤/a116/a116♣/a115✿✴✴❜♦♦❦/a115✳❣♦♦❣❧❡✳❝♦♠✴❜♦♦❦/a115❄✐❞❂✹✺❖♠❈◗❆❆◗❇❆❏

Robert L. Winkler. 1981. Combining Probability Distributions from Dependent Information Sources. Management Science 27, 4 (1981), 479-488. ❤/a116/a116♣✿✴✴✇✇✇✳❥/a115/a116♦/a114✳♦/a114❣✴/a115/a116❛❜❧❡✴✷✻✸✶✸✸✽

Julian Zimmert and Yevgeny Seldin. 2021. Tsallis-INF: An Optimal Algorithm for Stochastic and Adversarial Bandits. J. Mach. Learn. Res. 22 (2021), 28:1-28:49. ❤/a116/a116♣✿✴✴❥♠❧/a114✳♦/a114❣✴♣❛♣❡/a114/a115✴ ✈✷✷✴✶✾✲✼✺✸✳❤/a116♠❧

A Omitted Proofs

Efficiency of Algorithm 1 The only nontrivial step of the algorithm is finding the weight vector satisfying the equation on the last line of the algorithm. To do so, it is first necessary to compute the gradient of the loss. This gradient, given by Equation 8 below, can clearly be computed in time O ( mn ) . After that, it is necessary to find the weight vector w t +1 that satisfies the equation on the last line. This can be done efficiently through local search: the goal amounts to find weights ( w 1 , . . . , w m ) such that the vector ( w α -1 1 , . . . , w α -1 m ) is equal to a target vector (call it v ) plus a constant c times the all-ones vector. That is, we need to simultaneously solve the equation w α -1 i = v i + c for all i , with weights that add to 1 . (Here, the v i are knowns and the w i and c are unknowns.)

We start by finding c , by solving the equation ∑ i ( v i + c ) 1 / ( α -1) = 1 . Such a c exists because the left-hand side of this equation is continuous and monotone decreasing, going from infinity to zero as c ranges from -min i v i to infinity. We can solve for c very efficiently, e.g. with Newton's method. Once we know c , we know each w i : we have w i = ( v i + c ) 1 / ( α -1) . Thus, Algorithm 1 takes O ( mn ) time.

Proof of Lemma 4.4. Fix any t . Note that since the space of possible weights is ∆ m , it is most natural to think of ∇ R as a function from ∆ m to R m /T ( 1 m ) , i.e. R m modulo translation by the all-ones vector (which is orthogonal to ∆ m in R m ). That is, ∇ R ( w ) = -(( w 1 ) α -1 , . . . , ( w m ) α -1 ) , where this vector may be thought of as modulo translation by the all-ones vector. Nevertheless, we find it convenient to define ∂ i R ( w ) := -( w i ) α -1 . We define ∂ i L t ( w ) similarly (see Section 4.4).

Define h ∈ R m to have coordinates h i := ∂ i R ( w t ) -η t ∂ i L t ( w t ) . Per the update rule, we have that h i ≡ R ( w t +1 ) mod T ( 1 m ) . We have

Applying the first and last claims of Lemma A.1 (below) with a = α -1 , v = w t , κ = η t ζ , and g = -h , we have that there exists a unique c ∈ R such that

and in fact that -η t ζ ≤ c ≤ mη t ζ . (Equation 4 is relevant here because it is equivalent to the v a i -κ w t i ≤ g i ≤ v a i + κ conditions in Lemma A.1. This is also where we use that η t ζ ≤ (1 -α ) 2 ( w t i ) α , which is equivalent to κ ≤ a 2 v a +1 i .) The significance of this fact is that ( -h i + c ) 1 / ( α -1) is precisely w t +1 i , since (in R m ) we have that ( ∂ i R ( w t +1 ) , . . . , ∂ i R ( w t +1 ) ) = h -c · 1 for some c , and in particular this c must be such that ∑ i w t +1 i = 1 . In particular, this means that for all i , we have

Here, the inequality comes from the left inequality of Equation 4 and the fact that c ≤ 1 min k w k η t ζ . If we also have that η t ζ ≤ (1 -α ) 2 ( w t i ) α , then the last claim of Lemma A.1 gives us that

Similarly, we have

Lemma A.1. Let -1 < a < 0 and g ∈ R m . There is a unique c ∈ R such that ∑ i ( g i + c ) 1 /a = 1 . Furthermore, let v ∈ ∆ m and κ ≥ 0 . Then:

If g i ≤ v a i + κ for all i , then c ≥ -κ .

If g i ≥ v a i -κ v i for all i , then c ≤ κ min i v i .

-And if, furthermore, κ ≤ a 2 v a +1 i for all i , then c ≤ mκ .

Proof. Observe that ∑ i ( g i + c ) 1 /a is a continuous, monotone decreasing function on c ∈ ( -min i g i , ∞ ) ; the range of the function on this interval is (0 , ∞ ) . Therefore, there is a unique c ∈ ( -min i g i , ∞ ) such that the sum equals 1 .

We now prove the first bullet. Since x 1 /a decreases in x and g i ≤ v a i + κ , we have that

Suppose for contradiction that c < -κ . Then v a i + κ + c < v a i for all i , so

This is a contradiction, so in fact c ≥ -κ .

The first claim of the second bullet is analogous. Since x 1 /a decreases in x and g i ≥ v a i -κv i , we have that

Suppose for contradiction that c > κ v i for every i . Then v a i -κ v i + c > v a i for all i , so

This is a contradiction, so in fact c ≤ κ min i v i .

We now prove the second claim of the second bullet. To do so, we note the following technical lemma (proof below).

Lemma A.2. For -1 < a < 0 and κ, c ≥ 0 , the function f ( x ) = ( x a -κ x + c ) 1 /a is defined and concave at any value of x > 0 such that a 2 x a +1 ≥ κ .

Since for a general concave function f it holds that 1 m ∑ m i =1 f ( x i ) ≤ f ( 1 m ∑ m i =1 x i ) , the following inequality follows from Lemma A.2:

(Here we are using the fact that ∑ i v i = 1 .) Now, combining this fact with Equation 5, we have that

so c ≤ mκ , as desired.

Proof of Lemma A.2. To show that f is defined for any x such that a 2 x a +1 ≥ κ , we need to show that x a -κ x + c > 0 for such values of x . This is indeed the case:

Now we show concavity. We have

so we wish to show that

for every x such that a 2 x a +1 ≥ κ . Fix any such x , and let d = κ x a +1 (so 0 ≤ d ≤ a 2 ). We have 2

Note that the fifth line is justified by the fact that c ≥ 0 and a ( a -1) ≥ 2 d (because a 2 ≥ d and -a > a 2 ≥ d ). Now, multiplying both sides by ( x a -κ x + c ) 1 /a -2 completes the proof.

Proof of Corollary 4.5. Note that ηγ = 1 √ Tm (1+ α ) / 2 and also that η t ≤ η for all t ; we will be using these facts.

To prove (#1), we proceed by induction on t . In the case of t = 1 , all weights are 1 /m , so the claim holds for sufficiently large T . Now assume that the claim holds for a generic t < T ; we show it for t +1 .

By the small gradient assumption, we may use Lemma 4.4 with ζ = γ . By the inductive hypothesis (and the fact that η t ≤ η ), we may apply the second part of Lemma 4.4:

Since -1 2 < α -1 < 0 , this means that w t i ≥ 1 10 √ m T 1 / (2( α -1)) .

We also have that

for T sufficiently large, since α 2( α -1) > -1 2 . This completes the inductive step, and thus the proof of (#1).

To prove (#2), we use the following technical lemma (see below for the proof).

Lemma A.3. Fix x > 0 and -1 < a < 0 . Let f ( y ) = ( x a + y ) 1 /a . Then for all y > -x a , we have

and for all -1 < c ≤ 0 , for all cx a ≤ y ≤ 0 , we have

We apply Equation 6 to x = w t i , y = ( m +1) ηγ , and a = α -1 . This tells us that

The first step follows by the second part of Lemma 4.4 and the fact that η t ≤ η . The second step follows from Equation 6 and uses the fact that 1 1 -α > 2 .

For the other side of (#2), we observe that since by (#1) we have ( w t i ) α ≥ 4 ηγ , it follows that 1 2 ( w t i ) α ≥ ( w t i +1) ηγ , and so ( 1 w t i +1 ) ηγ ≤ 1 2 ( w t i ) α -1 . Therefore, we can apply Equation 7 to x = w t i , y = -( 1 w t i +1 ) ηγ , a = α -1 , and c = -1 2 . This tells us that

This completes the proof.

Proof of Lemma A.3. For all y > -x a , we have

so f ′ is increasing. Thus, for positive values of y we have

and for negative values of y we have

Regardless of whether y is positive or negative, this means that x -f ( y ) ≤ -yf ′ (0) = -1 x 1 -a y .

Now, let -1 < c ≤ 0 and suppose that cx ≤ y ≤ 0 . Since f is increasing, we have that a a ′

so

Proof of Lemma 4.8. We first derive an expression for ∂ i L ( w ) given expert reports p 1 , . . . , p m , where L ( w ) is the log loss of the logarithmic pool p ∗ ( w ) of p 1 , . . . , p m with weights w , and j is the realized outcome. We have 9

and

Equation (2) now follows fairly straightforwardly. Equation 8 tells us that ∂ i L ( w ) ≤ -ln p i J , where J is the random variable corresponding to the realized outcome. Therefore, we have

where the last step follows by the calibration property. This proves Equation (2).

We now prove Equation (3). The proof has a similar idea, but is somewhat more technical. We begin by proving the following lemma; we again use the calibration property in the proof.

Lemma A.4. For all q , we have

Proof. We have

Now, assuming that there is an ℓ such that for every k we have p k ℓ > q , the denominator is greater than q , in which case we have p ∗ j ( w ) < ( p i j ) wi q . Therefore, if p ∗ j ( w ) ≥ ( p i j ) wi q , it follows that for every ℓ there is a k such that p k ℓ ≤ q . By Lemma A.4, this happens with probability at most mnq .

We now use Corollary A.5 to prove Equation (3). Note that the equation is trivial for ζ < n , so we assume that ζ ≥ n . By setting q := e -ζ n , we may restate Equation (3) as follows: for any q ≤ 1 e , any i ∈ [ m ] , and any weight vector w ,

(Note that the condition q ≤ 1 e is equivalent to ζ ≥ n .) We prove this result. From Equation 8, we have

Proof. Let J be the random variable corresponding to the index of the outcome that ends up happening. We have

where the fact that P [ J = j | p i j ≤ q ] ≤ q follows by the calibration property.

Corollary A.5. For any reports p 1 , . . . , p m , weight vector w , i ∈ [ m ] , and j ∈ [ n ] , we have

Now, it suffices to show that for each j ∈ [ n ] , the probability that p j ( w ) ln p i j ≤ -ln 1 /q w i = ln q w i is at most mnq ; the desired result will then follow by the union bound. By Corollary A.5, for each j we have that

Additionally, we know for a fact that p j ( w ) ln p i j ≥ ln p i j (since p j ( w ) ≤ 1 ), so in fact

It remains only to show that max ( ( p i j ) wi q ln p i j , ln p i j ) ≥ ln q w i . If p i j ≥ q 1 /w i then this is clearly true, since in that case ln p i j ≥ ln q w i . Now suppose that p i j < q 1 /w i . Observe that x wi q ln x decreases on (0 , e -1 /w i ) , and that (since q ≤ 1 e ) we have q 1 /w i ≤ e -1 /w i . Therefore,

This completes the proof of Equation (3), and thus of Lemma 4.8.

The following lemma lower bounds the regret of Algorithm 1 as a function of ζ .

Lemma A.6. Consider a run of Algorithm 1. Let ζ be such that -ζ w t i ≤ ∂ i L t ( w t ) ≤ ζ for all i, t . The total regret is at most

Proof of Lemma A.6. We first bound w t i for all i, t . From Lemma 4.4, we have that

Here we use that 1 min i w i +1 ≤ 2 min i w i and that η t ≤ min i w i . Therefore, we have that

Thus, w t i ≥ ( m 1 -α +2 ζT ) 1 / ( α -1) ≥ Ω(( ζT ) 1 / ( α -1) ) for all i, t .

We now use the standard regret bound for online mirror descent, see e.g. (Orabona, 2021, Theorem 6.8):

where B R ( · ; · ) is the Bregman divergence of with respect to R , u is the optimal (overall lossminimizing) point, λ is a constant such that R is λ -strongly convex with respect to a norm of our choice over ∆ m , and ∥·∥ ∗ is the dual norm of the aforementioned norm.

Note that for any x ∈ ∆ m , we have

In the last step, we use the fact that -∇ R ( x ) = ( x α -1 1 , . . . , x α -1 m ) (all of these coordinates are positive), so -∇ R ( x ) · ( v -x ) ≤ ( x α -1 1 , . . . , x α -1 m ) · v , and that all coordinates of v are non-negative and add to 1 .

Therefore, given our bound on w t i , we have that this first component of our regret bound (9) is at most

To bound the second term, we choose to work with the ℓ 1 norm. To show that R is λ -convex it suffices to show that for all x , y ∈ ∆ m we have ( ∇ 2 R ( x ) y ) · y ≥ λ ∥ y ∥ 2 , where ∇ 2 R is the Hessian of R (Shalev-Shwartz, 2007, Lemma 14) (see also (Orabona, 2021, Theorem 4.3)). Equivalently, we wish to find a λ such that

Since x α -2 i ≥ 1 for all i , the left-hand side is at least (1 -α ) ∑ i y 2 i ≥ 1 -α m , so λ = 1 -α m suffices.

Now, given θ ∈ R m , we have ∥ θ ∥ ∗ = max x : ∥ x ∥≤ 1 θ · x . In the case of the ℓ 1 primal norm, the dual norm is the largest absolute component of θ . Thus, we have

Since η t ≤ O ( T -1 / 2 ) , we have that the second component of our regret bound (9) is at most

This component dominates our bound on the regret of the first component, in both ζ and T . This concludes the proof.

Proof of Corollary 4.9. Let Z be the minimum value of ζ such that -ζ w t i ≤ ∂ i L t ( w t ) ≤ ζ for all i, t . Note that by Lemma 4.8, we have that

Let µ be the constant hidden in the big-O of Lemma A.6, i.e. a constant (dependent on m , n , and α ) such that

Let r ( Z, T ) be the expression on the right-hand side. The small gradient assumption not holding is equivalent to Z > 12 n ln T , or equivalently, r ( Z, T ) > r (12 n ln T, T ) . The expected regret of our algorithm conditional on the small gradient assumption not holding, times the probability of this event, is therefore at most the expected value of r ( Z, T ) conditional on the value being greater than r (12 n ln T, T ) , times this probability. This is equal to

as desired. (The first inequality follows by matching the first term with the k = 11 summand and upper-bounding the integral with subsequent summands, noting that r (( k +1) n ln T, T ) ≥ 1 .)

Note that 5 -α 1 -α -10 ≤ 5 -1 / 2 1 -1 / 2 -10 = -1 . Therefore, the contribution to expected regret from the case that the small gradient assumption does not hold is ˜ O ( T -1 ) , which is negligible. Together with Corollary 4.7 (which bounds regret under the small gradient assumption), this proves Theorem 3.2.

We now extend Theorem 3.2 by showing that the theorem holds even if experts are only approximately calibrated.

Definition A.7. For τ ≥ 1 , we say that expert i is τ -calibrated if for all p ∈ ∆ n and j ∈ [ n ] , we have that P [ J = j | p j = p ] ≤ τp j . We say that P satisfies the τ -approximate calibration property if every expert is τ -calibrated.

Corollary A.8. For any τ , Theorem 3.2 holds even if the calibration property is replaced with the τ -approximate calibration property.

(Note that the τ is subsumed by the bigO notation in Theorem 3.2; Corollary A.8 does not allow experts to be arbitrarily miscalibrated.)

Technically, Corollary A.8 is a corollary of the proof of Theorem 3.2, rather than a corollary of the theorem itself. 10

Proof. We only used the calibration property in the proofs of Equations (2) and (3). In the proof of Equation (2), we used the fact that P [ J = j | p i j ≤ e -ζ ] ≤ e -ζ ; the right-hand side now becomes τe -ζ , and so the right-hand side of Equation (2) changes to τne -ζ . Similarly, in the proof of Equation (3), we use the calibration property in the proof of Lemma A.4; the right-hand side of the lemma changes to τmnq , and correspondingly Equation (3) changes to τmn 2 e -ζ/n .

Lemma 4.8 is only used in the proof of Corollary 4.9, where 2 m 2 n 2 T is replaced by 2 τm 2 n 2 T . Since τ is a constant, Corollary 4.9 holds verbatim.

B Ω( √ T ) Lower bound

We show that no OMD algorithm with a constant step size 11 substantially outperforms Algorithm 1.

Theorem B.1. For every strictly convex function R : ∆ m → R that is continuously twice differentiable at its minimum, and η ≥ 0 , online mirror descent with regularizer R and constant step size η incurs Ω( √ T ) expected regret.

Proof. Our examples will have m = n = 2 . The space of weights is one-dimensional; let us call w the weight of the first expert. We may treat R as a (convex) function of w , and similarly for the losses at each time step. We assume that R ′ (0 . 5) = 0 ; this allows us to assume that w 1 = 0 . 5 and does not affect the proof idea.

It is straightforward to check that if Experts 1 and 2 assign probabilities p and 1 2 , respectively, to the correct outcome, then

If roles are reversed (they say 1 2 and p respectively) then

We first prove the regret bound if η is small ( η ≤ T -1 / 2 ). Consider the following setting: Expert 1 always reports (50% , 50%) ; Expert 2 always reports (90% , 10%) ; and Outcome 1 happens with probability 90% at each time step. It is a matter of simple computation that:

L ′ ( w ) ≤ 2 no matter the outcome or the value of w .

If w ≥ 0 . 4 , then p ∗ 1 ( w ) ≤ 0 . 8 .

The first point implies that R ′ ( w t ) ≥ -2 ηt for all t . It follows from the second point that the algorithm will output weights that will result in an aggregate probability of less than 80% for values of t such that -2 ηt ≥ R ′ (0 . 4) , i.e. for t ≤ -R ′ (0 . 4) 2 η . Each of these time steps accumulates constant regret compared to the optimal weight vector in hindsight (which with high probability will be near 1 ). Therefore, the expected total regret accumulated during these time steps is Ω(1 /η ) = Ω( √ T ) .

Now we consider the case in which η is large ( η ≥ √ T ). In this case our example is the same as before, except we change which expert is 'ignorant" (reports (50% , 50%) and which is 'informed" (reports (90% , 10%) ). Specifically the informed expert will be the one with a lower weight (breaking ties arbitrarily).

We will show that our algorithm incurs Ω( η ) regret compared to always choosing weight 0 . 5 . Suppose without loss of generality that at a given time step t , Expert 1 is informed (so w t ≤ 0 . 5 ). Observe that

where p is the probability that Expert 1 assigns to the event that happens (so p = 0 . 9 with probability 0 . 9 and p = 0 . 1 with probability 0 . 1 ). This expression is (up to lower order terms) equal to c (0 . 5 -w t ) if p = 0 . 9 and -3 c (0 . 5 -w t ) if p = 0 . 1 , where c ≈ 0 . 55 . This means that an expected regret (relative to w = 0 . 5 ) of 0 . 6 c (0 . 5 -w t ) (up to lower order terms) is incurred.

Let D be such that R ′′ ( w ) ≤ D for all w such that | w -0 . 5 | ≤ √ T 4 D . (Such a D exists because R is continuously twice differentiable at 0 . 5 .) If | w t -0 . 5 | ≥ √ T 4 D , we just showed that an expected regret (relative to w = 0 . 5 ) of Ω ( √ T 4 D ) is incurred. On the other hand, suppose that | w t -0 . 5 | ≤ √ T 4 D . We show that ∣ ∣ w t +1 -0 . 5 ∣ ∣ ≥ √ T 4 D .

To see this, note that | L ′ ( w t ) | ≥ 0 . 5 , we have that ∣ ∣ R ′ ( w t +1 ) -R ′ ( w t ) ∣ ∣ ≥ 0 . 5 η . We also have that D ∣ ∣ w t +1 -w t ∣ ∣ ≥ ∣ ∣ R ′ ( w t +1 ) -R ′ ( w t ) ∣ ∣ , so D ∣ ∣ w t +1 -w t ∣ ∣ ≥ 0 . 5 η . Therefore, ∣ ∣ w t +1 -w t ∣ ∣ ≥ η 2 D ≥ √ T 2 D , which means that ∣ ∣ w t +1 -0 . 5 ∣ ∣ ≥ √ T 4 D .

This means that an expected regret (relative to w = 0 . 5 ) of Ω ( √ T 4 D ) is incurred on at least half of time steps. Since D is a constant, it follows that a total regret of at least Ω( √ T ) is incurred, as desired.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the logarithmic pooling method for prediction using expert advice. At each step, $m$ experts report distributions $p^1, p^2, \\ldots, p^m$ over a size-$n$ domain. The goal is to make predictions with a vanishing regret in terms of the log loss.\n\nThe usual logarithmic pooling returns an aggregated distribution $p^*$ such that each $p^*(j)$ is proportional to $\\prod_{i=1}^m[p^i(j)]^{1/m}$, the geometric mean of the predicted probabilities. This paper considers a generalization in which the exponent for each expert is replaced by $w_1, w_2, \\ldots, w_m$ that sum up to 1 (instead of $1/m$ identically), and the goal is to learn the right set of weights to achieve a low regret.\n\nFormally, each round of the prediction proceeds as follows: First, algorithm selects weights. Then, adversary chooses the experts' forecasts and the outcome subject to a *calibration* condition. The two steps are in sequential order, so that the forecaster is prevented from outputting an arbitrary distribution.\n\nThe main result of the paper (Theorem 3.2) states that the regret can be upper bounded by roughly $m^{3/2}n\\sqrt{T}\\log T$ in the $T \\gg m, n$ regime. The algorithm uses online mirror descent with the Tsallis entropy regularizer.\n\nSTRENGTHS:\nThe problem setting is quite natural and well-motivated. The authors did a good job in explaining why certain assumptions are necessary by providing illustrative examples. While the solution is based on online mirror descent, the analysis of the algorithm requires several new ideas that seem to be non-trivial and of independent interest.\n\nWEAKNESSES:\n- The scope of the setting is limited to logarithmic pooling and the log loss.\n- The tightness of the bound is unclear; an $\\Omega(\\sqrt{T})$ lower bound is proved for a restricted class of algorithms, while the polynomial dependence on $m$ and $n$ might be sub-optimal.\n- Several aspects of the setting don't seem sufficiently convincing; see questions below.\n\nQUESTIONS:\n- The setting assumes that the adversary chooses the forecasts and outcome after seeing the weights selected by the algorithm. If both parties act simultaneously, would the setting become significantly easier? (In particular, the obstacle of Example 1.1 still seems to be there?)\n- What if the weights don't need to sum up to 1? Would that change the expressivity of the aggregation? (Multiplying $w_1, \\ldots, w_m$ by the same factor seems to give a different distribution.)\n- Lines 63--70 argue that we need to keep the algorithm from seeing the experts' advice, so that the algorithm cannot \"output an essentially arbitrary function of the forecasts\". What would be the practical motivation/justification of this choice?\n\nLIMITATIONS:\nThe main limitations are the assumptions made regarding the setting (e.g., the adversarial choices of outcomes and forecasts must satisfy the calibration condition). These have been clearly pointed out in the abstract and introduction and, the assumptions are formally stated in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nSummary of the paper\n====================\n* The prediction setting explored in this work is harder than the usual \"prediction with experts advice\" (henceforth abbreviated PwE) setting (e.g. as in the Cesa-Bianchi and Lugosi book), since the learner is required to reveal the expert weights (w_t) *before* observing the expert advice (adversarially chosen by the environment) and the outcome (in the standard PwE model, the mixture prediction is to be done before observing the outcome but after observing the advice). This is crucial for the lower-bound in Example 1.1.\n* The form of weighted logarithmic pooling considered in this paper to aggregate the advice of the experts (Lines 47-48) is also natural and considered in the previous literature (as minimizing weighted average KL divergence to the expert advice, having external Bayesianality, maintaining log-concavity of densities etc).\n* The algorithm proposed is again efficient and well-studied in the literature; online mirror descent with the Tsallis entropy regularizer (as in Zimmert and Seldin, 2018).\n* To beat the lower-bound (Example 1.1) and achieve sublinear regret, the condition imposed on the adversarial joint-sampling of expert advices and outcome is that the all the experts must be calibrated (for each expert, the conditional probability of the outcome conditioned on that expert's advice must match the advice distribution).\n* With the calibration assumption and an appropriate step-size schedule, OMD+Tsallis is shown to achieve $\\tilde{O}(\\sqrt{T} \\log T)$ regret (for $T \\gg m$).\n\nSTRENGTHS:\n* The adversarial prediction setting explored in this work is harder than the usual PwE setting. The form of weighted logarithmic pooling considered is well-studied in the literature (especially in the context of combining Bayesian priors) and this work seems to give the first non-trivial online adversarial prediction regret bound for the logarithmic pooling with respect to the the log loss (to the best of my knowledge).\n* The algorithm proposed (OMD with Tsallis regularizer) is efficient and well-studied in the literature.\n* The calibration condition proposed for the experts is extensively studied in the Bayesian inference literature (in the sense that Bayesian posteriors are naturally calibrated).\n* Quite a bit of detail has been provided for the proofs in the main paper, and these are reasonably sound and well-written.\n\nWEAKNESSES:\n* Not much motivation is given for the harder adversarial setting (requiring the aggregator to predict the weights before seeing the expert forecasts).\n* While the calibration condition is well-motivated in the existing literature w.r.t Bayesian posteriors, the authors have not provided sufficient motivation for why it is practical when treating experts as learners (in a prediction-with-expert-advice setting). It may be particularly problematic when the experts are learners/prediction models which are decoupled from the generation of the adversarial outcomes. With the log loss, no-regret online predictors are of course possible even with adversarial outcomes (using the Laplace or Krichevsky-Trofimov estimators for instance, as in the Cesa-Bianchi and Lugosi book Chapter 9), but it may not be very useful to aggregate such no-regret experts with logarithmic pooling.\n* There are no experimental results in the paper, nor are any practical applications discussed.\n\nQUESTIONS:\n* Lines 66-70 show how forcing the aggregator to predict before seeing the expert forecasts disallows some form of manipulation by the aggregator (mimicking a linear mixture). Could you please elaborate on why such a situation should be disallowed (when using the aggregator for either theoretical or practical purposes)?\n* Could you please provide some discussion on why calibration is still practical in the PwE setting when the experts are learners which are decoupled from the adversarial generation of outcomes (i.e. the expert $i$ has fix the forecast distribution $p^i_t$ at time $t$ before seeing the adversarial outcome $j_t$)?\n\nLIMITATIONS:\nNot applicable.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates the logarithmic pooling method for minimizing log loss and introduces the OMD algorithm utilizing Tsallis entropy as a regularizer to update weights for the logarithmic pooling method. By assuming calibrated forecasts, the paper demonstrates that the proposed algorithm ensures a sub-linear regret. Additionally, this paper establishes a corresponding lower bound, indicating that their regret bound is optimal in terms of its dependence on the number of time steps $T$.\n\nSTRENGTHS:\n1. Learning weights for logarithmic pooling studied in this paper is novel as far as I know, which appears to be a crucial direction, particularly due to its natural alignment with log loss.\n2. The results and methodologies presented in this paper seem to be novel and the techniques used in this paper may be of independent interest to the community.\n3. This paper is well-written. The algorithm is well-motivated, and the theorems presented are rigorous and sound.\n\nWEAKNESSES:\n1. It is important to include a discussion of the relevant literature on online portfolio selection in the related work, as it addresses a similar problem to this paper, namely learning parameters using log loss.\n2. It is worth noting that the lower bound only applies to OMD with a constant step size, which may limit its effectiveness. Additionally, the optimality regarding the number of experts (m) and outcomes (n) should be discussed in this paper.\n3. Providing a more intuitive explanation for the choice of Tsallis entropy as the regularizer would enhance the paper's clarity and value.\n4. This paper can be improved by presenting a detailed example to illustrate the significance of the logarithmic pooling method with log loss.\n\nQUESTIONS:\n1. How about the efficiency of the algorithm? Can the algorithm be executed efficiently?\n2. Why there is no need to project our decision, as stated in footnote 5?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies no-regret learning in the setting of logarithmic pooling of experts with the logarithmic loss. In this setting, there is a set of experts each outputting a distribution $p_i^{t}$ over outcomes in some finite set $Y$. The task of the learner is to output a vector $w^{t}$. An outcome $y$ in $Y$ is then observed and the learner suffers loss $\\sum_i w^{t}_i  \\log p^{t}_i (y)$ where the sum is over the experts. The loss is then compared to the best choice of weights $w$ in hindsight. Under an additional assumption that the expert probabilities are \"calibrated\" (which the paper argues is necessary), the paper presents $\\sqrt{T}$ regret algorithm. Further, the paper is the first to consider regret in the logarithmic pooling setting and to connect calibration of experts to regret in this setting.\n\nSTRENGTHS:\nThe main contributions of the paper in my opinion is the initiation of study of logarithmic pooling + logarithmic loss to the study of online learning. The paper justifies this setting by noting various nice properties this method of aggregation is known to satisfy. Further the relationship of calibration to the regret in this setting is also interesting. It is the key condition under which their algorithm has non-trivial regret.\n\nWEAKNESSES:\nThe main issue I have with the setting is that beyond the presentation in terms of pooling, this setting seems like an instance of standard online linear optimization. As the paper notes the main difference here is the apriori unbounded sizes of the loss vectors. Viewed in this light calibration is the same as assuming that the losses are bounded ( ). The issue really is whether this is a priori clear or not. The point that makes me lean towards the fact that the calibration is \"just assuming\" boundedness is that no average notion of calibration (for example calibration on average across time steps) seems to be sufficient (it seems that the counterexample can be modified to be calibrated on average). Thus, the interpretation just as having \"good experts\" is less clear and seems more like assuming bounded losses.\n\nQUESTIONS:\n- It would be very helpful to elaborate the usual setting of learning with the log loss for comparison since that is well established area of online learning and information theory. The reason this would be helpful is that in the usual setting the actions are unrestricted but it turns out to be (essentially) optimal to consider \"linear pooling\". The fact that linear pooling comes up naturally in that setting would be a good constrast and good place to compare linear and logarithmic pooling.  \n- The comment about linear pooling and logarithmic regret needs to be considered a bit more carefully. In the usual notion, the learner does not compete against all convex combinations of experts. In fact, there can be polynomial difference between the two cases.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-09"}