Title: Truthfulness of Calibration Measures
Venue: NeurIPS 2024
Accepted: yes
Keywords: Calibration measures

online learning

sequential calibration
Abstract: We study calibration measures in a sequential prediction setup. In addition to rewarding accurate predictions (completeness) and penalizing incorrect ones (soundness), an important desideratum of calibration measures is *truthfulness*, a minimal condition for the forecaster not to be incentivized to exploit the system. Formally, a calibration measure is truthful if the forecaster (approximately) minimizes the expected penalty by predicting the conditional expectation of the next outcome, given the prior distribution of outcomes. We conduct a taxonomy of existing calibration measures. Perhaps surprisingly, all of them are far from being truthful. We introduce a new calibration measure termed the *Subsampled Smooth Calibration Error (SSCE)*, which is complete and sound, and under which truthful prediction is optimal up to a constant multiplicative factor. In contrast, under existing calibration measures, there are simple distributions on which a polylogarithmic (or even zero) penalty is achievable, while truthful prediction leads to a polynomial penalty.

Truthfulness of Calibration Measures

Abstract

We study calibration measures in a sequential prediction setup. In addition to rewarding accurate predictions (completeness) and penalizing incorrect ones (soundness), an important desideratum of calibration measures is truthfulness , a minimal condition for the forecaster not to be incentivized to exploit the system. Formally, a calibration measure is truthful if the forecaster (approximately) minimizes the expected penalty by predicting the conditional expectation of the next outcome, given the prior distribution of outcomes. We conduct a taxonomy of existing calibration measures. Perhaps surprisingly, all of them are far from being truthful. We introduce a new calibration measure termed the Subsampled Smooth Calibration Error (SSCE) , which is complete and sound, and under which truthful prediction is optimal up to a constant multiplicative factor. In contrast, under existing calibration measures, there are simple distributions on which a polylogarithmic (or even zero) penalty is achievable, while truthful prediction leads to a polynomial penalty.

1 Introduction

Probability forecasting is a central prediction task to a wide range of domains and applications, such as finance, meteorology, and medicine [MW84, DF83, WM68, JOKOM12, KSB21, VCV15, BF + 02, CAT16]. For forecasts to be useful, a common minimum requirement is that they are calibrated , i.e., the predictions are unbiased conditioned on the predicted value. Formally, for a sequence of T binary events, a forecaster who predicts probabilities in [0 , 1] is perfectly calibrated if for every α ∈ [0 , 1] , among the time steps on which α is predicted, an α fraction of the outcomes is indeed 1 . Since perfectly calibrated forecasts are often unachievable, calibration measures have been introduced to quantify some form of deviation from perfectly calibrated forecasts. Common examples of these measures include the expected calibration error (ECE) [FV98], the smooth calibration error [KF08], and the distance from calibration [BGHN23].

As these calibration measures are commonly used to evaluate the performance of forecasters, it is important that their use encourages forecasters to incorporate the highest quality information available to them (e.g., via their expert knowledge or side information) about the next outcome. This desideratum, formally referred to as truthfulness , requires that a calibration measure incentivizes the forecasters to predict truthfully when the true distribution of the next outcome is known to them. Lack of truthfulness can have severe consequences: it serves as a poor measure of quality of forecasts, tempts forecasters to make deliberately biased predictions in order to game the system, and erodes trust in predictions provided by third-party forecasters. Given the importance of truthfulness, we set out to identify calibration measures that demonstrate truthfulness.

While truthfulness of calibration measures has not been systematically investigated to date, evidence of the lack of truthfulness of some calibration measures has emerged in recent literature. For example, [FH21, QV21] noted that a forecaster can lower their ECE by predicting according to the past. This observation was applied in the algorithm of [FH21] and motivated the 'sidestepping' technique in the lower bound proof of [QV21]. More recently, [QZ24] highlighted a large gap in the truthfulness of a recently proposed calibration measure (called the distance from calibration [BGHN23]) by

showing that in a simple setup of predicting i.i.d. outcomes, the truthful forecaster incurs a distance of Ω( √ T ) from calibration but there is a forecasting algorithm that achieves polylog( T ) distance from calibration. We call this a polylog( T ) -Ω( √ T ) truthfulness gap. On the other hand, we say that a calibration measure is ( α, β ) -truthful if predicting the next outcome according to its conditional distribution incurs a measure that is no more than α OPT + β , where OPT is the minimum value of the calibration measure achievable by any forecaster. Faced with evidence that some calibration measures suffer from large truthfulness gaps, we will systematically examine the truthfulness (or a gap thereof) of a wide range of calibration measures.

For a truthful calibration measure to also be useful it must distinguish accurate predictions from inaccurate ones. After all, a measure that is uniformly 0 regardless of the quality of predictions is perfectly truthful (formally (1 , 0) -truthful) but provides no insights into the quality of the predictions. We formalize the minimum requirement for a measure to be useful by its completeness and soundness when predicting i.i.d. Bernoulli outcomes. The former requires that predicting the outcomes according to the correct parameter of the generating Bernoulli distribution incurs no or o ( T ) penalty, whereas the latter requires the penalty to be Ω( T ) when predictions systematically deviate from the correct parameter. An equally important feature of a calibration measure is that it defines an ideal that could be asymptotically achieved for all prediction tasks. This is formalized by the existence of forecasting algorithms with an o ( T ) penalty in the adversarial sequential prediction setting [FV98], where the sequence of outcomes is produced by an adaptive adversary.

With these desiderata in place (namely truthfulness, soundness, completeness, and asymptotic calibration), we ask whether there are calibration measures that simultaneously satisfy all these criteria? We answer this question in three parts:

Part I: We show that existing calibration measures do not simultaneously meet these criteria. We conduct a taxonomy of several existing calibration measures in terms of their completeness, soundness and truthfulness (formally defined in Section 2). We show that almost all of them have large truthfulness gaps : There are simple distributions on which an O (1) (or even zero) penalty is achievable, while truthful predictions lead to a poly( T ) penalty; see Table 1 for details.

Indeed, this lack of truthfulness is not limited to specific or contrived distributions. In the next theorem which we will prove in Appendix B, we strengthen these findings by showing that a commonly used notion of calibration systematically suffers large truthfulness gaps in most forecasting instances.

Theorem 1.1 (Informal) . For every product distribution with marginals bounded away from 0 and 1 , the truthful forecaster incurs Ω( √ T ) smooth calibration error but there exists a forecasting algorithm that incurs only polylog( T ) smooth calibration error.

A notable exception in Table 1 is the class of calibration measures induced by proper scoring rules , i.e., loss functions for probabilistic predictions that are optimized by truthful forecasts. By definition, these calibration measures are (1 , 0) -truthful. However, none of them is complete: as we show in Appendix A, even on i.i.d. Bernoulli trials, the optimal and truthful predictions incur an Ω( T ) penalty.

Part II: We introduce a new calibration measure, called SSCE, that is sound, complete, and approximately truthful. We do this using a simple adjustment to an existing notion of calibration measure: we subsample a subset of the time steps and evaluate the smooth calibration error [KF08] on this sampled set only. We call this the Subsampled Smooth Calibration Error (SSCE) and formally define it in Section 2. Our main result is that SSCE is ( O (1) , 0) -truthful.

Theorem 1.2 (Main Theorem) . There exists a universal constant c > 0 such that the SSCE is ( c, 0) -truthful. Furthermore, the SSCE is complete and sound.

As shown in Table 1, to the best of our knowledge, SSCE is the first calibration measure that simultaneously achieves completeness, soundness, and non-trivial truthfulness.

While our methodology for constructing this calibration measure is simple, the analytical steps required to establish the ( O (1) , 0) -truthfulness guarantee are far from simple. We dedicate most of the main body of this paper to illustrating the proof ideas in a series of warmups to Theorem 1.2.

√

Part III: There is a forecasting algorithm that achieves O ( T ) SSCE even in the adversarial setting. While our study of truthfulness of calibration measures is necessarily focused on when the

forecaster knows the conditional distribution of the next outcome, it is important to ensure that, even in the adversarial setting, a sublinear penalty can be achieved for this calibration measure. For this, we study the sequential calibration setting (e.g., [FV98]) where the outcome at time t is chosen by an adaptive adversary who has observed the sequence of earlier outcomes and predictions. We show that an O ( √ T ) SSCE is achievable.

Theorem 1.3. In the adversarial sequential calibration setting, there is a deterministic strategy for the forecaster that achieves an O ( √ T ) SSCE.

√

An interesting and important feature of this result is that it achieves an O ( T ) rate whereas an O ( √ T ) rate for the expected calibration error is known to be impossible to achieve [QV21]. Together our Theorems 1.2 and 1.3 establish that SSCE is a truthful and useful calibration measure.

1.1 Related Work

There is a large body of work on calibration, a notion that dates back to the 1950s [Bri50, Daw82, Daw85] and has been applied to game theory [FV97, HPY23], machine learning [GPSW17], and algorithmic fairness [KMR17, PRW + 17, HJKRR18, HJZ23]. We will restrict our discussion to sequential calibration and the systematic study of calibration measures, which are the closest to this work.

Sequential calibration. Foster and Vohra [FV98] first proved that one can achieve asymptotic calibration on arbitrary and adaptive outcomes. Formally, they gave a forecasting algorithm with an O ( T 2 / 3 ) ECE in expectation, when predicting T binary outcomes chosen by an adaptive adversary. Subsequent work gave alternative and simpler proofs of the result [FL99, Fos99, Har22], extended the result to other calibration measures [KF08, FH18, FH21, QZ24], and proved lower bounds on the optimal ECE [QV21]. Most closely related to our approach is the work of [FRST11], who studied a stronger notion that requires calibration on a family of checking rules , where each checking rule specifies a subset of the time horizon. Despite the apparent similarity, their notion is qualitatively different from the SSCE, since we take an expectation over the subsampled horizon, whereas they take the maximum. In particular, no forecaster can be calibrated in their definition if the checking rule family contains all subsets of [ T ] , since there always exists a checking rule that strongly correlates with the outcomes.

Calibration measures. The recent work of Błasiok, Gopalan, Hu and Nakkiran [BGHN23] initiated the rigorous study of calibration measures. Their work focused on the offline setup, where there is a known marginal distribution over the feature space, and each predictor maps the feature space to [0 , 1] . They proposed to use the distance from calibration -the ℓ 1 distance from the predictor to the closest predictor that is perfectly calibrated-as the ground truth, and studied whether existing

calibration measures are consistent with it. Note that completeness and soundness are defined differently in [BGHN23]: a calibration measure is called complete (resp., sound) if it is upper (resp., lower) bounded by a polynomial of the distance from calibration. Since the distance from calibration is far from being truthful in the online setup (as shown by [QZ24]), our definition of completeness and soundness set up minimal conditions for an error metric to be regarded as measuring calibration, rather than enforcing closeness to the distance from calibration.

Subsampling. Our new calibration measure is derived from subsampling the time horizon. This simple idea has been shown to be effective in various different contexts, including privacy amplification in differential privacy (e.g.,[Ste22, Section 6]), handling adversarial corruptions [BLMT22], as well as adaptive data analysis [Bla23].

Proper scoring rules. Proper scoring rules [WM68] are error metrics for probabilistic forecasts that are optimized when the forecaster predicts according to the true distribution. While the error metrics induced by proper scoring rules are (perfectly) truthful by definition, as we show in Appendix A, they are qualitatively different from the usual calibration measures and, in particular, do not meet the completeness criterion. We note that a recent line of work [CY21, NNW21, LHSW22, PW22, HSLW23] studied the optimization of scoring rules , namely, finding the proper scoring rule that maximally incentivizes the forecaster to exert effort to obtain additional information.

2 Preliminaries

Sequential prediction. Weconsider the following prediction setup: First, a sequence x ∈ { 0 , 1 } T is sampled from distribution D . At each step t ∈ [ T ] , the forecaster makes a prediction p t ∈ [0 , 1] , after which x t is revealed. Formally, a deterministic forecaster is a function A : ⋃ T t =1 { 0 , 1 } t -1 → [0 , 1] , where A ( b 1 , b 2 , . . . , b t -1 ) specifies the forecaster's prediction at step t if the first t -1 observations match b 1:( t -1) . Distribution D and forecaster A naturally induce a joint distribution of ( x, p ) ∈ { 0 , 1 } T × [0 , 1] T via sampling x ∼ D and predicting p t = A ( x 1 , x 2 , . . . , x t -1 ) .

Note that we could have defined the forecaster as a function of both the outcomes x 1:( t -1) and the predictions p 1:( t -1) in the past. This alternative definition is equivalent to ours, since p 1:( t -1) would be uniquely determined by x 1:( t -1) . We could also have considered randomized forecasters, which are specified by distributions over deterministic forecasters. However, as we will see later, restricting our attention to deterministic forecasters does not affect the subsequent definitions.

Calibration measures. The quality of the forecaster's predictions in the setting above is quantified by calibration measures. Formally, a calibration measure CM is a family of functions { CM T : T ∈ N } , where each CM T maps { 0 , 1 } T × [0 , 1] T to [0 , T ] . We will frequently omit the subscript T , since it is usually clear from the context. With respect to calibration measure CM , the expected penalty incurred by forecaster A on distribution D is defined as err CM ( D , A ) := E ( x,p ) ∼ ( D , A ) [ CM ( x, p )] , where ( x, p ) ∼ ( D , A ) denotes sampling a sequence x and predictions p from the joint distribution induced by D and A .

One example of calibration measures is the smooth calibration error introduced by [KF08] that is defined as smCE ( x, p ) := sup f ∈F ∑ T t =1 f ( p t )( x t -p t ) , where F is the family of 1 -Lipschitz functions from [0 , 1] to [ -1 , 1] . In this work, we introduce a new calibration measure called Subsampled Smooth Calibration Error (SSCE) that is defined by subsampling a subset of the time horizon, and evaluating the smooth calibration error on it. We will formally define this measure next. In the following, Unif ( S ) denotes the uniform distribution over a finite set S . For a T -dimensional vector x and S ⊆ [ T ] , x | S denotes the | S | -dimensional vector formed by the entries of x indexed by S .

Definition 2.1 (Subsampled Smooth Calibration Error) . For a sequence of outcomes x ∈ { 0 , 1 } T and predictions p ∈ [0 , 1] T , the Subsampled Smooth Calibration Error (SSCE) is defined as

Completeness and soundness. Wegive minimal conditions for a calibration measure to be regarded as complete (intuitively 'accurate' predictions have a small penalty) and sound (intuitively 'inaccurate' predictions have a large penalty).

̸

Definition 2.2 (Completeness and soundness) . A calibration measure CM is complete if: (1) For any x ∈ { 0 , 1 } T , CM T ( x, x ) = 0 ; (2) For any α ∈ [0 , 1] , E x 1 ,...,x T ∼ Bernoulli ( α ) [ CM T ( x, α · ⃗ 1 T ) ] = o α ( T ) . The calibration measure is sound if: (1) For any x ∈ { 0 , 1 } T , CM T ( x, ⃗ 1 T -x ) = Ω( T ) ; (2) For any α, β ∈ [0 , 1] such that α = β , E x 1 ,...,x T ∼ Bernoulli ( α ) [ CM T ( x, β · ⃗ 1 T ) ] = Ω α,β ( T ) . Here, o α ( · ) and Ω α,β ( · ) may hide constant factors that depend on the parameters in the subscript.

Truthfulness. To define the truthfulness of a calibration measure, we introduce the truthful forecaster and the optimal error for a distribution D .

Definition 2.3 (Truthful forecaster) . With respect to distribution D ∈ ∆( { 0 , 1 } T ) , the truthful forecaster is defined as A truthful ( D )( b 1 , b 2 , . . . , b t -1 ) := Pr x ∼D [ x t = 1 ∣ ∣ ∣ x 1:( t -1) = b 1:( t -1) ] .

Arguably, A truthful ( D ) is the only forecaster that makes the 'right' predictions on distribution D .

Definition 2.4 (Optimal error) . The optimal error on distribution D ∈ ∆( { 0 , 1 } T ) with respect to calibration measure CM is defined as OPT CM ( D ) := inf A err CM ( D , A ) , where A ranges over all deterministic forecasters.

Note that by an averaging argument, the definition of OPT CM ( D ) is unchanged if we take an infimum over randomized forecasters.

A calibration measure is truthful if, on every distribution, the truthful forecaster is near-optimal.

Definition 2.5 (Truthfulness of calibration measures) . A calibration measure CM is ( α, β ) -truthful if, for every D ∈ ∆( { 0 , 1 } T ) , err CM ( D , A truthful ( D )) ≤ α · OPT CM ( D ) + β . Conversely, CM is said to have an α -β truthfulness gap if, for some distribution D , OPT CM ( D ) ≤ α and err CM ( D , A truthful ( D )) ≥ β .

3 Technical Overview

In this section, we briefly discuss the main technical ideas and challenges behind the proofs of Theorems 1.1, 1.2, and 1.3. We provide more details on our main result, i.e., that SSCE is ( O (1) , 0) -truthful, in Sections 4 through 6. Theorem 1.3 follows from a recent result of [ACRS24] on minimizing the distance from calibration in the adversarial setup, along with a new result connecting SSCE to distance from calibration, and is proved in Section 7. We defer the proof of Theorem 1.1 to Appendix B.

A simple distribution that witnesses truthfulness gaps. Inspired by [QV21, Example 2], we consider the distribution D specified as follows: The time horizon is divided into T/ 3 blocks of length 3 , each with a uniformly random bit, followed by a zero and a one. Within each block, the truthful forecaster predicts 1 / 2 , 0 and 1 in order. Then, among the steps on which 1 / 2 is predicted, the frequency of ones is typically 1 / 2 ± Θ(1 / √ T ) . This deviation results in a Θ( √ T ) penalty under most calibration measures (concretely, all calibration measures in the first two rows of Table 1).

However, there is a different strategy that ensures perfect calibration, and thus a zero penalty under most calibration measures. Within each block, the forecaster predicts 1 / 2 on the first step. If the bit turns out to be 1 , the forecaster maintains perfect calibration by predicting 1 / 2 on the second step, on which the outcome is known to be 0 ; otherwise, the forecaster accomplishes the same by predicting 1 / 2 on the third step. Therefore, the distribution D witnesses a 0 -Ω( √ T ) truthfulness gap for every calibration measure in the first two rows of Table 1.

The importance of subsampling in the SSCE becomes apparent in light of the example above. On distribution D , the truthful forecaster has to pay a Θ( √ T ) cost for the mild deviation from the expectation, while a strategic forecaster avoids this deviation by correlating the predictions with the biases in the past. With the subsampling, however, the forecaster is no longer sure about the biases that factor into the penalty. This ensures that, compared to truth-telling, the benefit from predicting strategically is marginal, and thus makes the truthfulness guarantee in Theorem 1.2 possible.

Establishing truthfulness via martingale inequalities. We prove that the SSCE is ( O (1) , 0) -truthful in three steps: (1) Define a complexity measure σ ( D ) of distribution D ; (2) Show that err SSCE ( D , A truthful ( D )) = O ( σ ( D )) ; (3) Show that OPT SSCE ( D ) = Ω( σ ( D )) .

As we elaborate in Section 5, the crux of Step (2) is to control the expected deviation of a martingale ( M t ) 0 ≤ t ≤ T with respect to filtration ( F t ) by the its realized variance Var t := ∑ t s =1 Var [ M s | F s -1 ] , which is highly non-trivial as the two processes ( M t ) and (Var t ) are correlated. In more detail, the filtration ( F t ) corresponds to the randomness in x ∼ D , while ( M t ) tracks the biases in the predictions (on a subset of the time horizon) tested by a Lipschitz function. We note that such a bound would easily follow from 'off-the-shelf' concentration inequalities for martingales (e.g., Freedman's inequality [Fre75]), if the total realized variance Var T were uniformly bounded. However, in general, Var T may vary drastically, and directly applying these concentration inequalities would introduce an extra super-constant factor. Our workaround is a 'doubling trick' that divides the time horizon into epochs , the realized variances in which grow exponentially. We then apply Freedman's inequality to each epoch separately. In Section 5, we formulate a toy random walk problem that highlights this challenge and demonstrates our solution to it, which is of independent interest.

Similarly, as we show in Section 6, the crux of Step (3) is to establish another martingale inequality. We first show that for fixed x and p , we have SSCE ( x, p ) = Ω( √ N T ) , where N t := ∑ t s =1 ✶ [ | x s -p s | ≥ 1 / 2] . Furthermore, over the randomness in x ∼ D , the realized variance process (Var t ) defined above is shown to lower bound ( N t ) , i.e., ( N t -Var t ) is a sub-martingale. However, the desired result requires the lower bound E [ √ N T ] ≥ Ω(1) · E [ √ Var T ] , which does not follow from E [ N T -Var T ] ≥ 0 in general. This challenge necessitates a more careful analysis tailored to the specific properties of the processes ( N t ) and (Var t ) .

Deterministic forecasting strategy via reduction to smCE . We build on the result of [ACRS24] showing the existence of a deterministic forecasting strategy guaranteeing an O ( √ T ) bound on smCE . In particular, we show via a standard chaining argument that SSCE is upper bounded by smCE plus a variance term that can be upper bounded by O ( √ T ) . The result of [ACRS24] then implies a deterministic forecasting algorithm achieving an O ( √ T ) SSCE .

4 Warmup: The Product Distribution Case

As a warmup, in this section, we start by showing that SSCE is ( O (1) , O (log T )) -truthful for product distributions. This is a weaker version of Theorem 1.2 in terms of both the truthfulness parameters of SSCE and the restriction to product distributions. In Sections 5 and 6, we outline how we will remove these restrictions and improve the analysis of truthfulness.

For distribution D = ∏ T t =1 Bernoulli ( p ⋆ t ) , take σ 2 := Var x ∼D [ ∑ T t =1 x t ] = ∑ T t =1 p ⋆ t (1 -p ⋆ t ) as a complexity measure of the distribution of outcomes. We will show that err SSCE ( D , A truthful ( D )) = O ( σ +log T ) and OPT SSCE ( D ) = Ω( σ ) -O (1) .

4.1 Upper Bound the SSCE of the Truthful Forecaster

We first show that the truthful forecaster for D , which predicts p t = p ⋆ t at every step t , gives E x ∼D [ SSCE ( x, p ⋆ )] = O ( σ +log T ) . For this purpose, it suffices to prove

since for each fixed S ⊆ [ T ] , applying (1) to x | S and p ⋆ | S gives E x ∼D [ smCE ( x | S , p ⋆ | S )] ≤ O ( σ +log T ) , and taking an expectation over S ∼ Unif (2 [ T ] ) gives the desired bound on SSCE .

Recall that E [ smCE ( x, p ⋆ )] = E [ sup f ∈F ∑ T t =1 f ( p ⋆ t ) · ( x t -p ⋆ t ) ] . If we replace F with the family of constant functions from [0 , 1] to [ -1 , 1] , the right-hand side would reduce to

Therefore, to prove the upper bound in (1), we need to show that the family of one-dimensional Lipschitz functions is not significantly richer than constant functions.

At a high level, this is done by taking finite coverings of Lipschitz functions and using Dudley's chaining technique [Dud87] to upper bound the value of this stochastic process. In more detail, let F δ be the smallest δ -covering of F in the uniform norm, i.e., for each f ∈ F , there exists f δ ∈ F δ such that ∥ f -f δ ∥ ∞ ≤ δ . It is well-known that |F δ | = e O (1 /δ ) , and a chaining argument gives

where G δ := { f δ -f δ/ 2 : f δ ∈ F δ , f δ/ 2 ∈ F δ/ 2 , ∥ f δ -f δ/ 2 ∥ ∞ ≤ 3 δ/ 2 } .

It remains to bound the second term of (2). Note that for a fixed g , because of the independence of x t s, g ( p ⋆ t ) · ( x t -p ⋆ t ) is independent across t ∈ [ T ] . Therefore, we can control the tail probability of ∑ T t =1 g ( p ⋆ t ) · ( x t -p ⋆ t ) by Bernstein inequalities. For each fixed δ , using a Bernstein tail bound, taking a union bound over g ∈ G δ , and noting that |G δ | ≤ |F δ | · |F δ/ 2 | = e O (1 /δ ) , we have

Plugging this into (2) proves (1) and thus the desired bound E x ∼D [ SSCE ( x, p ⋆ )] = O ( σ +log T ) .

4.2 Lower Bound the Optimal SSCE

Next, we lower bound OPT SSCE ( D ) by showing that every forecasting strategy must incur an Ω( σ ) SSCE on D . Recall that SSCE ( x, p ) is given by

where we use the fact that F contains the constant functions 1 and -1 .

Fix x ∈ { 0 , 1 } T , p ∈ [0 , 1] T and let N := ∑ T t =1 ✶ [ | x t -p t | ≥ 1 / 2] . Over the randomness in y ∼ Unif ( { 0 , 1 } T ) , the quantity ∑ T t =1 y t · ( x t -p t ) , by the central limit theorem, is approximately distributed as a normal distribution with variance ∑ T t =1 1 4 ( x t -p t ) 2 ≥ ∑ T t =1 1 16 ✶ [ | x t -p t | ≥ 1 / 2] = Ω( N ) , so its expected absolute value is Ω( √ N ) .

Now it remains to lower bound the expectation of √ N induced by an arbitrary forecaster. Conditioning on x 1:( t -1) , x t always follows Bernoulli ( p ⋆ t ) . Thus, regardless of the choice of p t ∈ [0 , 1] , the condition | x t -p t | ≥ 1 / 2 holds with probability at least min { p ⋆ t , 1 -p ⋆ t } ≥ p ⋆ t (1 -p ⋆ t ) . Then, over the T steps, we expect that N ≥ Ω( ∑ T t =1 p ⋆ t (1 -p ⋆ t )) = Ω( σ 2 ) holds with probability Ω(1) , as long as σ = Ω(1) . This gives the desired lower bound E [ SSCE ( x, p )] ≳ E [ √ N ] = Ω( σ ) -O (1) .

5 Upper Bound the SSCE of the Truthful Forecaster

To extend the proof strategy sketched in Section 4 to non-product distributions, the first challenge is to define an appropriate complexity measure of a general distribution D . Consider the stochastic process (Var t ) 0 ≤ t ≤ T defined as Var t := ∑ t s =1 p ⋆ s (1 -p ⋆ s ) , where x ∼ D and p ⋆ t := E x ′ ∼D [ x ′ t ∣ ∣ ∣ x ′ 1:( t -1) = x 1:( t -1) ] is now a random variable that denotes the conditional expectation of x t after observing x 1:( t -1) . The 'right' definition turns out to be roughly σ ( D ) := E [ √ Var T ] . In this section, we prove the following weaker upper bound on the SSCE incurred by the truthful forecaster. We provide a stronger bound (Theorem C.1) in Appendix C.

Proof sketch. We begin by repeating the chaining argument in Section 4. Recall that, for any δ > 0 , there is a δ -covering F δ of F in the ∞ -norm that has size e O (1 /δ ) . Letting π δ ( f ) denote the mapping

of a function f onto the covering F δ such that ∥ f -π δ ( f ) ∥ ∞ ≤ δ , we can write for any M ∈ Z + :

To control the expectation of each W k , we note that the set G k := { π 2 -k ( f ) -π 2 1 -k ( f ) : f ∈ F} is of size at most |F 2 -k | · |F 2 1 -k | . Furthermore, every function g ∈ G k satisfies

for some f ∈ F . We apply the following technical lemma, which we prove in Appendix C.

Lemma 5.2. Given a function f : [0 , 1] → [ -1 , 1] and y ∈ { 0 , 1 } T , consider the martingale M t ( f, y ) := ∑ t s =1 y s · f ( p ⋆ s ) · ( x s -p ⋆ s ) where x ∼ D . Then, for any finite family G of functions from [0 , 1] to [ -1 , 1] and any y ∈ { 0 , 1 } T , we have

Applying Lemma 5.2 to each G k scaled up by a Θ(2 k ) factor and noting that log |G k | ≤ log |F 2 -k | + log |F 2 1 -k | = O (2 k ) gives

Choosing M = Θ(log T ) proves the theorem.

We remark that the proof of Lemma 5.2 is highly non-trivial. As mentioned in Section 3, such an upper bound would follow from Freedman's inequality, if Var T were always bounded by O ( ( E [ √ Var T ]) 2 ) . However, in general, applying Freedman's inequality to each M T ( f, y ) necessarily requires an additional union bound over possible values of Var T , and introduces a superconstant multiplicative factor.

The challenge in dealing with the randomness in Var T is captured by the following toy problem:

Random walk with early stopping: Let ( X t ) 0 ≤ t ≤ T be the random walk such that X 0 = 0 and each X t -X t -1 independently follows Unif ( {± 1 } ) . Let τ be an arbitrary stopping time with respect to ( X t ) . Prove that E [ | X τ | ] ≤ O (1) · E [ √ τ ] .

Indeed, the above corresponds to a special case of Lemma 5.2 in which: (1) the sequence p ⋆ starts with entry 1 / 2 , and may switch to entry 0 at any point, depending on the realization of x t s; (2) the family G consists of two constant functions 1 and -1 .

One might be tempted to prove E [ | X τ | ] ≤ O (1) · E [ √ τ ] by first proving E [ | X τ || τ = t ] = O ( √ t ) for all t ∈ [ T ] , and then applying the law of total expectation. Such an approach is doomed to fail, because the stopping time τ might significantly bias the conditional expectation of | X τ | on some event τ = t 0 , e.g., by stopping at time t 0 only if | X t 0 | ≫ √ t 0 .

Our workaround is inspired by the standard doubling trick in online learning. We break the time horizon into epochs of geometrically increasing lengths: the k -th epoch contains 2 k steps. We break | X τ | into the displacements accumulated in different epochs; their sum clearly upper bounds | X τ | .

Furthermore, we can show that, conditioning on reaching epoch k , the displacement within the epoch is O ( √ 2 k ) . This allows us to establish the desired inequality via

To prove Lemma 5.2, we extend this technique to a general martingale M T ( f, y ) by dividing the time horizon into epochs according to the doubling of Var t , and then applying Freedman's inequality to each epoch.

Towards a stronger upper bound. In our actual proof, we use a slightly different complexity measure σ γ ( D ) := E [ γ (Var T )] , where γ ( x ) = x if x < 1 and γ ( x ) = √ x otherwise. Roughly speaking, this definition accounts for the fact that a sum of independent Bernoulli random variables behaves quite differently when its mean is close to 0 . To remove the extra log 2 T term in Theorem 5.1, our actual proof also uses a variant of Lemma 5.2, Lemma C.9, which involves a more careful application of Freedman's inequality tailored to specific coverings of Lipschitz functions.

6 Lower Bound the Optimal SSCE

In this section, we outline a weaker lower bound on the optimal SSCE achievable on a distribution.

Similar to the product distribution case (Section 4), the key quantity in the proof is the stochastic process ( N t ) 0 ≤ t ≤ T defined as N t := ∑ t s =1 n s and n t := ✶ [ | x t -p t | ≥ 1 / 2] . This is formalized by the following lemma, which applies to any realization of x , p , and N T = ∑ T t =1 [ | x t -p t | ≥ 1 / 2] :

It remains to lower bound the quantity E [ √ N T ] induced by an arbitrary forecaster. As argued earlier, conditioning on x 1:( t -1) , we always have Pr [ n t = 1] ≥ p ⋆ t (1 -p ⋆ t ) = Var t -Var t -1 , where p ⋆ t and Var t are defined as in Section 5. Thus, ( N t -Var t ) is a sub-martingale, which implies E [ N T ] ≥ E [Var T ] . However, this does not imply that E [ √ N T ] ≥ Ω( E [ √ Var T ] ) . In fact, such an inequality does not hold in general: When p ⋆ 1 = ε ≪ 1 and p ⋆ t = 0 for all t ≥ 2 , E [ √ N T ] could be O ( ε ) , yet E [ √ Var T ] = Ω( √ ε ) ≫ O ( ε ) .

The following technical lemma circumvents this counterexample by subtracting a constant term from the right-hand side:

Note that Theorem 6.1 directly follows from Lemmas 6.2 and 6.3, which we prove in Appendix D.1. To avoid the extra -O (1) term in the lower bound, our actual proof (deferred to Appendix D.3) works with the slightly different complexity measure σ γ ( D ) := E [ γ (Var T )] defined in Section 5.

7 Forecasting with O ( √ T ) SSCE

In this section, we prove Theorem 1.3, which states the existence of a deterministic forecaster that incurs an O ( √ T ) SSCE against all adaptive adversaries. Recall the definition of the smooth calibration error ( smCE ) from Section 2. Using standard chaining arguments, we can show the following relation between SSCE and smCE , whose proof we defer to Appendix F.

Lemma 7.1. For any x ∈ { 0 , 1 } T and p ∈ [0 , 1] T ,

where the O ( · ) notation hides a universal constant that does not depend on T , x or p .

Theorem 1.3 follows from the lemma above and a recent result of [ACRS24].

Proof of Theorem 1.3. It was shown by [ACRS24] that there exists a deterministic forecaster with an O ( √ T ) distance from calibration ( CalDist ( x, p ) ) against every adaptive adversary in the adversarial sequential calibration setup. Lemma 7.1 together with the inequality 1 2 smCE ( x, p ) ≤ CalDist ( x, p ) from [BGHN23, Lemma 5.4 and Theorem 7.3] implies that

so the same forecaster incurs an SSCE of O ( √ T ) as well.

8 Discussion

We formulate three natural desiderata of calibration measures that evaluate the quality of probabilistic forecasts: truthfulness, completeness, and soundness. They serve as minimal requirements for an error metric to be considered as measuring calibration and not to create a significant incentive for forecasters to predict untruthfully. While existing calibration measures fail to simultaneously meet all these criteria, we propose the new calibration measure (SSCE) that is shown to be approximately truthful via a non-trivial analysis. In the following, we discuss two natural directions of future work.

Inherent trade-offs among different desiderata? As shown in Table 1, the SSCE and the error metrics induced by proper scoring rules give a trade-off between truthfulness and completeness: The former is complete and approximately truthful, while the latter is perfectly truthful but not complete. Is there a calibration measure that achieves the best of both worlds? Taking a step back, while our definition of truthfulness seems natural, the completeness and soundness criteria, as defined, only serve as minimal requirements. It still remains to explore ways to formally quantify the latter two, and investigate the inherent quantitative trade-offs among truthfulness, completeness and soundness.

Truthfulness against adaptive adversaries? One may wonder whether the truthfulness guarantee of SSCE can be extended to handle adaptive adversaries as well. Assuming that the forecaster is given an adversary's (randomized) strategy for choosing x t based on x 1:( t -1) and p 1:( t -1) , is it still approximately optimal to always predict the conditional probability? Here, 'adaptive' emphasizes that x t may depend on both x 1:( t -1) and p 1:( t -1) ; the formulation in Section 2 is equivalent to that x t only depends on x 1:( t -1) .

Unfortunately, as we show in Appendix G, such a guarantee does not hold for SSCE , and is unlikely to hold for any natural calibration measure: An adversary can 'force' the forecaster to predict untruthfully by 'threatening' to increase the variance of the subsequent bits. However, this adversary is highly contrived and unrealistic for practical scenarios. We may thus identify reasonable restrictions on the adaptive adversary to sidestep this counterexample.

References

[ACRS24] Eshwar Ram Arunachaleswaran, Natalie Collina, Aaron Roth, and Mirah Shi. An elementary predictor obtaining 2 √ T distance to calibration. arXiv preprint arXiv:2402.11410 , 2024.

[BF + 02] Henri Berestycki, Igor Florent, et al. Asymptotics and calibration of local volatility models. Quantitative finance , 2(1):61, 2002.

[BGHN23] Jarosław Błasiok, Parikshit Gopalan, Lunjia Hu, and Preetum Nakkiran. A unifying theory of distance from calibration. In Symposium on Theory of Computing (STOC) , pages 1727-1740, 2023.

[Bla23] Guy Blanc. Subsampling suffices for adaptive data analysis. In Symposium on Theory of Computing (STOC) , pages 999-1012, 2023.

[BLMT22] Guy Blanc, Jane Lange, Ali Malik, and Li-Yang Tan. On the power of adaptivity in statistical adversaries. In Conference on Learning Theory (COLT) , pages 5030-5061, 2022.

[Bri50] Glenn W. Brier. Verification of forecasts expressed in terms of probability. Monthly Weather Review , 78(1):1-3, 1950.

[CAT16] Cynthia S Crowson, Elizabeth J Atkinson, and Terry M Therneau. Assessing calibration of prognostic risk scores. Statistical methods in medical research , 25(4):1692-1706, 2016.

[CY21] Yiling Chen and Fang-Yi Yu. Optimal scoring rule design. arXiv preprint arXiv:2107.07420 , 2021.

[Daw82] A. P. Dawid. The well-calibrated bayesian. Journal of the American Statistical Association , 77(379):605-610, 1982.

[Daw85] A. P. Dawid. Calibration-based empirical probability. The Annals of Statistics , 13(4):1251-1274, 1985.

[DF83] Morris H DeGroot and Stephen E Fienberg. The comparison and evaluation of forecasters. Journal of the Royal Statistical Society: Series D (The Statistician) , 32(1-2):12-22, 1983.

[Dud87] R. M. Dudley. Universal donsker classes and metric entropy. The Annals of Probability , 15(4):1306-1326, 1987.

[FH18] Dean P. Foster and Sergiu Hart. Smooth calibration, leaky forecasts, finite recall, and nash dynamics. Games and Economic Behavior , 109:271-293, 2018.

[FH21] Dean P. Foster and Sergiu Hart. Forecast hedging and calibration. Journal of Political Economy , 129(12):3447-3490, 2021.

[FL99] Drew Fudenberg and David K. Levine. An easier way to calibrate. Games and Economic Behavior , 29(1-2):131-137, 1999.

[Fos99] Dean P. Foster. A proof of calibration via blackwell's approachability theorem. Games and Economic Behavior , 29(1-2):73-78, 1999.

[Fre75] David A. Freedman. On tail probabilities for martingales. The Annals of Probability , 3(1):100-118, 1975.

[FRST11] Dean P. Foster, Alexander Rakhlin, Karthik Sridharan, and Ambuj Tewari. Complexitybased approach to calibration with checking rules. In Conference on Learning Theory (COLT) , pages 293-314, 2011.

[FV97] Dean P. Foster and Rakesh V. Vohra. Calibrated learning and correlated equilibrium. Games and Economic Behavior , 21(1-2):40-55, 1997.

[FV98] Dean P. Foster and Rakesh V. Vohra. Asymptotic calibration. Biometrika , 85(2):379390, 1998.

[GPSW17] Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On calibration of modern neural networks. In International Conference on Machine Learning (ICML) , pages 1321-1330, 2017.

[Har22] Sergiu Hart. Calibrated forecasts: The minimax proof. arXiv preprint arXiv:2209.05863 , 2022.

[HJKRR18] Ursula Hébert-Johnson, Michael Kim, Omer Reingold, and Guy Rothblum. Multicalibration: Calibration for the (computationally-identifiable) masses. In International Conference on Machine Learning (ICML) , pages 1939-1948, 2018.

[HJZ23] Nika Haghtalab, Michael Jordan, and Eric Zhao. A unifying perspective on multicalibration: Game dynamics for multi-objective learning. In Advances in Neural Information Processing Systems (NeurIPS) , pages 72464-72506, 2023.

[HPY23] Nika Haghtalab, Chara Podimata, and Kunhe Yang. Calibrated Stackelberg games: Learning optimal commitments against calibrated agents. In Advances in Neural Information Processing Systems (NeurIPS) , pages 61645-61677, 2023.

[HSLW23] Jason D. Hartline, Liren Shan, Yingkai Li, and Yifan Wu. Optimal scoring rules for multi-dimensional effort. In Conference on Learning Theory (COLT) , pages 2624-2650, 2023.

[HW24] Lunjia Hu and Yifan Wu. Predict to minimize swap regret for all payoff-bounded tasks. arXiv preprint arXiv:2404.13503 , 2024.

[JOKOM12] Xiaoqian Jiang, Melanie Osl, Jihoon Kim, and Lucila Ohno-Machado. Calibrating predictive model estimates to support personalized medicine. Journal of the American Medical Informatics Association , 19(2):263-274, 2012.

[KF08] Sham M. Kakade and Dean P. Foster. Deterministic calibration and Nash equilibrium. Journal of Computer and System Sciences , 74(1):115-130, 2008.

[KLST23] Robert Kleinberg, Renato Paes Leme, Jon Schneider, and Yifeng Teng. U-calibration: Forecasting for an unknown agent. In Conference on Learning Theory (COLT) , pages 5143-5145, 2023.

[KMR17] Jon Kleinberg, Sendhil Mullainathan, and Manish Raghavan. Inherent trade-offs in the fair determination of risk scores. In Innovations in Theoretical Computer Science (ITCS) , pages 43:1-43:23, 2017.

[KSB21] Benjamin Kompa, Jasper Snoek, and Andrew L Beam. Second opinion needed: communicating uncertainty in medical machine learning. NPJ Digital Medicine , 4(1):4, 2021.

[KSJ18] Aviral Kumar, Sunita Sarawagi, and Ujjwal Jain. Trainable calibration measures for neural networks from kernel mean embeddings. In International Conference on Machine Learning (ICML) , pages 2805-2814, 2018.

[LHSW22] Yingkai Li, Jason D. Hartline, Liren Shan, and Yifan Wu. Optimization of scoring rules. In Economics and Computation (EC) , pages 988-989, 2022.

[MW84] Allan H Murphy and Robert L Winkler. Probability forecasting in meteorology. Journal of the American Statistical Association , 79(387):489-500, 1984.

[NNW21] Eric Neyman, Georgy Noarov, and S. Matthew Weinberg. Binary scoring rules that incentivize precision. In Economics and Computation (EC) , pages 718-733, 2021.

[NRRX23] Georgy Noarov, Ramya Ramalingam, Aaron Roth, and Stephan Xie. High-dimensional unbiased prediction for sequential decision making. In OPT 2023: Optimization for Machine Learning , 2023.

[PRW + 17] Geoff Pleiss, Manish Raghavan, Felix Wu, Jon Kleinberg, and Kilian Q. Weinberger. On fairness and calibration. Advances in Neural Information Processing Systems (NIPS) , pages 5680-5689, 2017.

[PW22] Maneesha Papireddygari and Bo Waggoner. Contracts with information acquisition, via scoring rules. In Economics and Computation (EC) , pages 703-704, 2022.

[QV21] Mingda Qiao and Gregory Valiant. Stronger calibration lower bounds via sidestepping. In Symposium on Theory of Computing (STOC) , pages 456-466, 2021.

[QZ24] Mingda Qiao and Letian Zheng. On the distance from calibration in sequential prediction. In Conference on Learning Theory (COLT) , pages 4307-4357, 2024.

[RS24] Aaron Roth and Mirah Shi. Forecasting for swap regret for all downstream agents. arXiv preprint arXiv:2402.08753 , 2024.

[She10] I. G. Shevtsova. An improvement of convergence rate estimates in the Lyapunov theorem. Doklady Mathematics , 82(3):862-864, 2010.

[Ste22] Thomas Steinke. Composition of differential privacy & privacy amplification by subsampling. arXiv preprint arXiv:2210.00597 , 2022.

[VCV15] Ben Van Calster and Andrew J Vickers. Calibration of risk prediction models: impact on decision-analytic performance. Medical decision making , 35(2):162-169, 2015.

[WM68] Robert L. Winkler and Allan H. Murphy. 'Good' probability assessors. Journal of Applied Meteorology and Climatology , 7(5):751-758, 1968.

A Taxonomy of Existing Calibration Measures

In this section, we prove that the existing calibration measures in Table 2 either have a large truthfulness gap or lack completeness.

In these proofs, the biases induced by specific outcomes and predictions will be frequently used: With respect to outcomes x ∈ { 0 , 1 } T and predictions p ∈ [0 , 1] T , the bias associated with value α ∈ [0 , 1] is defined as

A.1 Existing Calibration Measures

The expected calibration error. A common calibration measure is the sum of L 1 errors of each level set, known as the L 1 calibration error or the Expected Calibration Error (ECE) : On x ∈ { 0 , 1 } T and p ∈ [0 , 1] T , the expected calibration error is defined as

Note that the summand | ∆ α | is non-zero only if α ∈ { p 1 , p 2 , . . . , p T } , so the summations above are essentially finite and well-defined.

The smooth calibration error. The smooth calibration error [KF08] is defined as

̸

where F is the family of 1 -Lipschitz functions from [0 , 1] to [ -1 , 1] . Again, since ∆ α = 0 holds only if α ∈ { p 1 , p 2 , . . . , p T } , the summation above is finite and well-defined.

The distance from calibration. The distance from calibration , introduced by [BGHN23] and extended to the sequential setup by [QZ24], is defined as:

where is the set of predictions that are perfectly calibrated for x .

Interval calibration. The interval calibration error of [BGHN23] relaxes the ECE to a binned version while penalizing the use of long intervals. Formally, an interval partition I is a finite collection of intervals { I 1 , I 2 , . . . , I |I| } that form a partition of [0 , 1] . The interval calibration error is defined as:

where the infimum is over all interval partitions I , and len ( I ) denotes the length of interval I . Note that the first summation inside the infimum is analogous to the ECE, except that the biases associated with all values within the same interval are added together. The second summation gives the total lengths of the intervals into which the T predictions fall.

Laplace-kernel calibration. The Laplace-kernel calibration error [BGHN23] is a special case of the maximum mean calibration error introduced by [KSJ18]. It can be viewed as a variant of the smooth calibration error, in which the family F of Lipschitz functions is replaced by

where ∥ · ∥ 2 denotes the ℓ 2 norm of functions, and f ′ is the derivative of f . Namely,

U-calibration. The definition of the U-calibration error [KLST23] is based on proper scoring rules . A (bounded) scoring rule is a function S : { 0 , 1 } × [0 , 1] → [ -1 , 1] . A scoring rule is proper if it holds for every α ∈ [0 , 1] that

In other words, when the outcome x is drawn from follow Bernoulli ( α ) , predicting the true parameter α minimizes the expected loss. The U-calibration error is then defined as

where the supremum is over all proper scoring rules. Note that for each fixed S , the expression inside the supremum is exactly the external regret of the forecaster, i.e., the excess loss compared to the best fixed prediction in hindsight.

Maximum swap regret. A recent line of work [NRRX23, RS24, HW24] considers a strengthening of U-calibration, in which the external regret is replaced with the swap regret . In particular, [HW24] showed that the resulting calibration measure, termed the Maximum Swap Regret (MSR), is polynomially related to the ECE after scaling by a factor of 1 /T :

A.2 0 -Ω( T ) Truthfulness Gaps

We first prove the 0 -Ω( T ) truthfulness gaps of the ECE and the MSR.

Proposition A.1. Both the expected calibration error and the maximum swap regret have a 0 -Ω( T ) truthfulness gap.

To establish Proposition A.1, we follow a similar argument to the one in Section 3: We divide the time horizon into T/ 3 triples, each containing a random bit followed by a zero and a one. The truthful forecaster would predict the true probabilities for the T/ 3 random bits, which are designed to be close to 1 / 2 but distinct. This leads to a linear ECE. On the other hand, a strategic forecaster may always predict 1 / 2 on the random bit. Then, based on the realization of the random bit, they use the subsequent deterministic bits to offset the bias. The resulting predictions are perfectly calibrated, and thus have a zero ECE. Finally, the relation between the ECE and the MSR gives the same truthfulness gap for the MSR.

Proof of Proposition A.1. Consider the distribution D defined as follows:

Let ε 1 , ε 2 , . . . , ε T/ 3 be distinct values in [ -1 / 4 , 1 / 4] chosen arbitrarily.

For each k ∈ [ T/ 3] , set ( p ⋆ 3 k -2 , p ⋆ 3 k -1 , p ⋆ 3 k ) = (1 / 2 + ε k , 0 , 1) .

D is the product distribution ∏ T t =1 Bernoulli ( p ⋆ t ) .

By definition, the predictions made by the truthful forecaster are exactly given by p ⋆ . Then, for each k ∈ [ T/ 3] and α = 1 / 2 + ε k ∈ [1 / 4 , 3 / 4] , we have | ∆ α | = | x 3 k -2 -α | ≥ 1 / 4 . This shows err ECE ( D , A truthful ( D )) ≥ ( T/ 3) · (1 / 4) = Ω( T ) . By the inequality MSR ( x,p ) T ≥ [ ECE ( x,p ) T ] 2 , we also have err MSR ( D , A truthful ( D )) = Ω( T ) .

On the other hand, consider the following alternative forecaster for D :

For each k ∈ [ T/ 3] , predict p 3 k -2 = 1 / 2 .

If x 3 k -2 = 0 , predict p 3 k -1 = 0 and p 3 k = 1 / 2 ; otherwise, predict p 3 k -1 = 1 / 2 and p 3 k = 1 .

Clearly, for each k ∈ [ T/ 3] , the steps t ∈ { 3 k -2 , 3 k -1 , 3 k } have zero contribution to ∆ 0 , ∆ 1 and ∆ 1 / 2 . Therefore, this forecaster achieves a zero ECE on D . This proves OPT ECE ( D ) = 0 and establishes the 0 -Ω( T ) truthfulness gap for the ECE. Finally, the inequality MSR ( x,p ) T ≤ 2 ECE ( x,p ) T implies that the same forecaster achieves a zero MSR, which establishes OPT MSR ( D ) = 0 and the 0 -Ω( T ) truthfulness gap for the MSR.

A.3 0 -Ω( √ T ) Truthfulness Gaps

√

Next, we prove the 0 -Ω( T ) truthfulness gap for several calibration measures. The proof follows the argument outlined in Section 3.

Proposition A.2. The smooth calibration error, the distance from calibration, the interval calibration error, and the Laplace-kernel calibration error all have a 0 -Ω( √ T ) truthfulness gap.

Proof. The truthfulness gaps of the four calibration measures are witnessed by the same product distribution D = ∏ T t =1 Bernoulli ( p ⋆ t ) , where ( p ⋆ 3 k -2 , p ⋆ 3 k -1 , p ⋆ 3 k ) = (1 / 2 , 0 , 1) for every k ∈ [ T/ 3] .

Truthful forecaster has an Ω( √ T ) penalty. The truthful forecaster makes predictions that are identical to p ⋆ . As a result, we have ∆ 0 = ∆ 1 = 0 , while ∆ 1 / 2 is distributed as the difference between a sample from Binomial ( T/ 3 , 1 / 2) and its mean T/ 6 . It then follows that | ∆ 1 / 2 | ≥ Ω( √ T ) holds with probability Ω(1) . We will show that all four calibration measures evaluate to Ω( √ T ) in expectation.

For the smooth calibration error, we have

For the distance from calibration, by [BGHN23, Lemma 5.4 and Theorem 7.3], we have the inequality 1 2 smCE ( x, p ) ≤ CalDist ( x, p ) for any x ∈ { 0 , 1 } T and p ∈ [0 , 1] T , so the truthful forecaster also gives err CalDist ( D , A truthful ( D )) = Ω( √ T ) .

For interval calibration, let I be an arbitrary interval partition, and I ∈ I be the interval that contains 1 / 2 . If I contains either 0 or 1 , we must have len ( I ) ≥ 1 / 2 , and the term ∑ T t =1 ∑ |I| i =1 len ( I i ) · ✶ [ p t ∈ I i ] will be at least 2 T/ 3 · 1 / 2 = Ω( T ) . If I does not contain 0 or 1 , the summation ∑ T t =1 ( x t -p t ) · ✶ [ p t ∈ I ] will be exactly ∆ 1 / 2 , and the first term in the definition will be at least | ∆ 1 / 2 | . It follows that intCE ( x, p ) ≥ Ω( √ T ) with probability Ω(1) , so we have the lower bound err intCE ( D , A truthful ( D )) = Ω( √ T ) .

For Laplace-kernel calibration, let f 0 be an arbitrary function in ˜ F such that f 0 (1 / 2) > 0 , e.g., we can take f 0 ( x ) = ce -x 2 for a sufficiently small constant c > 0 . Then, we have

It follows that err kCE Lap ( D , A truthful ( D )) ≥ Ω(1) · E [ | ∆ 1 / 2 | ] = Ω( √ T ) .

Strategic forecaster has a zero penalty. Consider the same strategic forecaster as in the proof of Proposition A.1: For each k ∈ [ T/ 3] ,

Predict p 3 k -2 = 1 / 2 .

If x 3 k -2 = 0 , predict ( p 3 k -1 , p 3 k ) = (0 , 1 / 2) ; otherwise, predict ( p 3 k -1 , p 3 k ) = (1 / 2 , 1) .

Clearly, this guarantees that ∆ α = 0 holds for all α ∈ [0 , 1] . By definition, we have OPT smCE ( D ) = OPT CalDist ( D ) = 0 . It also easily follows that both intCE and kCE Lap evaluate to 0 . For intCE , we consider the interval partition I = {{ 0 } , (0 , 1 / 2) , { 1 / 2 } , (1 / 2 , 1) , { 1 }} , which witnesses intCE ( x, p ) = 0 . For kCE Lap , the summation ∑ T t =1 f ( p t ) · ( x t -p t ) = ∑ α ∈ [0 , 1] f ( α ) · ∆ α evaluates to 0 for all f ∈ ˜ F . This proves OPT intCE ( D ) = OPT kCE Lap ( D ) = 0 .

A.4 O (1) -Ω( √ T ) Truthfulness Gap of U-Calibration

For the U-calibration error, we prove a slightly smaller truthfulness gap of O (1) -Ω( √ T ) , via a more involved analysis.

Proposition A.3. The U-calibration error has an O (1) -Ω( √ T ) truthfulness gap.

Proof. We use a slightly different construction: the product distribution D = ∏ T t =1 Bernoulli ( p ⋆ t ) where p ⋆ t = 1 / 2 for t ≤ T/ 2 and p ⋆ t = 1 for t > T/ 2 .

Truthful forecaster has an Ω( √ T ) penalty. We first show that the truthful forecaster has an Ω( √ T ) U-calibration error. Let random variable X := ∑ T/ 2 t =1 x t denote the number of ones among the first T/ 2 random bits. Note that X follows Binomial ( T/ 2 , 1 / 2) . Consider the scoring rule defined as:

Note that S is proper, since for any α ∈ [0 , 1] , we have

which is always minimized at β = α .

The total loss (w.r.t. S ) incurred by the forecaster is then

On the other hand, the total loss incurred by a fixed prediction β ∈ [0 , 1] is given by:

By choosing β = 1 , we can obtain a total loss of -2 X . Therefore, whenever X ≥ T/ 4 , we have

When X < T/ 4 , we always have UCal ( x, p ) ≥ 0 , since the trivial scoring rule S ≡ 0 is proper. This shows that the truthful forecaster gives

Strategic forecaster with an O (1) penalty. We consider an alternative forecaster A , which is slightly more involved:

At every step t ≤ T/ 2 , predict p t = 5 / 8 .

For t = T/ 2 + 1 , T/ 2 + 2 , . . . , T , predict p t = 5 / 8 until | ∆ 5 / 8 | ≤ 1 at some time t . After that step, predict p t = 1 .

We first argue that the condition | ∆ 5 / 8 | ≤ 1 must hold at some point. Recall that X = ∑ T/ 2 t =1 x t . By a Chernoff bound, X falls into [ T/ 8 , 5 T/ 16] except with probability e -Ω( T ) . Assuming this, we have ∆ 5 / 8 = X -( T/ 2) · (5 / 8) ≤ 0 at time t = T/ 2 . Furthermore, if we hypothetically predict 5 / 8 for each of the last T/ 2 steps, we would have

after all the T steps. Since ∆ 5 / 8 changes by at most 1 at each step, we must hit the condition | ∆ 5 / 8 | ≤ 1 at some point.

Therefore, except with an e -Ω( T ) probability, we end up with ∆ 5 / 8 ∈ [ -1 , 1] . Furthermore, we predict at most two different values: 5 / 8 and 1 . For every fixed proper scoring rule S : { 0 , 1 } × [0 , 1] → [ -1 , 1] , we have

In the above, we divide the time horizon [ T ] into two parts, based on whether 5 / 8 or 1 is predicted. The inequality holds since the right-hand side allows different values of β for different parts. Clearly, the term corresponding to α = 1 has zero contribution, since it reduces to S (1 , 1) -inf β ∈ [0 , 1] S (1 , β ) times the number of times 1 is predicted, which evaluates to 0 by definition of proper scoring rules.

The term corresponding to α = 5 / 8 , on the other hand, is given by

where each N b denotes the number of steps on which 5 / 8 is predicted and the outcome is b ∈ { 0 , 1 } . By definition of proper scoring rules, the infimum is achieved by β ⋆ = N 1 N 0 + N 1 , and the above can be further simplified into

where S ( α, β ) := α · S (1 , β ) + (1 -α ) · S (0 , β ) is the linear extension of S to [0 , 1] 2 .

Let ℓ ( α ) := S ( α, α ) denote the uni-variate form of S . The following is a standard fact about proper scoring rules (see e.g., [KLST23, Lemma 1 and Corollary 2]).

Lemma A.4. For any proper scoring rule S : [0 , 1] 2 → [ -1 , 1] and its uni-variate form ℓ : [0 , 1] → [ -1 , 1] , it holds for all α, β ∈ [0 , 1] that

S ( α, β ) = ℓ ( β ) + ( α -β ) · ℓ ′ ( β )

| ℓ ′ ( α ) | ≤ 2 for all α ∈ [0 , 1] .

In particular, we have

and

It follows that, assuming X ∈ [ T/ 8 , 5 T/ 16] ,

When X ∈ [ T/ 8 , 5 T/ 16] does not hold (which happens with probability e -Ω( T ) ), the U-calibration error is trivially upper bounded by O ( T ) . It follows that

A.5 Lack of Completeness

Every scoring rule S : { 0 , 1 } × [0 , 1] → [0 , 1] induces a calibration measure CM ( S ) ( x, p ) := ∑ T t =1 S ( x t , p t ) . 1 When S is proper, it is easy to show that the resulting CM ( S ) is perfectly truthful, i.e., (1 , 0) -truthful.

A drawback of such calibration measures is that they all lack completeness. Concretely, consider the squared loss S ( x, p ) := ( x -p ) 2 . When the outcomes x 1 , x 2 , . . . , x T are independent and uniformly random bits, the 'right' prediction p t ≡ 1 / 2 gives a total penalty of T/ 4 , which is only a constant factor away from the maximum possible penalty of T . This violates the completeness property in Definition 2.2. In contrast, as shown in Table 2, almost all the other calibration measures would evaluate to ≪ T in this case. Such an asymptotic gap better justifies the intuition that p t ≡ 1 / 2 is a much better prediction than, say, p t ≡ 0 .

More generally, unless the proper scoring rule S is trivial, we may find ( x 0 , p 0 ) ∈ { 0 , 1 } × (0 , 1) such that S ( x 0 , p 0 ) > 0 . Then, on a sequence of independent samples from Bernoulli ( p 0 ) , we have

which violates the completeness condition in Definition 2.2.

We also note that smCE ( x, p ) + √ T gives a calibration measure that is trivially truthful: Implicit in the proof of [QZ24, Theorem 3], the truthful forecaster gives an O ( √ T ) smooth calibration error on every distribution D ∈ ∆( { 0 , 1 } T ) , so it immediately gives a constant approximation of the optimal

error, which is at least √ T . However, this metric is not complete in the sense of Definition 2.2, since it evaluates to √ T instead of 0 when p = x (i.e., the predictions are binary and perfect). While SSCE also discourages the forecaster from 'over-optimizing' the metric by introducing some additional noise, the subsampling procedure is arguably more 'organic' and better-justified than adding a √ T term.

B Proof of Theorem 1.1

We prove Theorem 1.1, which we formally restate below.

Theorem B.1 (Formal version of Theorem 1.1) . For every p ⋆ ∈ [0 , 1] T , on the product distribution D = ∏ T t =1 Bernoulli ( p ⋆ t ) , there is a forecaster that achieves an O (log 3 / 2 T ) smooth calibration error and distance from calibration. Moreover, assuming that p ⋆ ∈ [ δ, 1 -δ ] T for a fixed constant δ ∈ (0 , 1 / 2] , both OPT smCE ( D ) and OPT CalDist ( D ) are Ω( √ T ) .

B.1 The Upper Bound Part

We start by proving the upper bound part of Theorem B.1 by designing a forecasting algorithm.

The forecasting algorithm. Our proof is based on an algorithm of [QZ24] that works for the special case that p ⋆ t ≡ 1 / 2 . Their algorithm starts by predicting 1 / 2 on the first T/ 2 steps. Depending on the realization of these T/ 2 random bits, it predicts a slightly biased value for the next T/ 2 steps, until the total bias (i.e., the partial sum of x t -p t ) becomes close to 0 at some point. If there is still time left, the algorithm repeats the above strategy for the remainder of the time horizon.

Roughly speaking, [QZ24] shows that a polylog( T ) distance from calibration can be achieved by designing a sub-routine with the following three properties:

Small bias: With high probability, the total bias is O (1) in magnitude at some time t ∈ [ T/ 2 , T ] .

Proximity of predictions: During the sub-routine, the values being predicted lie in a short interval of length polylog( T ) / √ T .

Sparsity of predictions: During the sub-routine, only O (1) different values are predicted.

To handle the general case that p ⋆ ∈ [0 , 1] T is arbitrary, we design an alternative sub-routine, the behavior of which depends on whether the sequence p ⋆ is 'sufficiently stationary' in some sense. Let µ first := 1 T/ 2 ∑ T/ 2 t =1 p ⋆ t and µ second := 1 T/ 2 ∑ T t = T/ 2+1 p ⋆ t be the averages of the first and the second halves of the sequence, respectively. Let µ = ( µ first + µ second ) / 2 be the overall average.

Case 1: | µ first -µ | > polylog( T ) / √ T . When µ first and µ are far away, we predict α := µ first + µ 2 at every step. Without loss of generality, suppose that µ first < µ , in which case we have

where both inequalities hold with a margin > polylog( T ) / T . Then, with high probability the following two events happen: (1) The total bias is negative at time T/ 2 , i.e., it holds that ∑ T/ 2 t =1 x t < α · ( T/ 2) ; (2) If we (hypothetically) predict the same value α for the second half, the bias will be positive in the end with high probability, i.e., ∑ T t =1 x t > α · T . Therefore, with high probability, the bias must be close to 0 at some point in [ T/ 2 , T ] . In this case, this sub-routine has all the desired properties.

√

Case 2: | µ first -µ | ≤ polylog( T ) / T . When µ first and µ are close, we use a strategy that is more similar to the algorithm of [QZ24]. For the first half of the sequence, we predict α := µ first . Let ∆ first := ∑ T/ 2 t =1 ( x t -α ) denote the total bias at time T/ 2 . Say that ∆ first ≥ 0 . Then, we will predict β := µ second + ∆ first T/ 2 + polylog( T ) √ T in the second half of the sequence. The value of β is chosen such that we can offset the bias incurred in the first half (i.e., the ∆ first / ( T/ 2) term). We also introduce some additional bias (i.e.,

√

the polylog( T ) / √ T term), so that we can return to a zero bias with high probability. In this case, our sub-routine predicts two different values ( α and β ), and they only differ by polylog( T ) / √ T with high probability.

Formally, our algorithm is given in Algorithm 1. The actual algorithm is significantly more involved than the outline above. The complication is due to the constraint that all predictions must lie in [0 , 1] , while our choice of β in Case 2 above might be invalid. We circumvent this issue by noting that β can be invalid only if µ first is too close to either 0 or 1 . In that case, we will choose a different value of α (i.e., the prediction for the first half), so that the sign of the bias at time T/ 2 is more predictable, and the resulting choice of β will likely be valid.

Algorithm 1: Forecaster for Product Distributions

The analysis. We analyze Algorithm 1 and prove the upper bound in Theorem B.1 in the following three steps:

First, we break the execution of Algorithm 1 into different rounds of the while-loop, and show that each round brings a polylog( T ) smooth calibration error in expectation.

Then, using the simple observation that the smooth calibration error is sub-additive, we obtain an upper bound on the overall smooth calibration error.

Finally, we use a relation between smCE ( x, p ) and CalDist ( x, p ) when p only contains a few different values (shown by [QZ24]) to translate the upper bound to one on the distance from calibration.

The first step is the most technical. We fix r and condition on the value of t (equivalently, the value of T ( r ) ) at the beginning of the r -th round. Note that the event t = t 0 is solely determined by the realization of x 1 , x 2 , . . . , x t 0 , so conditioning on the value of t , the subsequent bits x t +1 through x T are still distributed according to D . Let sequences x ( r ) and p ( r ) denote the outcomes and predictions made in the r -th round. Note that the two sequences are of the same length, though the length might vary.

We classify the rounds into three different types as follows:

Type 1: The condition | µ ( r ) first -µ ( r ) | ≥ √ 2 ln T ( r ) H ( r ) holds in the if-statement on Line 7.

Type 2: | µ ( r ) first -µ ( r ) | < √ 2 ln T ( r ) H ( r ) , and α ( r ) is set to µ ( r ) first on either Line 16 or Line 19.

Type 3: | µ ( r ) first -µ ( r ) | < √ 2 ln T ( r ) H ( r ) , and α ( r ) is not set to µ ( r ) first .

Note that for fixed p ⋆ , the type of a round is deterministic given r and T ( r ) .

The three lemmas below give high-probability bounds on the smooth calibration error incurred during each round.

Lemma B.2. Conditioning on the value of T ( r ) , if the r -th round is Type 1, it holds with probability 1 -O (1 /T ( r ) ) that

Lemma B.3. Conditioning on the value of T ( r ) , if the r -th round is Type 2, it holds with probability 1 -O (1 /T ( r ) ) that

where ∆ ( r ) first denotes the value of ∆ ( r ) at the end of the first for-loop (on Line 25).

Lemma B.4. Conditioning on the value of T ( r ) , if the r -th round is Type 3, it holds with probability 1 -O (1 /T ( r ) ) that

where ∆ ( r ) first denotes the value of ∆ ( r ) at the end of the first for-loop (on Line 25).

We first prove the upper bound part of Theorem B.1 using the lemmas above.

Proof of Theorem B.1, the upper bound part. By Lemmas B.2 through B.4, regardless of the type of the r -th round, it holds with probability 1 -O ( 1 /T ( r ) ) that

where ∆ ( r ) first is regarded as 0 if the r -th round is Type 1. We say that the round fails if this upper bound on smCE does not hold. Conditioning on that T ( r ) = L , we always have smCE ( x ( r ) , p ( r ) ) ≤ L , since there are at most L steps in the r -th round. Therefore, we have the inequality

.

We will upper bound the value of E [ smCE ( x ( r ) , p ( r ) ) ] by taking an expectation over both sides of the above. Therefore, we examine the expectation of | ∆ ( r ) first | and [∆ ( r ) first ] 2 conditioning on T ( r ) = L When the round is Type 1, there is nothing to upper bound. For Type 2 rounds, ∆ ( r ) first is the difference between X first = ∑ t + H s = t +1 x s and its mean µ first H . Since the variance of X first is O ( L ) , we have E [∣ ∣ ∣ ∆ ( r ) first ∣ ∣ ∣ ] = O ( √ L ) and E [ [ ∆ ( r ) first ] 2 ] = O ( L ) .

Type 3 rounds are trickier. We assume that µ first ≤ 1 / 2 ; this is without loss of generality since the µ first > 1 / 2 case can be handled by a completely symmetric argument. Then, ∆ ( r ) first is the difference between X first = ∑ t + H s = t +1 x s and αH , and α may differ from µ first by at most √ 2 µ first ln L H . This gives

Now we use the fact that when µ first ≤ 1 / 2 , the round is Type 3 only if µ first < 10 √ ln T ( r ) H . This implies √

which is dominated by the O ( L ) term. It then follows from Jensen's inequality that

Put everything together. Therefore, we have the upper bound

The second step applies our earlier conclusion that E [ | ∆ ( r ) first | ] = O ( √ L ) and E [ [∆ ( r ) first ] 2 ] = O ( L ) conditioning on T ( r ) = L . Taking another expectation over the randomness in T ( r ) shows that smCE ( x ( r ) , p ( r ) ) = O ( √ log T ) for every r . Note that we have

Furthermore, there are at most O (log T ) rounds. It follows that E [ smCE ( x, p )] = O (log 3 / 2 T ) .

Finally, we note that in each round of the while-loop, the forecaster predicts at most 2 different values (namely, α ( r ) and β ( r ) ). Therefore, the predictions p 1 , p 2 , . . . , p T contain at most O (log T ) different values. By [QZ24, Theorem 2], we conclude that

Now we prove Lemmas B.2 through B.4. In the proofs below, we frequently drop the superscript ( r ) since we only refer to the r -th round.

Proof of Lemma B.2. Recall that a Type 1 round is one in which the condition | µ first -µ | ≥ √ 2 ln T ( r ) H holds in the if-statement. We say that the round succeeds , if we exit the for-loop using the 'break' statement on Line 12, i.e., the condition i > H and | ∆ ( r ) | ≤ 1 holds at some point (including in the last iteration where i = 2 H ); otherwise, the round fails .

Note that only one value (namely, α ( r ) ) is predicted within the round. Thus, if the round succeeds, we have

It remains to control the probability for a Type 1 round to fail. Consider random variables

Note that both are sums of independent Bernoulli random variables, with E [ X first ] = µ first H and E [ X ] = 2 µH . Also note that since α = ( µ first + µ ) / 2 , we have

Without loss of generality, suppose that µ first ≤ µ . By an additive Chernoff bound, we have

and

Therefore, except with probability O (1 /T ( r ) ) , we have both X first < αH and X > 2 αH . In other words, if the for-loop (hypothetically) runs all the 2 H iterations, we would have ∆ ( r ) < 0 at the end of the H -th iteration, and ∆ ( r ) > 0 at the end of the 2 H -th iteration. Since ∆ ( r ) changes by | x t -p t | ≤ 1 within each iteration, there must be an iteration i ∈ { H +1 , H +2 , . . . , 2 H } at the end of which ∆ ( r ) falls into [0 , 1] . By definition of Algorithm 1, we exit the for-loop at that time, and the r -th round succeeds.

Proof of Lemma B.3. Recall that in a Type 2 round, we have | µ first -µ | < √ 2 ln T ( r ) H and α = µ first . Without loss of generality, suppose that µ first ≤ 1 / 2 ; the case that µ first > 1 / 2 follows from a completely symmetric argument. We say that a Type 2 round succeeds if both conditions below are satisfied:

When β is chosen, the clipping (i.e., taking the minimum with 1 or taking the maximum with 0 ) is not effective.

We exit the second for-loop through the break statement on Line 30.

Otherwise, the round fails .

Again, we first upper bound the smooth calibration error incurred within a successful round, and then control the probability for a round to fail. Since only α and β are predicted in this round, we have

where ∆ α and ∆ β are defined with respect to x ( r ) and p ( r ) . The above is further given by

Note that ∆ α +∆ β is exactly the value of ∆ ( r ) at the end of the second for-loop, while ∆ α is its value after the first for-loop, i.e., ∆ ( r ) first . Then, assuming that the round succeeds, we have | ∆ α +∆ β | ≤ 1 and

Plugging the above back into the upper bound on smCE ( x ( r ) , p ( r ) ) shows that in a successful Type 2 round,

In the following, we show that a Type 2 round succeeds with probability 1 -O (1 /T ( r ) ) . Let X first := ∑ t + H s = t +1 x s . Note that X first is a sum of H independent Bernoulli random variables and E [ X first ] = µ first H . Furthermore, we have ∆ ( r ) first = X first -µ first H . By an additive Chernoff bound, we have

Recall that we need to argue that no clipping is applied when β is chosen. We analyze the following two cases:

Case 1. ∆ ( r ) first ≥ 0 . In this case, we need to show that

√

Recall that we assumed µ first ≤ 1 / 2 and | µ first -µ | < 2 ln T ( r ) H . The latter further implies | µ first -µ second | = 2 | µ first -µ | < √ 8 ln T ( r ) H . Thus, it suffices to prove that

When | ∆ ( r ) first | ≤ √ H ln T ( r ) 2 (i.e., the event in Equation (3) holds), the left-hand side above is O (√ log T ( r ) T ( r ) ) , which is below 1 / 2 as long as T ( r ) exceeds some universal constant T 0 .

Therefore, the probability that a clipping is applied is at most O (1 /T ( r ) ) , where we absorb the constraint T ( r ) ≥ T 0 into the hidden constant in O ( · ) .

Case 2. ∆ ( r ) first < 0 . In this case, we need to show that

Recall that the definition of Type 2 rounds implies µ first ≥ 10 √ ln T ( r ) H . Thus, it suffices to prove that

The above holds whenever the event in Equation (3) happens, since 10 -√ 2 -1 / √ 2 -1 / √ 2 > 0 .

Finally, we argue that, with high probability, we exit the second for-loop via the break statement. Let X second := ∑ t +2 H s = t + H +1 x s denote the total outcome in the second half. By symmetry, we only deal with the case that ∆ ( r ) first ≥ 0 , where we have β = µ second +∆ ( r ) first /H + √ ln T ( r ) 2 H . If the second for-loop runs all the H iterations in full, at the end of it, the value of ∆ ( r ) will be given by

Note that the above is non-negative only if X second ≤ µ second H + √ H ln T ( r ) 2 , which, by an additive Chernoff bound, holds with probability at most 1 /T ( r ) . Therefore, with probability 1 -1 /T ( r ) , the value of ∆ ( r ) must fall into [ -1 , 0] during the second for-loop, and we will take the break statement accordingly.

Proof of Lemma B.4. Again, without loss of generality, suppose that µ first ≤ 1 / 2 ; the other case follows from a completely symmetric argument. In contrast to Type 1 and Type 2 rounds, we say that a Type 3 round succeeds if all the following conditions hold simultaneously:

∆ ( r ) first ≥ 0 , i.e., ∆ ( r ) ≥ 0 holds at the end of the first for-loop (on Line 25).

When β is chosen, the clipping (i.e., taking the minimum with 1 ) is not effective.

We exit the second for-loop through the break statement on Line 30.

Otherwise, the round fails .

By the same argument as in the proof of Lemma B.3, in a successful Type 3 round, we have

The only change in the argument is the upper bound on | α -β | , since α is no longer equal to µ first . Nevertheless, we still have

and the rest of the analysis goes through.

Thus, it remains to show that a Type 3 round succeeds with probability 1 -O (1 /T ( r ) ) . Let X first := ∑ t + H s = t +1 x s . Note that X first is a sum of independent Bernoulli random variables and E [ X first ] = µ first H . By a multiplicative Chernoff bound, for any δ ≥ 0 , we have

In particular, plugging δ = √ 2 ln T ( r ) µ first H into the above gives

Recall that α is chosen as the maximum between µ first -√ 2 µ first ln T ( r ) H and 0 . Thus, with probability at least 1 -1 /T ( r ) , we have X first /H ≥ α , which is equivalent to ∆ ( r ) ≥ 0 at the end of the first for-loop.

Then, we need to argue that when β is chosen, we have µ second +∆ ( r ) /H + √ ln T ( r ) 2 H ≤ 1 . We will show the equivalent inequality:

For the first term, we note that since µ = ( µ first + µ second ) / 2 , the assumption | µ first -µ | < √ 2 ln T ( r ) H implies | µ first -µ second | = O (√ log T ( r ) T ( r ) ) . With the additional assumption that µ first ≤ 1 / 2 , we have

For the second term, we note that, at the end of the first for-loop, ∆ ( r ) /H is given by

By an additive Chernoff bound, X first H -µ first ≤ O (√ log T ( r ) T ( r ) ) holds with probability 1 -O (1 /T ( r ) ) . By our choice of α , µ first -α is always O (√ log T ( r ) T ( r ) ) . Finally, the last term is clearly O (√ log T ( r ) T ( r ) ) . Therefore, as long as T ( r ) is larger than a universal constant T 0 , the total O (√ log T ( r ) T ( r ) ) term is upper bounded by 1 / 2 . Again, we can absorb the condition T ( r ) ≥ T 0 into the bigO notation, so the second condition (that β is not clipped) is satisfied with probability 1 -O (1 /T ( r ) ) .

Finally, we argue that we exit the second for-loop via the break statement with high probability. Let X second := ∑ t +2 H s = t + H +1 x s denote the total outcome in the second half. Recall that we have ∆ ( r ) ≥ 0 at the end of the first for-loop, and that β = µ second +∆ ( r ) /H + √ ln T ( r ) 2 H . If the second for-loop runs all the H iterations in full, at the end of it, the value of ∆ ( r ) will be given by

Note that the above is non-negative only if X second ≤ µ second H + √ H ln T ( r ) 2 , which, by an additive Chernoff bound, holds with probability at most 1 /T ( r ) . Therefore, with probability 1 -1 /T ( r ) , the value of ∆ ( r ) must fall into [ -1 , 0] during the second for-loop, and we will take the break statement accordingly.

B.2 The Lower Bound Part

We prove the lower bound part of Theorem B.1 via a central limit theorem.

Proof of Theorem B.1, the lower bound part. On the product distribution D = ∏ T t =1 Bernoulli ( p ⋆ t ) , the truthful forecaster predicts p t = p ⋆ t at every step t . Then, we have

where we use the fact that F contains the constant functions f ≡ 1 and f ≡ -1 .

Applying the Berry-Esseen theorem [She10] to the random variable X := ∑ T t =1 ( x t -p ⋆ t ) gives:

where Φ( x ) is CDF of the standard normal distribution, C 0 ≤ 0 . 56 is a universal constant, and

In particular, taking x = -1 gives:

For all sufficiently large T , the O (1 / √ T ) term is dominated by the Ω(1) term, in which case we have

Finally, by the inequality 1 2 smCE ( x, p ) ≤ CalDist ( x, p ) [BGHN23, Lemma 5.4 and Theorem 7.3], the distance from calibration incurred by the truthful forecaster is also Ω( √ T ) .

C Supplemental Materials for Section 5

The following is a tighter version of Theorem 5.1.

Proof. Given a function f : [0 , 1] → [ -1 , 1] and binary vector y ∈ { 0 , 1 } T , we define the martingale M t ( f, y ) := ∑ t s =1 y s · f ( p ⋆ s ) · ( x s -p ⋆ s ) where x ∼ D and we use F t to denote the filtration describing the randomness of M T ( f, y ) up to time t and p ⋆ t := E [ x t | F t -1 ] . Note that, conditioned on F t -1 , x t is distributed as a Bernoulli with parameter p ⋆ t .

We can write the SSCE of a truthful forecaster in terms of M T ( f, y ) as

We now proceed via chaining and define the dyadic scale ε k = 2 1 -k for k = 0 , 1 , 2 , . . . . To cover the set of Lipschitz functions F , we will use the sets of piecewise constant functions {F δ } δ> 0 described in Lemma C.2. For each function f ∈ F , let π k ( f ) be a close function in F ε k such that d ( f, π k ( f )) ≤ 2 ε k . Observe that the covering F ε 0 is a singleton and that π k ( f ) always exists as F ε k is a 2 ε k -covering of F . Telescoping then gives

meaning that we have

First, we can use that d ( f ( p ⋆ t ) -π M ( f )( p ⋆ t )) ≤ 2 2 -M to deterministically bound Term A by

Second, we can observe that the image of π 0 ( f ) is a singleton: | { π 0 ( f ) | f ∈ F}| = 1 ; let this unique function be denoted by f ⋆ . Then, Term B reduces to E y ∼ Unif ( { 0 , 1 } T ) [ M T ( f ⋆ , y )] , which evaluates to 0 after taking an expectation over x ∼ D , since for every y ∈ { 0 , 1 } T , ( M t ( f ⋆ , y )) 0 ≤ t ≤ T forms a martingale. Third, we can observe that π i ( f ) -π i -1 ( f ) is a function from [0 , 1] → { -2 1 -i , 0 , 2 1 -i } that takes a constant value along the segments [( j -1)2 1 -i , j 2 1 -i ) for all j ∈ [2 i -1 ] . Thus, we can bound the summands of Term C by

Invoking Lemma C.9 with G = { x ↦→ 1 , x ↦→-1 } and I = [ j 2 1 -i , ( j +1)2 1 -i ) , we have that for all i ∈ [ M ] , j ∈ { 0 , 1 , . . . , 2 i -1 } , and y ∈ { 0 , 1 } T :

Plugging this into Term C, we have

Using Lemma C.3, we can simplify

Plugging this into (4) and observing that we can choose M to be arbitrarily large, we have as desired

C.1 Auxillary Lemmas

Covering Lipschitz functions. Let us first recall a standard covering of the class of Lipschitz functions F ⊆ [ -1 , 1] [0 , 1] . We will work with the metric d on the functions F induced by the ∞ -norm; that is, for any f, g ∈ F , d ( f, g ) := sup x ∈ [0 , 1] | f ( x ) -g ( x ) | . In this section, for δ > 0 and b > a where b -a δ ∈ Z , we will use the shorthand [ a, b ] δ := { a, a + δ, . . . , b } to denote endpoints of partitioning of [ a, b ] into segments of length δ . We will also use the shorthand ⌊ x ⌋ δ := max { iδ | iδ ≤ x, i ∈ Z } to denote rounding down to the nearest multiple of δ .

Lemma C.2. For δ > 0 where 1 δ ∈ Z , consider all functions f : [0 , 1] → [ -1 , 1] that satisfy conditions

∀ x ∈ [0 , 1] : f ( x ) ∈ [ -1 , 1]

∀ x ∈ [0 , 1] δ \ { 1 } : | f ( x + δ ) -f ( x ) | ≤ δ

∀ x ∈ [0 , 1] : f ( x ) = f ( ⌊ x ⌋ )

This set of functions, which we will denote by F δ , is a 2 δ -covering of the set of 1-Lipschitz functions F : [0 , 1] → [ -1 , 1] in the metric d .

Proof. Fix a 1-Lipschitz function f ∈ F . Let f ′ ∈ F δ be the function in our covering where, for all x ∈ [0 , 1] δ , f ′ ( x ) = ⌊ f ( x ) ⌋ δ . Note that f ′ is unique because the elements of F δ can be identified by their image on [0 , 1] δ . For any x ∈ [0 , 1] , we have

where the first inequality is the triangle inequality, the second inequality uses the 1 -Lipschitzness of f and that f ′ ( x ) = f ′ ( ⌊ x ⌋ δ ) , and the third inequality uses the fact that | f ( z ) -f ′ ( z ) | ≤ δ and | z -⌊ z ⌋ δ | ≤ δ for all z ∈ [0 , 1] .

Bounding sums of γ . Consider the piecewise function γ ( x ) := { x, x < 1 , √ x, x ≥ 1 .

Lemma C.3. For all values x 1 , . . . , x n ≥ 0 , we can upper bound ∑ n i =1 γ ( x i ) ≤ √ n · γ ( ∑ n i =1 x i ) .

Proof. First, suppose that ∑ n i =1 x i ≤ 1 . Then, γ ( ∑ n i =1 x i ) = ∑ n i =1 x i and x i ≤ 1 for all i ∈ [ n ] . The claim is therefore equivalent to the trivial statement ∑ n i =1 x i ≤ √ n · ∑ n i =1 x i .

Now suppose that ∑ n i =1 x i > 1 . The Cauchy-Schwarz inequality gives

By our assumption that ∑ n i =1 x i > 1 , we have γ ( ∑ n i =1 x i ) = √∑ n i =1 x i . We separately have that

because γ ( x ) = x ≤ √ x for x ∈ [0 , 1] and γ ( x ) = √ x = √ x for x ≥ 1 . Thus,

C.2 Epochs of Doubling Realized Variance

Definition C.4. For I ⊆ [0 , 1] , consider the stochastic process (Var t ( I )) 0 ≤ t ≤ T defined as

where x ∼ D and p ⋆ t := Pr x ′ ∼D [ x ′ t = 1 | x ′ 1:( t -1) = x 1:( t -1) ] . We define the epochs with respect to I as the sequence τ 0 , τ 1 , · · · ∈ N where τ 0 = 0 and, for each k ∈ [ ⌈ log 2 ( T ) ⌉ +2] ,

The epochs τ 0 , τ 1 , . . . defined in Definition C.4 partition the T time steps of a martingale into epochs such that the realized variance Var t ( I ) increases by approximately 2 k -1 within the k -th epoch. In particular, we can understand τ k as pointing to the last time step of the k th epoch. The definition of τ ensures that:

Epoch 1 starts from time step 1 , and ends at the earliest time step t such that Var t ( I ) ≥ 1 = 2 0 .

For k ≥ 2 , Epoch k starts from the time step after the last step of Epoch k -1 , and ends at the earliest time step such that the total variance within the epoch reaches 2 k -1 .

We have the following technical facts about the epochs τ .

Fact C.5. The ( ⌈ log 2 ( T ) ⌉ +2) -th epoch is never complete, i.e., τ ⌈ log 2 ( T ) ⌉ +2 = ∞ .

Proof. Our definition of Var t ( I ) clearly guarantees Var T ( I ) ≤ T , which implies

and therefore, τ ⌈ log 2 ( T ) ⌉ +2 = ∞ .

Fact C.6. For every epoch k ∈ [ ⌈ log 2 ( T ) ⌉ + 2] , the change in realized variance in epoch k is deterministically upper bounded by Var τ k ( I ) -Var τ k -1 ( I ) < 2 k -1 +1 .

Proof. By definition, Var τ k -1 ( I ) -Var τ k -1 ( I ) < 2 k -1 . Because p ⋆ t ∈ [0 , 1] for all t ∈ [ T ] , the realized variance increases by at most p ⋆ t (1 -p ⋆ t ) ≤ 1 in each timestep, i.e. Var τ k ( I ) -Var τ k -1 ( I ) ≤ 1 . The fact follows by summing the two inequalities.

Fact C.7. For any epoch k ∈ [ ⌈ log 2 ( T ) ⌉ +2] , the probability that the k th epoch ends is at most

Proof. The sequence of realized variances Var 1 ( I ) , . . . , Var T ( I ) is deterministically non-decreasing. Thus, for every epoch k ∈ [ ⌈ log 2 ( T ) ⌉ +2] ,

✶ with the second inequality following as τ 1 < ∞ implies Var T ( I ) ≥ 1 . We can next invoke Markov's inequality Pr [ X ≥ a ] ≤ E [ X ] a with a = √ 2 k -1 and X = [Var T ( I ) ≥ 1] · √ Var T ( I ) to recover

Fact C.8. The exponentially weighted sum of probabilities that each epoch ends is at most

Proof. We will prove the deterministic inequality

Let K = max { k | τ k < ∞} be the number of completed epochs. When K = 0 , we have Var T ( I ) < 1 , and both sides of the above reduce to 0 . Now, suppose that K ≥ 1 , in which case we have Var T ( I ) ≥ 1 . By telescoping, we can lower bound the realized variance by

Separately, by definition of K , we have

✶ with the second equality following from Var T ( I ) ≥ 1 . Combining the previous two inequalities gives the desired inequality

C.3 Random Walks with Early Stopping

We now prove a technical result that the magnitude of a random walk with random variance can be upper bounded by its (expected) standard deviation. Compared to Lemma 5.2, the lemma below gives a bound that depends on γ (Var T ( I )) (rather than the square root), and avoids the extra log |G| · log T term. While the leading factor ( ≈ log |G| ) is larger than the one in Lemma 5.2 ( ≈ √ log |G| ), we will only apply the bound to the case that |G| = O (1) , where the difference between the two is only a constant factor.

Lemma C.9. Given a function f : [0 , 1] → [ -1 , 1] , y ∈ { 0 , 1 } T , and set I ⊆ [0 , 1] , consider the martingale M t ( f, y, I ) := ∑ t s =1 y t · f ( p ⋆ s ) · ( x s -p ⋆ s ) · ✶ [ p ⋆ s ∈ I ] , where x ∼ D , and p ⋆ t = Pr x ′ ∼D [ x ′ t = 1 | x ′ 1:( t -1) = x 1:( t -1) ] . Then, for any finite family G of functions from [0 , 1] to [ -1 , 1] , any y ∈ { 0 , 1 } T , and any I ⊆ [0 , 1] , we have

where Var t ( I ) := ∑ t s =1 p ⋆ s (1 -p ⋆ s ) · ✶ [ p ⋆ s ∈ I ] is the realized variance restricted to subset I , and γ ( x ) := { x, x < 1 , √ x, x ≥ 1 .

Proof. Let us decompose the horizon into epochs of doubling realized variance with respect to the subset I as per Definition C.4. Using τ as defined in (5), we will write I k := [ τ k -1 +1 : min { T, τ k } ] to denote the time steps composing epoch k and write K := max { k | τ k < ∞} to denote the number of completed epochs.

We will separately handle the contributions of epoch 1 and those of later epochs.

First epoch. Since y t ∈ { 0 , 1 } and ∥ f ∥ ∞ ≤ 1 holds for every f ∈ G , we can bound the expected contribution from the first epoch as follows:

Note that for any p ∈ [0 , 1] and Bernoulli random variable x ∼ Bernoulli ( p ) ,

It thus follows that the process ( X t ) 0 ≤ t ≤ T where

is a martingale, as conditioning on any realization of x 1:( t -1) , we have

By the optional stopping theorem, we have

Plugging this identity into (6) gives

where the third step applies Fact C.6 with k = 1 .

Later epochs. Applying a triangle inequality and the law of total expectation gives

where we define the process

In the above, the third step uses Fact C.5, namely that τ ⌈ log 2 ( T ) ⌉ +2 = ∞ . We can use Freedman's inequality to obtain a maximal inequality for each of these M k,f T processes.

Fact C.10. For every y ∈ { 0 , 1 } T and k ≥ 2 , we can uniformly bound the process M k,f T defined in (9) over a finite class G of functions from [0 , 1] to [ -1 , 1] by

Applying Fact C.10 to each of the martingales M k,f T in (8) gives us

To upper bound the right-hand side above, we use Fact C.8 to bound and use Fact C.7 to bound

Combine bounds. Combining (7) and (11) and recalling the definition of γ , we recover our main claim

The second step above applies Inequalities (7) and (11). The third holds since min { 1 , x } ≤ γ ( x ) and [ x ≥ 1] √ x ≤ γ ( x ) hold for all x ≥ 0 .

Lemma C.11. Consider a martingale M n ∼ D with filtration ( F t ) where | M t -M t -1 | ≤ 1 for all t ∈ [ n ] . For all x, y > 0 , we have the following high-probability bound on M n :

✶ Let us recall Freedman's inequality [Fre75].

Fact C.10. For every y ∈ { 0 , 1 } T and k ≥ 2 , we can uniformly bound the process M k,f T defined in (9) over a finite class G of functions from [0 , 1] to [ -1 , 1] by

Proof. Fix any f ∈ G . For t / ∈ I k , we have trivially that for any x 1: t -1 ∈ { 0 , 1 } t -1 :

This means that M k,f T is a martingale even conditioned on the event that τ k -1 < ∞ .

Our construction of epoch k in (5) further guarantees that the realized variance of M k,f T is deterministically upper bounded by Var τ k ( I ) -Var τ k -1 ( I ) ≤ 2 k -1 +1 (Fact C.6). Thus,

where the first equality uses the definition of a Bernoulli's variance; the second equality uses that, conditioned on F t -1 , x t ∼ Bernoulli ( p ⋆ t ) ; and the second inequality uses that y 2 t ≤ 1 and | f ( x ) | ≤ 1 for all x ∈ [0 , 1] .

We can thus use Freedman's inequality to bound the deviation of each martingale M k,f T . First, observe that the quadratic formula gives the inequality exp ( -x 2 2( x + y ) ) ≤ p if x ≥ log(1 /p ) +

to show that

Applying (12), we can simplify this to

We can then take a union bound over G for

Using the layer cake representation of expectation, we can convert this high-probability bound into the expectation bound through a change of variables

where the last equality follows by Fact C.12. When |G| > 1 , we can compute the integral to be

where the first inequality uses that erfc( z ) < exp( -z 2 ) z √ π . When |G| = 1 , we again have

Fact C.12. For k, n ∈ Z + , the following integral equality holds

where erfc denotes the complementary error function.

Proof. Let us first separate the integral into two parts:

We can bound the second integral easily. Since log( n/p ) = log n -log p ,

Now we consider the first integral. Let u = log( n/p ) . Then p = ne -u and d p = -ne -u d u . When p = 1 , u = log n . When p = 0 , u goes to ∞ . Thus, the integral becomes:

The integral involving the error function erfc( x ) can be recognized:

Thus, the integral ∫ 1 0 √ 2 k +1 log( n/p ) d p is given by

Summing (13) and (14) gives the claim.

C.4 Proof of Lemma 5.2

Lemma 5.2. Given a function f : [0 , 1] → [ -1 , 1] and y ∈ { 0 , 1 } T , consider the martingale M t ( f, y ) := ∑ t s =1 y s · f ( p ⋆ s ) · ( x s -p ⋆ s ) where x ∼ D . Then, for any finite family G of functions from [0 , 1] to [ -1 , 1] and any y ∈ { 0 , 1 } T , we have

Proof. Let us decompose the martingale M T ( f, y ) into epochs of doubling realized variance with respect to I = [0 , 1] as per Definition C.4. Using τ as defined in (5), we will write I k := [ τ k -1 + 1 , min { T, τ k } ] to denote the time steps composing epoch k and write K := max { k | τ k < ∞} to denote the number of completed epochs.

Applying a triangle inequality and the law of total expectation gives

where we define the process M k,f T := ∑ T t =1 y t · f ( p ⋆ t ) · ( x t -p ⋆ t ) · ✶ [ t ∈ I k ] . In the above, the second equality uses Fact C.5, namely that τ ⌈ log 2 ( T ) ⌉ +2 = ∞ . Applying Fact C.10 to each of the martingales M k,f T in (15) gives us

We can upper bound some of the summands in the right-hand side by using Fact C.8 to bound

This gives that

D Supplemental Materials for Section 6

Notation. For all stochastic processes ( X t ) , we use X t 1 : t 2 = X min { t 2 ,T } -X t 1 to denote the increment within the time interval ( t 1 , t 2 ] (with X 0 = 0 by default).

D.1 Proof of the Weaker Lower Bound

We restate and prove Lemmas 6.2 and 6.3.

Proof. Recall that SSCE is defined using smCE , which is in turn a supremum over the family F of Lipschitz functions. Since both f ≡ 1 and f ≡ -1 are included in F , for any realized sequences x and p , we can lower bound SSCE ( x, p ) as follows:

where we have defined z t := ( y t -0 . 5)( x t -p t ) to be zero-mean independent random variables, and µ := ∑ T t =1 0 . 5( x t -p t ) . Now we partition [ T ] into T 1 and T 2 , where T 1 includes the all time steps such that | x t -p t | ≥ 1 2 , and T 2 = T \ T 1 contains the remaining time steps. From the definition of N T , it immediately follows that N T = | T 1 | . Letting Z 1 := ∑ t ∈ T 1 z t and Z 2 := ∑ t ∈ T 2 z t , it remains to lower bound E [ | Z 1 + Z 2 + µ | ] by Ω( √ N T ) .

We will first prove that E [ | Z 1 | ] ≥ C √ N T for a universal constant C > 0 . From the Berry-Esseen theorem (e.g. from [She10]), the CDF of Z 1 can be approximated by the CDF of the standard normal distribution as follows:

where Φ( x ) is the standard Gaussian CDF, C 0 is a universal constant no larger than 0 . 56 , and

As a result, we can lower bound the probability of | Z 1 | ≥ 0 . 05 √ N T as follows:

Since C 0 ≤ 0 . 56 and Φ(0 . 2) ≤ 0 . 58 , we can guarantee Pr [ | Z 1 | ≥ 0 . 05 √ N T ] ≥ Ω(1) for all N T ≥ 8 . When N T ≤ 7 , we have | Z 1 | = N T / 2 ≥ 0 . 05 √ N T when all { z t | t ∈ T 1 } are positive, which happens with probability 2 -N T ≥ 2 -7 = Ω(1) . Therefore, we can always conclude that

for some universal constant C > 0 .

Finally, we consider the randomness of Z 2 and show that E [ | Z 1 + Z 2 + µ | ] ≥ C 2 √ N T . Applying the tower property of expectations, we have

Consider the following two cases for the conditional expectation inside:

When | Z 2 + µ | ≥ C 2 √ N T , we use Jensen's inequality and E [ Z 1 ] = 0 to obtain E [ | Z 1 + Z 2 + µ | | Z 2 ] ≥ | E [ Z 1 + Z 2 + µ | Z 2 ] | = | Z 2 + µ | ≥ C 2 √ N T .

When | Z 2 + µ | < C 2 √ N T , we apply the triangle inequality and have E [ | Z 1 + Z 2 + µ | | Z 2 ] ≥ E [ | Z 1 | ] -| Z 2 + µ | > C √ N T -C √ N T = C √ N T

Therefore, regardless of the realization of Z 2 , we always have E [ | Z 1 + Z 2 + µ 0 | | Z 2 ] ≥ C 2 √ N T . Taking an expectation over the randomness of Z 2 gives the desired bound SSCE ( x, p ) ≥ C 2 √ N T .

Lemma 6.3. The stochastic process ( N t ) t ∈ [ T ] satisfies E [ √ N T ] ≥ Ω( E [ √ Var T ] ) -O (1) .

Proof. Since N T ≥ Var T / 16 implies √ N T ≥ √ Var T / 4 , we have

Therefore, to establish the inequality E [ √ N T ] ≥ Ω( E [ √ Var T ] ) -O (1) , it suffices to prove that the expectation of the second term-which we denote with M -is upper bounded by O (1) , i.e.,

✶ We proceed by partitioning the range of Var T into subintervals of geometrically increasing length and enumerating all possibilities for which subinterval Var T falls into. If Var T ≤ 1 , its contribution to M is clearly O (1) . Otherwise, we must have Var T ∈ [2 l , 2 l +1 ) for some l ∈ N , which implies that N T < Var T / 16 < 2 l -3 . Therefore, we bound M by taking a union bound over all such l 's:

Now we bound Pr [ N T < k/ 8 ∧ Var T ≥ k ] for any fixed value of k (that plays the role of 2 l ) by constructing a sub-martingale. We start by partitioning the time horizon [ T ] into blocks based on the realized variance Var t , such that each block B j := ( b j -1 , b j ] terminates upon the realized variance Var B j first exceeds 1 . Formally, using notation X t 1 : t 2 := X min { t 2 ,T } -X t 1 to denote the increment of any process ( X t ) in ( t 1 , t 2 ] (with X 0 = 0 by default), the endpoints b j are defined recursively as:

We show in the following lemma that for each block B j , the expected increment N B j within B j is lower bounded by a constant as long as B j terminates before T .

Lemma D.1. For the constant c = 1 -1 /e , E [ ✶ [ N B j ≥ 1 ] -c · ✶ [ b j < ∞ ] ∣ ∣ ∣ F b j -1 ] ≥ 0 . We prove Lemma D.1 in Appendix D.2. This lemma justifies that if we define A j as

A 0 := 0 , A j -A j -1 := ✶ [ N B j ≥ 1 ] -c · ✶ [ b j < ∞ ] ( j ≥ 1) , then ( A j ) j ≥ 0 forms a sub-martingale of bounded increment | A j -A j -1 | ≤ 1 , making it unlikely for any A j to deviate significantly below 0 . However, if N T < k/ 8 and Var T ≥ k , then A k/ 2 must witness a large deviation: on the one hand, block B k/ 2 should terminate properly because the variance in each block cannot exceed 2 ; on the other hand, N T < k/ 8 implies that at most k/ 8 of these blocks can have a nonzero increment N B j . As a result,

✶ ✶ By applying the Azuma-Hoeffding inequality for submartingales, we can quantitatively bound the probability of such a large deviation by

Finally, plugging the above bound back into equation (17) gives us

We have thus established the inequality (16), which in turn proves the lemma.

D.2 Proof of Lemma D.1

Now we prove Lemma D.1, which we restate below.

Proof. We first show that for all t ∈ [ T ] , we have Pr [ n t = 1 | F t -1 ] ≥ p ⋆ t (1 -p ⋆ t ) , where F t -1 denotes the filtration generated by all the randomness up to time t -1 . Note that conditioning on F t -1 , x t is distributed according to Bernoulli ( p ⋆ t ) . If the forecaster chooses p t ≥ 1 2 , the condition

| x t -p t | ≥ 1 2 holds when x t = 0 , which happens with probability 1 -p ⋆ t ; otherwise it holds when x t = 1 , which happens with probability p ⋆ t . Therefore, regardless of the choice of p t , we have

This allows us to invoke Lemma D.5 with q t := n t , r t := p ⋆ t (1 -p ⋆ t ) , and θ = 1 , where we only consider the random process inside block B j . In this context, the stopping time τ 1 corresponds to the end of the block, i.e., b j . Therefore, by applying Lemma D.5 at time step b j -1 , we obtain

D.3 A Stronger Lower Bound

In this section, we state and prove the stronger SSCE lower bound for all forecasters.

Theorem D.2. For any D ∈ ∆( { 0 , 1 } T ) , OPT SSCE ( D ) = Ω( E [ γ (Var T )]) , where the function γ is defined as γ ( x ) := x · [0 ≤ x < 1] + √ x · [ x ≥ 1] .

✶ ✶ Proof of Theorem D.2. The theorem holds by combining Lemma 6.2, which lower bounds the SSCE by Ω( √ N T ) , and the stronger lower bound on E [ √ N T ] shown in Lemma D.3.

Lemma D.3. There exists a universal constant C > 0 such that E [ √ N T ] ≥ C · E [ γ (Var T )] , where the function γ is defined as γ ( x ) := x · [0 ≤ x < 1] + √ x · [ x ≥ 1] .

✶ ✶ Proof of Lemma D.3. The proof is also based on partitioning the time horizon into blocks B j = ( b j -1 , b j ] -each with approximately unit variance-similar to the approach used in proving Lemma 6.3. However, this proof involves a more careful analysis of the growth of √ N t by further grouping blocks into 'epochs' and giving special treatment to the first epoch, where the cumulative variance is very small.

Specifically, consider the blocks B j = ( b j -1 , b j ] defined by

Recall that the increment of Var t satisfies Var t -Var t -1 = p ⋆ t (1 -p ⋆ t ) ≤ 1 / 4 . Thus, every block j satisfies Var B j = Var b j -Var b j -1 = (Var b j -1 -Var b j -1 ) + (Var b j -Var b j -1 ) ≤ 1 + 1 / 4 = 5 / 4 . We further group blocks into epochs such that the k -th epoch T k := ( τ k -1 , τ k ] contains ≈ 2 k blocks:

In addition, we define ˜ N t as the sum of n s capped by 1 in each block:

Clearly, for all the realized sequences we have N T ≥ ˜ N T and ˜ N τ k ≤ 2 k , where the latter is because each block contributes at most 1 to ˜ N t . In the following, we will first analyze the growth of √ ˜ N t in epochs k ≥ 1 , then provide a different analysis for the zeroth epoch.

In each epoch T k with k ≥ 1 . We start by establishing the following lemma, which extends the characterization of Lemma D.1 into epochs.

Lemma D.4 (Lower bound on ˜ N T k ) . For any k ≥ 1 , we have

Proof of Lemma D.4. According to Lemma D.1, we have that in each block B j = ( b j -1 , b j ] ,

Summing over all the blocks in epoch T k , we obtain

We have thus established Lemma D.4.

With Lemma D.4, we obtain a lower bound by linearizing the increment of √ ˜ N t in each block.

The first step above can be alternatively justified by √ a -√ b = a -b √ a + √ b ≥ a -b 2 √ a , which holds for all a ≥ b ≥ 0 .

In epoch T 0 . We now analyze √ ˜ N T 0 in epoch 0 . Note that the T 0 contains only the first block B 1 , so this value is either 0 or 1 , depending on whether there exists a t ∈ B 1 such that n t = ✶ [ | x t -p t | ≥ 1 2 ] = 1 . Recall that in the proof of Lemma D.1, we have shown that regardless of the choice of p t ,

Therefore, in the special case of product distributions (i.e., the sequence ( p ⋆ t ) is deterministic and each outcome x t ∼ p ⋆ t is independent of other time steps), we can directly bound the probability that √ ˜ N T 0 = 1 as follows:

where the last step follows from the inequality 1 -e -x ≥ x/ 2 when 0 ≤ x ≤ 5 / 4 , and the fact that Var B 1 ≤ 5 / 4 .

However, in the general case where the sequence ( p ⋆ t ) is itself random and depends on the history of x t 's, such a direct argument fails. Instead, we use Lemma D.6 that extends the above analysis to this more general setting. Lemma D.6 is itself a similar but more general statement than Lemma D.5, as it is applicable even when the cumulative variance is smaller than the hard threshold θ . Invoking Lemma D.6 with q t := n t , r t := p ⋆ t (1 -p ⋆ t ) , and the stopping time τ as the earlier time step between the end of block B 1 and the first time where n t = 1 , we have

where the last step again uses 1 -e -x ≥ x/ 2 for x ∈ [0 , 5 / 4] . Moreover, since 5 4 · ✶ [ N τ ≥ 1] ≥ Var τ : b 1 , we also have

Combining the two inequalities, we obtain

Putting everything together. Combining the lower bounds for epoch 0 and epochs k ≥ 1 , we obtain

where the last step follows from the observation that the cumulative variance in each block cannot exceed 2 , so τ k = ∞ implies that Var T < 2 k +1 , i.e., 2 k/ 2 ≥ √ Var T / √ 2 . Finally, since τ 1 = ∞ is equivalent to Var T < 1 , we have established that

The lemma follows from the fact that N T ≥ ˜ N T always holds, which implies E [ √ N T ] ≥ E [√ ˜ N T ] ≥ 1 16 E [ γ (Var T )] .

D.4 Auxiliary Lemmas

Lemma D.5. Let Q t = ∑ s ≤ t q s , R t = ∑ s ≤ t r s be two (coupled) stochastic processes such that q t ∈ { 0 , 1 } , r t ∈ [0 , 1] for all t ∈ [ T ] . Let F t denote the filtration generated by all the randomness up to time t . Suppose r t is a deterministic function on F t -1 , and s t := Pr [ q t = 1 | F t -1 ] ≥ r t .

For any constant θ > 0 , define τ θ to be a stopping time chosen as the first time that R t reaches θ , i.e., τ θ := min {∞} ∪ { t ∈ [ T ] | R t ≥ θ } .

Let Q + t := Q t : τ θ be the sum of q s in the future until the stopping time τ θ . If t > τ θ , then we let Q + t := 0 . Consider random variables A t 's defined on the filtration F t as follows:

Then we have A t ≥ 0 for every t ≤ T and every event in F t .

Proof of Lemma D.5. It suffices to prove the inequality conditioning on events in F t that are 'atomic' in the sense that they uniquely determine the values of q 1: t and r 1: t . The general case would follow from the law of total probability. In particular, in the following proof, we may view the value of R t as fixed when we analyze the quantity A t .

We perform a backwards induction from t = T to t = 0 . Consider the base case of t = T . If R T ≥ θ , we have

Otherwise when R T < θ , we have Pr [ τ θ < ∞| F T ] = 0 , which also implies A T = 0 ≥ 0 .

We then assume A t ≥ 0 , and show that the same holds for A t -1 , where t ≤ T . If R t -1 ≥ θ , we clearly have A t -1 ≥ 0 , as the factor -( 1 -e -( θ -R t -1 ) ) would be non-negative. Therefore, it suffices to consider the case that R t -1 < θ . In this case, the stopping time τ θ should be ≥ t , so we have Q + t -1 = q t + Q + t . We bound A t -1 by breaking the event Q + t -1 ≥ 1 into two cases: either q t = 1 , or q t = 0 but Q + t ≥ 1 . We have

For the second term, we apply the induction hypothesis of A t ≥ 0 and get

where the second step uses the fact that conditioning on F t -1 , R t = R t -1 + r t . As a result, we obtain

Combining the bounds in (18) and (19), we obtain

We have thus proved that the claim also holds for t -1 . This completes the induction.

Lemma D.6. Let Q t = ∑ s ≤ t q s , R t = ∑ s ≤ t r s be two (coupled) stochastic processes such that q t ∈ { 0 , 1 } , r t ∈ [0 , 1] for all t ∈ [ T ] . Let F t denote the filtration generated by all the randomness up to time t . Suppose r t is a deterministic function on F t -1 , and s t := Pr [ q t = 1 | F t -1 ] ≥ r t .

For any constant θ > 0 , define τ to be a stopping time chosen as the first time that either R t reaches 1 or q t = 1 , i.e.,

Let Q + t := Q t : τ and R + t := Q t : τ be the sum of q s and r s in the future until the stopping time τ , respectively. We also let Q + t = R + t = 0 when t > τ . Consider random variables A t 's defined on the filtration F t as follows:

Then we have A t ≥ 0 for every t ≤ T and every event in F t .

Proof of Lemma D.6. Using a similar approach to that for Lemma D.5, we prove this claim via a backwards induction from t = T to t = 0 . Again, we only consider the 'atomic' events in F t that uniquely determines the values of q 1: t and r 1: t , and thus whether τ ≤ t ; the general case follows from the law of total probability.

For the base case of t = T , we have

Now for t ≤ T , we assume the claim holds for t and analyze A t -1 . If τ ≤ t -1 , we immediately obtain A t -1 ≥ 0 since Q + t -1 = R + t -1 = 0 . It remains to consider the case of τ ≥ t . For the first term of A t -1 (the conditional probability), we have

where the inequality step follows from the induction hypothesis A t ≥ 0 .

On the other hand, the second term of A t -1 (the conditional expectation) can be bounded as

∣

Finally, combining the bounds for both terms of A t -1 , we obtain

We have proved that A t -1 ≥ 0 . As a result, A t ≥ 0 for all t ≤ T and all events in F t .

E Proof of Theorem 1.2

In this section, we prove our main theorem (Theorem 1.2) by combining the theorems established in the previous sections, and then verifying the completeness and soundness of the SSCE.

Proof of Theorem 1.2. Let D ∈ ∆( { 0 , 1 } T ) be an arbitrary distribution and define the random variable

over x ∼ D , where p ⋆ t := Pr x ′ ∼D [ x ′ t = 1 | x ′ 1:( t -1) = x 1:( t -1) ] . By Theorems C.1 and D.2, the truthful forecaster gives

In the above, the O ( · ) and Ω( · ) notations hide universal constants that do not depend on D . Therefore, there exists a universal constant c > 0 such that the SSCE is ( c, 0) -truthful.

Completeness. Now we verify that the SSCE is complete. For any x ∈ { 0 , 1 } T , we have

For any α ∈ [0 , 1] , the upper bound

follows from applying Theorem C.1 to the product distribution D = ∏ T t =1 Bernoulli ( α ) and the fact that γ ( x ) ≤ √ x for all x ≥ 0 .

Soundness. To show that the SSCE is sound, we first consider the case that x ∈ { 0 , 1 } T is arbitrary and the predictions are p = ⃗ 1 T -x . Noting that the function x ↦→ 1 / 2 -x is in the family F of 1 -Lipschitz functions from [0 , 1] to [ -1 , 1] , we have

̸

where the third step holds since ( x -1 / 2) · (2 x -1) = 1 / 2 holds for every x ∈ { 0 , 1 } . Finally, we fix α, β ∈ [0 , 1] such that α = β . For fixed x, y ∈ { 0 , 1 } T , we have

whereas

Taking an expectation over x 1 , . . . , x T ∼ Bernoulli ( α ) and y ∼ Unif ( { 0 , 1 } T ) gives

where the third step follows from Jensen's inequality E [ | X | ] ≥ | E [ X ] | .

F Proof of Lemma 7.1

Lemma 7.1. For any x ∈ { 0 , 1 } T and p ∈ [0 , 1] T ,

where the O ( · ) notation hides a universal constant that does not depend on T , x or p .

We prove Lemma 7.1 via a standard chaining argument.

Proof of Lemma 7.1. We decompose the SSCE as follows:

(

x, p

)

Note that the second term is exactly 1 smCE ( x, p ) , so it suffices to bound the first term by O ( √ T ) .

2

For notational convenience, let M ( f ) T := ∑ T t =1 ( y t -1 2 ) · f ( p t ) · ( x t -p t ) for function f ∈ F . We will establish the following bound for any N ≥ 1 and functions f 1 , f 2 , . . . , f N from [0 , 1] to [ -1 , 1] :

Assuming Inequality (20), applying Dudley's chaining technique [Dud87] to the δ -covering F δ defined in Lemma C.2 would give

which implies the lemma.

Therefore, it remains to establish Inequality (20). We prove this using Hoeffding's inequality and a union bound. For each i ∈ [ N ] and every ε > 0 , we have

Finally, the bound (20) holds by taking an integral over ε > 0 : shorthanding X := sup i ∈ [ N ] M ( f i ) T , we have

This completes the proof.

G Supplemental Materials for Section 8

We justify the claim in Section 8 that it is impossible for the SSCE (and most natural calibration measures) to incentivize truthful prediction against all adaptive adversaries.

Suppose that the adversary draws x 1 from Bernoulli (1 / 2) . If the forecaster predicts p 1 = 0 , all the subsequent bits are zeros; otherwise, the adversary keeps producing independent samples from Bernoulli (1 / 2) .

Clearly, the truthful forecaster predicts p t = 1 / 2 at every step t ∈ [ T ] , and the resulting outcome sequence x is uniform over { 0 , 1 } T . The resulting SSCE is then Θ( T 1 / 2 ) in expectation. If the forecaster keeps predicting p t = 0 instead, the expectation of SSCE ( x, p ) is only O (1) . Note that this impossibility holds for any calibration measure CM that satisfies

and where ◦ denotes concatenation.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The claims match the theoretical results that we prove in the paper.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: This is a theoretical work, so the results hold for the specific problem setups and formulations, which we formally state in Section 2.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: All the theorems and claims are formally proved, either in the main paper or in the appendix.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [NA]

Justification: The paper does not include experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [NA]

Justification: The paper does not include experiments requiring code.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [NA]

Justification: The paper does not include experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [NA]

Justification: The paper does not include experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [NA]

Justification: The paper does not include experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: This is a theoretical work, and there is no societal impact of the work performed to the best of our knowledge.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification: The paper does not use existing assets.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors evaluate a variety of existing calibration measures according to their completeness, soundness, along with truthfulness which is their main focus, by theoretically showing how the measures satisfy these aforementioned criteria. They discover that existing measures, despite being complete and sound, have large truthfulness gaps. They introduce a new method, SSCE, that makes a simple modification of smooth calibration error by subsampling time steps, that is also approximately truthful and the authors provide proofs for this.\n\nSTRENGTHS:\n- The paper is the first systematic, theoretical investigation of the truthfulness of a range of calibration measures.  \n\n- Extensive and rigorous proofs are provided for all claims.  \n\n- The paper introduces a novel calibration measure, the Subsampled Smooth Calibration Error (SSCE), which is both complete and sound while also being approximately truthful.  The proposed SSCE has the potential to be widely used.\n\nWEAKNESSES:\n- Given your definition of truthfulness that you base your analysis on, is it consistent with what previous works have defined, and do other works stress the crucialness for this aspect of calibration? I’d like to see more discussion on related works tackling truthfulness in calibration and its importance, particularly to help establish why such an approach like SSCE should be used in practice in the future over ECE or proper scoring rules that have trade offs.  \n\n\n- Despite the introduction claims that Part III of the paper (about the adversarial setting and theorem 1.3) is one of the main contributions, it is not featured prominently in the rest of the main body of the paper, nor is much written about it, so it is difficult to ascertain its importance in the context of the work. It would be beneficial to discuss this more, as most of the paper is devoted to Part II. \n\n- The paper lacks many practical examples which can make it hard to follow at times, particularly when the formulation (of notions such as soundness) differs from previous works. It would be helpful to have more examples such as in [FRST11].\n\nQUESTIONS:\n- What are the main implications of adaptive adversary results? If a sublinear penalty for SSCE is achievable, how would it perform on average or in a typical use case? \n\n- The subsampling time steps technique is applied to smooth calibration error to improve its truthfulness. Can it be more generalized and applied to different calibration measures to improve their truthfulness? Discussing this could help shed light on the generalizability of the approach.\n\nLIMITATIONS:\nyes.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper initiates the study of truthful calibration measures. It introduces a set of requirements --- truthfulness, completeness, soundness, and asymptotic calibration --- that together define a novel and fruitful set of requirements on calibration measures. It explores the classic sequential binary prediction setting, and proves in that setting both that (1) an entire array of existing calibration metrics fail to satisfy at least one of these requirements, and (2) that a simple novel metric called SSCE (which is a smooth and subsampled calibration measure) actually satisfies all of these, in particular being only a constant factor away from perfect truthfulness.\n\nSTRENGTHS:\nThis paper is in my opinion strong both on the modeling (principled definitions leading to nontrivial theory) and on the technical side.\n\nThe paper presents a new array of results that evaluate calibration measures under a new angle not explicitly considered before: truthfulness (along with several other dimensions that make this requirement nontrivial, as without further restrictions even constant measures can be truthful as the authors remark). It is very appealing that this work successfully tackles both sides of the issue — on the one hand, it carefully analyzes an entire array of existing (including recent) calibration measures and derives principled bounds showcasing their lack of (optimal) truthfulness, but it also proposes a novel — even if simple — calibration metric that “smooths out” lack of truthfulness in prior measures via subsampling. In this way, it presents a principled framework on which future work could build to both explore and exploit truthful calibrated elicitation.\n\nThe technique, with which the main result — O(1) truthfulness of SSCE — is derived, is quite impressive; despite a simple formulation, the proof requires a careful martingale-based argument which necessitates careful twists on standard concentration/anticoncentration/other measure-theoretic results. Based on familiarity with related work, I believe this complexity is fully justified even the “elementary” nature of the statement; and I concur with the authors that this technique may be useful for further work as well.\n\nWEAKNESSES:\nOverall, this is a strong paper. It is not a flaw per se given that this is the first paper to model the “truthful” setting — and thus any and all reasonable modeling assumptions are great to have at this stage — but the main question I am not fully certain about is: how principled is the fashion in which “soundness” — i.e. the property that “bad” predictions should lead to linear (in T) calibration measure value — is defined? Meaning, the requirements imposed (that predicting the complement of the truth, or consistently predicting a bias different from the true underlying bias, are both bad) seem necessary but possibly not sufficient to correspond to what may be viewed as “consistently bad predictions”. It would be great if the authors could shed further light on this. (On the other hand, the other dimensions of inquiry (truthfulness/completeness/asymptotic calibration), I would argue, are indeed defined in a very principled way.)\n\nQUESTIONS:\nCan you give some crude but reasonable constant bound on the truthfulness constant c of SSCE? If that is possible, along with an explanation of how this bounding propagates through the outlined proof, that might also serve as additional explanation of the proof pipeline for the reader.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies calibration measures in a sequential prediction setup. In addition to rewarding accurate predictions (completeness) and penalizing incorrect ones (soundness), the paper formalizes another desideratum of calibration measures — truthfulness. A calibration measure is truthful if the forecaster (approximately) minimizes the expected penalty by predicting the conditional expectation of the next outcome, given the prior distribution of outcomes. \n\nThe paper first shows that the existing calibration measures fail to simultaneously meet all these criteria (completeness, soundness, truthfulness). The paper then proposes a new calibration measure — Subsampled Smooth Calibration Error (SSCE) — that is shown to be approximately truthful via a non-trivial analysis.\n\nSTRENGTHS:\nThe proposed truthfulness desideratum for the calibration measure is well-motivated. The authors have provided several rigorous technical results to support the need of the proposed calibration measure (SSCE), including the failure of being truthful of existing calibration measures. The analysis and the results are non-trivial.\n\nWEAKNESSES:\nThe proposed Subsampled Smooth Calibration Error is essentially the expected of the smooth calibration error over the uniformly randomly selected events. This seems to be related to the definition of “Event-conditional unbiasedness” proposed in “High-Dimensional Prediction for Sequential Decision Making”. Could authors comment a bit on this connection?\n\nIt is a bit unclear on how to understand the truthfulness in an adversary setting, especially given that in Definition 2.5 (Truthfulness of calibration measures), it requires a distribution over the sequence of the events. It might be helpful to clarify here a bit.\n\nDoes the authors have intuitions on how the current results could be extended if one consider multi-outcome space, instead of being binary?\n\nQUESTIONS:\nplease see above.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 1, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes revisiting the calibration measure by emphasizing three main desirable properties for the metric's behavior: rewarding accurate predictions, penalizing incorrect ones, and ensuring truthfulness. The authors define truthfulness as the ability to accurately predict the conditional expectation of future outcomes. The paper's core findings are twofold: first, it demonstrates that most calibration metrics do not adequately capture truthfulness; second, it argues that a small modification to the classical Expected Calibration Error (ECE)—specifically, subsampling the window in which the ECE is calculated and then averaging the results—is sufficient to meet all the desired properties. The whole article is a series of theoretical arguments supporting these claims.\n\nSTRENGTHS:\nThe claims appear to be supported by rigorous mathematical arguments, which, I found,  are also quite lengthy and challenging to follow.\n\nWEAKNESSES:\nThe theoretical argument for including the notion of truthfulness is intriguing, but I couldn't discern any practical implications for omitting it. At the very least, we would expect some numerical demonstrations, even on simple toy examples.\n\nQUESTIONS:\n- Could you numerically highlight any meaningful difference between smECE and the proposed SSCE?\n\n- Does the introduced subsampling breaks to possibility to easily draw a reliability diagram as in (Blasiok & Nakkiran, 2023)?\n\n\n- The definition in line 41-42 is crucial to the paper and deserves to be more clearly introduced. It is formalized in page 5. This in general makes the paper quite convoluted (most of the point -even simple argument or definition- are understood after reading a long detour. For a > 50 pages technical paper it is quite painful and I personally just gave up reading it.)\n\nLIMITATIONS:\nOk", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 1, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-26"}