SlowGuess/NeurIPS_2023_OCR huggingface dataset:
where  $t \coloneqq \ln (v / (1 - v))$  is what we call the dual prediction corresponding to  $v$ . While a standard terminology for  $t$  is logit, we say  $t$  is the dual prediction because this notion generalizes to arbitrary proper loss functions. One can conversely obtain a prediction  $v$  from its dual prediction  $t$  by taking the sigmoid transformation:  $v = \sigma(t) \coloneqq 1 / (1 + e^{-t})$ . The superscript  $\psi$  in the logistic loss  $\ell^{(\psi)}$  can be understood to indicate its relationship to the dual prediction and the exact meaning is made clear in Section 4. Based on (2), optimizing the cross-entropy loss  $\ell_{\mathrm{xent}}$  over predictions  $v$  is equivalent to optimizing the logistic loss  $\ell^{(\psi)}$  over dual predictions  $t$ .
Usually, a neural network that aims to minimize the cross-entropy loss has a last layer that computes the sigmoid  $\sigma$  (the binary version of softmax), so the value computed by the network before the sigmoid (the "logit") is the dual prediction. If we want to enhance the neural network by adding more layers, these layers are typically inserted before the final sigmoid transformation. It is thus more natural to consider post-processings on the dual predictions (logits) rather than on the predictions themselves.
For a predictor  $f$ , we define the dual post-processing gap  $\mathsf{pGap}^{(\psi, 1/4)}(g)$  of its dual predictor  $g(x) = \ln(f(x) / (1 - f(x)))$  to be the difference between the expected logistic loss  $\mathbb{E}[\ell^{(\psi)}(y, g(x))]$  of  $g$  and the minimum expected logistic loss after we post-process  $g(x) \mapsto g(x) + \eta(g(x))$  for some 1-Lipschitz  $\eta: \mathbb{R} \to [-4, 4]$ , where the constant 4 comes from the fact that the logistic loss is 1/4-smooth in  $t$ .
Definition 2.5 (Dual post-processing gap for cross-entropy loss, special case of Definition 4.4). Let  $K$  denote the family of all post-processing functions  $\kappa : \mathbb{R} \to \mathbb{R}$  such that the update function  $\eta(t) \coloneqq \kappa(t) - t$  is 1-Lipschitz and bounded  $|\eta(t)| \leq 4$ . Let  $\ell^{(\psi)}$  be the logistic loss. Let  $\mathcal{D}$  be a distribution over  $\mathcal{X} \times \{0,1\}$ . We define the dual post-processing gap of a function  $g: \mathcal{X} \to \mathbb{R}$  to be
$$
\mathsf {p G a p} _ {\mathcal {D}} ^ {(\psi , 1 / 4)} (g) := \mathbb {E} _ {(x, y) \sim \mathcal {D}} \ell^ {(\psi)} (y, g (x)) - \inf  _ {\kappa \in K} \mathbb {E} _ {(x, y) \sim \mathcal {D}} \ell^ {(\psi)} \big (y, \kappa (g (x)) \big).
$$
We define the dual smooth calibration error  $\mathsf{smCE}^{(\psi, 1/4)}(g)$  to be the maximum of  $|\mathbb{E}[(y - f(x))\eta(g(x))]|$  over all  $1/4$ -Lipschitz functions  $\eta: \mathbb{R} \to [-1, 1]$ . Like with smooth calibration, it is 0 if and only if the predictor  $f$  is perfectly calibrated.
Definition 2.6 (Dual smooth calibration for cross-entropy loss, special case of Definition 4.5). Let  $H$  be the family of all  $1/4$ -Lipschitz functions  $\eta: \mathbb{R} \to [-1,1]$ . For a function  $g: \mathcal{X} \to \mathbb{R}$ , define predictor  $f: \mathcal{X} \to [0,1]$  such that  $f(x) = \sigma(g(x))$  for every  $x \in \mathcal{X}$  where  $\sigma$  is the sigmoid transformation. Let  $\mathcal{D}$  be a distribution over  $\mathcal{X} \times \{0,1\}$ . We define the dual calibration error of  $g$  as
$$
\mathsf {s m C E} _ {\mathcal {D}} ^ {(\psi , 1 / 4)} (g) := \sup  _ {\eta \in H} | \mathbb {E} _ {(x, y) \sim \mathcal {D}} [ (y - f (x)) \eta (g (x)) ] |.
$$
We show a result similar to Theorem 2.4 that the dual post-processing gap and the dual smooth calibration error are also quadratically related. Moreover, we show that a small dual smooth calibration error implies a small (standard) smooth calibration error.
Corollary 2.7 (Corollary of Theorem 4.6 and Lemma 4.7). Let  $\mathsf{pGap}^{(\psi,1/4)}$  and  $\mathsf{smCE}^{(\psi,1/4)}$  be defined as in Definition 2.5 and Definition 2.6 for the cross-entropy loss. For any function  $g: \mathcal{X} \to \mathbb{R}$  and any distribution  $\mathcal{D}$  over  $\mathcal{X} \times \{0,1\}$ ,
$$
2 \operatorname {s m C E} _ {\mathcal {D}} ^ {(\psi , 1 / 4)} (g) ^ {2} \leq \mathrm {p G a p} _ {\mathcal {D}} ^ {(\psi , 1 / 4)} (g) \leq 4 \operatorname {s m C E} _ {\mathcal {D}} ^ {(\psi , 1 / 4)} (g). \tag {3}
$$
Moreover, let predictor  $f: \mathcal{X} \to [0,1]$  be given by  $f(x) = \sigma(g(x))$  for the sigmoid transformation  $\sigma$ . Its (standard) smooth calibration error  $\mathsf{smCE}_{\mathcal{D}}(f)$  defined in Definition 2.3 satisfies
$$
\operatorname {s m C E} _ {\mathcal {D}} (f) \leq \operatorname {s m C E} _ {\mathcal {D}} ^ {(\psi , 1 / 4)} (g). \tag {4}
$$
The constants in the corollary above are optimal as we show in Lemmas D.3 and D.4 in the appendix. Both results (3) and (4) generalize to a wide class of proper loss functions as we show in Section 4.
Remark 2.8. In a subsequent work, Blasiok and Nakkiran [2023] proved the reverse direction of (4) for the cross-entropy loss:  $\mathsf{smCE}_{\mathcal{D}}(f)\geq \Omega \big(\mathsf{smCE}_{\mathcal{D}}^{(\psi ,1 / 4)}(g)^2\big)$ . Hence both  $\mathsf{smCE}^{(\psi ,1 / 4)}(g)$  and  $\mathsf{pGap}^{(\psi ,1 / 4)}(g)$  are consistent calibration measures, a notion introduced by Blasiok et al. [2023b].
Our result shows that achieving a small dual post-processing gap when optimizing a cross-entropy is necessary and sufficient for good calibration guarantees. It sheds light on the examples where logistic regression fails to yield a calibrated predictor; in those instances the cross-entropy loss can be further reduced by some Lipschitz post-processing of the logit. This is intuitive because logistic regression only optimizes cross-entropy within the restricted class where the logit is a linear combination of the features plus a bias term, and this class is not closed under Lipschitz post-processing.
3 Calibration and Post-processing Gap for the Squared Loss
In this section we prove our main result Theorem 2.4 relating the smooth calibration error of a predictor to its post-processing gap with respect to the squared loss.
Proof of Theorem 2.4. We first prove the upper bound on  $\mathsf{pGap}_{\mathcal{D}}(f)$ . For any  $\kappa \in K$  in the definition of  $\mathsf{pGap}$  (Definition 2.2), there exists a 1-Lipschitz function  $\eta :[0,1]\to [-1,1]$  such that  $\kappa (v) = v + \eta (v)$  for every  $v\in [0,1]$ . For the squared loss  $\ell$ ,
$$
\begin{array}{l} \mathbb {E} _ {(x, y) \sim \mathcal {D}} [ \ell (y, \kappa (f (x))) ] = \mathbb {E} [ (y - f (x) - \eta (f (x))) ^ {2} ] \\ = \mathbb {E} [ (y - f (x)) ^ {2} ] - 2 \mathbb {E} [ (y - f (x)) \eta (f (x)) ] + \mathbb {E} [ \eta (f (x)) ^ {2} ]. \tag {5} \\ \end{array}
$$
The three terms on the right hand side satisfy
$$
\begin{array}{l} \mathbb {E} \left[ (y - f (x)) ^ {2} \right] = \mathbb {E} \left[ \ell (y, f (x)) \right], \\ \mathbb {E} \left[ (y - f (x)) \eta (f (x)) \right] \leq \operatorname {s m C E} _ {\mathcal {D}} (f), \\ \mathbb {E} [ \eta (f (x)) ^ {2} ] \geq 0. \\ \end{array}
$$
Plugging these into (5), we get
$$
\mathbb {E} [ \ell (y, f (x)) ] - \mathbb {E} [ \ell (y, \kappa (f (x))) ] \leq 2 \operatorname {s m} \mathsf {C E} _ {\mathcal {D}} (f).
$$
Since this inequality holds for any  $\kappa \in K$ , we get  $\mathsf{pGap}_{\mathcal{D}}(f) \leq 2\mathsf{smCE}_{\mathcal{D}}(f)$  as desired.
Now we prove the lower bound on  $\mathsf{pGap}_{\mathcal{D}}(f)$ . For any 1-Lipschitz function  $\eta :[0,1]\to [-1,1]$ , define  $\beta \coloneqq \mathbb{E}[(y - f(x))\eta (f(x))] \in [-1,1]$  and define post-processing  $\kappa :[0,1]\to [0,1]$  such that
$$
\kappa (v) = \operatorname {p r o j} _ {[ 0, 1 ]} (v + \beta \eta (v)) \quad \text {f o r e v e r y} v \in [ 0, 1 ],
$$
where  $\operatorname{proj}_{[0,1]}(u)$  is the value in  $[0,1]$  closest to  $u$ , i.e.,  $\operatorname{proj}_{[0,1]}(u) = \min(\max(u, 0), 1)$ . By Lemma H.2, we have  $\kappa \in K$ . The expected squared loss after we apply the post-processing  $\kappa$  can be bounded as follows:
$$
\begin{array}{l} \mathbb {E} _ {(x, y) \sim \mathcal {D}} [ \ell (y, \kappa (f (x))) ] \leq \mathbb {E} [ (y - f (x) - \beta \eta (f (x))) ^ {2} ] \\ = \mathbb {E} [ (y - f (x)) ^ {2} ] - 2 \beta \mathbb {E} [ (y - f (x)) \eta (f (x)) ] + \beta^ {2} \mathbb {E} [ \eta (f (x)) ^ {2} ] \\ = \mathbb {E} [ \ell (y, f (x)) ] - 2 \beta^ {2} + \beta^ {2} \mathbb {E} [ \eta (f (x)) ^ {2} ] \\ \leq \mathbb {E} [ \ell (y, f (x)) ] - 2 \beta^ {2} + \beta^ {2}. \\ \end{array}
$$
Re-arranging the inequality above, we have
$$
\mathbb {E} \left[ (y - f (x)) \eta (f (x)) \right] ^ {2} = \beta^ {2} \leq \mathbb {E} \left[ \ell (y, f (x)) \right] - \mathbb {E} \left[ \ell (y, \kappa (f (x))) \right] \leq p G a p _ {\mathcal {D}} (f).
$$
Since this inequality holds for any 1-Lipschitz function  $\eta :[0,1]\to [-1,1]$ , we get  $\mathsf{smCE}_{\mathcal{D}}(f)^2\leq$ $\mathsf{pGap}_{\mathcal{D}}(f)$ , as desired.
In Appendix D we provide examples showing that the constants in Theorem 2.4 are optimal.
4 Generalization to Any Proper Loss
When we aim to minimize the squared loss, we have shown that achieving a small post-processing gap w.r.t. Lipschitz post-processings ensures a small smooth calibration error and vice versa. In this section, we extend this result to a wide class of proper loss functions including the popular cross-entropy loss.
Definition 4.1. Let  $V \subseteq [0,1]$  be a non-empty interval. We say a loss function  $\ell : \{0,1\} \times V \to \mathbb{R}$  is proper if for every  $v \in V$ , it holds that  $v \in \mathrm{argmin}_{v' \in V} \mathbb{E}_{y \sim \mathrm{Ber}(v)}[\ell(y, v')]$ .
One can easily verify that the squared loss  $\ell(y, v) = (y - v)^2$  is a proper loss function over  $V = [0, 1]$ , and the cross entropy loss  $\ell(y, v) = -y \ln v - (1 - y) \ln (1 - v)$  is a proper loss function over  $V = (0, 1)$ .
It turns out that if we directly replace the squared loss in Definition 2.2 with an arbitrary proper loss, we do not get a similar result as Theorem 2.4. In Appendix C we give a simple example where the post-processing gap w.r.t. the cross-entropy loss can be arbitrarily larger than the smooth calibration error. Thus new ideas are needed to extend Theorem 2.4 to general proper loss functions. We leverage a general theory involving correspondence between proper loss functions and convex functions [Shuford et al., 1966, Savage, 1971, Schervish, 1989, Buja et al., 2005]. We provide a detailed description of this theory in Appendix E. There we include a proof of Lemma E.4 which implies the following lemma:
Lemma 4.2. Let  $V \subseteq [0,1]$  be a non-empty interval. Let  $\ell : \{0,1\} \times V \to \mathbb{R}$  be a proper loss function. For every  $v \in V$ , define dual(v) :=  $\ell(0,v) - \ell(1,v)$ . Then there exists a convex function  $\psi : \mathbb{R} \to \mathbb{R}$  such that
$$
\ell (y, v) = \psi (\operatorname {d u a l} (v)) - y \operatorname {d u a l} (v) \quad f o r e v e r y y \in \{0, 1 \} a n d v \in V. \tag {6}
$$
Moreover, if  $\psi$  is differentiable, then  $\nabla \psi (t)\in [0,1]$
Lemma 4.2 says that every proper loss induces a convex function  $\psi$  and a mapping dual:  $V\to \mathbb{R}$  that maps a prediction  $v\in V\subseteq [0,1]$  to its dual prediction  $\mathrm{dual}(v)\in \mathbb{R}$ . Our main theorem applies whenever the induced function  $\psi$  is differentiable, and  $\nabla \psi$  is  $\lambda$ -Lipschitz (equivalently  $\psi$  is  $\lambda$ -smooth) — those are relatively mild conditions that are satisfied by all proper loss functions of interest.
The duality relationship between  $v$  and  $\mathrm{dual}(v)$  comes from the fact that each  $(v, \mathrm{dual}(v))$  pair makes the Fenchel-Young divergence induced by a conjugate pair of convex functions  $(\varphi, \psi)$  take its minimum value zero, as we show in Appendix E. Equation (6) expresses a proper loss as a function that depends on  $\mathrm{dual}(v)$  rather than directly on  $v$ , and thus minimizing a proper loss over predictions  $v$  is equivalent to minimizing a corresponding dual loss over dual predictions  $t = \mathrm{dual}(v)$ :
Definition 4.3 (Dual loss). For a function  $\psi : \mathbb{R} \to \mathbb{R}$ , we define a dual loss function  $\ell^{(\psi)} : \{0,1\} \times \mathbb{R} \to \mathbb{R}$  such that
$$
\ell^ {(\psi)} (y, t) = \psi (t) - y t \quad f o r e v e r y y \in \{0, 1 \} a n d t \in \mathbb {R}.
$$
Consequently, if a loss function  $\ell : \{0,1\} \times V \to \mathbb{R}$  satisfies (6) for some  $V \subseteq [0,1]$  and dual:  $V \to \mathbb{R}$ , then
$$
\ell (y, v) = \ell^ {(\psi)} (y, \operatorname {d u a l} (v)) \quad f o r e v e r y y \in \{0, 1 \} a n d v \in V. \tag {7}
$$
The above definition of a dual loss function is essentially the definition of the Fenchel-Young loss in the literature [see e.g. Duchi et al., 2018, Blondel et al., 2020]. A loss function  $\ell^{(\psi)}$  satisfying the relationship in (7) has been referred to as a composite loss [see e.g. Buja et al., 2005, Reid and Williamson, 2010].
For the cross-entropy loss  $\ell(y, v) = -y \ln v - (1 - y) \ln (1 - v)$ , the corresponding dual loss is the logistic loss  $\ell^{(\psi)}(y, t) = \ln (1 + e^t) - yt$ , and the relationship between a prediction  $v \in (0, 1)$  and its dual prediction  $t = \text{dual}(v) \in \mathbb{R}$  is given by  $v = \sigma(t)$  for the sigmoid transformation  $\sigma(t) = e^t / (1 + e^t)$ .
For a predictor  $f: \mathcal{X} \to V$ , we define its dual post-processing gap by considering dual predictions  $g(x) = \mathrm{dual}(f(x))$  w.r.t. the dual loss  $\ell^{(\psi)}$  as follows:
Definition 4.4 (Dual post-processing gap). For  $\lambda >0$ , let  $K_{\lambda}$  denote the family of all post-processing functions  $\kappa :\mathbb{R}\to \mathbb{R}$  such that there exists a 1-Lipschitz function  $\eta :\mathbb{R}\rightarrow [-1 / \lambda ,1 / \lambda ]$  satisfying  $\kappa (t) = t + \eta (t)$  for every  $t\in \mathbb{R}$ . Let  $\psi$  and  $\ell^{(\psi)}$  be defined as in Definition 4.3. Let  $\mathcal{D}$  be a distribution over  $\mathcal{X}\times \{0,1\}$ . We define the dual post-processing gap of a function  $g:\mathcal{X}\to \mathbb{R}$  to be
$$
\mathsf {p G a p} _ {\mathcal {D}} ^ {(\psi , \lambda)} (g) := \mathbb {E} _ {(x, y) \sim \mathcal {D}} \ell^ {(\psi)} (y, g (x)) - \inf  _ {\kappa \in K _ {\lambda}} \mathbb {E} _ {(x, y) \sim \mathcal {D}} \ell^ {(\psi)} \big (y, \kappa (g (x)) \big).
$$
Definition 4.4 is a generalization of Definition 2.2 from the squared loss to an arbitrary proper loss corresponding to a function  $\psi$  as in (6). The following definition generalizes the definition of smooth calibration in Definition 2.3 to an arbitrary proper loss in a similar way. Here we use the fact proved in Appendix E (equation (20)) that if the function  $\psi$  from Lemma 4.2 for a proper loss  $\ell : \{0,1\} \times V \to \mathbb{R}$  is differentiable, then  $\nabla \psi(\mathrm{dual}(v)) = v$  holds for any  $v \in V$ , where  $\nabla \psi(\cdot)$  denotes the derivative of  $\psi$ . This means that  $\nabla \psi$  transforms a dual prediction to its original prediction.
Definition 4.5 (Dual smooth calibration). For  $\lambda >0$ , let  $H_{\lambda}$  be the family of all  $\lambda$ -Lipschitz functions  $\eta :\mathbb{R}\to [-1,1]$ . Let  $\psi :\mathbb{R}\rightarrow \mathbb{R}$  be a differentiable function with derivative  $\nabla \psi (t)\in [0,1]$  for every  $t\in \mathbb{R}$ . For a function  $g:\mathcal{X}\to \mathbb{R}$ , define predictor  $f:\mathcal{X}\to [0,1]$  such that  $f(x) = \nabla \psi (g(x))$  for every  $x\in \mathcal{X}$ . Let  $\mathcal{D}$  be a distribution over  $\mathcal{X}\times \{0,1\}$ . We define the dual calibration error of  $g$  to be
$$
\mathsf {s m} \mathsf {C E} _ {\mathcal {D}} ^ {(\psi , \lambda)} (g) := \sup  _ {\eta \in H _ {\lambda}} | \mathbb {E} _ {(x, y) \sim \mathcal {D}} [ (y - f (x)) \eta (g (x)) ] |.
$$
In Theorem 4.6 below we state our generalization of Theorem 2.4 to arbitrary proper loss functions. Theorem 4.6 shows that achieving a small dual post-processing gap is equivalent to achieving a small dual calibration error. We then show in Lemma 4.7 that a small dual calibration error implies a small (standard) smooth calibration error.
Theorem 4.6. Let  $\psi : \mathbb{R} \to \mathbb{R}$  be a differentiable convex function with derivative  $\nabla \psi(t) \in [0,1]$  for every  $t \in \mathbb{R}$ . For  $\lambda > 0$ , assume that  $\psi$  is  $\lambda$ -smooth, i.e.,
$$
\left| \nabla \psi (t) - \nabla \psi \left(t ^ {\prime}\right) \right| \leq \lambda \left| t - t ^ {\prime} \right| \quad f o r e v e r y t, t ^ {\prime} \in \mathbb {R}. \tag {8}
$$
Then for every  $g:\mathcal{X}\to \mathbb{R}$  and any distribution  $\mathcal{D}$  over  $\mathcal{X}\times \{0,1\}$ ,
$$
\operatorname {s m C E} _ {\mathcal {D}} ^ {(\psi , \lambda)} (g) ^ {2} / 2 \leq \lambda \operatorname {p G a p} _ {\mathcal {D}} ^ {(\psi , \lambda)} (g) \leq \operatorname {s m C E} _ {\mathcal {D}} ^ {(\psi , \lambda)} (g).
$$
Lemma 4.7. Let  $\psi : \mathbb{R} \to \mathbb{R}$  be a differentiable convex function with derivative  $\nabla \psi(t) \in [0,1]$  for every  $t \in \mathbb{R}$ . For  $\lambda > 0$ , assume that  $\psi$  is  $\lambda$ -smooth as in (8). For  $g: \mathcal{X} \to \mathbb{R}$ , define  $f: \mathcal{X} \to [0,1]$  such that  $f(x) = \nabla \psi(g(x))$  for every  $x \in \mathcal{X}$ . For a distribution  $\mathcal{D}$  over  $\mathcal{X} \times \{0,1\}$ , define  $\mathsf{smCE}_{\mathcal{D}}(f)$  as in Definition 2.3. Then
$$
\operatorname {s m C E} _ {\mathcal {D}} (f) \leq \operatorname {s m C E} _ {\mathcal {D}} ^ {(\psi , \lambda)} (g).
$$
We defer the proofs of Theorem 4.6 and Lemma 4.7 to Appendix F.1 and Appendix F.2. Combining the two results, assuming  $\psi$  is  $\lambda$ -smooth, we have
$$
\operatorname {s m C E} _ {\mathcal {D}} (f) ^ {2} / 2 \leq \lambda \operatorname {p G a p} _ {\mathcal {D}} ^ {(\psi , \lambda)} (g).
$$
This means that a small dual post-processing gap for a proper loss implies a small (standard) smooth calibration error. The cross-entropy loss  $\ell$  is a proper loss where the corresponding  $\psi$  from Lemma 4.2 is given by  $\psi(t) = \ln(1 + e^t)$  and it is  $1/4$ -smooth. Setting  $\lambda = 1/4$  in Theorem 4.6 and Lemma 4.7, we get (3) and (4) for the cross-entropy loss.
5 Optimization Algorithms and Implicit Regularization
As simple consequences of our results, there are several natural algorithms which explicitly minimize loss, but implicitly achieve good calibration. Here, we focus on one such algorithm: structural risk minimization, and discuss additional such algorithms in Appendix G. We also give informal intuitions connecting each algorithm to training DNNs in practice.
We show that structural risk minimization (SRM) on the population distribution is guaranteed to output well-calibrated predictors, under mild assumptions on the function family and regularizer. Recall, structural risk minimization considers a function family  $\mathcal{F}$  equipped with some "complexity measure"  $\mu : \mathcal{F} \to \mathbb{R}_{\geq 0}$ . For example, we may take the family of bounded-width neural networks, with depth
as the complexity measure. SRM then minimizes a proper loss, plus a complexity-regularizer given by  $\mu$ :
$$
f^{*} = \operatorname *{argmin}_{f\in \mathcal{F}}\underset {\mathcal{D}}{\mathbb{E}}[\ell (y,f(x)] + \lambda \mu (f).
$$
The complexity measure  $\mu$  is usually designed to control the capacity of the function family for generalization reasons, though we will not require such assumptions about  $\mu$ . Rather, we only require that  $\mu(f)$  does not grow too quickly under composition with Lipschitz functions. That is,  $\mu(\kappa \circ f)$  should be at most a constant greater than  $\mu(f)$ , for all Lipschitz functions  $\kappa$ . Now, as long as the function family  $\mathcal{F}$  is also closed under composition, SRM is well-calibrated:
Claim 5.1. Let  $\mathcal{F}$  be a class of functions  $f:\mathcal{X}\rightarrow [0,1]$  closed under composition with  $K$ , where we define  $K$  as in Definition 2.2. That is, for  $f\in \mathcal{F},\kappa \in K$  we have  $\kappa \circ f\in \mathcal{F}$ . Let  $\mu :\mathcal{F}\to \mathbb{R}_{\geq 0}$  be any complexity measure satisfying, for all  $f\in \mathcal{F},\kappa \in K:\quad \mu (\kappa \circ f)\leq \mu (f) + 1$ . Then the minimizer  $f^{*}$  of the regularized optimization problem
$$
f^{*} = \operatorname *{argmin}_{f\in \mathcal{F}}\underset {\mathcal{D}}{\mathbb{E}}\ell_{\mathbf{sq}}(y,f(x)) + \lambda \mu (f).
$$
satisfies  $\mathsf{pGap}_{\mathcal{D}}(f^{*})\leq \lambda$  , and thus by Theorem 2.4 has small calibration error: smCE  $\mathcal{D}(f^{*})\leq \sqrt{\lambda}$
Proof. The proof is almost immediate. Let  $\mathrm{MSE}_{\mathcal{D}}(f)$  denote  $\mathbb{E}_{\mathcal{D}}[\ell_{\mathrm{sq}}(y, f(x))]$ . Consider the solution  $f^{*}$  and arbitrary  $\kappa \in K$ . Since  $\kappa \circ f^{*} \in \mathcal{F}$ , we have
$$
\mathrm {M S E} _ {\mathcal {D}} \left(f ^ {*}\right) + \lambda \mu \left(f ^ {*}\right) \leq \mathrm {M S E} _ {\mathcal {D}} \left(\kappa \circ f ^ {*}\right) + \lambda \mu \left(\kappa \circ f ^ {*}\right) \leq \mathrm {M S E} _ {\mathcal {D}} \left(\kappa \circ f ^ {*}\right) + \lambda \mu \left(f ^ {*}\right) + \lambda .
$$
After rearranging, this is equivalent to
$$
\operatorname {M S E} _ {\mathcal {D}} \left(f ^ {*}\right) - \operatorname {M S E} _ {\mathcal {D}} \left(\kappa \circ f ^ {*}\right) \leq \lambda ,
$$
and since  $\kappa \in K$  was arbitrary, we get  $\mathsf{pGap}_{\mathcal{D}}(f) \leq \lambda$ . as desired.
Discussion: Implicit Regularization. This aspect of SRM connects our results to the "implicit regularization" hypothesis from deep learning theory community [Neyshabur, 2017]. The implicit regularization hypothesis is an informal belief that SGD on neural networks implicitly performs minimization on a "complexity-regularized" objective, which ensures generalization. The exact form of this complexity regularizer remains elusive, but many works have attempted to identify its structural properties (e.g. Arora et al. [2019], Neyshabur et al. [2014], Neyshabur [2017], Zhang et al. [2021]). In this context, our results imply that if the implicit regularization hypothesis is true, then as long as the complexity measure  $\mu$  doesn't increase too much from composition, the final output of SGD will be well-calibrated.
Inspired by recent empirical observations, we studied formal conditions under which optimizing a proper loss also happens to yield calibration "for free." We identified a certain local optimality condition that characterizes distance to calibration, in terms of properties of the (proper) loss landscape. Our results apply even to realistic optimization algorithms, which optimize over restricted function families, and may not reach global minima within these families. In particular, our formal results suggest an intuitive explanation for why state-of-the-art DNNs are often well-calibrated: because their test loss cannot be improved much by adding a few more layers. It also offers guidance for how one can achieve calibration simply by proper loss minimization over a sufficiently expressive family of predictors.
Limitations. The connection between our theoretical results and DNNs in practice is heuristic—for example, we do not have a complete proof that training a DNN with SGD on natural distributions will yield a calibrated predictor, though we have formal results that give some intuition for this. Part of the difficulty here is due to known definitional barriers [Nakkiran, 2021], which is that we would need to formally specify what "DNN" means in practice (which architectures, and sizes?), what "SGD" means in practice (what initialization, learning-rate schedule, etc), and what "natural distributions" means in practice. Finally, we intuitively expect the results from our work to generalize beyond the binary label setting; we leave it to future work to formalize this intuition.