SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\ell (y, v) = \ell (0, v) - y \operatorname {d u a l} (v) = \psi_ {0} (\operatorname {d u a l} (v)) - y \operatorname {d u a l} (v). \tag {16}
$$
By the assumption that  $\ell$  is proper, for every  $v\in V$
$$
v\in \operatorname *{argmin}_{v^{\prime}\in V}\underset {y\sim \mathsf{Ber}(v)}{\mathbb{E}}\ell (y,v^{\prime}) = \operatorname *{argmin}_{v^{\prime}\in V}\Bigl(\psi_{0}(\mathsf{dual}(v^{\prime})) - v  \mathsf{dual}(v^{\prime})\Bigr),
$$
$$
\operatorname {dual}(v)\in \operatorname *{argmin}_{t\in T}\Bigl(\psi_{0}(t) - vt\Bigr).
$$
For every  $v\in V$  , we define
$$
\varphi (v) := v \operatorname {d u a l} (v) - \psi_ {0} (\operatorname {d u a l} (v)) = \max  _ {t \in T} \left(v t - \psi_ {0} (t)\right).
$$
The definition of  $\varphi$  immediately implies the following:
$$
D _ {\varphi , \psi_ {0}} (v, t) \geq 0 \quad \text {f o r e v e r y} v \in V \text {a n d} t \in T, \tag {17}
$$
$$
D _ {\varphi , \psi_ {0}} (v, \operatorname {d u a l} (v)) = 0 \quad \text {f o r e v e r y} v \in V. \tag {18}
$$
By Claim E.3,  $\mathrm{dual}(v)$  is a sub-gradient of  $\varphi$  at  $v$ . This proves that  $\varphi$  is convex. Now we define  $\psi : \mathbb{R} \to \mathbb{R}$  such that  $\psi(t) = \sup_{v \in V} vt - \varphi(v)$  for every  $t \in \mathbb{R}$ . This definition ensures that inequality (13) holds. By Lemma H.3,  $\psi$  is a convex function and (15) holds. Moreover, for every  $v \in V$ , we have shown that  $\mathrm{dual}(v)$  is a sub-gradient of  $\varphi$  at  $v$ , so by (11),  $\psi(\mathrm{dual}(v)) = v \mathrm{dual}(v) - \varphi(v) = \psi_0(\mathrm{dual}(v))$ . This implies that (12) and (14) hold because of (16) and (18).
Lemma E.5 (Proper loss from convex functions). Let  $V \subseteq [0,1]$  be a non-empty interval. Let  $\varphi : V \to \mathbb{R}$ ,  $\psi : \mathbb{R} \to \mathbb{R}$ , and dual:  $V \to \mathbb{R}$  be functions satisfying (13) and (14). Define loss function  $\ell : \{0,1\} \times V \to \mathbb{R}$  as in (12). Then
1.  $\ell$  is a proper loss function;
2.  $\varphi$  is convex;
3. for every  $v \in V$ , dual(v) is a sub-gradient of  $\varphi$  at  $v$ , and  $v$  is a sub-gradient of  $\psi$  at dual(v);
4.  $\varphi (v) = v\mathrm{dual}(v) - \psi (\mathrm{dual}(v)) = -\mathbb{E}_{y\sim \mathrm{Ber}(v)}[\ell (y,v)];$
5. if we define  $\psi'(t) \coloneqq \sup_{v \in V} (vt - \varphi(v))$ , then  $\psi'$  is convex and  $\psi'(\mathrm{dual}(v)) = \psi(\mathrm{dual}(v))$  for every  $v \in V$ . Moreover, for any two distinct real numbers  $t_1, t_2$ ,
$$
0 \leq \frac {\psi^ {\prime} \left(t _ {2}\right) - \psi^ {\prime} \left(t _ {1}\right)}{t _ {2} - t _ {1}} \leq 1. \tag {19}
$$
Proof. For every  $v \in V$ , equations (13) and (14) imply that
$$
\operatorname {d u a l} (v) \in \operatorname * {a r g m i n} _ {t \in \mathbb {R}} \left(\psi (t) - v t\right),
$$
$$
v\in \operatorname *{argmin}_{v^{\prime}\in V}\Bigl(\psi (\mathsf{dual}(v^{\prime})) - v  \mathsf{dual}(v^{\prime})\Bigr) = \operatorname *{argmin}_{v^{\prime}\in V}\underset {y\sim \mathsf{Ber}(v)}{\mathbb{E}}\ell (y,v^{\prime}).
$$
This proves that  $\ell$  is proper (Item 1). Item 3 follows from (13), (14) and Claim E.3. Item 2 follows from Item 3. The first equation in Item 4 follows from (14), and the second equation follows from (12). For Item 5, the convexity of  $\psi'$  and inequality (19) follow from Lemma H.3, and  $\psi'(\mathrm{dual}(v)) = \psi(\mathrm{dual}(v))$  holds because  $\psi'(\mathrm{dual}(v)) = v\mathrm{dual}(v) - \varphi(v)$  by Item 3 and (11), and  $\psi(\mathrm{dual}(v)) = v\mathrm{dual}(v) - \varphi(v)$  by Item 4.
Our discussion above gives a correspondence between a proper loss function  $\ell : \{0,1\} \times V \to \mathbb{R}$  and a tuple  $(\varphi, \psi, \mathrm{dual})$  satisfying (13) and (14). By Lemma E.5, if  $\varphi, \psi$  and dual satisfy (13) and (14), then every  $v \in V$  is a sub-gradient of  $\psi$  at  $\mathrm{dual}(v)$ . Therefore, assuming  $\psi$  is differentiable and using  $\nabla \psi : \mathbb{R} \to \mathbb{R}$  to denote its derivative, if  $t = \mathrm{dual}(v)$  for some  $v \in V$ , then we can compute  $v$  from  $t$  by
$$
v = \nabla \psi (t). \tag {20}
$$
For the cross entropy loss  $\ell(y, v) = -y \ln v - (1 - y) \ln (1 - v)$ , we can find the corresponding  $\varphi, \psi$ , dual and  $\ell^{(\psi)}$  (see Definition 4.3) as follows:
$$
\varphi (v) = - \underset {y \sim \operatorname {B e r} (v)} {\mathbb {E}} [ \ell (y, v) ] = v \ln v + (1 - v) \ln (1 - v),
$$
$$
\operatorname {d u a l} (v) = \ell (0, v) - \ell (1, v) = \ln (v / (1 - v)),
$$
$$
\psi (t) = \sup  _ {v \in (0, 1)} \left(v t - \varphi (v)\right) = \ln (1 + e ^ {t}),
$$
$$
\ell^ {(\psi)} (y, t) = \psi (t) - y t = \ln \left(1 + e ^ {t}\right) - y t, \quad (\text {l o g i s t i c}
$$
$$
\nabla \psi (t) = e ^ {t} / \left(1 + e ^ {t}\right). \quad (\text {s i g m o i d t r a n s f o r m a t i o n})
$$
For the squared loss  $\ell(y, v) = (y - v)^2$ , we can find the corresponding  $\varphi, \psi$ , dual and  $\ell^{(\psi)}$  as follows:
$$
\varphi (v) = - \underset {y \sim \operatorname {B e r} (v)} {\mathbb {E}} [ \ell (y, v) ] = - v (1 - v) ^ {2} - (1 - v) v ^ {2} = v (v - 1),
$$
$$
\operatorname {d u a l} (v) = \ell (0, v) - \ell (1, v) = 2 v - 1,
$$
$$
\psi (t) = \sup  _ {v \in [ 0, 1 ]} \big (v t - \varphi (v) \big) = \left\{ \begin{array}{l l} 0, & \text {i f} t <   - 1; \\ (t + 1) ^ {2} / 4, & \text {i f} - 1 \leq t \leq 1; \\ t, & \text {i f} t > 1. \end{array} \right.
$$
$$
\ell^ {(\psi)} (y, t) = \psi (t) - y t = \left\{ \begin{array}{l l} - y t, & \text {i f} t <   - 1; \\ (y - (t + 1) / 2) ^ {2}, & \text {i f} - 1 \leq t \leq 1; \\ (1 - y) t, & \text {i f} t > 1. \end{array} \right.
$$
$$
\nabla \psi (t) = \left\{ \begin{array}{l l} 0, & \text {i f} t <   - 1; \\ (t + 1) / 2, & \text {i f} - 1 \leq t \leq 1; \\ 1, & \text {i f} t > 1. \end{array} \right.
$$
For the squared loss, one can alternatively choose  $\psi(t) = (t + 1)^2 / 4$  for every  $t \in \mathbb{R}$  and accordingly define  $\ell^{(\psi)}(y, t) = (y - (t + 1) / 2)^2$  for every  $y \in \{0, 1\}$  and  $t \in \mathbb{R}$ . This still ensures (12), (13) and (14), but (15) no longer holds.
F Generalized Dual (Multi)calibration and Generalized Dual Post-processing Gap
Below we generalize the notions of dual post-processing gap and dual smooth calibration error from Section 4 by replacing the Lipschitz functions in those definitions with functions from a general class. Here we allow the functions to not only depend on the dual prediction  $g(x)$ , but also depend directly on  $x$ . With such generality, the calibration notion we consider in Definition F.2 below captures the notion of multicalibration [Hébert-Johnson et al., 2018]. We connect this generalized calibration notion with a generalized notion of post-processing gap (Definition F.1) in Theorem F.3.
Definition F.1 (Generalized dual post-processing gap). Let  $\psi$  and  $\ell^{(\psi)}$  be defined as in Definition 4.3. Let  $W$  be a class of functions  $w: \mathcal{X} \times \mathbb{R} \to \mathbb{R}$ . Let  $\mathcal{D}$  be a distribution over  $\mathcal{X} \times \{0,1\}$ . We define the generalized dual post-processing gap of a function  $g: \mathcal{X} \to \mathbb{R}$  w.r.t. class  $W$  and distribution  $\mathcal{D}$  to be
$$
\operatorname {g e n G a p} _ {\mathcal {D}} ^ {(\psi , W)} (g) := \mathbb {E} _ {(x, y) \sim \mathcal {D}} \ell^ {(\psi)} (y, g (x)) - \inf  _ {w \in W} \mathbb {E} _ {(x, y) \sim \mathcal {D}} \ell^ {(\psi)} \big (y, w (x, g (x)) \big).
$$
Definition F.2 (Generalized dual calibration). Let  $\psi : \mathbb{R} \to \mathbb{R}$  be a differentiable function. For a function  $g: \mathcal{X} \to \mathbb{R}$ , define predictor  $f: \mathcal{X} \to \mathbb{R}$  such that  $f(x) = \nabla \psi(g(x))$  for every  $x \in \mathcal{X}$ .<sup>4</sup> Let
<sup>4</sup>It is possible here for  $f(x)$  to lie outside the interval [0, 1], in which case  $f$  is not a valid predictor. However, if  $\psi$  satisfies (15), then  $\nabla \psi(t) \in [0, 1]$  for every  $t \in \mathbb{R}$  and thus  $f(x) \in [0, 1]$  for every  $x \in \mathcal{X}$ . Also, one can
$W$  be a class of functions  $w: \mathcal{X} \times \mathbb{R} \to \mathbb{R}$ . Let  $\mathcal{D}$  be a distribution over  $\mathcal{X} \times \{0,1\}$ . We define the generalized dual calibration error of  $g$  w.r.t. class  $W$  and distribution  $\mathcal{D}$  to be
$$
\operatorname {g e n C E} _ {\mathcal {D}} ^ {(\psi , W)} (g) := \sup  _ {w \in W} | \mathbb {E} _ {(x, y) \sim \mathcal {D}} [ (y - f (x)) w (x, g (x)) ] |.
$$
Theorem F.3. Let  $\psi : \mathbb{R} \to \mathbb{R}$  be a differentiable convex function. For  $\lambda > 0$ , assume that  $\psi$  is  $\lambda$ -smooth as defined in (8). Let  $W$  be a class of bounded functions  $w: \mathcal{X} \times \mathbb{R} \to [-1,1]$ . Define  $W'$  to be the class of functions  $w': \mathcal{X} \times \mathbb{R} \to \mathbb{R}$  such that there exist  $w \in W$  and  $\beta \in [-1/\lambda, 1/\lambda]$  satisfying
$$
w ^ {\prime} (x, t) = t + \beta w (x, t) \quad f o r e v e r y x \in \mathcal {X} a n d t \in \mathbb {R}.
$$
$$
\operatorname {g e n C E} _ {\mathcal {D}} ^ {(\psi , W)} (g) ^ {2} / 2 \leq \lambda \operatorname {g e n G a p} _ {\mathcal {D}} ^ {(\psi , W ^ {\prime})} (g) \leq \operatorname {g e n C E} _ {\mathcal {D}} ^ {(\psi , W)} (g).
$$
Proof. By the definition of  $\ell^{(\psi)}$  in Definition 4.3,
$$
\begin{array}{l} \underset {(x, y) \sim \mathcal {D}} {\mathbb {E}} \ell^ {(\psi)} (y, g (x)) - \underset {(x, y) \sim \mathcal {D}} {\mathbb {E}} \ell^ {(\psi)} \left(y, w ^ {\prime} (x, g (x))\right) \\ = \mathbb {E} [ \psi (g (x)) - y g (x) - \psi \left(w ^ {\prime} (x, g (x))\right) + y w ^ {\prime} (x, g (x)) ] \\ = \mathbb {E} [ \psi (g (x)) - \psi \left(w ^ {\prime} (x, g (x))\right) + y \beta w (x, g (x)) ]. \tag {21} \\ \end{array}
$$
By the convexity of  $\psi$  and the  $\lambda$ -smoothness property, for every  $t, t' \in \mathbb{R}$ ,
$$
\begin{array}{l} \nabla \psi (t) (t ^ {\prime} - t) \leq \psi (t ^ {\prime}) - \psi (t) = \int_ {t} ^ {t ^ {\prime}} \nabla \psi (\tau) \mathrm {d} \tau \\ \leq \int_ {t} ^ {t ^ {\prime}} (\nabla \psi (t) + \lambda (\tau - t)) \mathrm {d} \tau = \nabla \psi (t) \left(t ^ {\prime} - t\right) + \frac {\lambda}{2} \left(t ^ {\prime} - t\right) ^ {2}. \tag {22} \\ \end{array}
$$
For  $x \in X$ , setting  $t$  to be  $g(x)$  and setting  $t'$  to be  $w'(x, g(x)) \coloneqq g(x) + \beta w(x, g(x))$  for some  $w \in W$  and  $\beta \in [-1/\lambda, 1/\lambda]$ , we have
$$
t ^ {\prime} - t = w ^ {\prime} (x, g (x)) - g (x) = \beta w (x, g (x)).
$$
Plugging this into (22) and defining  $f(x)\coloneqq \nabla \psi (g(x)) = \nabla \psi (t)$ , we have
$$
f (x) \beta w (x, g (x)) \leq \psi \left(w ^ {\prime} (x, g (x))\right) - \psi (g (x)) \leq f (x) \beta w (x, g (x)) + \frac {\lambda \beta^ {2}}{2} w (x, g (x)) ^ {2}.
$$
Plugging this into (21) and using the fact that  $|w(x, g(x))| \leq 1$ , we have
$$
\begin{array}{l} \mathbb {E} \left[ (y - f (x)) \beta w (x, g (x)) \right] - \frac {\lambda \beta^ {2}}{2} \\ \leq \underset {(x, y) \sim \mathcal {D}} {\mathbb {E}} \ell^ {(\psi)} (y, g (x)) - \underset {(x, y) \sim \mathcal {D}} {\mathbb {E}} \ell^ {(\psi)} \left(y, w ^ {\prime} (x, g (x))\right) (23) \\ \leq \mathbb {E} [ (y - f (x)) \beta w (x, g (x)) ]. (24) \\ \end{array}
$$
For any  $w \in W$ , choosing  $\beta = \mathbb{E}[(y - f(x))w(x,g(x))] / \lambda$  and choosing  $w' \in W'$  such that  $w'(x,t) = t + \beta w(x,t)$ , by (23) we have
$$
\frac {1}{2 \lambda} \mathbb {E} [ (y - f (x)) w (x, g (x)) ] ^ {2} \leq \underset {(x, y) \sim \mathcal {D}} {\mathbb {E}} \ell^ {(\psi)} (y, g (x)) - \underset {(x, y) \sim \mathcal {D}} {\mathbb {E}} \ell^ {(\psi)} \big (y, w ^ {\prime} \big (x, g (x) \big) \big).
$$
$$
\operatorname {g e n C E} _ {\mathcal {D}} ^ {(\psi , W)} (g) ^ {2} / 2 \leq \lambda \operatorname {g e n G a p} _ {\mathcal {D}} ^ {(\psi , W ^ {\prime})} (g).
$$
For any  $w' \in W'$ , there exists  $\beta \in [-1/\lambda, 1/\lambda]$  and  $w \in W$  such that  $w'(x,t) = t + \beta w(x,t)$ . By (24),
$$
\begin{array}{l} \underset {(x, y) \sim \mathcal {D}} {\mathbb {E}} \ell^ {(\psi)} (y, g (x)) - \underset {(x, y) \sim \mathcal {D}} {\mathbb {E}} \ell^ {(\psi)} \left(y, w ^ {\prime} (x, g (x))\right) \\ \leq \mathbb {E} [ (y - f (x)) \beta w (x, g (x)) ] \\ \leq (1 / \lambda) | \mathbb {E} [ (y - f (x)) w (x, g (x)) ] |. \\ \end{array}
$$
$$
\lambda \operatorname {g e n G a p} _ {\mathcal {D}} ^ {(W ^ {\prime})} (g) \leq \operatorname {g e n C E} _ {\mathcal {D}} ^ {(W)} (g).
$$
start from a valid predictor  $f: \mathcal{X} \to [0,1]$  and define  $g(x) = \mathrm{dual}(f(x))$ . By Lemma E.5, this ensures that  $f(x) = \nabla \psi(g(x))$  assuming that  $(\varphi, \psi, \mathrm{dual})$  satisfy (13) and (14).
F.1 Proof of Theorem 4.6
We restate and prove Theorem 4.6.
$$
\mathsf {s m C E} _ {\mathcal {D}} ^ {(\psi , \lambda)} (g) ^ {2} / 2 \leq \lambda \mathsf {p G a p} _ {\mathcal {D}} ^ {(\psi , \lambda)} (g) \leq \mathsf {s m C E} _ {\mathcal {D}} ^ {(\psi , \lambda)} (g).
$$
Proof. Let  $W$  be the family of all  $\lambda$ -Lipschitz functions  $\eta : \mathbb{R} \to [-1, 1]$  and define  $W'$  as in Theorem F.3. We have
$$
\operatorname {g e n} \mathrm {C E} _ {\mathcal {D}} ^ {(\psi , W)} (g) = \operatorname {s m} \mathrm {C E} _ {\mathcal {D}} ^ {(\psi , \lambda)} (g), \quad \text {a n d}
$$
$$
\operatorname {g e n G a p} _ {\mathcal {D}} ^ {(\psi , W ^ {\prime})} (g) = \mathrm {p G a p} _ {\mathcal {D}} ^ {(\psi , \lambda)} (g).
$$
Theorem 4.6 then follows immediately from Theorem F.3.
F.2 Proof of Lemma 4.7
We restate and prove Lemma 4.7.
Proof. For a 1-Lipschitz function  $\eta : [0,1] \to [-1,1]$ , define  $\eta': \mathbb{R} \to [-1,1]$  such that  $\eta'(t) = \eta(\nabla \psi(t))$  for every  $t \in \mathbb{R}$ . For  $t_1, t_2 \in \mathbb{R}$ , by the  $\lambda$ -smoothness property of  $\eta$ ,
$$
\left| \eta^ {\prime} \left(t _ {1}\right) - \eta^ {\prime} \left(t _ {2}\right) \right| = \left| \eta \left(\nabla \psi \left(t _ {1}\right)\right) - \eta \left(\nabla \psi \left(t _ {2}\right)\right) \right| \leq \left| \nabla \psi \left(t _ {1}\right) - \nabla \psi \left(t _ {2}\right) \right| \leq \lambda \left| t _ {1} - t _ {2} \right|.
$$
This proves that  $\eta^{\prime}$  is  $\lambda$ -Lipschitz. Therefore,
$$
\left| \mathbb {E} _ {(x, y) \sim \mathcal {D}} [ (y - f (x)) \eta (f (x)) ] \right| = \left| \mathbb {E} _ {(x, y) \sim \mathcal {D}} (y - f (x)) \eta^ {\prime} (g (x)) \right| \leq \operatorname {s m C E} _ {\mathcal {D}} ^ {(\psi , \lambda)} (g).
$$
Taking supremum over  $\eta$  completes the proof.
G Optimization Algorithms Which Achieve Calibration
Continuing the discussion of structural risk minimization in Section 5, here we describe additional families of learning algorithms which explicitly minimize loss, but implicitly achieve good calibration.
G.1 Iterative Risk Minimization
One intuition for why state-of-the-art DNNs are often calibrated is that they are "locally optimal" (in our sense) by design. For a state-of-the-art network, if it were possible to improve its test loss significantly by just adding a layer (and training it optimally), then the human practitioner training the DNN would have added another layer and re-trained. Thus, we expect the network eventually produced will be approximately loss-optimal with respect to adding a layer, and thus also with respect to univariate Lipschitz post-processings (so  $\mathsf{pGap}_{\mathcal{D}}(f)\approx 0$ ).
This intuition makes several strong assumptions, for example, the practitioner must be able to retrain the last two layers globally optimally<sup>5</sup>. However, the same idea can apply in settings where optimization is tractable. We give one way of formalizing this via the Iterative Risk Minimization algorithm.
We work with the following setup:
Technically, we only require a weaker statement of the form: For all networks  $f_{d}$  of depth  $d$  in the support of SGD outputs, if the loss of  $\kappa \circ f_{d}$  is  $\varepsilon$ , then training a network  $f_{d+1}$  of depth  $d+1$  will reach loss at most  $\varepsilon$ .
- We are interested in models  $f \in \mathcal{F}$  with a complexity measure  $\mu : \mathcal{F} \to \mathbb{Z}$ . Think of size for decision trees or depth for neural nets with a fixed architecture. We assume that  $\mathcal{F}$  contains all constant functions and that  $\mu(c) = 0$  for any constant function  $c$ .
- We consider post-processing functions  $\kappa : [0,1] \to [0,1]$  belonging to some family  $K$ , which have bounded complexity under  $\mu$ . Formally, there exists  $b = b(K) \in \mathbb{Z}$  such that for every  $f \in \mathcal{F}$  and  $\kappa \in K$ ,
$$
\mu (\kappa \circ f) \leq \mu (f) + b
$$
- We have a loss function  $\ell : [0,1] \times \mathbb{R} \to \mathbb{R}$ . Let
$$
\ell (f, \mathcal {D}) = \underset {(x, y) \sim \mathcal {D}} {\mathbb {E}} [ \ell (y, f) ].
$$
For any complexity bound  $s$ , we have an efficient procedure  $\mathrm{Min}_{\mathcal{D}}(s)$  that can solve the loss minimization problem over models of complexity  $s$ :
$$
\min_{\substack{f\in \mathcal{F}\\ \mu (f)\leq s}}\ell (f,\mathcal{D}).
$$
The running time can grow with  $s$ . For each  $s \in \mathbb{Z}$ , let  $\mathrm{OPT}_s$  denote the minimum of this program and let  $f_s^*$  denote the model found by  $\operatorname{Min}_{\mathcal{D}}(s)$  that achieves this minimum.
We use this to present an algorithm that produces a model  $f$  which is smoothly calibrated, has loss bounded by  $\mathrm{OPT}_s$ , and complexity not too much larger than  $s$ .
Algorithm 1 Local search for small pGap.
Input  $s_0$ $s\gets s_0$ $h\gets 0$    
while True do   
 $f_{s}^{*} = \mathrm{Min}(s)$ $f_{s + b}^{*} = \mathrm{Min}(s + b)$    
if  $\mathrm{OPT}_{s + b}\geq \mathrm{OPT}_s - \alpha$  then Return  $f_{s}^{*}$    
else   
 $h\gets h + 1$ $s\gets s + b$    
end if   
end while
Theorem G.1. On input  $s_0$ , Algorithm 1 terminates in  $h \leq 1 / \alpha$  steps and returns a model  $f \in \mathcal{F}$  which satisfies the following guarantees:
1.  $\mu (f) = t\leq s_0 + hb$
2.  $\ell_2(f,\mathcal{D}) = \mathrm{OPT}_t\leq \mathrm{OPT}_{s_0} - h\alpha .$
3.  $\mathsf{pGap}(f)\leq \alpha$  , hence smCE  $(f)\leq \sqrt{\alpha}$
Proof. The bound on the number of steps follows by observing that each update decreases  $\ell_2(f, \mathcal{D})$  by  $\alpha$ . Since  $\mathrm{OPT}_{s_0} \leq \mathrm{OPT}_0 \leq 1$ , this can only happen  $1 / \alpha$  times. This also implies items (1) and (2), since each update increases the complexity by no more than  $b$ , while decreasing the loss by at least  $\alpha$ .
To prove claim (3), assume for contradiction that  $\mathsf{pGap}(f) > \alpha$ . Then there exists  $\kappa \in K$  such that if we consider  $f' = \kappa \circ f$ , then
$$
\mu \left(f ^ {\prime}\right) \leq t + b, \ell_ {2} (f, \mathcal {D}) <   \mathrm {O P T} _ {t} - \alpha .
$$
But this  $f^{\prime}$  provides a certificate that
$$
\mathrm {O P T} _ {t + b} \leq \ell_ {2} \left(f ^ {\prime}, \mathcal {D}\right) <   \mathrm {O P T} _ {t} - \alpha .
$$
Hence the termination condition is not satisfied for  $f$ . So by contradiction, it must hold that  $\mathsf{pGap}(f) \leq \alpha$ , hence  $\mathsf{smCE}(f) \leq \sqrt{\alpha}$ .
We now present an alternative algorithm that is close in spirit to algorithm 1, but can be viewed more directly as minimizing a regularized loss function. We assume access to the same minimization procedure  $\operatorname{Min}_{\mathcal{D}}(s)$  as before. We define the regularized loss
$$
\ell^ {\lambda} (f, \mathcal {D}) = \ell (f, \mathcal {D}) + \lambda \mu (f).
$$
$\lambda$  quantifies the tradeoff between accuracy and model complexity that we accept. In other words, we accept increasing  $\mu(f)$  by 1 if it results in a reduction of at least  $\lambda$  in the loss.
Algorithm 2 Regularized loss minimization for small pGap.
$r\gets \lfloor 1 / \lambda \rfloor$    
while  $s\in \{0,\dots ,r\}$  do  $f_{s}^{*}\gets \mathrm{Min}(s)$ $\mathrm{OPT}_s\gets \ell_2(f_s^*,\mathcal{D}).$    
end while  
 $t\gets \mathrm{argmin}_{s\in \{0,\ldots ,r\}}\mathrm{OPT}_s + \lambda s.$    
Return  $f_{t}^{*}$
Theorem G.2. Let  $\lambda = \alpha /b$ . Algorithm 2 returns a model  $f$  with  $\mu (f) = t\leq \lfloor 1 / \lambda \rfloor$  where
1.  $f = \operatorname{argmin}_{f' \in \mathcal{F}} \ell^{\lambda}(f', \mathcal{D})$  and  $\ell_2(f, \mathcal{D}) = \mathrm{OPT}_t$ .
2. For any  $s \neq t$ ,
$$
\mathrm {O P T} _ {s} \geq \mathrm {O P T} _ {t} - (s - t) \lambda .
$$
3.  $\mathsf{pGap}(f)\leq \alpha$  , hence smCE  $(f)\leq \sqrt{\alpha}$
Proof. We claim that
$$
\min  _ {f ^ {\prime} \in \mathcal {F}} \ell^ {\lambda} (f ^ {\prime}, \mathcal {D}) \leq 1.
$$
This follows by taking the constant  $1/2$ , which results in  $\ell_2(0, \mathcal{D}) \leq 1$  and  $\mu(f) = 0$ . This means that the optimum is achieved for  $f$  where  $\mu(f) \leq 1/\lambda$ . For  $f$  such that  $\mu(f) \leq t$ , it is easy to see that Algorithm 2 finds the best model.
Assume that  $s \neq t$ . By item (1), we must have
$$
\mathrm {O P T} _ {t} + \lambda t \leq \mathrm {O P T} _ {s} + \lambda s
$$
Rearranging this gives the claimed bound.
Assume that  $\mathsf{pGap}(f) = \alpha' > \alpha$ . Then there exists  $f' = \kappa(f)$  such that  $\mu(f') \leq \mu(f) + b$ , but  $\ell(f', \mathcal{D}) \leq \ell_2(f, \mathcal{D}) - \alpha'$ . But this means that
$$
\begin{array}{l} \ell^ {\lambda} \left(f ^ {\prime}\right) = \ell \left(f ^ {\prime}, \mathcal {D}\right) + \lambda \mu \left(f ^ {\prime}\right) \\ \leq \ell_ {2} (f, \mathcal {D}) - \alpha^ {\prime} + \lambda (\mu (f) + b) \\ = \ell^ {\lambda} (f) + (\lambda b - \alpha^ {\prime}) \\ <   \ell^ {\lambda} (f) \\ \end{array}
$$
where the inequality holds by our choice of  $\lambda = \alpha /b$ . But this contradicts item (1).
Note that if we take  $s - t = hb$ , then  $(s - t)\lambda = h\alpha$ , so Item (2) in Theorem G.2 gives a statement matching Item (2) in Theorem G.1. The difference is that we now get a bound for any  $s \neq t$ . When  $s > t$ , item (3) upper bounds  $\mathrm{OPT}_t - \mathrm{OPT}_s$ , which measures how much the  $\ell_2$  loss decreases with increased complexity. When  $s < t$ ,  $\mathrm{OPT}_s > \mathrm{OPT}_t$ , and item (3) lower bounds how much the loss increases when shrink the complexity of the model.
Discussion: Importance of depth. These two algorithms highlight the central role of composition in function families. Specifically, if we are optimizing over a function family that is not closed under univariate Lipshitz compositions (perhaps approximately), then we have no reason to expect  $\mathsf{pGap}(f) \approx 0$ : because there could exist a simple (univariate, Lipshitz) transformation which reduces the loss, but which our function family cannot model. This may provide intuition for why certain models which preceded DNNs, like logistic regression and SVMs, were often not calibrated. Moreover, it emphasizes the importance of depth in neural-networks: depth is required to model composing Lipshitz transforms, which improves both loss and calibration.
G.2 Heuristic Structure of SGD
Finally, there is an informal conjecture which could explain why certain DNNs in practice are calibrated, even if they are far from state-of-the-art: Informally, if it were possible to improve the loss by a simple update to the network function (post-processing), then one may conjecture SGD would have exploited this by the end of training. That is, updating the parameters from computing the function  $f$  to computing  $(\kappa \circ f)$  may be a "simple" update for SGD on a deep-enough network, and SGD would not leave this gain on the table by the end of training. We stress that this conjecture is informal and speculative, but we consider it a plausible intuition.
H Helper Lemmas
Claim H.1. Let  $\eta_1, \eta_2 : [0, 1] \to \mathbb{R}$  be two 1-Lipschitz functions. Then  $\max(\eta_1(v), \eta_2(v))$  and  $\min(\eta_1(v), \eta_2(v))$  are also 1-Lipschitz functions of  $v \in [0, 1]$ .
Proof. Fix arbitrary  $v, v' \in [0,1]$ . Let  $i, i' \in \{1,2\}$  be such that  $\max(\eta_1(v), \eta_2(v)) = \eta_i(v)$  and  $\max(\eta_1(v'), \eta_2(v')) = \eta_{i'}(v')$ . We have