SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Part of this work was performed while LH was interning at Apple. LH is also supported by Moses Charikar's and Omer Reingold's Simons Investigators awards, Omer Reingold's NSF Award IIS-1908774, and the Simons Foundation Collaboration on the Theory of Algorithmic Fairness. JB is supported by Simons Foundation.
Moloud Abdar, Farhad Pourpanah, Sadiq Hussain, Dana Rezazadegan, Li Liu, Mohammad Ghavamzadeh, Paul Fieguth, Xiaochun Cao, Abbas Khosravi, U Rajendra Acharya, et al. A review of uncertainty quantification in deep learning: Techniques, applications and challenges. Information Fusion, 76:243-297, 2021.
Sanjeev Arora, Nadav Cohen, Wei Hu, and Yuping Luo. Implicit regularization in deep matrix factorization. Advances in Neural Information Processing Systems, 32, 2019.
Jarosław Błasiok and Preetum Nakkiran. Smooth ECE: Principled reliability diagrams via kernel smoothing. arXiv preprint arXiv:2309.12236, 2023.
Jarosław Blasiot, Parikshit Gopalan, Lunjia Hu, Adam Tauman Kalai, and Preetum Nakkiran. Loss minimization yields multicalibration for large neural networks. arXiv preprint arXiv:2304.09424, 2023a. URL https://doi.org/10.48550/arXiv.2304.09424.
Jarosław Blasiot, Parikshit Gopalan, Lunjia Hu, and Preetum Nakkiran. A unifying theory of distance from calibration. In Proceedings of the 55th Annual ACM Symposium on Theory of Computing, page 1727-1740, New York, NY, USA, 2023b. Association for Computing Machinery. ISBN 9781450399135.
Mathieu Blondel, André F. T. Martins, and Vlad Niculae. Learning with Fenchel-Young losses. Journal of Machine Learning Research, 21(35):1-69, 2020. URL http://jmlr.org/papers/v21/19-021.html.
Glenn W Brier et al. Verification of forecasts expressed in terms of probability. Monthly weather review, 78(1):1-3, 1950.
Andreas Buja, Werner Stuetzle, and Yi Shen. Loss functions for binary class probability estimation and classification: Structure and applications. Manuscript, 2005. URL https://sites.stat.washington.edu/wxs/Learning-papers/paper-proper-scoring.pdf.
Annabelle Carrell, Neil Mallinar, James Lucas, and Preetum Nakkiran. The calibration generalization gap. arXiv preprint arXiv:2210.01964, 2022.
Lucas Clarté, Bruno Loureiro, Florent Krzakala, and Lenka Zdeborova. On double-descent in uncertainty quantification in overparametrized models. In Francisco Ruiz, Jennifer Dy, and Jan-Willem van de Meent, editors, Proceedings of The 26th International Conference on Artificial Intelligence and Statistics, volume 206 of Proceedings of Machine Learning Research, pages 7089-7125. PMLR, 25-27 Apr 2023a. URL https://proceedings.mlr.press/v206/clarte23a.html.
Lucas Clarté, Bruno Loureiro, Florent Krzakala, and Lenka Zdeborova. Theoretical characterization of uncertainty in high-dimensional linear classification. Machine Learning: Science and Technology, 4(2):025029, jun 2023b. doi: 10.1088/2632-2153/acd749. URL https://dx.doi.org/10.1088/2632-2153/acd749.
George Cybenko. Approximation by superpositions of a sigmoidal function. Mathematics of control, signals and systems, 2(4):303-314, 1989.
A Philip Dawid. The well-calibrated bayesian. Journal of the American Statistical Association, 77 (379):605-610, 1982.
Emir H. Shuford, Arthur Albert, and H. Edward Massengill. Admissible probability measurement procedures. Psychometrika, 31(2):125-145, 1966. doi: 10.1007/BF02289503. URL https://doi.org/10.1007/BF02289503.
Chulhee Yun, Srinadh Bhojanapalli, Ankit Singh Rawat, Sashank J Reddi, and Sanjiv Kumar. Are transformers universal approximators of sequence-to-sequence functions? arXiv preprint arXiv:1912.10077, 2019.
Bianca Zadrozny and Charles Elkan. Obtaining calibrated probability estimates from decision trees and naive bayesian classifiers. In Icml, volume 1, pages 609-616. CiteSeer, 2001.
Bianca Zadrozny and Charles Elkan. Transforming classifier scores into accurate multiclass probability estimates. In Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining, pages 694-699. ACM, 2002.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning (still) requires rethinking generalization. Communications of the ACM, 64(3):107-115, 2021.
Ding-Xuan Zhou. Universality of deep convolutional neural networks. Applied and computational harmonic analysis, 48(2):787-794, 2020.
Calibration without Bayesian Modeling. Our results show that it is formally possible to obtain calibrated classifiers via pure loss minimization, without incorporating Bayesian priors into the learning process. We view this as encouraging for the field of uncertainty quantification in machine learning, since it shows it is possible to obtain accurate uncertainties without incurring the computational and conceptual overheads of Bayesian methods [Abdar et al., 2021, Graves, 2011, MacKay, 1995, Gal, 2016].
Calibration Methods. There have been many methods proposed to achieve accurate models that are also well-calibrated. The classic two-stage approach is to first learn an accurate classifier, and then post-process for calibration (via e.g. isotonic regression or Platt scaling as in Platt [1999], Zadrozny and Elkan [2001, 2002], Niculescu-Mizil and Caruana [2005]). Recent papers on neural networks also suggest optimizing for both calibration and accuracy jointly, by adding a "calibration regularizer" term to the standard accuracy-encouraging objective function [Kumar et al., 2018, Karandikar et al., 2021]. In contrast, we are interested in the setting where loss-minimization happens to yield calibration "out of the box", without re-calibration. There is a growing body of work on principled measures of calibration [Kakade and Foster, 2008, Gopalan et al., 2022b, Błasiok et al., 2023b], which we build on in our paper. Clarté et al. [2023a,b] analyze calibration for empirical risk minimization and highlight the role played by regularization.
Multicalibration. There has recently been great interest in a generalization of calibration known as multicalibration [Hébert-Johnson et al., 2018], see also Kearns et al. [2017], motivated by algorithmic fairness considerations. A series of works explores the connection between multicalibration and minimization of convex losses, through the notion of omniprediction [Gopalan et al., 2022a, 2023a,b, Hu et al., 2022, Globus-Harris et al., 2023]. A tight characterization of multicalibration in terms of squared loss minimization is presented in Gopalan et al. [2023b], which involves a novel notion of loss minimization called swap agnostic learning. Another related result is the recent work of Błasiok et al. [2023a] which shows that minimizing squared loss for sufficiently deep neural nets yields multicalibration with regard to smaller neural nets of a fixed complexity. This result is similar in flavor since it argues that a lack of multicalibration points to a good way to reduce the loss, and relies on the expressive power of DNNs to do this improvement. But it goes only in one direction. Our results apply to any proper loss, and give a tight characterization of calibration error. We generalize our results to multicalibration in Appendix F.
B Miscalibration Example from Logistic Regression
Figure 2 gives an example of a 1-dimensional distribution for which logistic regression (on the population distribution itself) is severely mis-calibrated. This occurs despite logistic regression optimizing a proper loss. Note that the constant predictor  $f(x) = 0.5$  is perfectly calibrated in this example, but logistic regression encourages the solution to deviate from it in order to minimize the proper cross-entropy loss. Similar examples of logistic regression being miscalibrated are found in Kull et al. [2017].
C Post-processing Gap for Cross Entropy Loss
Here we show that directly replacing the squared loss in the definition of post-processing gap (Definition 2.2) with the cross-entropy loss does not give us a similar result as Theorem 2.4. Specifically, we give an example where the smooth calibration error of a predictor is small but the post-processing gap w.r.t. the cross entropy loss can be arbitrarily large.
We choose  $\mathcal{X} = \{x_0, x_1\}$  and let  $\mathcal{D}$  be the distribution over  $\mathcal{X} \times \{0, 1\}$  such that the marginal distribution over  $\mathcal{X}$  is uniform,  $\mathbb{E}_{(x,y) \sim \mathcal{D}}[y|x = x_0] = 0.1$  and  $\mathbb{E}_{(x,y) \sim \mathcal{D}}[y|x = x_1] = 0.9$ . Consider predictor  $f: \mathcal{X} \to [0,1]$  such that  $f(x_0) = \varepsilon$  and  $f(x_1) = 1 - \varepsilon$  for some  $\varepsilon \in (0,0.1)$ . As  $\varepsilon \to 0$ , it is easy to verify that  $\operatorname{smCE}_{\mathcal{D}}(f) \to 0.05$ . However, the post-processing gap w.r.t. the cross-entropy loss tends to  $+\infty$  because  $\ell(1,\varepsilon) = \ell(0,1 - \varepsilon) \to +\infty$  as  $\varepsilon \to 0$  for the cross-entropy loss  $\ell$ .
Figure 2: Logistic Regression can be Miscalibrated. We illustrate a 1-dimensional distribution on which logistic regression is severely miscalibrated: consider two overlapping uniform distributions, one for each class. Specifically, with probability  $1/2$  we have  $y = 0$  and  $x \sim \mathrm{Unif}([-3,1])$ , and with the remaining probability  $1/2$ , we have  $y = 1$  and  $x \sim \mathrm{Unif}([-1,3])$ . This distribution is shown on the left, as well as the optimal logistic regressor  $f$  on this distribution. In this setting, the logistic regressor is miscalibrated (as shown at right), despite logistic regression optimizing a proper loss. Note that the constant predictor  $f(x) = 0.5$  is perfectly calibrated in this example, but logistic regression encourages the solution to deviate from it in order to minimize the proper cross-entropy loss. Similar examples of logistic regression being miscalibrated are also found in Kull et al. [2017].
D Tight Examples
We give examples showing that the constants in Theorem 2.4 and in Corollary 2.7 are all optimal. Specifically Lemma D.1 shows that the constants in the upper bound of  $\mathsf{pGap}$  in Theorem 2.4 are optimal, whereas Lemma D.2 shows that the constants in the lower bound of  $\mathsf{pGap}$  are also optimal. Lemma D.3 shows that the constants in the upper bound of  $\mathsf{pGap}^{(\psi ,1 / 4)}$  in (3) are optimal. Lemma D.4 simultaneously shows that the constants in the lower bound of  $\mathsf{pGap}^{(\psi ,1 / 4)}$  in (3) and the constants in (4) are optimal.
Lemma D.1. Let  $\mathcal{X} = \{x_0, x_1\}$  be a set of two individuals. For every  $\varepsilon \in (0, 1/4)$ , there exists a distribution  $\mathcal{D}$  over  $\mathcal{X} \times \{0, 1\}$  and a predictor  $f: \mathcal{X} \to [0, 1]$  such that
$$
\mathsf {p G a p} _ {\mathcal {D}} (f) = \varepsilon - 3 \varepsilon^ {2}, \quad \text {w h e r e a s} \quad \mathsf {s m C E} _ {\mathcal {D}} (f) = \varepsilon / 2 - \varepsilon^ {2}.
$$
Proof. We simply choose  $\mathcal{D}$  to be the uniform distribution over  $\{(x_0,0),(x_1,1)\}$ , and we choose  $f(x_0) = 1 / 2 - \varepsilon$  and  $f(x_{1}) = 1 / 2 + \varepsilon$ . For any 1-Lipschitz function  $\eta :[0,1]\to [-1,1]$ , we have
$$
\begin{array}{l} \left| \mathbb {E} _ {(x, y) \sim \mathcal {D}} [ (y - f (x)) \eta (f (x)) ] \right| = \left| - (1 / 2) (1 / 2 - \varepsilon) \eta (f (x _ {0})) + (1 / 2) (1 / 2 - \varepsilon) \eta (f (x _ {1})) \right| \\ = (1 / 2) (1 / 2 - \varepsilon) | \eta (1 / 2 - \varepsilon) - \eta (1 / 2 + \varepsilon) |. \\ \end{array}
$$
Clearly, the supremum of the above quantity is  $\varepsilon / 2 - \varepsilon^2$  which is achieved when  $|\eta(1/2 - \varepsilon) - \eta(1/2 + \varepsilon)| = 2\varepsilon$ . Therefore,
$$
\operatorname {s m} \mathsf {C E} _ {\mathcal {D}} (f) = \varepsilon / 2 - \varepsilon^ {2}.
$$
For any predictor  $f^{\prime}:\mathcal{X}\to [0,1]$  and the squared loss  $\ell$
$$
\begin{array}{l} \mathbb {E} _ {(x, y) \sim \mathcal {D}} [ \ell (y, f ^ {\prime} (x)) ] = (1 / 2) (0 - f ^ {\prime} (x _ {0})) ^ {2} + (1 / 2) (1 - f ^ {\prime} (x _ {1})) ^ {2} \\ = f ^ {\prime} (x _ {0}) ^ {2} / 2 + f ^ {\prime} (x _ {1}) ^ {2} / 2 - f ^ {\prime} (x _ {1}) + 1 / 2 \\ = \frac {1}{4} (f ^ {\prime} (x _ {0}) + f ^ {\prime} (x _ {1}) - 1) ^ {2} + \frac {1}{4} (f ^ {\prime} (x _ {1}) - f ^ {\prime} (x _ {0}) - 1) ^ {2} \\ \end{array}
$$
When  $f'(x) = f(x) + \eta(f(x))$  for a 1-Lipschitz function  $\eta : [0,1] \to [-1,1]$ , the infimum of the above quantity is achieved when  $f'(x_0) + f'(x_1) - 1 = 0$  and  $f'(x_1) - f'(x_0) = 4\varepsilon$ , and the infimum is  $(1/4)(1 - 4\varepsilon)^2$ . If we choose  $f' = f$  instead, we get  $\mathbb{E}[\ell(y, f(x))] = (1/4)(1 - 2\varepsilon)^2$ . Therefore,
$$
\mathsf {p G a p} _ {\mathcal {D}} (f) = (1 / 4) (1 - 2 \varepsilon) ^ {2} - (1 / 4) (1 - 4 \varepsilon) ^ {2} = \varepsilon - 3 \varepsilon^ {2}.
$$
Lemma D.2. Let  $\mathcal{X} = \{x_0\}$  be a set consisting of only a single individual  $x_0$ . Let  $\mathcal{D}$  be the uniform distribution over  $\mathcal{X} \times \{0,1\}$  and for  $\varepsilon \in (0,1/2)$ , let  $f: \mathcal{X} \to [0,1]$  be the predictor such that  $f(x_0) = 1/2 + \varepsilon$ . Then
$$
\mathsf {p G a p} _ {\mathcal {D}} (f) = \varepsilon^ {2}, \quad \text {w h e r e a s} \quad \mathsf {s m C E} _ {\mathcal {D}} (f) = \varepsilon .
$$
Proof. For any 1-Lipschitz function  $\eta :[0,1]\to [-1,1]$
$$
| \mathbb {E} _ {(x, y) \sim \mathcal {D}} [ (y - f (x)) \eta (f (x)) ] | = \varepsilon | \eta (1 / 2 + \varepsilon) |.
$$
The supremum of the quantity above over  $\eta$  is clearly  $\varepsilon$ , implying that  $\mathsf{smCE}_{\mathcal{D}}(f) = \varepsilon$ .
$$
\begin{array}{l} \mathbb {E} _ {(x, y) \sim \mathcal {D}} [ \ell (y, f ^ {\prime} (x)) ] = (1 / 2) (0 - f ^ {\prime} (x _ {0})) ^ {2} + (1 / 2) (1 - f ^ {\prime} (x _ {0})) ^ {2} \\ = f ^ {\prime} (x _ {0}) ^ {2} - f ^ {\prime} (x _ {0}) + 1 / 2 \\ = 1 / 4 + \left(f ^ {\prime} \left(x _ {0}\right) - 1 / 2\right) ^ {2}. \\ \end{array}
$$
The infimum of the above quantity over a function  $f': \mathcal{X} \to [0,1]$  satisfying  $f'(x) = f(x) + \eta(f(x))$  for some 1-Lipschitz  $\eta: [0,1] \to [-1,1]$  is clearly  $1/4$ . If we choose  $f' = f$  instead, we get  $\mathbb{E}_{(x,y) \sim \mathcal{D}}[\ell(y, f(x))] = 1/4 + \varepsilon^2$ . Therefore,
$$
\mathsf {p G a p} _ {\mathcal {D}} (f) = (1 / 4 + \varepsilon^ {2}) - 1 / 4 = \varepsilon^ {2}.
$$
Lemma D.3. Define  $\mathsf{pGap}^{(\psi, 1/4)}$  and  $\mathsf{smCE}^{(\psi, 1/4)}$  as in Definition 2.5 and Definition 2.6 for the cross-entropy loss. Let  $\mathcal{X} = \{x_0, x_1\}$  be a set of two individuals. For every  $\varepsilon \in (0, 1/4)$ , there exists a distribution  $\mathcal{D}$  over  $\mathcal{X} \times \{0, 1\}$  and a function  $g: \mathcal{X} \to \mathbb{R}$  such that
$$
\mathsf {p G a p} _ {\mathcal {D}} ^ {(\psi , 1 / 4)} (g) \geq 2 \varepsilon + O (\varepsilon^ {2}), \quad \text {w h e r e a s} \quad \mathsf {s m C E} _ {\mathcal {D}} ^ {(\psi , 1 / 4)} (g) \leq \varepsilon / 2 + O (\varepsilon^ {2}).
$$
Proof. We simply choose  $\mathcal{D}$  to be the uniform distribution over  $\{(x_0,0),(x_1,1)\}$ , and we choose  $g(x_0) = -4\varepsilon$  and  $g(x_{1}) = 4\varepsilon$ . Let  $\sigma$  denote the sigmoid transformation given by  $\sigma (t) = e^{t} / (1 + e^{t})$ . The Taylor expansion of the sigmoid transformation  $\sigma$  around  $t = 0$  is  $\sigma (t) = 1 / 2 + O(t)$ . Thus the predictor  $f$  we get from applying  $\sigma$  to  $g$  satisfies  $f(x_{1}) = 1 - f(x_{0}) = 1 / 2 + O(\varepsilon)$ . For any  $1 / 4$ -Lipschitz function  $\eta :\mathbb{R}\to [-1,1]$ , we have
$$
\begin{array}{l} \mathbb {E} _ {(x, y) \sim \mathcal {D}} [ (y - f (x)) \eta (g (x)) ] = (1 / 2) (0 - f (x _ {0})) \eta (g (x _ {0})) + (1 / 2) (1 - f (x _ {1})) \eta (g (x _ {1})) \\ = - (1 / 2) \left(1 - f \left(x _ {1}\right)\right) \eta \left(g \left(x _ {0}\right)\right) + (1 / 2) \left(1 - f \left(x _ {1}\right)\right) \eta \left(g \left(x _ {1}\right)\right) \\ = (1 / 2) \left(1 - f \left(x _ {1}\right)\right) \left(\eta \left(g \left(x _ {1}\right)\right) - \eta \left(g \left(x _ {0}\right)\right)\right) \\ \leq (1 / 2) (1 - f (x _ {1})) | g (x _ {0}) - g (x _ {1}) | / 4 \\ \leq (1 / 2) (1 / 2 + O (\varepsilon)) \cdot (8 \varepsilon) / 4 \\ = \varepsilon / 2 + O (\varepsilon^ {2}). \\ \end{array}
$$
This implies that  $\mathsf{smCE}_{\mathcal{D}}^{(\psi, 1/4)}(g) \leq \varepsilon/2 + O(\varepsilon^2)$ .
For any function  $g^{\prime}:\mathcal{X}\to \mathbb{R}$  and the logistic loss  $\ell^{(\psi)}$
$$
\begin{array}{l} \mathbb {E} _ {(x, y) \sim \mathcal {D}} \left[ \ell^ {(\psi)} \left(y, g ^ {\prime} (x)\right) \right] = (1 / 2) \ln \left(1 + e ^ {g ^ {\prime} \left(x _ {0}\right)}\right) + (1 / 2) \ln \left(1 + e ^ {g ^ {\prime} \left(x _ {1}\right)}\right) - (1 / 2) g ^ {\prime} \left(x _ {1}\right) \\ = (1 / 2) \ln \left(1 + e ^ {g ^ {\prime} \left(x _ {0}\right)}\right) + (1 / 2) \ln \left(1 + e ^ {- g ^ {\prime} \left(x _ {1}\right)}\right). \\ \end{array}
$$
We use the Taylor expansion  $\ln (1 + e^{t}) = \ln 2 + t / 2 + O(t^2)$  to get
$$
\mathbb {E} _ {(x, y) \sim \mathcal {D}} \left[ \ell^ {(\psi)} \left(y, g ^ {\prime} (x)\right) \right] = \ln 2 + (1 / 4) \left(g ^ {\prime} \left(x _ {0}\right) - g ^ {\prime} \left(x _ {1}\right)\right) + O \left(g ^ {\prime} \left(x _ {0}\right) ^ {2} + g ^ {\prime} \left(x _ {1}\right) ^ {2}\right). \tag {9}
$$
Consider the case when  $g'(x_0) = -8\varepsilon$  and  $g'(x_1) = 8\varepsilon$ . Clearly,  $g'$  can be written as  $g'(x) = g(x) + \eta(g(x))$  for a 1-Lipschitz function  $\eta: \mathbb{R} \to [-4, 4]$ . By (9), the expected loss achieved by  $g'$  is
$$
\mathbb {E} _ {(x, y) \sim \mathcal {D}} \left[ \ell^ {(\psi)} (y, g ^ {\prime} (x)) \right] = \ln 2 - 4 \varepsilon + O (\varepsilon^ {2}).
$$
If  $g' = g$  instead, the expected loss is  $\ln 2 - 2\varepsilon + O(\varepsilon^2)$ . Taking the difference between the two, we get
$$
\mathsf {p G a p} _ {\mathcal {D}} ^ {(\psi , 1 / 4)} (g) \geq 2 \varepsilon + O (\varepsilon^ {2}).
$$
Lemma D.4. Define  $\mathsf{pGap}^{(\psi,1/4)}$  as in Definition 2.5 for the cross-entropy loss. Let  $\mathcal{X} = \{x_0\}$  be a set consisting of only a single individual  $x_0$ . Let  $\mathcal{D}$  be the uniform distribution over  $\mathcal{X} \times \{0,1\}$  and for  $\varepsilon \in (0,1/2)$ , let  $g: \mathcal{X} \to \mathbb{R}$  be the function such that  $g(x_0) = 4\varepsilon$ . Then
$$
\mathsf {p G a p} _ {\mathcal {D}} ^ {(\psi , 1 / 4)} (g) = 2 \varepsilon^ {2} + O (\varepsilon^ {4}), \quad \text {w h e r e a s} \quad \mathsf {s m C E} _ {\mathcal {D}} (f) = \varepsilon + O (\varepsilon^ {3}),
$$
where  $f: \mathcal{X} \to (0,1)$  is given by  $f(x) = \sigma(g(x))$  for the sigmoid transformation  $\sigma(t) = e^t / (1 + e^t)$ .
Proof. The Taylor expansion of  $\sigma$  around  $t = 0$  is  $\sigma(t) = 1/2 + t/4 + O(t^3)$ . Thus  $f(x_0) = \sigma(g(x_0)) = 1/2 + \varepsilon + O(\varepsilon^3)$ . For any 1-Lipschitz function  $\eta: [0,1] \to [-1,1]$ ,
$$
| \mathbb {E} _ {(x, y) \sim \mathcal {D}} [ (y - f (x) \eta (f (x)) ] | = | (1 / 2 - f (x _ {0})) \eta (f (x _ {0})) | = (\varepsilon + O (\varepsilon^ {3})) | \eta (f (x _ {0})) |.
$$
The supremum of the quantity above over  $\eta$  is clearly  $\varepsilon + O(\varepsilon^3)$ , implying that  $\mathsf{smCE}_{\mathcal{D}}(f) = \varepsilon + O(\varepsilon^3)$ .
For any function  $g^{\prime}:\mathcal{X}\to [0,1]$  and the logistic loss  $\ell^{(\psi)}$
$$
\begin{array}{l} \mathbb {E} _ {(x, y) \sim \mathcal {D}} \left[ \ell^ {(\psi)} \left(y, g ^ {\prime} (x)\right) \right] = \ln \left(1 + e ^ {g ^ {\prime} \left(x _ {0}\right)}\right) - (1 / 2) g ^ {\prime} \left(x _ {0}\right) \tag {10} \\ = \ln \left(e ^ {g ^ {\prime} \left(x _ {0}\right) / 2} + e ^ {- g ^ {\prime} \left(x _ {0}\right) / 2}\right). \\ \end{array}
$$
The infimum of the above quantity over a function  $g^{\prime}:\mathcal{X}\to \mathbb{R}$  satisfying  $g^{\prime}(x) = g(x) + \eta (g(x))$  for some 1-Lipschitz  $\eta :\mathbb{R}\rightarrow [-4,4]$  is  $\ln 2$  achieved when  $g^{\prime}(x_0) = 0$ . If we choose  $g^{\prime} = g$  instead, we get  $\mathbb{E}_{(x,y)\sim \mathcal{D}}[\ell^{(\psi)}(y,g(x))] = \ln 2 + 2\varepsilon^2 +O(\varepsilon^4)$  by plugging the Taylor expansion  $\ln (1 + e^{t}) = \ln 2 + t / 2 + t^{2} / 8 + O(t^{4})$  into (10). Therefore,
$$
\mathsf {p G a p} _ {\mathcal {D}} ^ {(\psi , 1 / 4)} (g) = 2 \varepsilon^ {2} + O (\varepsilon^ {4}).
$$
E Proper Loss and Convex Functions
It is known that there is a correspondence between proper loss functions and convex functions [Shuford et al., 1966, Savage, 1971, Schervish, 1989, Buja et al., 2005]. Here we demonstrate this correspondence, which is important for extending the connection between smooth calibration and post-processing gap (Theorem 2.4) to general proper loss functions in Section 4.
Definition E.1 (Sub-gradient). For a function  $\varphi : V \to \mathbb{R}$  defined on  $V \subseteq \mathbb{R}$ , we say  $t \in \mathbb{R}$  is a sub-gradient of  $\varphi$  at  $v \in V$  if
$$
\varphi (v ^ {\prime}) \geq \varphi (v) + (v ^ {\prime} - v) t \quad f o r e v e r y v ^ {\prime} \in V,
$$
$$
v t - \varphi (v) = \max  _ {v ^ {\prime} \in V} \left(v ^ {\prime} t - \varphi \left(v ^ {\prime}\right)\right). \tag {11}
$$
Definition E.2 (Fenchel-Young divergence). For a pair of functions  $\varphi : V \to \mathbb{R}$  and  $\psi : T \to \mathbb{R}$  defined on  $V, T \subseteq \mathbb{R}$ , we define the Fenchel-Young divergence  $D_{\varphi, \psi} : V \times T \to \mathbb{R}$  such that
$$
D _ {\varphi , \psi} (v, t) = \varphi (v) + \psi (t) - v t \quad f o r e v e r y v \in V a n d t \in T.
$$
The following claim follows immediately from the two definitions above.
Claim E.3. Let  $\varphi : V \to \mathbb{R}$  and  $\psi : T \to \mathbb{R}$  be functions defined on  $V, T \subseteq \mathbb{R}$ . Then  $t \in T$  is a sub-gradient of  $\varphi$  at  $v \in V$  if and only if  $D_{\varphi, \psi}(v, t) = \min_{v' \in V} D_{\varphi, \psi}(v', t)$ . Similarly,  $v$  is a sub-gradient of  $\psi$  at  $t$  if and only if  $D_{\varphi, \psi}(v, t) = \min_{t' \in T} D_{\varphi, \psi}(v, t')$ . In particular, assuming  $D_{\varphi, \psi}(v', t') \geq 0$  for every  $v' \in V$  and  $t' \in T$  whereas  $D_{\varphi, \psi}(v, t) = 0$  for some  $v \in V$  and  $t \in T$ , then  $t$  is a sub-gradient of  $\varphi$  at  $v$ , and  $v$  is a sub-gradient of  $\psi$  at  $t$ .
Lemma E.4 (Convex functions from proper loss). Let  $V \subseteq [0,1]$  be a non-empty interval. Let  $\ell : \{0,1\} \times V \to \mathbb{R}$  be a proper loss function. For every  $v \in V$ , define dual  $(v) := \ell(0,v) - \ell(1,v)$ . There exist convex functions  $\varphi : V \to \mathbb{R}$  and  $\psi : \mathbb{R} \to \mathbb{R}$  such that
$$
\ell (y, v) = \psi (\operatorname {d u a l} (v)) - y \operatorname {d u a l} (v) \quad f o r e v e r y y \in \{0, 1 \} a n d v \in V, \tag {12}
$$
$$
D _ {\varphi , \psi} (v, t) \geq 0 \quad \text {f o r e v e r y} v \in V \text {a n d} t \in \mathbb {R}, \tag {13}
$$
$$
D _ {\varphi , \psi} (v, \operatorname {d u a l} (v)) = 0 \quad f o r e v e r y v \in V, \tag {14}
$$
$$
0 \leq \frac {\psi \left(t _ {2}\right) - \psi \left(t _ {1}\right)}{t _ {2} - t _ {1}} \leq 1 \quad f o r e v e r y d i s t i n c t t _ {1}, t _ {2} \in \mathbb {R}. \tag {15}
$$
Before we prove the lemma, we remark that (12) and (14) together imply the following:
$$
\ell (y, v) = - \varphi (v) + (v - y) \text {d u a l} (v) \quad \text {f o r e v e r y} y \in \{0, 1 \} \text {a n d} v \in V.
$$
Proof. Any loss function  $\ell : \{0,1\} \times V \to \mathbb{R}$ , proper or not, can be written as  $\ell(y,v) = \ell(0,v) - y \text{ dual}(v)$ . When  $\ell$  is proper, it is easy to see that for  $v, v' \in V$  with  $\text{dual}(v) = \text{dual}(v')$ , it holds that  $\ell(0,v) = \ell(0,v')$ ; otherwise,  $\ell(y,v)$  is always smaller or always larger than  $\ell(y,v')$  for every  $y \in \{0,1\}$ , making it impossible for  $\ell$  to be proper. Therefore, for proper  $\ell$ , there exists  $\psi_0: T \to \mathbb{R}$  such that  $\ell(0,v) = \psi_0(\text{dual}(v))$  for every  $v \in V$ , where  $T := \{\text{dual}(v): v \in V\}$ . Therefore,