SlowGuess/NeurIPS_2023_OCR huggingface dataset:
which we will minimize with AdamW optimizer. The first term in eq. (12) is the likelihood loss which we use for the NF-based NPE (or its corresponding cross-entropy loss for the NRE) and the second is the uniformity enforcing regularizer. The index  $(\cdot)$  operator gives the index of its argument in the sequence of sorted elements of  $\{\hat{\alpha}_{\mathrm{HPDR}}^{L,IS}(\hat{p},\theta_i,x_i)\mid i = 1,\dots ,N\}$ . When regularizing for conservativeness we accept  $\alpha_{\mathrm{HPDR}}(\hat{p},\theta^{*},x^{*})$ s to first-order stochastically dominate [39] the target uniform distribution, thus the differences in the uniformity regularizer are additionally activated with a rectifier  $\max (\cdot ;0)$  analogously to eq. (8). In appendix A.1 we sketch a proof of how minimizing such regularizer affects the learned model. The two main hyper-parameters of the objective function are the number of samples  $L$  we use to evaluate eq. (11), and the weight of the regularizer  $\lambda$  in eq. (12). In algorithm 1 we provide the pseudo-code to compute the uniformity regularizer.
To allow backpropagation through indicator functions, we use a Straight-Through Estimator [4] with the Hard Tanh [37] backward relaxation. In order to backpropagate through sorting operation we use its differentiable implementation from Blondel et al. [6].
In our experiments<sup>1</sup>, we basically follow the experimental protocol introduced in Hermans et al. [20] for evaluating SBI methods. We focus on two prevailing amortized neural inference methods, i.e. NRE approximating the likelihood-to-evidence ratio and NPE using conditional NF as the underlying model. In the case of NRE, there has been already a regularization method proposed [11] and we include it for comparison with our results, and mark it as BNRE. In the main text of the paper, we focus on the conservativeness objective. For the calibration objective, the results can be found in
<sup>1</sup>The code is available at https://github.com/DMML-Geneva/calibrated-posterior.
Figure 1: Evaluation on six benchmark problems (for each column) in the form of coverage curves estimated on a set of  $10\mathrm{k}$  test instances. If the curve is on (above) the diagonal line, then  $\hat{p} (\theta |x)$  is calibrated (conservative). Expected coverage of HPDRs at 19 evenly spaced levels on the [0.05; 0.95] interval is estimated following eq. (2) by solving an optimization problem to find  $\Theta_{\hat{p} (\theta |x)}^{\mathrm{HPDR}}(1 - \alpha)$  for every test instance. The average of retraining five times from random initialization is shown. Top there rows marked as NRE, BNRE, and CalNRE (ours) show the NRE trained without regularization, with balancing regularizer, and the proposed regularizer with conservativeness objective, respectively. The bottom rows marked as NPE and CalNPE show the NPE trained without regularization and with the proposed regularizer with conservativeness objective, respectively. Best viewed in color.
appendix C where we report positive outcomes only for part of the studied SBI problems. As the proposal distribution for IS, we use the prior  $p(\theta)$ . In the main experiments, we set the weight of the regularizer  $\lambda$  to 5, and the number of samples  $L$  to 16 for all benchmarks. In section 4.3 we provide insights into the sensitivity of the method with respect to its hyper-parameters.
4.1 Principal experiments
In fig. 1, we present a cross-sectional analysis of the results of the proposed method, based on six benchmark SBI problems described in detail in Hermans et al. [20], for each considering eight different simulation budgets. Expected coverage of HPDRs at 19 evenly spaced levels on the  $[0.05; 0.95]$  interval is estimated following eq. (2) by solving an optimization problem to find  $\Theta_{\hat{p}(\theta|x)}^{\mathrm{HPDR}}(1 - \alpha)$  for every test instance. We intentionally use a different ECP estimation method than in the regularizer to avoid accidental overfitting to the evaluation metric. The top three rows show empirical expected coverage curves on the test set obtained by training the same model with three different objectives – NRE: classification loss; BNRE: classification loss with balancing regularizer [11]; CalNRE: classification loss with the proposed regularizer. The results for CalNRE show that the proposed method
Figure 2: Expected value of the approximate log posterior density of the nominal parameters  $\mathbb{E}_{p(\theta ,x)}[\log \hat{p} (\theta |x)]$  of the SBI approaches in fig. 1 estimated over  $10\mathrm{k}$  test instances. The solid line is the median over five random initializations, and the boundaries of the shaded area are defined by the minimum and maximum values. The horizontal black line indicates the performance of the prior distribution. Best viewed in color.
Figure 3: The computational overhead of the proposed method applied to NRE on the SLCP problem. The Y-axis indicates how many times training with the regularizer lengthens relative to training without it. For every number of samples experiments were conducted on both CPU (left chart) and GPU (right chart). Averaged over five runs, error bars show a double standard deviation range. Best viewed in color.
systematically results in conservative posteriors - with SLCP simulation budget 1024 standing out, due to too weak regularization weight  $\lambda$ . Moreover, in fig. 2 it can be seen that CalNRE typically outperforms BNRE in terms of  $\mathbb{E}_{p(\theta ,x)}[\log \hat{p} (\theta |x)]$ , being close to the performance of NRE (which is not conservative), and demonstrating less variance across initializations.
The bottom two rows in fig. 1 show the same type of results for NPE-based models, with CalNPE being trained with the proposed regularizer. For all budgets in the Gravitational Waves benchmark, CalNPE does not meet the expected target, but this can easily be fixed by adjusting the regularization strength  $\lambda$ . In fig. 2 one finds that CalNPE sometimes yields a higher  $\mathbb{E}_{p(\theta ,x)}[\log \hat{p} (\theta |x)]$  than the non-regularized version, which is a counterintuitive result. We hypothesize that by regularizing the model we mitigate the classical problem of over-fitting on the training dataset.
4.2 Computation time
In fig. 3 we show how applying the proposed regularizer affects the training time in the case of NRE for the SLCP benchmark, with the distinction to CPU and GPU. As a reference, we take the training time of the non-regularized model. While on CPU, the training time grows about linearly with  $L$ , on GPU due to its parallel computation property the overhead is around x2, with some variation depending on the simulation budget and the particular run. However, this is only true as long as the GPU memory is not exceeded, while the space complexity is of order  $\mathcal{O}(N\dim(\theta)L)$ , where  $N$  is the batch size. Comparing with fig. 2, we conclude that applying the regularizer can result in a similar  $\mathbb{E}_{p(\theta,x)}[\log \hat{p}(\theta|x)]$  as in the case of increasing the simulation budget (what extends training time) while keeping comparable computational time, and avoiding over-confidence. Moreover, in situations where the acquisition of additional training instances is very expensive or even impossible, the proposed method allows for more effective use of the available data at a cost sub-linear in  $L$ .
4.3 Sensitivity analysis
In fig. 4 we show the sensitivity of the method with respect to its hyper-parameters on the example of NRE applied for the Weinberg benchmark. In fig. 4a, the analysis of regularization strength indicates a tradeoff between the level of conservativeness and  $\mathbb{E}_{p(\theta ,x)}[\log \hat{p} (\theta |x)]$ . Therefore, choosing the
(a) Sensitivity analysis with respect to regularizer's weight  $\lambda$
(b) Sensitivity analysis with respect to the number of samples  $L$  in the regularizer.
Figure 4: Sensitivity analysis study of CalNRE on the Weinberg benchmark with the smallest simulation budget of 1024 training instances with respect to  $\lambda$  in (a) and  $L$  in (b). The yellow color in each chart shows the results of non-regularized NRE on the highest simulation budget of 131072 training instances. Results in the first two charts show the performance of an ensemble model built from five randomly initialized members. The vertical line in the first chart marks the nominal parameter value. Points in the third chart indicate the average AUC over five randomly initialized models, with two times the standard deviation marked with the shaded area. The shaded area and the error bars in the fourth chart span between minimum and maximum results over the five randomly initialized models with the median indicated by point (line). Best viewed in color.
right configuration for practical applications is a multi-objective problem. The lowest possible  $\lambda$  value that yields a positive AUC ( $\lambda = 5$  in the figure) should be used to maximize predictive power as measured by  $\mathbb{E}_{p(\theta, x)}[\log \hat{p}(\theta|x)]$ . From fig. 4b we can read that increasing the number of samples in the regularizer brings the coverage curve closer to the diagonal while increasing  $\mathbb{E}_{p(\theta, x)}[\log \hat{p}(\theta|x)]$ . This suggests that a more accurate estimation of conservativeness errors during training allows better focus on the main optimization objective.
In this work, we introduced a regularizer designed to avoid overconfident posterior distributions, which is an undesirable property in SBI. It is applicable in both the NRE and NPE settings, whenever the underlying model allows for amortized conditional density evaluation. It is derived directly from the evaluation metric based on Expected Coverage Probability, used to assess whether the posterior distribution is calibrated or conservative, depending on the target. We formally show that minimizing the regularizer to 0 error under infinite sampling guarantees the inference model to be calibrated (conservative).
We empirically show that applying the regularizer in parallel to the main optimization objective systematically helps to avoid overconfident inference for the studied benchmark problems. We admit that calculating the additional term adds computation cost at training time, but this can be mitigated with the use of modern hardware accelerators well suited for parallel computation.
The presented method is introduced in the context of simulation-based inference, but it is suitable for application in general conditional generative modeling, where calibration (conservativeness) can for example help mitigate mode collapse [50].
As a future direction, we identify the need to evaluate the proposed method in problems where the parameters' space has a high dimension. Moreover, currently, the method is restricted to neural models acting as density estimators, but not general generative models that only allow for sampling from the
learned distribution. A sampling-based relaxation of the regularizer would extend its applicability to a variety of models including Generative Adversarial Networks [33], score/diffusion-based models [52], and Variational Autoencoders [24].
We acknowledge the financial support of the Swiss National Science Foundation within the MIGRATE project (grant no. 209434). Antoine Wehenkel and Arnaud Delaunoy are recipients of an F.R.S.-FNRS fellowship and acknowledge the financial support of the FNRS (Belgium). Naoya Takeishi was supported by JSPS KAKENHI Grant Number JP20K19869. The computations were performed at the University of Geneva on "Baobab" and "Yggdrasil" HPC clusters.
[1] G. Aad, T. Abajyan, B. Abbott, J. Abdallah, S. A. Khalek, A. A. Abdelalim, R. Aben, B. Abi, M. Abolins, O. AbouZeid, et al. Observation of a new particle in the search for the Standard Model Higgs boson with the ATLAS detector at the LHC. Physics Letters B, 716(1):1-29, 2012.
[2] V. Akbarzadeh, J.-C. Lévesque, C. Gagné, and M. Parizeau. Efficient sensor placement optimization using gradient descent and probabilistic coverage. Sensors, 14(8):15525-15552, 2014.
[3] M. A. Beaumont, J.-M. Cornuet, J.-M. Marin, and C. P. Robert. Adaptive approximate Bayesian computation. Biometrika, 96(4):983-990, 2009.
[4] Y. Bengio, N. Léonard, and A. Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv:1308.3432, 2013.
[5] V. W. Berger and Y. Zhou. Kolmogorov-Smirnov Test: Overview. John Wiley & Sons, Ltd, 2014.
[6] M. Blondel, O. Teboul, Q. Berthet, and J. Djolonga. Fast differentiable sorting and ranking. In International Conference on Machine Learning, pages 950-959. PMLR, 2020.
[7] S. Brooks, A. Gelman, G. Jones, and X.-L. Meng. Handbook of Markov chain Monte Carlo. CRC press, 2011.
[8] A. Cividini, L. Jurina, and G. Gioda. Some aspects of 'characterization' problems in geomechanics. In International Journal of Rock Mechanics and Mining Sciences & Geomechanics Abstracts, volume 18, pages 487-503. Elsevier, 1981.
[9] K. Cranmer, J. Brehmer, and G. Louppe. The frontier of simulation-based inference. Proceedings of the National Academy of Sciences, 117(48):30055-30062, 2020.
[10] F. M. Dekking, C. Kraaikamp, H. P. Lopuhaa, and L. E. Meester. A Modern Introduction to Probability and Statistics: Understanding why and how, volume 488, page 219. Springer, 2005.
[11] A. Delaunoy, J. Hermans, F. Rozet, A. Wehenkel, and G. Louppe. Towards reliable simulation-based inference with balanced neural ratio estimation. In A. H. Oh, A. Agarwal, D. Belgrave, and K. Cho, editors, Advances in Neural Information Processing Systems, 2022.
[12] A. Delaunoy, B. K. Miller, P. Forre, C. Weniger, and G. Louppe. Balancing simulation-based inference for conservative posteriors. arXiv preprint arXiv:2304.10978, 2023.
[13] B. Dey, D. Zhao, J. A. Newman, B. H. Andrews, R. Izbicki, and A. B. Lee. Calibrated predictive distributions via diagnostics for conditional coverage. arXiv preprint arXiv:2205.14568, 2022.
[14] P. J. Diggle and R. J. Gratton. Monte Carlo methods of inference for implicit statistical models. Journal of the Royal Statistical Society. Series B (Methodological), 46(2):193-227, 1984.
[15] C. Durkan, I. Murray, and G. Papamakarios. On contrastive learning for likelihood-free inference. In International Conference on Machine Learning, pages 2771-2781. PMLR, 2020.
[16] T. Geffner, G. Papamakarios, and A. Mnih. Score modeling for simulation-based inference. In NeurIPS 2022 Workshop on Score-Based Methods, 2022.
[17] J. Geweke. Bayesian inference in econometric models using Monte Carlo integration. *Econometrica: Journal of the Econometric Society*, pages 1317-1339, 1989.
[39] J. P. Quirk and R. Saposnik. Admissibility and measurable utility functions. The Review of Economic Studies, 29(2):140-146, 1962.
[40] D. B. Rubin. Bayesianly justifiable and relevant frequency calculations for the applied statistician. The Annals of Statistics, 12(4):1151 - 1172, 1984.
[41] L. Sharrock, J. Simons, S. Liu, and M. Beaumont. Sequential neural score estimation: Likelihood-free inference with conditional score based diffusion models. arXiv preprint arXiv:2210.04872, 2022.
[42] S. A. Sisson, Y. Fan, and M. M. Tanaka. Sequential Monte Carlo without likelihoods. Proceedings of the National Academy of Sciences, 104(6):1760-1765, 2007.
[43] S. A. Sisson, Y. Fan, and M. Beaumont. Handbook of approximate Bayesian computation. CRC Press, 2018.
[44] M. Sunnäker, A. G. Busetto, E. Numminen, J. Corander, M. Foll, and C. Dessimoz. Approximate Bayesian computation. PLOS Computational Biology, 9(1):1-10, 01 2013.
[45] S. Talts, M. Betancourt, D. Simpson, A. Vehtari, and A. Gelman. Validating Bayesian inference algorithms with simulation-based calibration. arXiv preprint arXiv:1804.06788, 2018.
[46] A. Tarantola. Inverse Problem Theory and Methods for Model Parameter Estimation. Society for Industrial and Applied Mathematics, 2005.
[47] J. M. Tomczak. Deep generative modeling. Springer, 2022.
[48] T. Toni and M. P. H. Stumpf. Simulation-based model selection for dynamical systems in systems and population biology. Bioinformatics, 26(1):104-110, 10 2009.
[49] F. Topsøe. On the Glivenko-Cantelli theorem. Zeitschrift für Wahrscheinlichkeitstheorie und verwandte Gebiete, 14(3):239-250, 1970.
[50] L. Vaitl, K. A. Nicoli, S. Nakajima, and P. Kessel. Gradients should stay on path: better estimators of the reverse- and forward KL divergence for normalizing flows. Machine Learning: Science and Technology, 3 (4):045006, oct 2022.
[51] V. Volodina and P. Challenor. The importance of uncertainty quantification in model reproducibility. Philosophical Transactions of the Royal Society A, 379(2197):20200071, 2021.
[52] L. Yang, Z. Zhang, Y. Song, S. Hong, R. Xu, Y. Zhao, W. Zhang, B. Cui, and M.-H. Yang. Diffusion models: A comprehensive survey of methods and applications. ACM Comput. Surv., sep 2023. Just Accepted.
[53] D. Zhao, N. Dalmasso, R. Izbicki, and A. B. Lee. Diagnostics for conditional density models and Bayesian inference algorithms. In Uncertainty in Artificial Intelligence, pages 1830–1840. PMLR, 2021.
Proof of Lemma 1.
$$
\begin{array}{l} 1 - \alpha_ {\mathrm {H P D R}} (\hat {p}, \theta^ {*}, x ^ {*}) = \int_ {\Theta_ {\hat {p} (\theta | x)} ^ {\mathrm {H P D R}} (1 - \alpha_ {\mathrm {H P D R}} (\hat {p}, \theta^ {*}, x ^ {*}))} \hat {p} (\theta | x) d \theta = \\ \int_ {\left\{\theta \mid \hat {p} \left(\theta \mid x ^ {*}\right) \geqslant \hat {p} \left(\theta^ {*} \mid x ^ {*}\right) \right\}} \hat {p} (\theta | x) d \theta = \\ \int \hat {p} (\theta | x) \mathbb {1} [ \hat {p} (\theta | x ^ {*}) \geqslant \hat {p} (\theta^ {*} | x ^ {*}) ] \mathrm {d} \theta , \tag {13} \\ \end{array}
$$
$$
\int \hat {p} (\theta | x) \mathbb {1} [ \hat {p} (\theta | x ^ {*}) \geqslant \hat {p} \left(\theta^ {*} \mid x ^ {*}\right) ] + \int \hat {p} (\theta | x) \mathbb {1} [ \hat {p} (\theta | x ^ {*}) <   \hat {p} \left(\theta^ {*} \mid x ^ {*}\right) ] = 1 \tag {14}
$$
because  $\hat{p} (\theta |x)$  is a probability density function.
Proof of Remark 1. We start by observing that RHS of eq. (5) can be reformulated as  $\mathbb{E}_{p(\theta ,x)}[\mathbb{1}[1 - \alpha_{\mathrm{HPDR}}(\hat{p},\theta ,x)\leqslant 1 - \alpha ]]$  which is estimated using MC in the following way:
$$
\frac {1}{N} \sum_ {i = 1} ^ {N} \mathbb {1} \left[ 1 - \alpha_ {\mathrm {H P D R}} (\hat {p}, \theta_ {i}, x _ {i}) \leqslant 1 - \alpha \right], \tag {15}
$$
and for a calibrated model it is equal to  $1 - \alpha$ . The empirical ECP expressed above, takes the form of the ECDF of  $1 - \alpha_{\mathrm{HPDR}}(\hat{p},\theta_i,x_i)$  at  $1 - \alpha$ . Thus, following Glivenko-Cantelli theorem we conclude that for a calibrated model,  $1 - \alpha_{\mathrm{HPDR}}(\hat{p},\theta_i,x_i)$  has a standard uniform distribution because it is bounded in  $[0,1]$  interval by construction. Thus, the same holds for  $\alpha_{\mathrm{HPDR}}(\hat{p},\theta_i,x_i)$ .
The opposite direction is trivial to show.
A.1 Stochastic dominance of rank statistics over  $\mathcal{U}[0,1]$  as over-confidence aversion
Let us consider the following minimization objective
$$
D _ {N} = \sup  _ {\alpha} | \max  \left(F _ {N} (\alpha) - F (\alpha), 0\right) |, \tag {16}
$$
which leads to first-order stochastic dominance of  $F_{N}(\alpha)$  (ECDF of  $\{\alpha_{\mathrm{HPDR}}(\hat{p},\theta_i,x_i)\}_i^N$ ) over  $F(\alpha)$  (target standard uniform distribution). At the minimum  $D_N = 0$  we know that
$$
\forall_ {\alpha} \alpha \geqslant \sum_ {j = 1} ^ {N} \mathbb {1} \left[ \alpha_ {\mathrm {H P D R}} (\hat {p}, \theta_ {j}, x _ {j}) \leqslant \alpha \right] / N. \tag {17}
$$
But the LHS can be reformulated as ECDF of infinitely many samples generated with the ground-truth distribution. Without the loss of generality, assume the number of samples is equal to  $N \to \infty$  on both sides.
$$
\forall_ {\alpha} \sum_ {j = 1} ^ {N} \mathbb {1} \left[ \alpha_ {\mathrm {H P D R}} \left(p, \theta_ {j}, x _ {j}\right) \leqslant \alpha \right] \geqslant \sum_ {j = 1} ^ {N} \mathbb {1} \left[ \alpha_ {\mathrm {H P D R}} (\hat {p}, \theta_ {j}, x _ {j}) \leqslant \alpha \right]. \tag {18}
$$
With equality between LHS and RHS, this means calibrated  $\hat{p} (\theta |x)$  which is a special case of conservativeness in our formulation. Let us focus only on the inequality. Then, the condition above translates into
$$
\forall_ {\alpha} \exists_ {j} \alpha_ {\mathrm {H P D R}} \left(p, \theta_ {j}, x _ {j}\right) \leqslant \alpha \& \alpha_ {\mathrm {H P D R}} \left(\hat {p}, \theta_ {j}, x _ {j}\right) > \alpha , \tag {19}
$$
$$
\forall_ {\alpha} \exists_ {j} \alpha_ {\mathrm {H P D R}} \left(p, \theta_ {j}, x _ {j}\right) <   \alpha_ {\mathrm {H P D R}} \left(\hat {p}, \theta_ {j}, x _ {j}\right). \tag {20}
$$
Now, insert the definition of  $\alpha_{\mathrm{HPDR}}(\cdot ,\theta_j,x_j)$
$$
\forall_ {\alpha} \exists_ {j} \sum_ {k = 1} ^ {L} \mathbb {1} \left[ \hat {p} \left(\theta_ {(j) k} ^ {*} \mid x _ {j}\right) <   \hat {p} \left(\theta_ {j} \mid x _ {j}\right) \right] <   \sum_ {k = 1} ^ {L} \mathbb {1} \left[ \hat {p} \left(\theta_ {(j) k} \mid x _ {j}\right) <   \hat {p} \left(\theta_ {j} \mid x _ {j}\right) \right], \tag {21}
$$
where  $\theta_{(j)k}^{*}$  are sampled from the ground truth posterior  $p(\theta |x)$ , and  $\theta_{(j)k}$  from the approximate posterior  $\hat{p} (\theta |x)$ . The following two steps show equivalent conditions:
$$
\forall_ {\alpha} \exists_ {j} \exists_ {k} \hat {p} \left(\theta_ {(j) k} ^ {*} \mid x _ {j}\right) \geqslant \hat {p} \left(\theta_ {j} \mid x _ {j}\right) \& \hat {p} \left(\theta_ {(j) k} \mid x _ {j}\right) <   \hat {p} \left(\theta_ {j} \mid x _ {j}\right), \tag {22}
$$
$$
\forall_ {\alpha} \exists_ {j} \exists_ {k} \hat {p} \left(\theta_ {(j) k} ^ {*} \mid x _ {j}\right) > \hat {p} \left(\theta_ {(j) k} \mid x _ {j}\right). \tag {23}
$$
The condition in eq. (23) says that ground-truth posterior generates at least one sample with higher density measured by the approximate posterior  $\hat{p} (\theta |x)$  than that generated by the approximate posterior. Thus, we have shown that a model generating  $\alpha_{\mathrm{HPDR}}(\hat{p},\theta_i,x_i)\mathrm{s}$  stochastically dominating standard uniform distribution has an aversion to over-confidence.
A.2 Convergence to uniform
Let us<sup>2</sup> consider a discrete uniform distribution over  $L + 1$  evenly spaced values on the [0, 1] interval, which has the following CDF:
$$
F _ {L} (x) = \left\{ \begin{array}{l l} 0, & x <   0 \\ \lfloor x L \rfloor / L, & 0 \leqslant x <   1 \\ 1, & x \geqslant 1 \end{array} \right.
$$
The CDF of the standard continuous uniform distribution has the following form:
$$
F (x) = \left\{ \begin{array}{l l} 0, & x <   0 \\ x, & 0 \leqslant x <   1 \\ 1, & x \geqslant 1 \end{array} \right.
$$
For  $0 \leqslant x < 1$  we have
$$
| F (x) - F _ {L} (x) | = | x - \lfloor x L \rfloor / L | = | x L - \lfloor x L \rfloor | / L <   1 / L,
$$
where the inequality is a property of the floor  $\lfloor \cdot \rfloor$  operator.
Taking the limit of  $L\to \infty$  we have
$$
\lim _ {L \to \infty} | F (x) - F _ {L} (x) | = \lim _ {L \to \infty} 1 / L = 0,
$$
what proofs convergence in the distribution of  $F_{L}(x)$  to  $F(x)$ .
A.3 GC theorem inverse direction
We<sup>3</sup> are given a set of iid samples  $\alpha_{1},\ldots ,\alpha_{N}$  from a  $G$  distribution implicitly given by the trained model. With the regularizer optimized to 0 loss, we assume that
$$
D _ {N} = \sup  _ {\alpha} | F _ {N} (\alpha) - F (\alpha) | \stackrel {{a. s.}} {{\rightarrow}} 0.
$$
<sup>2</sup>We follow the derivation from https://stats.stackexchange.com/a/373350 (version: 2018-10-23).
<sup>3</sup>We follow the derivation from https://stats.stackexchange.com/q/485148 (version: 2020-08-29).
Figure 5: AUC scores of the coverage curves presented in fig. 1. Points indicate the average, while the shaded area spans over the two standard deviation range. Best viewed in color.
From the Glivenko-Cantelli theorem, we have that
$$
D _ {N} = \sup  _ {\alpha} | F _ {N} (\alpha) - G (\alpha) | \stackrel {{a. s.}} {{\rightarrow}} 0.
$$
The triangle inequality is satisfied for any norm, so: