SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\sup  _ {\alpha} | F _ {N} (\alpha) - F (\alpha) | + \sup  _ {\alpha} | F _ {N} (\alpha) - G (\alpha) | > \sup  _ {\alpha} | G (\alpha) - F (\alpha) |,
$$
$$
\sup  _ {\alpha} | G (\alpha) - F (\alpha) | \stackrel {a. s.} {\rightarrow} 0,
$$
where the LHS does not depend on  $N$ , and we conclude that  $G = F$ , what was to be shown.
B Additional conservative posterior results
In this section, we supplement the results shown in section 4.1 by presenting the coverage AUC scores in fig. 5. We show explicitly, that applying the proposed method with a conservativeness objective leads to conservative posteriors. For the two non-conservative cases – CalNRE for SLCP, and CalNPE for Gravitational Waves – we expect a positive result when choosing the appropriate value of the regularization weight, as already stated in the main text.
Figure 6: Evaluation on six benchmark problems (for each column) in the form of coverage curves estimated on a set of  $10\mathrm{k}$  test instances. If the curve is on (above) the diagonal line, then  $\hat{p} (\theta |x)$  is calibrated (conservative). Expected coverage of HPDRs at 19 evenly spaced levels on the [0.05; 0.95] interval is estimated following eq. (2) by solving an optimization problem to find  $\Theta_{\hat{p} (\theta |x)}^{\mathrm{HPDR}}(1 - \alpha)$  for every test instance. The average of retraining five times from random initialization is shown. Top two rows marked as NRE, and CalNRE (ours) show the NRE model trained without regularization, and the proposed regularizer with calibration objective, respectively. The bottom rows marked as NPE and CalNPE show the NPE model trained without regularization and with the proposed regularizer with calibration objective, respectively. Best viewed in color.
C Regularization for calibrated posterior
In this section, we present the results of experiments analogous to those in section 4.1, but with the proposed regularizer configured as calibration error (see eq. (7)). Obtaining a calibrated posterior is a task way more difficult than in the case of a conservative one, and this is what we empirically confirm.
In fig. 6 we find a characteristic S-shaped coverage curve emerging for the regularized models. For the low credibility levels (small credible regions) posteriors become overconfident, while with the high levels (big credible regions) they become conservative, by a certain margin. Overall, this results in an approximately 0 AUC as shown in fig. 8. However, it is only by chance that it becomes positive (on average conservative), and not negative (on average overconfident). From fig. 7 we also see that the predictive power of the regularized models measured by  $\mathbb{E}_{p(\theta ,x)}[\log \hat{p} (\theta |x)]$  estimated on the test set degrades compared to the conservativeness setting in fig. 2.
Figure 7: Expected value of the approximate log posterior density of the nominal parameters  $\mathbb{E}_{p(\theta ,x)}[\log \hat{p} (\theta |x)]$  of the models in fig. 6 estimated over 10k test instances. The solid line is the median over five random initializations, and the boundaries of the shaded area are defined by the minimum and maximum values. The horizontal black line indicates the performance of the prior distribution. Best viewed in color.
Figure 8: AUC scores of the coverage curves presented in fig. 6. Points indicate the average, while the shaded area spans over the two standard deviation range. Best viewed in color.
D Intuition about coverage analysis
In this section, we build a toy example to intuitively introduce the concept of Expected Coverage Probability and the over-/under-confidence that results from it. We rely on non-conditional one-dimensional distributions for clarity. In fig. 9 we introduce a black multimodal ground-truth distribution and a red one intentionally under-dispersed, but with the same modes, acting as the approximate distribution. We find the  $1 - \alpha = 0.9$  Highest Density Region for the red distribution, which consists of two segments concentrated around the modes. To mimic the real-world scenario we sample  $N = 1000$  points from the ground-truth distribution and use them to estimate the expected coverage probability, which is trivial in one dimension knowing the segments' boundaries. According to the plan, the coverage analysis showed that the red distribution is overconfident for evidence from the data.
E Implementation details
For the details of the architecture of Neural Ratio Estimation models please refer to Delaunoy et al. [11]. The SELU function [25] is used as an activation function in the feed-forward networks.
For the details of the architecture of Neural Posterior Estimation models please refer to Hermans et al. [20].
All the models with the proposed regularizer are trained for 500 epochs with 128 batch size with AdamW optimizer with a 0.001 learning rate.
Additional remarks:
- We use the following PyTorch implementation of Blondel et al. [6]: https://github.com/teddykoker/torchsort. We use it with the default parameters.
- In order to stabilize the training of regularized NRE (CalNRE) we introduced Gradient Norm Clipping.
- Training of CalNPE for the M/G/1 in the conservativeness setting with the last four simulation budgets was challenging due to extreme values in observations. It is reflected in the decrease of  $\mathbb{E}_{p(\theta ,x)}[\log \hat{p} (\theta |x)]$  in fig. 2 as the simulation budget increases.
- When computing the regularizer the same  $\hat{p}(\cdot | x^{*})$  is repeatedly evaluated, therefore, our implementation evaluates  $x^{*}$ s embedding only once (per regularizer evaluation) and reuses it for all samples.
Figure 9: Plots of two probability density functions of 2-mixture of Gaussians (0.7/0.3 ratio), with the red one  $(\sigma_{1} = 0.7, \sigma_{2} = 0.2)$  being under-dispersed compared to the black one  $(\sigma_{1} = 0.9, \sigma_{2} = 0.4)$ . The blue region marks 0.9 Highest Posterior Density Region of the red distribution. The black points on the x-axis are random samples from the black distribution. Below the graph, the empirical Expected Coverage Probability of the red distribution against  $N = 1000$  samples of the black distribution is computed. Best viewed in color.