SlowGuess/NeurIPS_2023_OCR huggingface dataset:
[72] Rothfuss, J., Ferreira, F., Walther, S., and Ulrich, M. (2019). Conditional density estimation with neural networks: Best practices and benchmarks. arXiv preprint arXiv:1903.00954.
[73] Schmitt, M., Bürkner, P.-C., Köthe, U., and Radev, S. T. (2021). Detecting model misspecification in amortized bayesian inference with neural networks. arXiv e-prints.
[74] Schmon, S. M., Cannon, P. W., and Knoblauch, J. (2020). Generalized posteriors in approximate bayesian computation. arXiv preprint arXiv:2011.08644.
[75] Sharrock, L., Simons, J., Liu, S., and Beaumont, M. (2022). Sequential neural score estimation: Likelihood-free inference with conditional score based diffusion models. arXiv preprint arXiv:2210.04872.
[76] Sisson, S. A. (2018). Handbook of Approximate Bayesian Computation. Chapman and Hall/CRC.
[77] Thomas, O., Dutta, R., Corander, J., Kaski, S., and Gutmann, M. U. (2022). Likelihood-free inference by ratio estimation. Bayesian Analysis, 17(1).
[78] Turin, G. L., Clapp, F. D., Johnston, T. L., Fine, S. B., and Lavry, D. (1972). A statistical model of urban multipath propagation. IEEE Transactions on Vehicular Technology, 21(1):1-9.
[79] van Opheusden, B., Acerbi, L., and Ma, W. J. (2020). Unbiased and efficient log-likelihood estimation with inverse binomial sampling. PLoS Computational Biology, 16(12):e1008483.
[80] Vasist, M., Rozet, F., Absil, O., Mollière, P., Nasedkin, E., and Louppe, G. (2023). Neural posterior estimation for exoplanetary atmospheric retrieval. *Astronomy & Astrophysics*, 672:A147.
[81] Ward, D., Cannon, P., Beaumont, M., Fasiolo, M., and Schmon, S. M. (2022). Robust neural posterior estimation and statistical model criticism. In Advances in Neural Information Processing Systems.
[82] Weinstein, E. N. and Miller, J. W. (2023). Bayesian data selection. Journal of Machine Learning Research, 24(23):1-72.
[83] Wiqvist, S., Frellsen, J., and Picchini, U. (2021). Sequential neural posterior and likelihood approximation. arXiv preprint arXiv:2102.06522.
[84] Wiqvist, S., Mattei, P.-A., Picchini, U., and Frellsen, J. (2019). Partially exchangeable networks and architectures for learning summary statistics in approximate Bayesian computation. In International Conference on Machine Learning, volume 97, pages 6798-6807.
[85] Wood, S. N. (2010). Statistical inference for noisy nonlinear ecological dynamic systems. Nature, 466(7310):1102-1104.
[86] Zaheer, M., Kottur, S., Ravanbakhsh, S., Poczos, B., Salakhutdinov, R. R., and Smola, A. J. (2017). Deep sets. Advances in neural information processing systems, 30.
[87] Zbair, M., Qaffou, A., and Hilal, K. (2022). Approximate bayesian estimation of parameters of an agent-based model in epidemiology. In Lecture Notes in Networks and Systems, pages 302-314. Springer International Publishing.
The appendix is organized as follows:
- In Appendix A, we present the results on detecting model misspecification.
- In Appendix B, we provide further implementation details and results for the numerical experiment in Section 4.
- Appendix B.1: Implementation details
- Appendix B.2: Additional posterior plots
- Appendix B.3: Results for  $\mathcal{D}$  being the Euclidean distance
- Appendix B.4: Computational cost analysis
- Appendix B.5: Adversarial training on Gaussian linear model
- Appendix B.6: Experiment with neural likelihood estimator
- In Appendix C, we provide details of the radio propagation experiment of Section 5.
A Detecting misspecification of simulators
Considering that existing SBI methods can yield unreliable results under misspecification and that real-world simulators are probably not able to fully replicate observed data in most cases, detecting whether the simulator is misspecified becomes necessary for generating confidence in the results given by these methods. As misspecification can lead to observed statistics or features falling outside the distribution of training statistics, detecting for it essentially boils down to a class of out-of-distribution detection problems known as novelty detection, where the aim is to detect if the test sample  $\mathbf{s}_{\mathrm{obs}}$  comes from the training distribution induced by  $\{s_i\}_{i=1}^m$ . This two-label classification problem can potentially be solved by adapting any of the numerous novelty detection methods from the literature. We propose the following two simple novelty detection techniques for detecting misspecification:
Distance-based approach. We assign a score to the observed statistic based on the value of the margin upper bound, as introduced in the main text. We use the MMD as the choice of distance  $\mathcal{D}$  and estimate the MMD between the set of simulated statistics  $\{s_i\}_{i = 1}^m$  and the observed statistic  $\mathbf{s}_{\mathrm{obs}}$ . This MMD-based score can be used in a classification method to detect misspecification.
Density-based approach. In this method, the training samples  $\{s_i\}_{i = 1}^m$  are used to fit a generative model  $q$ , and the log-likelihood of the observed statistics under  $q$  are used as the classification score. We use a Gaussian mixture model (GMM) with  $k$  components as  $q$ , having the distribution
$$
q (s) = \sum_ {i = 1} ^ {k} \xi_ {i} \varphi (s | \mu_ {i}, \Sigma_ {i}), \tag {11}
$$
where  $\xi_{i}$ ,  $\mu_{i}$ , and  $\Sigma_{i}$  are the weight, the mean and the covariance matrix associated with the  $i^{\text{th}}$  component, and  $\varphi$  denotes the Gaussian pdf. The score  $\ln q(\mathbf{s}_{\mathrm{obs}})$  can then be used to classify it as either being from in or out of the training distribution.
Experimental set-up. We test the performance of the proposed detection methods on the Ricker model and the OUP with the same contamination model as given in the main text. For each of these simulators, we first train the NPE method on  $m = 1000$  training data points, and fit a GMM with  $k = 2$  components to them. We then generate 1000 test datasets or points, half of them from the well-specified model and the other half from the misspecified model, and compute their score. The area under the receiver operating characteristic (AUROC) is used as the performance metric.
**Baseline.** We construct a baseline for comparing performance of the proposed detection methods. The baseline is based on the insight that under model misspecification, the NPE posterior moves away from the true parameter value (even going outside the prior range). Therefore, we take the root mean squared error (RMSE), defined as  $(1/N\sum_{i=1}^{N}(\theta_i - \theta_{\mathrm{true}})^2)^{\frac{1}{2}}$  where  $\{\theta_i\}_{i=1}^N$  are posterior samples, as the classification score.
Figure 7: Misspecification detection experiment. AUROC of the proposed detection methods (GMM and MMD) versus misspecification level for the Ricker model and the OUP. The RMSE-based baseline is shown in blue.
Results. The AUROC of the classifiers for different levels of misspecification ( $\epsilon$  in the main text) is shown in Fig. 7 for both the models. The proposed GMM-based detection method performs the best, followed by the MMD-based method. The RMSE-based baseline performs the worst at the classification task. We conclude that it is possible to detect model misspecification in the space of summary statistics using simple to use novelty detection methods.
B Additional details and results of the numerical experiments
We implement our NPE-RS models based on publicly available implementations from https://github.com/mackelab/sbi. We use the NPE-C model [41] with Masked Autoregressive Flow (MAF) [60] as the backbone inference network, and adopt the default configuration with 50 hidden units and 5 transforms for MAF. The batch size is set to 50, and we maintain a fixed learning rate of  $5 \times 10^{-4}$ . The implementation for RNPE is sourced directly from the original repository at https://github.com/danielward27/rnpe.
Regarding the summary network in NPE tasks, for the Ricker model, we employ three 1D convolutional layers with 4 hidden channels, and we set the kernel size to 3. For the OUP model, we combine three 1D convolutional layers with one bidirectional LSTM layer. The convolutional layers have 8 hidden channels and a kernel size equal to 3, while the LSTM layer has 2 hidden dimensions. We pass the data separately through the convolutional layers and the LSTM layer and then concatenate the resulting representations to obtain our summary statistics. For the Turin model in Section 5, we utilize five 1D convolutional layers with hidden units set to [8, 16, 32, 64, 8], and the kernel size is set to 3. Across all three summary networks, we employ the mean operation as our aggregator to ensure permutation invariance among realizations.
In ABC tasks, we incorporate autoencoders as our summary network. For the Ricker model, the encoder consists of three 1D convolutional layers with 4 hidden channels, where the kernel size is set to 3. The decoder comprises of three 1D transposed convolutional layers with the same settings as the encoder's convolutional layers, allowing for data reconstruction. For the OUP model, we adopt a similar summary network as the one used for the Ricker model but with a smaller stride.
In NPE tasks, we use 1000 samples for the training data, along with 100 realizations of both observed and simulated data for each value of  $\theta$ . We also use 1000 samples for training the autoencoders. For ABC, we use 4000 samples from the prior and accept  $n_{\delta} = 200$  samples giving a tolerance rate of  $5\%$ . We take  $\rho$  to be Euclidean distance in the rejection ABC and normalize the statistics by the median absolute deviation before computing the distance to account for the difference in their magnitude.
B.2 Additional posterior plots
We now present examples of the remaining posterior plots, apart from the one shown in the main text. The posterior plots for OUP using the NPE-based methods is shown in Figure 8. The observations
are similar to the Ricker model example in the main text: we see that our NPE-RS method yields similar posterior as NPE in the well-specified case, whereas RNPE posteriors are underconfident. When the model is misspecified, NPE posterior goes far from the true parameter value. The NPE-Rs posteriors, however, are still around  $\theta_{\mathrm{true}}$ , demonstrating robustness to misspecification.
(c) Misspecified  $(\epsilon = 20\%)$
Figure 8: Ornstein-Uhlenbeck process. Posteriors obtained from our method (NPE-RS), RNPE, and NPE for different degrees of model misspecification.
Similar behavior is observed in the ABC case for both the Ricker model and OUP in Figure 9 and Figure 10, respectively. The ABC posteriors go outside the prior range under misspecification, while ABC with our robust statistics yields posteriors closer to  $\theta_{\mathrm{true}}$ . In Table 1, we report the sample mean and standard deviations for the results shown in Figure 2 of the main text.
(c) Misspecified  $(\epsilon = 20\%)$
Figure 9: Ricker model. Posteriors obtained from our method (ABC-RS) and ABC for different degrees of model misspecification.
B.3 Results for  $\mathcal{D}$  being the Euclidean distance
We present results for  $\mathcal{D}$  being the Euclidean distance in the well-specified case of the Ricker model in Figure 11(a). As mentioned in Section 3 of the main text, this choice leads to very underconfident posteriors. This is because the Euclidean distance is not a robust distance: it becomes large even if a few points are far from the observed statistic. As a result, using this as the regularization term penalizes most choices of summarizer  $\eta$ , and we learn statistics that are very concentrated around the observed statistic (orange dot). Although a good choice for being robust, Euclidean distance leads to statistics that are not informative about the model parameters, yielding posterior that is similar to the uniform prior. Hence, we used the MMD as the distance in the margin upper bound, which provides a better trade-off between robustness and efficiency (in terms of learning about model parameters).
(c) Misspecified  $(\epsilon = 20\%)$
Figure 10: Ornstein-Uhlenbeck process. Posteriors obtained from our method (ABC-RS) and ABC for different degrees of model misspecification.
Table 1: Performance of the SBI methods in terms of RMSE and MMD for both Ricker and OUP. We report the average (±1 std. deviation) values across 100 runs for varying levels of misspecification.
Ricker	NPE	RMSE (↓)	MMD (↓)
ε = 0%	ε = 10%	ε = 20%	ε = 0%	ε = 10%	ε = 20%
RNPE	2.16 (3.07)	7.86 (1.57)	11.2 (1.70)	0.04 (0.07)	0.74 (0.09)	1.06 (0.17)
NPE-RS (ours)	3.27 (0.35)	5.51 (0.58)	7.14 (1.15)	0.06 (0.05)	0.51 (0.19)	0.79 (0.25)
2.18 (2.66)	2.19 (1.01)	4.66 (4.15)	0.09 (0.14)	0.21 (0.16)	0.42 (0.37)
ABC	1.46 (0.44)	6.95 (0.25)	9.79 (0.96)	0.01 (0.01)	0.85 (0.02)	1.18 (0.04)
ABC-RS (ours)	1.20 (0.51)	3.16 (1.08)	2.99 (1.28)	0.01 (0.02)	0.17 (0.15)	0.18 (0.16)
OUP	NPE	0.79 (0.62)	1.26 (1.18)	2.59 (2.75)	0.01 (0.01)	0.34 (0.15)	0.63 (0.29)
RNPE	0.78 (0.09)	0.87 (0.10)	0.98 (0.15)	0.01 (0.01)	0.22 (0.13)	0.49 (0.26)
NPE-RS (ours)	0.74 (0.70)	0.62 (0.33)	0.63 (0.36)	0.02 (0.05)	0.09 (0.09)	0.21 (0.17)
ABC	0.50 (0.07)	1.20 (0.40)	5.16 (2.39)	0.05 (0.03)	0.88 (0.21)	0.92 (0.23)
ABC-RS (ours)	0.44 (0.06)	0.62 (0.23)	0.88 (0.48)	0.02 (0.02)	0.26 (0.17)	0.50 (0.38)
B.4 Computational cost analysis
We now present a quantitative analysis of the computational cost of training with and without our MMD regularization term. The results, presented in Table 2, are calculated on an Apple M1 Pro CPU. As expected, we observe a higher runtime for our method due to the computational cost of estimating the MMD from 200 samples of simulated data. The total runtime also depends on the number of batchsize  $N_{\mathrm{batch}}$ , hence, as  $N_{\mathrm{batch}}$  increases, the proportion of runtime used for estimating MMD reduces. As a result, we see that for large  $N_{\mathrm{batch}}$ , the increase in the computational cost of our method with robust statistics is not significant.
B.5 Adversarial training on Gaussian linear model
To verify the robustness of our method on higher dimensional parameter space, we run an experiment on the Gaussian linear model, where the data  $\mathbf{x} \in \mathbb{R}^{10}$  is sampled from  $\mathcal{N}(\theta, 0.1 \cdot I_{10})$ . A uniform prior  $\mathcal{U}([-1,1])^{10}$  is placed on the parameters  $\theta \in \mathbb{R}^{10}$ . We take  $\theta_{\mathrm{true}} = [0.5, \dots, 0.5]^{\top}$ ,  $\theta_c = [2, \dots, 2]^{\top}$ . To introduce contamination to the observed data, we employ the same approach as outlined in the main text of our paper. However, there is a slight divergence in our experimental setup. In this example, we employ adversarial training, meaning that the model is trained on observed data with a high degree of misspecification ( $\epsilon = 20\%$ ), while we perform inference on data with a lower misspecification degree ( $\epsilon = 10\%$ ). For the summary network, we utilize the DeepSet [86] architecture. The encoder comprises two linear layers, each with a width of 20 hidden units, paired with the ReLU activation function. The decoder is constructed with a single linear layer of 20 hidden units.
The results are shown in Table 3. NPE-RS outperforms NPE in terms of MMD between the posterior predictive distribution and the observed data, which highlights the effectiveness of our approach in
(a) Posteriors
(b) Summary statistics
Figure 11: Ricker model. Posteriors and summary statistics for  $\mathcal{D}$  being the Euclidean distance.
Table 2: Comparison of computational costs across different models on Ricker model. We report the mean value (standard deviation) derived from 20 updates. We use different batch size  $N_{\mathrm{batch}}$  and generate 100 realizations for each  $\theta$ .
Runtime (seconds)
Nbatch = 50	Nbatch = 100	Nbatch = 200
NPE	0.22 (0.03)	0.46 (0.04)	0.87 (0.03)
NPE-RS (ours)	1.26 (0.05)	1.53 (0.14)	1.92 (0.10)
ABC	0.68 (0.04)	1.41 (0.04)	3.29 (0.27)
ABC-RS (ours)	1.79 (0.04)	2.71 (0.25)	4.25 (0.46)
high-dimensional parameter spaces, even though the observed data was not used in the training of NPE-RS. This points towards the possibility that by employing adversarial training, we might achieve robustness against lower levels of misspecification whilst still being amortized.
Table 3: Performance comparison between NPE and NPE-RS for Gaussian linear model. We use MMD between the posterior predictive distribution and the observed data as our metric. We report the average (±1 std. deviation) values across 100 runs.
λ	NPE	NPE-RS
-	20	50	100
MMD	0.26 (0.02)	0.19 (0.04)	0.18 (0.06)	0.21 (0.08)
B.6 Experiment with neural likelihood estimators
In this section, we explore the performance of our method when paired with Neural Likelihood Estimators (NLE). NLE are a class of methods that leverage neural density estimators to directly estimate likelihood functions, bridging the gap between simulators and statistical models.
For this experiment, we adopt NLE-A as proposed by [61]. The original implementation can be found at https://github.com/mackelab/sbi. Similar to our approach with ABC, we adapt our method to NLE by pre-emptively training an autoencoder with our regularization term to learn the summary statistics. We refer to our adapted method as NLE-RS. The configurations for our summary network and simulator are consistent with those described in Appendix B.1.
Figure 12 presents the posterior plots for the Ricker model using the NLE-based methods. Consistent with our observations in the previous experiments, NLE-RS still demonstrates robustness to misspecification, while the NLE posterior tends to deviate away from the true parameters.
Figure 12: Ricker model. Posteriors obtained from our method (NLE-RS) and NLE. We set  $\epsilon = 10\%$  for this experiment.
C Details of the radio propagation experiment
In this section, we describe the data and the Turin model used in Section 5 of the main text.
Data and model description. Let  $B$  be the frequency bandwidth used to measure radio channel data at  $K$  equidistant points, leading to a frequency separation of  $\Delta f = B / (K - 1)$ . The measured transfer function at  $k$ th point,  $Y_{k}$ , is modelled as
$$
Y _ {k} = H _ {k} + W _ {k}, \quad k = 0, 1, \dots , K - 1,
$$
where  $H_{k}$  is the transfer function at the  $k$ th frequency, and  $W_{k}$  is additive zero-mean complex circular symmetric Gaussian noise with variance  $\sigma_W^2$ . Taking the inverse Fourier transform, the time-domain signal  $y(t)$  can be obtained as
$$
y (t) = \frac {1}{K} \sum_ {k = 0} ^ {K - 1} Y _ {i} \exp (j 2 \pi k \Delta f t).
$$
The Turin model defines the transfer function as  $H_{k} = \sum_{l}\alpha_{l}\exp (-j2\pi \Delta f k\tau_{l})$ , where  $\tau_{l}$  is the time-delay and  $\alpha_{l}$  is the complex gain of the  $l^{\mathrm{th}}$  component. The arrival time of the delays is modeled as one-dimensional homogeneous Poisson point processes, i.e.,  $\tau_{l}\sim \mathrm{PPP}(\mathbb{R}_{+},\nu)$ , with  $\nu >0$ . The gains conditioned on the delays are modeled as iid zero-mean complex Gaussian random variables with conditional variance  $\mathbb{E}[|\alpha_l|^2 |\tau_l] = G_0\exp (-\tau_l / T) / \nu$ . The parameters of the model are  $\theta = [G_0,T,\nu ,\sigma_W^2 ]^\top$ . The prior ranges used for the parameters are given in Table 4.
Table 4: Prior distributions for the parameters of the Turin model.
G0	T	ν	σ2W
Prior	U(10-9, 10-8)	U(10-9, 10-8)	U(107, 5 × 109)	U(10-10, 10-9)
The radio channel data from [44] is collected in a small conference room of dimensions  $3 \times 4 \times 3 \, \mathrm{m}^3$ , using a vector network analyzer. The measurement was performed with a bandwidth of  $B = 4$  GHz, and  $K = 801$ . Denote each complex-valued time-series by  $\tilde{\mathbf{y}} \in \mathbb{R}^{K}$ , and the whole dataset by  $\tilde{\mathbf{y}}_{1:n}$ , where  $n = 100$  realizations. We take the input to the summary network to be  $\mathbf{y}_{1:n} = 10\log_{10}(|\tilde{\mathbf{y}}_{1:n}|^2)$ .
Scatter-plot of learned statistics. In Figure 13 and Figure 14, we show the scatter-plots of the learned statistics using the NPE and our NPE-RS method, respectively. We observe that the observed statistics (shown in orange) is often outside the set of simulated statistics (shown in blue) for the NPE method. Hence, the inference network is forced to generalize outside its training distribution, which leads to poor fit of the model, as shown in Section 5 of the main text. On the other hand, the observed statistic is always inside the set of simulated statistics (or the training distribution) for our method in Figure 14, which leads to robustness against model misspecification.
Figure 13: Pairwise scatter-plots of summary statistics learned using NPE method for the Turin model. Each blue dot corresponds to simulated statistic obtained from a parameter value sampled from the prior. The orange dot represents the observed statistic.
Figure 14: Pairwise scatter-plots of summary statistics learned using our NPE-RS method for the Turin model. Each blue dot corresponds to simulated statistic obtained from a parameter value sampled from the prior. The orange dot represents the observed statistic.