SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Other algorithms As a comparison against SBI methods that target the standard Bayesian posterior (but which nevertheless might produce good predictive samples), we also tested approximate Bayesian computation (ABC), neural posterior estimation (NPE, [7]), and neural likelihood estimation (NLE, [4, 26]) on the same tasks. NPE and NLE were trained on the same 10,000 simulations, and 5000 approximate posterior samples were obtained for each  $\mathbf{x}_o$ . We used the amortized (single-round) variants of both as a fair comparison against ACE. Additionally, to rule out the possibility that ACE is simply benefiting from the additional inverse temperature hyperparameter, we implemented a "tempered" version of NLE by scaling the NLE-approximated log-likelihood term (by the same  $\beta$ s) during MCMC sampling. For ABC, we used the 10,000 training samples as a reference set, from which 50 were drawn as posterior samples with probability scaling inversely with the distance between their corresponding simulation and the observation, i.e., ABC with acceptance kernel [27].
4.2 Benchmark results
Overall, we see that for well-specified  $\mathbf{x}_o$  (i.e., observations for which the simulator is well-specified), ACE obtains GBI posterior samples that achieve low average posterior predictive simulation distance across all four tasks, especially at high values of  $\beta$  (Fig. 3, 1st column). In comparison, ABC is worse for the Linear Gaussian task (which has a higher parameter dimensionality than all other tasks), whereas NPE, NLE, and tempered NLE achieve similarly low posterior predictive distances.
On misspecified observations, across all tasks and simulation-budgets (with the exception of Gaussian mixture on 10k simulations) we see that ACE achieves lower or equally low average posterior predictive simulation distance as neural SBI methods, even at moderate values of  $\beta$  (Fig. 3, 2nd column, Figs. A4, A5). This is in line with our intuition that ACE returns a valid and accurate cost even if the simulator is incapable of producing data anywhere near the observation, while Bayesian likelihood and posterior probabilities estimated by NLE and NPE are in these cases nonsensical [28, 29, 30, 31]. Furthermore, simply concentrating the approximate Bayesian posterior via tempering does not lead to more competitive performance than ACE on such misspecified observations, and is sometimes even detrimental (e.g., tempered NLE at high  $\beta$  on Uniform 1D and Linear Gaussian tasks). Therefore, we see that ACE can perform valid inference for a broad range of simulators, obtaining a distribution of posterior samples with predictive simulations close to observations, and is automatically robust against model-misspecification as a result of directly targeting the cost function.
For both well-specified and misspecified observations, ACE-GBI samples achieve posterior predictive distance very close to ground-truth (GT)-GBI samples, at all values of  $\beta$  (Fig. 3, 1st and 2nd column), suggesting that ACE is able to accurately predict the expected distance. Indeed, especially for low to moderate values of  $\beta$ , the ACE-predicted cost closely matches the true cost (Fig. 3, 3rd column, light blue for well-specified  $\mathbf{x}_o$ , Fig. A2 for misspecified). For higher values of  $\beta$ , ACE-predicted cost is still similar to true cost, although the error is, as expected, larger for very large  $\beta$  (Fig. 3, 3rd column, dark blue).
As a result, highly concentrated generalized posteriors are estimated with larger (relative) discrepancies, which is reflected in the classifier 2-sample score (C2ST) between ACE and GT GBI posteriors (Fig. 3, 4th column): ACE posterior samples are indistinguishable from GT samples at low  $\beta$ , even for the 10D Linear Gaussian task, but becomes less accurate with increasing  $\beta$ . Nevertheless, predictive simulation distance dramatically increases with  $\beta$  even when ACE is less accurate, suggesting that sampling to explicitly minimize a cost function which targets parameters with data-similar simulations is a productive goal. Relative performance results across algorithms are qualitatively similar when using a training simulation budget of 200 (Fig. A4) and 1000 (Fig. A5), but ABC required a sufficiently high simulation budget and performed poorly for 1000 training simulations or less.
5 Hodgkin-Huxley inference from Allen Cell Types Database recordings
Finally, we applied ACE to a commonly used scientific simulator and real data: we used a single-compartment Hodgkin-Huxley (HH) simulator from neuroscience and aimed to infer eight parameters of the simulator given electrophysiological recordings from the Allen Cell Types Database [32, 33, 34]. While this simulator can generate a broad range of voltage traces, it is still a crude approximation to real neurons: it models only a subset of ion channels, it ignores the spatial structure of neurons, and it ignores many intracellular mechanisms [35]. It has been demonstrated that parameters of the HH-model given synthetic recordings can be efficiently estimated with standard NPE [10], but estimating parameters given experimental recordings has been challenging [36] and has required ad-hoc changes to the inference procedure (e.g., Bernaerts et al. [37] added noise to the summary statistics, and Gonçalves et al. [10] used a custom multi-round scheme with a particular choice of density estimator). We will demonstrate that ACE can successfully perform simulation-amortized inference given experimental recordings from the Allen Cell Types Database (Fig. 4a).
We trained NPE and ACE given 100K prior-sampled simulations (details in Appendix A4.4). After training, ACE accurately predicts the true cost of parameters given experimental observations (Fig. 4b). We then used slice sampling to draw samples from the GBI posterior for three different values of  $\beta = \{25, 50, 100\}$  and for ten observations from the Allen Cell Types Database. Interestingly, the marginal distributions between NPE and ACE posteriors are very similar, especially for rather low values of  $\beta$  (Fig. 4c, cornerplot in Appendix Fig. A7). The quality of posterior predictive samples, however, strongly differs between NPE and ACE: across the ten observations from the Allen
e
Figure 4: Applicaton of ACE to Allen data. (a) Three observations from the Allen Cell Types Database. (b) True cost (evaluated as Monte-Carlo average over 10 simulations) per  $\theta$  vs ACE-predicted cost. Colors are different observations. (c) Marginals of posterior distributions for NPE (orange) and ACE (shades of blue. Light blue:  $\beta = 25$ , medium blue:  $\beta = 50$ , dark blue:  $\beta = 100$ ). (d) Top: Two GBI predictive samples for each observation. Bottom: Two NPE predictive samples. Additional samples in Appendix A8-A11. (e) Average predictive distance to observation for NPE and ACE with  $\beta = \{25, 50, 100\}$ .
Cell Types database, only  $35.6\%$  of NPE posterior predictions produced more than five spikes (all observations have at least 12 spikes), whereas the ACE posterior predicts closely match the data, even for low values of  $\beta$  (Fig. 4d, samples for all observations and all  $\beta$  values in Figs. A8,A9,A10 and for NPE in Fig. A11.  $66\%$ $(\beta = 25)$ $87\%$ $(\beta = 50)$ $96\%$ $(\beta = 100)$  of samples have more than five spikes). Indeed, across all ten observations, the average posterior predictive distance of ACE was significantly smaller than that of NPE, and for large values of  $\beta$  the distance is even less than half (Fig. 4e). Finally, for rejection-ABC, only the top 35 samples (out of the full training budget of 100K simulations) had a distance that is less than the average posterior predictive distance achieved by ACE.
To investigate these differences between NPE and ACE, we also evaluated NPE posterior predictive performance on synthetic data (prior predictions) and found that it had an average predictive distance of 0.189, which roughly matches the performance of ACE on the experimental observations (0.174 for  $\beta = 50$ ). This suggests that, in line with previous results [37], NPE indeed struggles with experimental observations, for which the simulator is inevitably imperfect. We then trained NPE with 10 times more simulations (1M in total). With this increased simulation budget, NPE performed significantly better than with 100K simulations, but still produced poorer predictive samples than ACE trained with 100K simulations (for  $\beta = \{50, 100\}$ ), although the marginals were similar between NPE (1M) and ACE (100K) (Fig. A6, samples for all observations in Appendix Fig. A12).
Overall, these results demonstrate that ACE can successfully be applied to real-world simulators on which vanilla NPE fails. On the Hodgkin-Huxley simulator, ACE generates samples with improved predictive accuracy despite an order of magnitude fewer simulations and despite the marginal distributions being similar to those of NPE.
We presented ACE, a method to perform distance-aware inference for scientific simulators within the Generalized Bayesian Inference (GBI) framework. Contrary to 'standard' simulation-based inference (SBI), our method does not target the Bayesian posterior, but replaces the likelihood function with a cost function. For real-world simulators, doing so can provide practical advantages over standard Bayesian inference:
First, the likelihood function quantifies the probability that a parameter generates data which exactly matches the data. However, in cases where the model is a rather crude approximation to the real system being studied, scientists might well want to include parameters that can generate data that is sufficiently close (but not necessarily identical) in subsequent analyses. Our method makes this possible, and is advantageous over other GBI-based methods since it is amortized over observations and the inverse temperature  $\beta$ . Second, many simulators are formulated as noise-free models, and it can be hard to define appropriate stochastic extensions (e.g., [38]). In these cases, the likelihood function is ill-defined and, in practice, this setting would require 'standard' SBI methods, whose density estimators are generally built to model continuous distributions, to model discrete jumps in the posterior density. In contrast, our method can systematically and easily deal with noise-free simulators, and in such situations more closely resembles parameter-fitting algorithms. Lastly, standard Bayesian inference is challenging when the model is misspecified, and the performance of neural network-based SBI methods can suffer drastically in this scenario [30].
6.1 Related work
GBI for Approximate Bayesian Computation Several studies have proposed methods that perform GBI on simulators with either an implicit (i.e., simulation-based) likelihood or an unnormalized likelihood. Wilkinson et al. [39] argued that rejection-ABC performs exact inference for a modified model (namely, one that appends an additive uniform error) instead of approximate inference for the original model. Furthermore, ABC with arbitrary probabilistic acceptance kernels can also be interpreted as having different error models, and Schmon et al. [17] integrate this view to introduce generalized posteriors for ABC, allowing the user to replace the hard-threshold kernel (i.e.,  $\epsilon$ -ball of acceptance) with an arbitrary loss function that measures the discrepancy between  $\mathbf{x}$  and  $\mathbf{x}_o$  for MCMC-sampling of the approximate generalized posterior.
Other recent GBI methods require a differentiable simulator [16, 15] or build tractable cost functions that can be sampled with MCMC [14, 18], but this still requires running simulations at inference time (i.e., during MCMC) and does not amortize the cost of simulations and does not reuse already simulated datapoints.
Finally, Bayesian Optimization for Likelihood-free Inference (BOLFI, [40]) and error-guided LFI-MCMC [41] are not cast as generalized Bayesian inference approaches, but are related to ACE. Similarly as in ACE, they train models (for BOLFI, a Gaussian process and, for error-guided LFI-MCMC, a classifier) to estimate the discrepancy between observation and simulation. In BOLFI, the estimator is then used to iteratively select new locations at which to simulate. However, contrary to ACE, neither of these two methods amortizes the cost of simulations over observations.
Misspecification-aware SBI Several other methods have been proposed to overcome the problem of misspecification in SBI: For example, Bernaerts et al. [37] add noise to the summary statistics in the training data, Ward et al. [42] use MCMC to make the misspecified data well-specified, and Kelly et al. [43] introduce auxiliary variables to shift the (misspecified) observation towards being well-specified. All of these methods, however, maintain that the inference result should be an 'as close as possible' version of the posterior distribution. Contrary to that, our method does not aim to obtain the Bayesian posterior distribution (which, for misspecified models, can often be nonsensical or even undefined if the evidence is zero), but is specifically targeted towards parameter regions that are a specified distance from the observation. More broadly, recent advances in uncertainty quantification in deep neural networks, where standard mean-predicting regression networks are supplemented with a uncertainty- or variance-predicting network [44, 45], may serve to further connect loss-minimizing deep learning with (misspecification-aware) SBI.
While our method amortizes the cost of simulations and of training, it still requires another method to sample from the posterior distribution. We used multi-chain slice-sampling [46] for efficiency, but any other MCMC algorithm, as well as variational inference, could also be employed [47, 48]. While sampling incurs an additional cost, this cost is generally small in comparison to potentially expensive simulations.
In addition, our method can perform inference for distance functions which can be written as expectations over the likelihood. As we demonstrated, this applies to many popular and widely used distances. Our method can, however, not be applied to arbitrary distance functions (e.g., the minimum distance between all simulator samples and the observation). While the distances we investigated here are certainly useful to tractioners, they do not necessarily fulfill the criterion of being 'proper' scoring rules [22, 18]. Furthermore, we note that the cost functions considered here by default give rise to unnormalized generalized likelihoods. Therefore, depending on whether the user aims to approximate the generalized posterior given the normalized or unnormalized likelihood, different MCMC schemes should be used in conjunction with ACE (standard MCMC vs. doubly-intractable MCMC, e.g., the Exchange algorithm [49]).
Compared to 'standard' SBI, GBI introduces an additional hyperparameter to the inference procedure, the inverse temperature  $\beta$ . This hyperparameter has to be set by the user and its choice strongly affects inference behaviour: low values of  $\beta$  will include regions of parameter-space whose data do not necessarily match the observation closely, whereas high values of  $\beta$  constrain the parameters to only the best-fitting parameter values. While we provide heuristics for selecting  $\beta$ , we acknowledge the inconvenience of an additional hyperparameter. However, our method is amortized over  $\beta$ , which makes exploration of different  $\beta$  values possible, and which could simplify automated methods for setting  $\beta$ , similar to works where  $\beta$  is taken as the exponent of the likelihood function [50].
Finally, as with any method leveraging deep neural networks, including neural density estimator-based SBI methods (such as NPE), sensitivity to the number of training samples and the dimensionality of the task should always be considered. As we demonstrate above, increasing simulation budget improves the performance of any algorithm, and a reasonable number of training simulations yielded improved performance on a real-world neuroscience application, while the amortization property shifts the cost of simulations up front. In addition, we consider tasks up to 10 dimensions here, as most existing SBI methods have been benchmarked and shown to perform adequately on such tasks [24], though it remains to be seen how ACE can extend to higher dimensional parameter-space and data-space and whether embedding networks will be similarly helpful.
We presented a method that performs generalized Bayesian inference with amortized cost estimation. Our method produces good predictive samples on several benchmark tasks, especially in the case of misspecified observations, and we showed that it allows amortized parameter estimation of Hodgkin-Huxley models given experimental recordings from the Allen Cell Types Database.
RG is supported by the European Union's Horizon 2020 research and innovation program under the Marie Skłodowska-Curie grant agreement No. 101030918 (AutoMIND). MD is supported by the International Max Planck Research School for Intelligent Systems (IMPRS-IS). The authors are funded by the Machine Learning Cluster of Excellence, EXC number 2064/1-390727645. This work was supported by the Tübingen AI Center (Agile Research Funds), the German Federal Ministry of Education and Research (BMBF): Tübingen AI Center, FKZ: 01IS18039A, and the German Research Foundation (DFG): SFB 1233, Robust Vision: Inference Principles and Neural Mechanisms, project number: 276693517.
We would like to thank Jan Boelts, Janne Lappalainen, and Auguste Schulz for feedback on the manuscript, Julius Vetter for feedback and discussion on proper scoring rules, Poornima Ramesh and Mackelab members for extensive discussions throughout the project, as well as Francois-Xavier Briol for suggestions on sampling doubly-intractable posteriors, and the reviewers for their constructive comments on the readability of the manuscript and suggestions for additional analyses.
[56] Alvaro Tejero-Cantero, Jan Boelts, Michael Deistler, Jan-Matthis Lueckmann, Conor Durkan, Pedro J. Gonçalves, David S. Greenberg, and Jakob H. Macke. sbi: A toolkit for simulation-based inference. Journal of Open Source Software, 5(52):2505, 2020.
[57] Conor Durkan, Artur Bekasov, Iain Murray, and George Papamakarios. Neural spline flows. Advances in neural information processing systems, 32, 2019.
[58] Michael Deistler, Pedro J. Goncalves, and Jakob H. Macke. Truncated proposals for scalable and hassle-free simulation-based inference. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022.
A1 Software and Data
We used PyTorch for all neural networks [51] and hydra to track all configurations [52]. Code to reproduce results is available at https://github.com/mackelab/neuralgbi.
A2 Common cost functions as expectations of the likelihood
As described in Sec. 3, our framework includes all cost functions which can be written as expectations over the likelihood:  $\ell (\pmb {\theta};\mathbf{x}_o) = \mathbb{E}_{p(\mathbf{x}|\pmb {\theta})}[d(\mathbf{x},\mathbf{x}_o)]$  . Below, we demonstrate that the average mean-squared error (MSE), the maximum mean discrepancy  $(\mathrm{MMD}^2)$  [23], and the energy score [22] fall into this category.
Average Mean-squared error The average mean-squared error between samples from the likelihood  $p(\mathbf{x}|\boldsymbol{\theta})$  and an observation  $\mathbf{x}_o$  is defined as:
$$
\mathrm {M S E} (p (\mathbf {x} | \boldsymbol {\theta}), \mathbf {x} _ {o}) = \mathbb {E} _ {\mathbf {x} \sim p (\mathbf {x} | \boldsymbol {\theta})} [ | | \mathbf {x} - \mathbf {x} _ {o} | | ^ {2} ]
$$
which is trivially of the form required for our framework.
Maximum mean discrepancy Throughout this paper, we use the maximum mean discrepancy squared (MMD²) [23]. The MMD² between the likelihood  $p(\mathbf{x}|\boldsymbol{\theta})$  and the data distribution  $p(\mathbf{x}_o)$  is defined as:
$$
\begin{array}{l} \mathrm {M M D} ^ {2} (p (\mathbf {x} | \boldsymbol {\theta}), p (\mathbf {x} _ {o})) = \\ \mathbb {E} _ {\mathbf {x}, \mathbf {x} ^ {\prime} \sim p (\mathbf {x} | \boldsymbol {\theta})} [ k (\mathbf {x}, \mathbf {x} ^ {\prime}) ] + \mathbb {E} _ {\mathbf {x} _ {o}, \mathbf {x} _ {o} ^ {\prime} \sim p (\mathbf {x} _ {o})} [ k (\mathbf {x} _ {o}, \mathbf {x} _ {o} ^ {\prime}) ] + \mathbb {E} _ {\mathbf {x} \sim p (\mathbf {x} | \boldsymbol {\theta}), \mathbf {x} _ {o} \sim p (\mathbf {x} _ {o})} [ k (\mathbf {x}, \mathbf {x} _ {o}) ] \\ \end{array}
$$
Assume we are given  $K$  iid samples as observations  $\mathbf{x}_o^{(1,\dots,K)}$ . Thus, the MMD² becomes:
$$
\begin{array}{l} \mathrm {M M D} ^ {2} (p (\mathbf {x} | \boldsymbol {\theta}), p (\mathbf {x} _ {o})) = \\ \mathbb {E} _ {\mathbf {x}, \mathbf {x} ^ {\prime} \sim p (\mathbf {x} | \boldsymbol {\theta})} [ k (\mathbf {x}, \mathbf {x} ^ {\prime}) ] + \frac {1}{K ^ {2}} \sum_ {i = 1} ^ {K} \sum_ {j = 1} ^ {K} k (\mathbf {x} _ {o} ^ {i}, \mathbf {x} _ {o} ^ {j}) + \mathbb {E} _ {\mathbf {x} \sim p (\mathbf {x} | \boldsymbol {\theta})} \frac {1}{K} \sum_ {i} ^ {K} k (\mathbf {x}, \mathbf {x} _ {o} ^ {i}) \\ \end{array}
$$
which can be written as a single expectation over  $p(\mathbf{x}|\boldsymbol{\theta})$
$$
\begin{array}{l} \mathrm {M M D} ^ {2} (p (\mathbf {x} | \boldsymbol {\theta}), p (\mathbf {x} _ {o})) = \\ \mathbb {E} _ {\mathbf {x} \sim p (\mathbf {x} | \boldsymbol {\theta})} \left[ \mathbb {E} _ {\mathbf {x} ^ {\prime} \sim p (\mathbf {x} | \boldsymbol {\theta})} [ k (\mathbf {x}, \mathbf {x} ^ {\prime}) ] + \frac {1}{K ^ {2}} \sum_ {i = 1} ^ {K} \sum_ {j = 1} ^ {K} k (\mathbf {x} _ {o} ^ {i}, \mathbf {x} _ {o} ^ {j}) + \frac {1}{K} \sum_ {i} ^ {K} k (\mathbf {x}, \mathbf {x} _ {o} ^ {i}) \right] \\ \end{array}
$$
and follows the form required for our framework.
Energy score Following Gneiting et al. [22], the (negative, in order to fit our notation) energy score (ES) is defined as
$$
\operatorname {E S} (p (\mathbf {x} | \boldsymbol {\theta}) | \mathbf {x} _ {o}) = - \frac {1}{2} \mathbb {E} _ {\mathbf {x}, \mathbf {x} ^ {\prime} \sim p (\mathbf {x} | \boldsymbol {\theta})} [ | | \mathbf {x} - \mathbf {x} ^ {\prime} | | ^ {\beta} ] + \mathbb {E} _ {\mathbf {x} \sim p (\mathbf {x} | \boldsymbol {\theta})} [ | | \mathbf {x} - \mathbf {x} _ {o} | | ^ {\beta} ].
$$
This can straight-forwardly be written as a single expectation over the likelihood
$$
\operatorname {E S} (p (\mathbf {x} | \boldsymbol {\theta}) | \mathbf {x} _ {o}) = \mathbb {E} _ {\mathbf {x} \sim p (\mathbf {x} | \boldsymbol {\theta})} [ \frac {1}{2} \mathbb {E} _ {\mathbf {x} ^ {\prime} \sim p (\mathbf {x} | \boldsymbol {\theta})} [ | | \mathbf {x} - \mathbf {x} ^ {\prime} | | ^ {\beta} ] - | | \mathbf {x} - \mathbf {x} _ {o} | | ^ {\beta} ]
$$
and, thus, falls within our framework.
A3 Convergence proofs
The proofs closely follow standard proofs that regression converges to the conditional expectation.
A3.1 Proposition 1 and proof
Proposition 1. Let  $p(\theta, \mathbf{x})$  be the joint distribution over parameters and data. Let  $\ell(\theta; \mathbf{x}_o)$  be a cost function that can be written as  $\ell(\theta; \mathbf{x}_o) = \mathbb{E}_{p(\mathbf{x}|\theta)}[d(\mathbf{x}, \mathbf{x}_o)] = \int_{\mathbf{x}} d(\mathbf{x}, \mathbf{x}_o)p(\mathbf{x}|\theta)d\mathbf{x}$  and let  $f_\phi(\cdot)$  be a function parameterized by  $\phi$ . Then, the loss function  $\mathcal{L} = \mathbb{E}_{p(\theta, \mathbf{x})}[(f_\phi(\theta) - d(\mathbf{x}, \mathbf{x}_o))^2]$  is minimized if and only if, for all  $\theta \in \operatorname{supp}(p(\theta))$ ,  $f_\phi(\theta) = \mathbb{E}_{p(\mathbf{x}|\theta)}[d(\mathbf{x}, \mathbf{x}_o)]$ .
Proof. We aim to prove that