A Crisis In Simulation-Based Inference? Beware, Your Posterior Approximations Can Be Unfaithful
Authors: Joeri Hermans
Arnaud Delaunoy
François Rozet
Antoine Wehenkel
Volodimir Begy
Gilles Louppe
Venue: Accepted by TMLR ()

Abstract: We present extensive empirical evidence showing that current Bayesian simulation-based inference algorithms can produce computationally unfaithful posterior approximations. Our results show that all benchmarked algorithms -- (S)NPE, (S)NRE, SNL and variants of ABC -- can yield overconfident posterior approximations, which makes them unreliable for scientific use cases and falsificationist inquiry. Failing to address this issue may reduce the range of applicability of simulation-based inference. For this reason, we argue that research efforts should be made towards theoretical and methodological developments of conservative approximate inference algorithms and present research directions towards this objective. In this regard, we show empirical evidence that ensembling posterior surrogates provides more reliable approximations and mitigates the issue.

Published in Transactions on Machine Learning Research (11/2022)
A Trust Crisis In Iimulation-Basd Inference? Your
Posterior Approximations Can Be Unfaithful
Joeri Hermans* Unaffiliated
joeri com
Arnaud Delaunoy University of Liège
Shalieg. be
François Rozet University of Liège
Antoine Wehenkel University of Liège
antoine. wwwpsewwwwwweeeeeeege....
Volodimir Begy University of Vienna
Gilles Louppe University of Liège
Louppe be
Reviewed on OpenReview: https: //
// openreview. net/ forum? id- 6
Abstract
We present extensive empirical evidence showing that current Bayesian simulation-based inference algorithms can produce computationally unfaithful posterior approximations. Our results show that all benchmarked algorithms (Sequential) Neural Posterior Estimation, (Sequential) Ncural Ratio Estimation, Sequential Neural Likelihood and variants of Approxi- mate Bayesian Computation can yield overconfident posterior approximations, which makes them unreliable for scientific cases and falsificationist inquiry. Failing to address this issue may reduce the range of applicability of simulation-bees inference. For this reason, argue that research efforts should be made towards theoretical and methodological developments of conservative approximate inference algorithms and present research directions towards this objective. In this regard, we show empirical evidence that ensembling postcrior surrogates provides more reliable approximations and mitigales the issue.
1 Introduction
Many scientific disciplines rely computer simulations to study complex phenomena under various conditions. Although modern simulators can generate realistic synthetic observables through detailed descriptions of their data generating processes, they make statistical inference more challenging. The computer code describing the data generating processes defines the likelihood function p(æ: 0) only implicitly, and its direct evaluation stochastic execution paths. In this problem setting, statistical
requires the often intractable integration inference based on the likelihood becomes impractical. However, approximate inference remains possible by relying on likelihood-free approximations thanks to the increasingly accessible and effective suite of methods and software from the field of simulation- based inference (Cranmer et al. 2020).
While simulation-based inference targets domain sciences, advances in the field are mainly driven from a machine learning perspective. The field, therefore, inherits the quality assessments (Lueckmann et al.
"Equa contribution

---

Published in Transactions on Machine Learning Research (11/2022)
2021 customary to the machine learning literature, primarily targeting the exactness of the approximation. In fact, domain sciences, and more specifically the physical sciences, are not necessarily interested in the exactness of an approximation. Instead, in the tradition of Popperian falsification, they often seek to constrain paramcters of interest as much as possible at a given confidence level. Scientific examples include frequentist confidence intervals on the mass of the Higgs boson (Aad et al.) 2012), Bayesian credible regions on cosmological parameters (Gilman al.) 2018 Aghanim et al. (020), constraints on the intrinsic parameters of binary black hole coalescences (Abbott et al., 2016), or characterizing the space of circuit configurations giving rise to rhythmic activity in the crustacean stomatogastric ganglion (Gonçalves et al. 2020). Wrongly excluding plausible values could drive the scientific inquiry in the wrong direction, whereas failing to exclude implausible values because of too conservative estimations is much less detrimental. This implies that statistical approximations ions in simulation-based inference should ideally come with conservative guarantees to not produce credible regions smaller than they should be, even when the approximations are not faithful. Despite recent developments of post hoc diagnostics
inspect the quality of likelihood-fie
[Crammer et al., 2015 Brehmer el. al., 2018 2019 Hermans et al., 2021 Lueckmann et al.; 2021
Talls el. al.,
2018
Dalmasso et al. ) and likelihood based (Geweke et al. 1991 Gelman & Rubin
Raftery & approximations, assessing whether approxinate inference
1992
Lewis
1991 Dixit & Roy 2017 Talts et al.
2018
results are sufficiently reliable for scientific inquiry remains largely unanswered whenever fitting criteria are not globally optimized or whenever the data is limited.
In this work, we measure and discuss the quality of the credible regions produced by various algorithms for Bayesian simulation based inference. We frame our main contribution as the collection of extensive empirical evidence requiring months of computation, demonstrating that all benchmarked techniques may produce non-consorvative credible regions. Our results emphasize the need for new class of methods: conservative ap- proximate inference algorithms. In addition, we provide empirical evidence that using an ensemble of models in place of a single model tends to produce more reliable approximations. The structure of the paper is outlined as follows. Section describes the statistical formalism, necessary background and includes a thorough motivation for coverage. Section highlights our main results. Section presents several avenues of fut-ure research to en- able drawing reliable scientific conclusions with simulation-buased inference. All code related this manuscript. is available at http:////tttcccccooo/////eeee//a
2
3
2 Background
2.1 Statistical formalism
We evaluate posterior estimators that produce approximations p(0)z) with the following semantics.
Target parameters 0 denote the parameters of interest of simulation model, and are sometimes referred to as free or model parameters. We make the reasonable assumption that the prior p(0) is tractable.
An observable æ denotes a synthetic realization of the simulator, or the observed data v, we would like do inference on. We assume that the simulation model is correctly specified and hence is an accurate representation of the real data generation process.
The likelihood model p(x(0) is implicitly defined by the simulator's computer code. While we cannot evaluate the density p(z)0). we can draw samples through simulation.
The ground truth 0* specified to the simulation model whose forward evaluation produced the observable Coo i... H, ~ p(x = 0-).
A credible region a space θ within the target parameters domain that satisfics
[ p(0]== x..) do a
(1)
for some observable x, and confidence level 1 - a. Because many such regions exist, we compute the credible region with the smallest volume. In the literature this credible region is known as the highest posterior density region (Box & Tiao 1973 Hyndman 1996). In our evaluations, we determine the credible regions by evaluating the approximated posterior density function in discretized and empirically normalized grid
2

---

Published in Transactions on Machine Learning Research (11/2022)
0.8
Overoonfident
AUROC & 0.7
Gromdttuth
Conservative
016
Posterior density
0.4
0.2
AUROC R 0.7
0.0
Figure 1: A classifier-basee metric measures the divergence between posterior approximations and a ground truth by means of evaluating the classifier's discriminative performance through Area Under the Receiver Operating Characteristics curve (AUROC). The metric indicates that both the overconfident and the conser- vative approximations are equally accurate as yiclds AUROC 0.7 for both of them. From an inference perspective however, the conservative approximation is more suitable because it produces credible regions larger than they should be.
of the parameter space. The credible region is the set of parameters whose density is greater than a given threshold fitted to achieve the desired credibility level 1 - a. Additional details are described in Appendix A
2.2
Statistical quality assessment
Common metrics for evaluating the quality a posterior surrogate include the Classifier Two-sample Test (Lhhmann & Romano 2006 Lopez-Paz Paz & Oquab 2017] and Maximum Mean Discrepancy (Gretton al.] 2012 Bengio et al. 2014 Dziugaite al., (2015). The main problem with these metrics is that they assess exactness of approximation through divergence with respoct to the posterior. All approximations will diverge from the posterior and there are criteria to what constilutes an acceptable estimator. For this reason, we argue that metrics evaluating the reliability for scientific inquiry should be used alongside the divergence evaluation when ovaluating estimators.
To clarify this point, consider the demonstration Figure I binary classifier is trained to discriminate between samples from a posterior approximation and the true posterior, as in a classifier two-sample test. The discriminative performance of the classifier is expressed through Arca Under the Receiver Operating Characteristics curve (AUROC) and serves measure for divergence between both densities. An AUROC 0.5 suggests an approximation that is indistinguishable from the true posterior, while AUROC = 1.0 implies that both distributions do not overlap. Although both the overconfident and the conservative approximations are of equal quality according to the AUROC metric (AUROC = 0.7), credible regions that are biased or smaller than they should be could result. in the wrong exclusion of act plausible parameter values for a given significance level, and hence erroncous scientific conclusions. By contrast, conservative approximations, which leads to credible regions that larger than they should be, are more scientifically reliable since they would not wrongly reject plausible parameters values but only fail to reject actually implausible parameter values. For this reason, we take the position that posterior approximations should produce overdispersed credible regions for any simulation budget. Posterior approximations do not have to closely match the true posterior to draw meaningful inferences, but they should however be conservative.
Instcad of mcasuring the exactness of an approximation, this work directly assesses the quality of credible regions through the notion of expected coverage which probes the consistency of the posterior approximations and can be used to diagnose conservative and overconfident. approximations. Similar usages of coverage
3

---

Published in Transactions on Machine Learning Research (11/2022)
diagnosties can be found in the context of standard statistical inference (Schall 2012) and approximate Bayesian computation (Prangle et al. 2014).
Definition 2.1. The expected coverage probability of the a highest posterior density regions derived from the posterior estimator ((0)z)) is
[1l (0 E (1-aa)
(2)
where the function (p(l(a)) (1 a) yields the 1 highest posterior density region of p(0)z).
Note that Equation can be expressed either as
E T [1 (0 (1
(3)
which is the expected frequentist coverage probability, or alternatively as the expected Bayesian credibility
E (el [1 (0 p1 - a))]]
(4)
whose inner expectation reduces to 1 whenever the posterior estimator p(0)z) is well-calibrated.
The expected coverage probability can be estimated empirically given set of n ii.d. samples (0,,z;) ) ~ p(0,x) as
n
1 (0: E (1
(5)
Definition 2.2. A conservative posterior estimator is an estimator that has coverage the credibility level of interest, i.e., whenever the expected coverage probability is larger or equal to the credibility level.
While coverage is necessary to assess conservativeness, it is limited in its ability to determine the information gain a posterior (approximation) has over its prior. Consider an estimator whose posteriors are identical the prior. In this case, there is no gain in information and the expected coverage probability is equal to the credibility level. For this reason, complete analysis should be complemented with measures such the expected information gain 22)(0.2) [logr(0] a=) log (()) (as illustrated in Appendix D classifier two-sample tests when the ground-truth posterior is available. This work is concerned with conservative inference and will therefore mainly focus on the evaluation of expected coverage. Finally, it should be noted that expected coverage is a statement about the credible regions in expectation and therefore does not provide any guarantec on the quality of a single posterior approximation. However, the quality of a single postcrior approximation can itself be approximated with a local coverage test (Zhao et al., 2021
3 Empirical observations
This section covers our main contribution: the collection of empirical evidence to determine whether some simulation-based inference algorithms are conservative by nature, We are particularly interested in determining whether certain approaches should be favoured over others. We do so by estimating the expected coverage of posterior estimators produced by these approaches across a broad range of hyperparameters and benchmarks of varying complexity, including two real problems. As in real use cases, the true posteriors are effectively intractable and therefore unknown.
3.1 Methods
We make the distinction between two paradigms. Non-amortized approaches are designed to approximate a single posterior, while amortized methods aim to learn general purpose estimator that attempts approximate all posteriors supported by the prior. The architectures used for each inference algorithm, including hyperparameters, listed in Appendix C
4

---

Published in Transactions on Machine Learning Research (11/2022)
3.1.1 Amortized
Neural Ratio Estimation (NRE) is an established approach in the simullatoon-ass inference litcrature both from a issquentist (Cranmer et al. 2015) and Bayesian (Thomas et al. 2016; Hermans et al.] 2020
Durkan et al. 2020) perspective. In a Bayesian analysis, an amortized estimator f(x(0) of the intractabl ble ratio r(z]0) can be learned by training a binary classifier d(0, x) to distinguish between samples of the joint p(0,z) x) with class label 1 and samples of the product of marginals p(0)p(x)) with class label O, with equal label marginal probability. Similar to the density-ratio trick (Sugiyama al., 2012
Goodfellow et al. 2014 Cranmer et al. 2013 Hermans et al. 2020), the Bayes optimal classifier d(0, x) for
the cross-enl.rop loss is
p(0, d(0,z)) p(0, a:) p(0)((x))
of (1og
log p(0,2) x) p(0)p(zz)
(6)
where (.) is the sigmoid function. Given a largel parameter 0 and observable æ supported by p(0) and p(æ)) respectively, the learned classifier d(00, æ:) approximates the log likelihood-to-evideeee ratio log rarr(a (x:(0
through the logit function because logit( (a((()))))))))))))) logr(z The approximate log posterior density function is logp(0) p(0) + log (a):)))
Neural Posterior Estimation (NPE) [Rezende & Mohamed 2015 is concerned with directly learning
amortized posterior estimator p.. (0/z) with normalizing flows. Normalizing flows define a class of probability distributions P (-) built from neural network- based bijective transformations (Rezende & Mohamed, 2015
Dinh et al., 2015 2017 parameterized by th. They are usually optimized using variational inference, by solving (0(z))) which equivalent lo arg max Ep(e.x) [log [logg+(( p. (0(x))) Once trained, the density of the modeled distribution can directly be evaluated and sampled from.
arg ming Pp()) (f(lz) ||
Ensembles of models constitute standard method to improve predictive performance. In this work, we consider an ensemble model that averages the approximated posteriors of n posterior estimators that are either trained independently on the same dataset (deep ensemble) Lakshminarayanau et al. (2017) or on bootstrap replicates of the learning set (bagging) Breiman (196).. While this formulation is natural for NPE, averaging liceihhoo-to--viideeee ratios is equivalent since E (0)z) = p(0) En= (=(0)) We show in Section 3 that ensembles lead average to a higher expected coverage than individual models and hence constitute a straightforward mitigation strategy against overconfidence.
3.1.2 Non-amortized
Pritchard et al. 1999h nu- merically estimates a single postcrior by collecting samples O ~ p(0) whenever x ~ is similar to a.- Similarity is expressed by means of a distance function p. For high-dimensional observables, the probability density of simulating an observable æ such that æ = N. extremely small. For this reason, ABC uses a summary statistic s and an acceptance threshold e. Using these components, ABC accepts samples into the approximate posterior whenever p(s(z)) s(x.)) I c. Improvements include regression adjustment (Beaumont) 2002) of the sampled parameters using local linear regression, combining ABC with MCMC (Marjoram et al. 2003 and the automatic construction of summary statistics [Fearnhead & Prangle, 2012). In our experiments, we apply REJ-ABC in its simplest form and use the identity function as a sufficient summary statistic, use no regression adjustment and set e such that max(100, simulation max(m0i,simnltton budget/100) samples are accepted.
Rejection Approximate Bayesian Computation (REJ-ABC) (Rubin 1984
et al.
Sequential methods for simulation-based inference aim to approximate a single posterior by iteratively improving posterior approximation. These methods alternate between simulation and an exploitation phase. The latter is designed to take current knowledge into account such that subsequent simulations can be focused on parameters that are more likely to produce observables a similar to To.
Sequential Monte-Carlo ABC (SMC-ABC) (Ton & Stumpf 2009 Sisson et al. 2007 Beaumont et al., 2009) iteratively updates a set of proposal states to match the posterior distribution. At each itcration, accepted proposals are ranked by distance. The rankings determine whether a proposal is propagated to the next iteration. New candidates are generated by perturbing the selected ranked proposals.
5

---

Published in Transactions on Machine Learning Research (11/2022)
Sequential Neural Posterior Estimation (SNPE) Papamakarios & Murray 2016 Lueckmann et al. 2017
Greenberg et al. 2019) iteratively improves norma izing flow that models the posterior. Our eval evaluations will specifically use the SNPE-C (Greenberg et al. 2019) variant.
Sequential Neural Likelihood (SNL) Papamakarios et al., 2019 models the likelihood p(z[0). A mumerical approximation of the posterior obtained by plugging the learned likelihood estimator into a Markov Chain Monte Carlo (MCMC) sampler as a surrogate likelihood. Similarly, Price et al. (2018); Frazier et al. (2022) construct synthetic normal approximation of the likelihood over summary statistics.
Sequential Neural Ratio Estimation (SNRE) (Hermans et al. improves the modelled likelihood-to--videeeee ratio.
2020: Durkan et al. 2020 iteratively
3.2 Benchmarks
We consider 7 benchmarks, ranging from a toy problem to real scientific use cases covering various disciplines. All benchmarks and priors are available in the codebase.
The SLCP simulator modols a fictive problem with paramcters. The observable æ E IR represents the 2D-cooddinates of 4 points. The coordinate of each point is sampled from the same multivariate Gaussian whose mean and covariance matrix are parametrized by O. We consider an alternative version of the original task (Papamakarios et al., 2019) by inferring the marginal posterior density of 2 of those parameters. In contrast to its original formulation, the likelihood is not tractable due to the marginalization.
The Weinberg problem (Cranmer et al. 2017 concerns simulation of high energy particle collisions ete- + + n*= The angular distribution of the particles can be used to measure the Weinberg angle æ in the standard model of particle physics. From the scattering angle, we are interested in inferring Fermi's constant 0.
The Spatial SIR model involves a grid-world of susceptible, infected, and recovered individuals. Based initial conditions and the infection and recovery rate 0, the model describes the spatial evolution of an infection. The observable is snapshot of the grid- wrrldworld after some fixed amount of time.
M/G/1 (Blum& & François 2010) models a processing and arrival queue. The problem is described by 3 parameters that influence the time it lakes lo serve a customer, and the time between their arrivals. The observable æ is composed of 5 equally spaced quantiles of inter-departure times.
The Lotka-Volterra population model (Lotka) 1920 Volterra 1926 describes a process of interactions between predator and a prey species. The model is conditioned 4 parameters O which influence the reproduction and mortality rate of the predators and preys. We infer the marginal posterior of the predator parameters from time series reprosenting the evolution of both populations over time.
Stellar Streams form due to the disruption of spherically packed clusters of stars by the Milky Way. Because of their distance from the galactic center and other visible matter, distant stellar streams are considered to ideal probes to detect gravitational interactions with dark matter. The model (Banik et al. 2018) evolves the stellar density æ of a stream over several billion years and perturbs the stream over its evolution through gravitational interactions with dark matter subhaloes parameterized by the dark matter mass 0.
Gravitational Waves (GW) are ripples in space-time emitted during events such as the collision of two holes. They can be detected through interferometry measurements æ and convey information about celestial bodies, unlocking new ways to study the universe. We consider inferring the masses of two black-holes colliding through the observation of the gravitational wave as measured by LIGO'S dual detectors ILIO Scientific Collaboration, 2018 Biwer et al., 2019).
3.3 Setup
Our evaluations consider simulation budgets ranging from 200 up to 217 samples and confidence levels from 0.05 up to 0.95. Within the amortized sctting we train, for every simulation budget, 5 postcrior estimators for 100 epochs. The expected coverage probability is estimated as described in Appendix A on at least 5,000 unseen samples from the joint, p(0, x)) and for all confidence levels under consideration. In addition, we
6

---

Published in Transactions on Machine Learning Research (11/2022)
repeat the expected coverage evaluation for ensembles of 5 estimators well. Special care for non-amortized approaches is necessary because they approximate single posterior and can therefore not reasonably evaluate expected coverage in the same way. Our experiments for non-amortized approaches estimate the expected coverage by repeating the inference procedure on 300 distinct observables for every simulation budget. The expected coverage probabilities are subsequently estimated based on the resulting posterior approximations. Our experiments with NPE, SNPE, SNI, SNRE, REJ-ABC and SMC-ABC rely on the implementation in the sbi package Cantero et al. 2020), while custom implementation for NRE is used.
Computational cost We emphasize the computational requirements necessary to generate our main contribution: the experimental observations, whose generation took months of computation. The average CPU time for evaluating an amortized procedure on all common benchmark problems is in order of 200 CPU days, while for a non-amortized approach this increases to 2800 CPU days. The bulk of the cost was associated with the repeated optimization procedure and exccution of the simulator. While we considered benchmarks with low dimensional paramcter spaces (up to 3 parameters), we stress that the cost for computing crodible regions would grow exponentially with the number of parameters, making coverage diagnostics expensive for high-dimensional problems. Note that when using NPE, the procedure for computing the coverages could be modified to scale to higher dimensions (Rozet & Louppe 2021a by. However, this procodure cannot be applied to all simulation-ban based inference algorithms. Nevertheless, we expect the observed behaviours to be similar or accentuated in a high-dimensional setting.
3.4 Results
Figures 2 and 8 highlight our main results. Through these plots, we can directly assess whether a posterior estimator is consorvative at given confidence level and simulation budget. The figures should be interpretod as follows: a perfectly calibrated posterior has an expected coverage probability equal to the credibility level. Plotting this relation produces diagonal line. Conservative estimators the other hand produce curves above the diagonal and overconfident models underneath. The plots highlight an unsettling observation: all benchmarked approaches produce non-conservative posterior approximations on at least one problem setting. In general, this pathology is especially prominent in non-amortized approaches with a small simulation budget; a regime they have been specifically designed for. A large simulation budget does not guarantee that posterior estimator is conservative either. More importantly, as the reliability of the approximations computed through sequential approaches cannot practically be determined, it leaves practitioners uncertain about the reliability of their approximations. To complement this analysis, an evaluation of the predictive performance of the approximate posteriors is provided in Appendix [ We observe that, on most benchmark/metee method pairs, the predictive performance increases with the simulation budget. However, this is not the case for SNRE applied to MG1 which explains the strange behaviour of the coverage curves.
In sequential approaches, overconfidence could be explained by the alternating simulation and exploitation phases. One potential failure mode is that a non-conservative posterior approximation at a previous iteration forces the subsequent simulation phases to not produce observables that should in fact be associated with a higher posterior density, causing the posterior estimator to increase its overconfidence at each iteration.
Despite the fact that all ABC approaches use a sufficient summary statistic (by definition, the identity function), our results demonstrate that this alone is no guarantee for conservative posterior approximations. In fact, using a sufficient summary statistic with 6 > 0 does not always correspond to conservative approximations either. In such cases, ABC accepts samples with larger distances, permitting the procodure to shift the mass of the approximated posterior elsewhere. In addition, a limited number of posterior samples can negatively affect the quality of the credible regions, e.g., when approximating the posterior density function with kernel density estimation. Both cases can cause the observed behaviour. ABC should therefore be applicd with caution to scientific applications. Even though a handcrafted, albeit sufficient, summary statistic provides some insight inlo the approximated posterior, it does not imply that ABC approximations are conservative whenever the threshold E > 0.
In Figure 3 we observe that the expected coverage probability of deep ensemble models is consistently larger than the expected coverage probability of individual posterior estimator. This highlights the fact that
7

---

Published in Transactions on Machine Learning Research (11/2022)
NRR
NRF ensemble
NPR
NPE marmble
SNPE
SNI.
SN RR
Raj-ABC
SMC-ARC
MCSI
Weinbeng
Lotao Volterna
Spotial
CON
Elspttial conerage
Strams
Chepe with wha
Chnen wandh de hene
Cadibiiity
1000 somples
8190 sumplex
39788 samples
131072 somples
semples
ramples
samples
Caliooaed
1300 samplox
Figure Evolution of the expected coverage w.r. the simulation budget. perfectly calibrated posterior has expected coverage probability equal to the credibility level and produces diagonal line. Conservative estimators the other hand produce curves above the diagonal and overconfident models underneath. All algorithms can lead to non-conservative estimators. This pathology tends to be accentuated for small simulation budgets and non-amortized methods. Finally, the computationally prohibitive results indicate that the computational requirements did not allow for a coverage analysis. the case of SNL, this was mostly due high dimensional observables. For the astronomy benchmarks, the simulation model was simply too expensive reasonably evaluate coverage for non-amortized methods.
8

---

Published in Transactions on Machine Learning Research (11/2022)
NRE
NPE
samples
8192 samples
sacppees
131072 samples
1021 samples
8192 somplee
samples
131072 samples
SLOP
20G:
Weitherg
Locia Vottera
0.5
Spetial
GW
0.5
Earairia coweregg
Canartatho
Steearze
65
Crodibility
Dop ensemble
Individunl
Bagging
Callbested oxtinnaion
Figure 3: Analysis of coverage between ensemble and individual models w.r.t the various simulation budgets. The blue line represents the mcan expected coverage of individual models over runs, the shaded area represents its standard deviation. The black line represents the expected coverage of single ensemble composed of 5 models. We observe that deep ensembles consistently have higher expected coverage probability compared to the average individual model. A similar effect is not always observed with bagging, indicated by the red line. Ensembles are only evaluated for amortized approaches such as NPE and NRE.

---

Published in Transactions on Machine Learning Research (11/2022)
ensembling constitutes an immediately applicable and easy way to mitigate the overconfidence issue and build more reliable posterior estimators. However, the deep ensemble model can still be non-conservative. We hypothesize that the increase in coverage is linked to the added uncertainty captured by the ensemble model, leading to inflated credible regions. In fact, individual estimators only capture data uncertainly, while an ensemble is expected lo partially capture the epistemic uncertainty as well. Surprisingly, we find that ensembles built using bagging do not always produce higher coverage than individual models while they should also capture part of the epistemic uncertainby. We could potentially altribute this behaviour to the reduced effective dataset size used to train each member of the ensemble. Appendix E shows positive effect with respect to ensemble size.
Not evident from figures 2 and 2 are the computational consequences of coverage analysis on non-amortized methods. Although the figures mention a certain simulation budget, the total number of simulations for non-amortized methods should be multiplied by the number of approximated posteriors (300) to estimate the coverage. This highlights the simulation cost associated with diagnosing non-amortized approaches. This issue is not limited to coverage. Simulation-Based Calibration (SBC) (Talts et al., (018)) relies on samples of well. Diagnosing non-amortized estimators with SBC therefore requires a similar approach as we have taken in our coverage analyscs. In fact, other benchmark papers such as Lueckmann et al. (2021) restrain from evaluating SBC because the diagnostic is computationally prohibitive for non-amortized approaches.
arbitrary posterior approximations
Our results illustrate a clear distinction between the amortized and non-amortized paradigms. Amortized methods do not require retraining or new simulations to determine the empirical expected coverage probability of posterior estimator, while non-amortized methods do. For this reason, global coverage analysis of non-amortized approaches is computationally prohibitive and mostly impractical. More importantly, the coverage analysis of a non-amortized approach only measures the quality of the training procedure, whereas a coverage analysis of an amortized approach diagnoses the posterior estimator itself. In addition, global coverage analysis not only serves as diagnostic but also allows to partially alleviate the issue by performing post-training calibration. A simple way for calibrating level a credible regions is to replace those by credible regions at level that has the desired expected coverage. Finally, non-amortized sequential algorithms have to repeat the entire simulation-training pipcline whenever architectural or hyperparameter changes are made, while amortized methods reuse previously simulated datasets. All of the above lead us to conclude that while sequential methods have the benefit of being faster to train, amortized methods should be considered for sensitive applications requiring detailed statistical validation.
Observation 1 All benchmarked algorithms may produce non-conservative postcrior approximations. This pathology tends to be accentuated with small simulation budgets in both paradigms.
Observation 2 Amortized approaches tend to be more conservative in contrast to non-amortized ap- proaches.
Observation 3 The expected coverage probability of an ensemble model is larger than the average individual model. The ensemble size positively affects the expected coverage probability well.
Observation 4 Amortized methods are simulation-eflicienn,, especially when taking hyper-paramedde tuning and the evaluation of the expected coverage diagnostic into account.
4 Discussion
As demonstrated empirically, simulation- inference can be unreliable, especially whenever its approxima- tions cannot be diagnosed. The problem of determining whether a posterior approximation is correct, rather, suitable to a use case, is in fact not restricted to simulation-based inference specifically; the concern occurs in all of approximale Bayesian inference. The MCMC literature deals with this exact same problem in the form of determining whether a set of Markov chain samples have converged to the target distribution Lin 2014 Hogg & Foreman-Mackor (0088). In this regard, empirical diagnostic tools have been proposed over the years (Geweke et al. 1991 Gelman & Rubin 1992. Raftery & Lewis 1991] Dixit Roy 2017
Talts al.] 2018) and have helped practitioners to apply MCMC reliably. While, we are not aware of any
10

---

Published in Transactions on Machine Learning Research (11/2022)
coverage studies similar to ours for MCMC, would expect similar results to be found. Nonetheless, there is currently no clear solution to determine convergence with absolute certainty (Dixit, 2018 Roy 2020), even if the likelihood function is tractable. This issue has also been studied in the ABC literature. Prangle et al. (2014) present a way for approximating the coverage in ABC by reusing the same simulations in different rums. Xing et al. (2020) provides a diagnostic for the quality of the approximation for a given observation by the mean of distortion maps. Finally, Frazier et al. (2018) provide condition on the threshold c such as produce posterior estimates that asymptotically have proper frequentist coverage. However, the quality of the posterior estimates remains uncertain in a practical non-asymptotic regime.
We are of the opinion that theoretical and methodological advances within the field of simulation-based inference will strengthen its reliability and thereby promote its applicability in sciences. First, although all benchmarked algorithms recover the true posterior under specific optimal conditions, it is generally not possible to know whether those conditions are satisfied in practice. Therefore, the study of new objective functions that would force posterior estimators to always be conservative, regardless of optimality conditions, constitutes valuable research avenue. From a Bayesian perspective, Rozet & Louppe| (2021b) propose using the focal and the peripheral losses to weigh down strongly classified samples as means to tune the conservativeness of posterior estimator. However, the technique is empirical and roquires tuning to attain the desired properties in practice. Dalmasso et al. (2020) on the other hand consider the frequentist setting and introduce a theoretically-grounde guaranteed to have calibrated coverage, regardless of the quality of the used statistic. Dalmasso et al. (2021 extends this work with finite sample guarantoes.
algorithm for the construction of confidence intervals that are
Second, in light of our results that ensembles produce more conservative posteriors, model averaging constitutes another promising direction of study as simple and directly applicable method to estimators. However, a deeper understanding of the behaviour we observe is certainly first required to further develop these methods.
produce reliable posterior
Third, post-training calibration can be used to improve the reliability of posterior estimators and should certainly be considered as a way toward more conservative inference. To some exlent, this has already been considered for amortized methods @Cranmer et al., 2015 Brehmer et al., 2018 Hermans et al., 2021 and would be worth exploring further, espocially for non-amortized approaches.
In this work, we assumed that the simulator perfectly models the real data generating process and focused on the computational faithfulness of the inference engine. In practice, there might be gap between the simulation and reality. This issue, orthogonal to our work, should also be taken into account for making
reliable simulation-basel inference, as already initiated by several authors (Schmon et al. 2020 Matsubara
et al. 2022 Pacchiardi & Dutta, 2022. Dellaporta et al. 2022).
In summary, show that current algorithms for simulation-basee inference can produce overconfident posterior approximations, making them possibly unreliable for scientific use cases and falsificationist inquiry. Nevertheless, we remain confident and optimistic and advocate that our results are only stepping stone toward more reliable simulation -based inference and its wider adoption in the sciences.
References
Georges Aad, Tatevik Abajyan, B Abbott, J Abdallah, S Abdel Khalek, Ahmed Ali Abdelalim, R. Aben, B Abi, M Abolins, OS AbouZeid, et al. Observation of a new particle in the search for the standard model higgs boson with the atlas detector at the lhe. Physics Letters B, 716(1):1 29, 2012.
Benjamin P Abbott, R Abbott, TD Abbott, MR Abernathy, F Acernese, K Ackley, C Adams, T Adams, P Addesso, RX Adhikari, et al. Gw151226: observation of gravitational waves from a 22-solar-mass binary black hole coalescence. Physical review lellers, 116(24):241103, 2016.
N. Aghanim et al. Planck 2018 results. VI. Cosmological parameters. Astron. Astrophys., 641:A6, 2020. doi: 00001/00004-631/200000
[Erratum: Astron.Astrophys. 652, C4 (2021)).
Nilanjan Banik, Gianfranco Bertone, Jo Bovy, and Nassim Bozorgnia. Probing the nature of dark matter particles with stellar reams. Journal of Cosmology and Astroparticle Physies, 2018(07):061, 2018.
11

---

Published in Transactions on Machine Learning Research (11/2022)
Mark A Beaumont, Wenyang Zhang, and David J Balding. Approximate bayesian computation in population genetics. Genelies, 162(4))2025-200) 2035, 2002.
Mark A Beaumont, Mern-Marie Cornuet, Jean-Michel Marin, and Christian P Robert. Adaptive approximate bayesian computation. Biometrika, 96(4) 990, 2009.
Yoshua Bengio, Li Yao, and Kyunghyun Cho. Bounding the test log- log-likel of generative models. In International Conference on Learning Representations, 2014.
C. M. Biwer, Collin D. Capano, Soumi De, Miriam Cabero, Duncan A. Brown, Alexander H. Nitz, and V. Raymond. PyCBC Inference: A Python-based parameter estimation toolkit for compact binary coalescence signals. Publ. Astron. Soc. Pac., 131 (996), 2019. doi: (10088/533-387//aam
Michael GB Blum and Olivier François. Non-linear regression models for approximate bayesian computation. Statistics and computing, 20(1):63 73, 2010.
George EP Box and George C Tiao. Bayesian inference in statistical analysis, volume 40. John Wiley & Sons, 1973.
Johann Brehmer, Kyle Cranmer, Gilles Louppe, and Juan Pavez. A Guide to Constraining Effeetive Field Theories with Machine Learning. Phys. Rev. D, 98(5):052004, 2018. doi: 10.1103/ 10.1103/PhysRevD 98.0042044
Johann Brehmer, Siddharth Mishra-Sharma, Jocri Hermans, Gilles Louppe, and Kyle Cranmer. Mining for dark matter substructure: Inferring subhalo population properties from strong lenses with machine learning. The Astrophysical Journal, 886(1):49, 2019.
Leo Breiman. Bagging predictors. Machine learnóng, 24(2):123 140, 1996.
Kyle Cranmer, Juan Pavez, and Gilles Louppe. Approximating likelihood ratios with calibrated discriminative classifiers. arXiv preprint ar arXiv:1506.0911
2015.
Kyle Cranmer, Lukas Heinrich, Tim Head, and Gilles Louppe. "Active Sciencing" with Reusable Workflows. 2017.
https://githul:
Kyle Crammer, Johann Brehmer, and Gilles Louppe. The frontier of simulation-based inference. Proceedings of the National Academy of Sciences, 2020.
Niccolò Dalmasso, Rafael Izbicki, and Ann Lee. Confidence sets and hypothesis testing in a likelihood-free inference setting. In International Conference on Machine Learning, pp. 2323 2334. PMLR, 2020.
Niccolò Dalmasso, David Zhao, Rafael Izbicki, and Ann B Lee. Likelihood free frequentist inference: Bridging classical statistics and machine learning in simulation and uncertainty quantification. arXiv preprint arXiu:2107.039200 2021.
Charita Dellaporta, Jeremias Knoblauch, Theodoros Damoulas, and François-Xaan Xavier Briol. Robust bayesian inference for simulator-based models via the mind posterior bootstrap. In International Conference Artificial Intelligence and Statistics, pp. 943 970. PMLR, 2022.
Laurent Dinh, David Krueger, and Yoshua Bengio. NICE: lin-linear independent components estimation. In 3rd International Conference on Learning Representations, TCLR 2015, San Diego, CA, USA, May 7-9, 2015, Workshop Track Proceedings, 2015.
Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real NVP. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings, 2017.
Anand Dixit and Vivekananda Roy. Meme diagnostics for higher dimensions using kullback leibler divergence. Journal of Statistical Computation and Simulation, 87(13):262) 222222238,2 2017.
Anand Ulhas Dixit. Developments in MCMC diagnostics and sporse Bayesion learning models. PhD thesis, Towa State University, 2018
12

---

Published in Transactions on Machine Learning Research (11/2022)
Conor Durkan, lain Murray, and George Papamakarios. On contrastive learning for likelihood-free inference. In International Conference on Machine Learning, pp. 2771-2781 PMLR, 2020.
Gintare Karolina Dziugaite, Daniel M Roy, and Zoubin Ghahramani. Training generative neural networks via maximum mean discrepancy optimization. In Proceedings of the Thirty-Fian First Conference on Uncertainty in Artificial Intelligence, pp. 258 267, 2015.
Paul Fearnhead and Dennis Prangle. Constructing summary statistics for approximate bayesian computation: semi-aulomatic approximate bayesian computation. Journal of the Royal Statistical Sociely: Series B (Statistical Methodology), 74(3):419 474, 2012.
David T Frazier, Gael M Martin, Christian P Robert, and Judith Rousseau. Asymptotic properties of approximate bayesian computation. Biometrika, 105(3):593 607, 2018.
David T Frazier, David J Nott, Christopher Drovandi, and Robert Kohn. Bayesian inference using synthetic likelihood: asymptotics and adjustments. Journal of the American Statistical Association, (just-cccpteed): 1-28, 2022.
Andrew Gelman and Donald B Rubin. Inference from iterative simulation using multiple sequences. Stalisticnl science, 7(4):457-477 472, 1992.
John F Geweke et al. Evaluating the accuracy of sampling- based approaches to the calculation of posterior moments. Technical report, Federal Reserve Bank of Minneapolis, 1991.
Daniel Gilman, Simon Birrer, Tommaso Treu, Charles R Kecton, and Anna Nierenberg. Probing the nature of dark matter by forward modelling flux ratios in strong gravitational lenses. Monthly Notices of the Royal Astronomical Society, 481 (81(1) (1): 834, 2018.
Pedro J Gonçalves, Jan-Matthis Lueckmann, Michael Deistler, Marcel Nonnenmacher, Kaan Öcal, Giacomo Bassetto, Chaitanya Chintaluri, William F Podlaski, Sara A Haddad, Tim P Vogels, et al. Training deep neural density estimators to identify mechanistic models of neural dynamics. Elife, 9ce56261, 2020.
Ian Goodfellow, Jean Pouget-Abadie, Mchdi Mirza, Bing Xu, David Fardey Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.
David Greenberg, Marcel Nonnenmacher, and Jakob Macke. Automatic posterior transformation for likelihood- free inference. In International Conference on Machine Learning, pp. 2404 2414. PMLR, 2019.
Arthur Gretton, Karsten M Borgwardt, Malte J Rasch, Bernhard Schölkopf, and Alexander Smola. kernel (wo-sample test. The Journal of Machine Learning Research, 13(1):723 773, 2012.
Jocri Hermans, Volodimir Begy, and Gilles Louppe. Likelihood-free MCMC with amortized approximate ratio estimators. In Hal Daumé IIT and Aarti Singh (eds.), Proceedings of the 97th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 4239 4248. PMLR, 13 18 Jul 2020.
Joeri Hermans, Nilanjan Banik, Christoph Weniger, Gianfranco Bertone, and Gilles Louppe. Towards constraining warm dark matter with stellar streams through neural simulation-baln inference. Monthly Notices of the Royal Astronomical Society, 507(2):1999 2011, 2021.
David W Hogg and Daniel Foreman-Mackey. Data analysis recipes: Using markov chain monte carlo. The Astrophysical Journal Supplement Series, 236(1):11, 2018
Rob J Hyndman. Computing and graphing highest density regions. The American Statistician, 5002):120-126, 1996.
Diederick P Kingma and Jimmy Ba. Adam: method for stochastic optimization. In International Conference Learning Representations (ICL.), 2015.
13

---

Published in Transactions on Machine Learning Research (11/2022)
Günter Klambauer, Thomas Unterthiner, Andreas Mayr, and Sepp Hochreiter. Self-normalizing neural networks. In Proceedings of the 31st. international conference on neural information processing systems, pp- 972 981, 2017.
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in neural information processing systems, 30, 2017.
Erich L Lehmann and Joseph P Romano. Testing statistical typotheses. Springer Science & Business Media, 2006.
LIGO Scientific Collaboration. LIGO Algorithm Library - LALSuite. free software (GPI.), 2018.
Jing Lin. An integrated procedure for bayesian reliability inference using meme. Journal of Quality and Reliability Engineering, 2014, 2014.
David Lopez-Paz. and Maxime Oquab. Revisiting classifier two-sample tests. In International Conference Learning Representations, 2017.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference Learning Representations, 2019.
Alfred J Lotka. Analytical note certain rhythmic relations in organic systems. Proceedings of the National Academy of Sciences, 6(7):410-415, 1920.
Jan-Matthis Lucckmann, Pedro Goncalves, Giacomo Bassetto, Kaan Öcal, Marcel Nonnenmacher, and Jakob H Macke. Flexible statistical inference for mechanistic models of neural dynamics. Advances in Neural Information Processing Systems, 30, 2017.
Jan-Matthis Lueckmann, Jan Boclts, David Greenberg, Pedro Goncalves, and Jakob Macke. Benchmarking simulation- inference. In Arindam Banerjee and Kenji Fukumizu (eds.), Proceedings of The 24th International Conference Artificial Intelligence and Statistics, volume 130 of Proceedings of Machine Learning Research, pp. 343 351. PMLR, 13 15 Apr 2021.
Paul Marjoram, John Molitor, Vincent Plagnol, and Simon Tavaré. Markov chain monte carlo without likelihoods. Proceedings of the National Academy of Sciences, 100(26):15324-15328, 2003.
Takuo Matsubara, Jeremias Knoblauch, François-Xava Xavier Briol, and Chris Oates. Robust generalised bayesian inference for intractable likelihoods. Journal of the Royal Statistical Society: Series B, 2022.
Lorenzo Pacchiardi and Ritabrata Dutta. Score matched neural exponential familics for likelihood-freo inference. Journal of Machine Learning Research, (3(3)) 71, 2022.
George Papamakarios and lain Murray. Fast e-free inference of simulation models with bayesian conditional density estimation. In Advonces in neural information processing systems, pp. 1036-1036, 2016.
George Papamakarios, David Sterralt, and Jain Murray. Sequential neural likelihood: Fast likelihood-free inference with autoregressive flows. In The 22nd International Conference on Artificial Intelligence and Statistics, pp. 837-848. PMLR, 2019.
Dennis Prangle, Michael GB Blum, G Popovic, and SA Sisson. Diagnostic tools for approximate bayesian computation using the coverage property. Australian & New Zealand Journal of Statistics, 56(4):309-329, 2014.
Leah F Price, Christopher C Drovandi, Anthony Lee, and David J Nott. Bayesian synthetic likelihood. Journal of Computational and Graphical Statistics, 27(1))1-11 2018.
Jonathan K Pritchard, Mark T Seielstad, Anna Perez-Lezaun, and Marcus W Feldman. Population growth of human y chromosomes: study of chromosome microsatellites. Molecular biology and evolution, 16
(11):179, 1999.
Adrian E. Raftery and Steven Lewis. How many iterations in the gibbs sampler? Technical report, 1991.
14

---

Published in Transactions on Machine Learning Research (11/2022)
Danilo Rezende and Shakir Mohamed. Variational inference with normalizing flows. In International conference machine learning, pp. 1530-1538. PMLR, 2015.
Vivekananda Roy. Convergence diagnostics for markov chain monte carlo. Annual Review of Statistics and Its Application, 7:387-412, 2020.
François Rozet and Gilles Louppe. Arbitrary marginal neural ratio estimation for simulation-based inference. arXiv preprint 10X0::210000449, 2021a.
François Rozet and Gilles Louppe. Arbitrary marginal neural ratio estimation for likelihood-free inference. Master's thesis, University of Liège, Belgium, 2021b.
Donald B. Rubin. Bayesianly Justifiable and Relevant Frequency Calculations for the Applied Statistician. The Annals of Statistics, 12(4):1151 1172, 1984.
Robert Schall. The empirical coverage of confidence intervals: Point estimates and confidence intervals for confidence levels. Biometrical journal, 54(())5337553 2012.
Sebastian M Schmon, Patrick W Cannon, and Jeremias Knoblauch. Generalized posteriors in approximate bayesian computation. arXiv preprinl Xrx:22001 1. 1.08644 2020.
Scott A Sisson, Yanan Fan, and Mark M Tanaka. Sequential monte carlo without likelihoods. Proceedings of the National Academy of Sciences, 104(6):1760-177) 17600-176,2200 2007.
Masashi Sugiyama, Taiji Suzuki, and Takafumi Kanamori. Density-ratio matching under the bregman diver- gence: a unified framework of density-ratio estimation. Annals of the Institute of Statistical Mathematics, 64(55:1009-10444 2012.
Scan Talts, Michael Betancourt, Daniel Simpson, Aki Vehtari, and Andrew Gelman. Validating baycsian inference algorithms with simulation-bused calibration. arviv preprinl urXiv:1804.0677 2018.
Alvaro Tejero-Cantero, Jan Boelts, Michael Deistler, Jan-Matthis Lucekmann, Conor Durkan, Pedro Gonçalves, David Greenberg, and Jakob Macke. sbi: A toolkit for simulation- based inference. The Journal of Open Source Software, 2(52):2505, 2020.
Owen Thomas, Ritabrata Dutta, Jukka Corander, Samuel Kaski, Michael U Gutmann, et al. Likelihood free inference by ratio estimation. Bayesion Analysis, 2016.
Tina Toni and Michael H. Stumpf. Simulation-basee model selection for dynamical systems in systems and population biology. Bioinformatics, 26(1):104 110, 10 2009. ISSN 1367-4803. doi: 10.1093/ 1..1093/bioinformatiss btp619.
Vito Volterra. Fluctuations in the abundance of a species considered mathematically. Nature, 118(2972): 558-50 558-560, 1926.
Hanwen Xing, Geoff Nicholls, and Jeong Kate Lee. Distortion estimates for approximate bayesian inference. In Conference on Uncertainty in Artificial Intelligence, pp. 1208-1217, PMLR, 2020.
David Zhao, Niccolò Dalmasso, Rafael Izbicki, and Ann B Lee. Diagnostics for conditional density models and bayesian inference algorithms. In Uncertainty in Artificial Intelligence, pp. 1830 1840. PMLR, 2021.
15

---

Published in Transactions on Machine Learning Research (11/2022)
A Estimation of the empirical expected coverage probability
We describe in this section the methodology used to compute the empirical expected coverage probability
1 (0 @p( - -a)
(7)
We consider n simulations (07, v.) ~ p(0)p(zz() and compute their associated approximate posteriors p(0)a);) in discretized and empirically normalized grid of the parameter space. The associated credible region is the highest density credible region, i.e. credible region of the form
pp (1 - a) = (0: p:0 a.)) 27.
The threshold is computed using dichotomic search to produce a credible region of level 1 1-a We then estimate the empirical expected coverage probability by the proportion of nominal parameters 0, that falls in their associated credible region (1-0)
B Expected simulation times
SLCP
M/G/1
Weinberg
Lotka-V. V.
Spatial SIR
GW
Streams
0.22 + 0.002 0.20 + 0.002
0.20 + 0.002
19.08 + 0.96
9.18 + 0.28
545.13 + 23.63
39, 369 584
Expected simulation time to produce 1000 simulations for all benchmark problems on a single CPU core. The expected time and standard deviation are reported in seconds.
Table 1:
C Architectures & hyperparameters
In this section we describe the neural architectures and hyperparameters associated with our experiments. Our descriptions are complemented with the actual number of coverage evaluations. As evident from the tables describing both amortized and non-amortized approaches, the number of coverage evaluations for amortized approaches is substantially larger. should be noted that, coverage analysis consisting of 300 posteriors of the non-amortized approaches took months on these relatively simple problems. While for the amortized methods, coverage analysis of 100,000 samples matter of hours to few days depending the dimensionality of 0.
C.1 Amortized
C.1.1 Neural Posterior Estimation
The MLP embeddings are 3 layer MLP's with 64 hidden units and a final latent space of 10, which is fed the normalizing flow. The CNN architecture in the Gravitational Waves benchmark consists of 13-layer deep convolutional head of ID convolutions with dilation factor of 2d, Where d corresponds to the depth of the convolutional head. The SELU [Klambaner et al. 2017) function is used as an activation function.
C.1.2 Neural Ratio Estimation
Our experiments use the ADAMW (Kingma & Ba 2015 Loshchilov & Hutter 2019) optimizer. Accross all benchmarks, the MLP architectures constitute of 3 hidden layers with 128 units and SELU KKammauuer et al.) 2017) activations. The Gravitational Waves benchmark uses the same convolutional architecture as NPE. The resulting embedding is flattened and fed to a MLP in which the dependence on the target parameter 0 is added. As before, the MLP consists of 3 hidden layers with 128 units.
16

---

Published in Transactions on Machine Learning Research (11/2022)
SLCP
M/G/1
Weinberg
Lotka-V.
Spatial SIR
GW
Streams
Embedding
MLP
MLP
MLP
MLP
MLP
CNN
MLP
Batch-size
128
128
128
128
128
64
128
Coverage samples individual
100,000
5,000
100,000
100,000
100,000
10,000
100,000
Coverage samples ensemble
20,000
5,000
20,000
20,000
20,000
5,000
20,000
Epochs
100
100
100
100
100
100
100
Model
NSF
NSF
NSF
NSF
NSF
NSF
NSF
Tronsforms
3
1
3
3
Learning-rate
0.001
0.001
0.001
0.001
0.001
0.001
0.001
Table 2: Architectures and hyperparameters associated with Neural Posterior Estimation.
SLCP
M/G/1
Weinberg
Lotka-V.
Spatial SIR
GW
Streams
Architecture
MLP
MLP
MLP
MLP
MLP
CNN
MLP
Batch-size
128
128
128
128
128
64
128
Coverage sumples individual
100,000
100,000
100,000
100,000
100,000
10,000
100,000
Coverage samples ensemble
20,000
20,000
20,000
20,000
20,000
10,000
20,000
Epochs
100
100
100
100
100
100
100
Leatning-rate
0.001
0.001
0.001
0.001
0.001
0.001
0.001
Table Architectures and hyperparameters associated with Neural Ratio Estimation.
C.2
Non-amortized
All our implementations of non-amortized approaches rely on the reference implementation in sbi Tejero Cantero al.) 2020). We use the recommended defaults unless stated otherwise. Whenever available, the same MLP embedding network is used. It consists of 3 hidden layers with 64 units and SELU (Klambauer eel al. 2017) activations. The latent space has a dimensionality of 10 features. For all sequential methods, we use 10 rounds to iteratively improve the posterior approximation. Tasks that are tagged with the prohibitive keyword are computationally prohibitive, but technically not intractable because the computational cost is tied to the learning of the posterior approximation and not to the underlying (intractable) likelihood model.
C.2.1 SNPE
Our evaluations with SNPE specifically use the SNPE-C (Greenberg et al. 2019 variant, suggested by sbi Cantero et al. 2020 To minimize inconsistencies between experiments, by the sbi authors unless states otherwise. Specific changes are highlighted in Table
the defaults suggested
SLCP
M/G/1
Weinberg
Lotka-V V.
Spatial SIR
GW
Streams
Batch-size
128
128
128
128
32
Prohibitive
Prohibitive
Coverage samples
300
300
300
300
300
Prohibitive
Prohibitive
Embedding
ML.P
MLP
MLP
MLP
MLP
Prohibitive
Prohibitive
Epochs
100
100
100
100
100
Prohibitive
Prohibitive
Fealures
64
64
64
64
64
Prohibitive
Prohibitive
Model
NSF
NSF
NSF
NSF
NSF
Prohibitive
Prohibitive
Transforms
3
3
3
Prohibitive
Prohibitive
Rounds
10
10
10
10
Prohibitive
Prohibitive
Learning-rate
0.001
0.001
0.001
0.001
0.001
Prohibitive
Prohibitive
Table 4: Architectures and hyperparameters associated with Sequential Neural Posterior Estimation.
17

---

Published in Transactions on Machine Learning Research (11/2022)
C.2.2 SNL
In contrast to other sequential methods, our evaluations with SNL (Papamakarios et al., 2019 add two addi- tional computationally prohibitive or Prohibitive benchmarks. the root of this issue lies the dimensionality of the observable. In both cases, the dimensionality of observables caused memory issues in SNL. In addition, training seperate embedding model (that requires additional simulations) is outside of the scope of this work. For this reason, consider the Lotka- Volterra en Spatial SIR benchmark to be Prohibitive.
SLCP
M/G/1
Weinberg
Lotka-V.
Spatial SIR
GW
Streams
Batch-size
128
128
128
Prohibitive
Prohibitive
Prohibitive
Prohibitive
Coverage samples
300
300
300
Prohibitive
Prohibitive
Prohibitive
Prohibitive
Epocks
100
100
100
Prohibitive
Prohibitive
Prohibitive
Prohibitive
Features
64
64
Prohibitive
Prohibitive
Prohibitive
Prohibitive
Model
NSF
NSF
NSF
Prohibitive
Prohibitive
Prohibitive
Prohibitive
Transforms
3
1
Prohibitive
Prohibitive
Prohibitive
Prohibitive
Rounds
10
10
10
Prohibitive
Prohibitive
Prohibitive
Prohibitive
Learning-rate
0.001
0.001
0.001
Prohibitive
Prohibitive
Prohibitive
Prohibitive
Table Architectures and hyperparameters associated with Sequential Neural Likelihood.
18

---

Published in Transactions on Machine Learning Research (11/2022)
C.2.3 SNRE
SLCP
M/G/1
Weinberg
Lotka-V.
Spatial SIR
GW
Streams
Architecture
MLP
MLP
MLP
MLP
MLP
Prohibitive
Prohibitive
Biteh-size
128
128
128
128
128
Prohibitive
Prohibitive
Coverage samples
300
300
300
300
300
Prohibitive
Prohibitive
Epochs
100
100
100
100
100
Prohibitive
Prohibitive
Fentures
64
64
64
64
64
Prohibitive
Prohibitive
Rounds
10
10
10
10
10
Prohibitive
Prohibitive
Learning-rate
0.001
0.001
0.001
0.001
0.001
Prohibitive
Prohibitive
Table 6: Architectures and hyperparameters associated with Sequential Neural Ratio Estimation.
C.2.4 Approximate Bayesian Computation
Our ABC implementation relies on the MCABC and SMCABC classes the sbi (Tejero-Canters Canerro et al. 2020) package. The specific settings from Rejection ABC and SMC-ABC are described in Tables 7 and S respectively. The quantile specifically refers to the proportion of closest samples that were kept in the final posterior. Because our specific implementation of coverage requires the ability to describe the posterior density function, we relied on Kernel Density Estimation to estimate the posterior density from the accepted samples.
SLCP
M/G/1
Weinberg
Lotka-V.
Spatial SIR
GW
Streams
Coverage samples
300
300
300
300
300
Prohibitive Prohibitive
Prohibitive Prohibitive
Quantile
0.01
0.01
0.01
0.01
0.01
Table Hyperparameters associated with Rejection Approximate Bayesian Computation.
SLCP
M/G/1
Weinberg
Lotka-V
Spatial SIR
GW
Streams
Coverage samples
300
300
300
300
300
Prohibitive
Prohibitive
decoy
0.5
0.5
0.5
0.5
0.5
Prohibitive
Prohibitive
Quantile
0.01
0.01
0.01
0.01
0.01
Prohibitive
Prohibitive
Table 8: Hyperparameters associated with Sequential Monte Carlo Approximate Bayesian Computation.
19

---

Published in Transactions on Machine Learning Research (11/2022)
D Predictive performance of the approximate posteriors
NB8
NIR snweriiie
NBR baggeayy
NPE
XPS semmmille
XPP hagghig
SNIPE
S5L.
SNGE
Da-ARR
SMC-ARC
SIATE
Londo Vidtom
pastidio
Spatial SER
Gecyedirds
pridiatie
CWV
Corpetetsnaly
Capela sndy
pailidie
priles
paitis
217
200
18
10
10
11
C.73
Figure 4: Evolution of the predicted log posterior probability of the nominal parameters with respect to the simulation budget. We evaluate the log posterior probability 100, 000 test samples for amortized algorithms and 300 for non-amortized algorithms. We report the 0.25; 0.5; 0.75 quantiles of this quantity over the test set. We observe that the predictive performance increases in general with the simulation budget. Exceptions are the MG1 benchmark with the SNRE algorithm and the lowest quantile of the streams benchmark.
20

---

Published in Transactions on Machine Learning Research (11/2022)
E Additional results
0
11.36
0,000
11120
0.50
0,80
0.50
0.94
0.875
0.52
0.850
0.00
0.75
0.70
0.75
0.7
0.63
0.6
1180
0.75
0.70
01.50
0.75
0.70
01:30
02.70
0.63
01.00
0.63
0.60
0.55
0.500
0.00
0.50
0.00
0.45
20
10
21
20
0.55
0.5
0.45
0.4
1..0
0.00
0..5
0.65
1..0
0.55
1..0
0.50
0.55
0.50
0.43
0.50
0.45
0.00
0.45
0.40
0333
0.40
0.35
0.30
0.50
0.45
0.30
0.45
0.40
0.35
0.40
0.35
0.30
0.25
0..35
0.30
0.25
0.25
0.00
0.35
0.20
0.15
20
1004
2008
4006
S122
1634
37768
65536
180072
Figure 5: Evolution of the empirical expected coverage of deep ensembles with respect to ensemble size for various confidence levels. The results are obtained by training ratio istioos (NRE) on the SLCP benchmark. A positive effect is observed in terms of empirical expected coverage and ensemble size, i.c., larger ensemble size correlates with larger empirical expected coverage. This is unsurprising, because a larger ensemble is expected to capture more of the uncertainty that stems from the training procedure.
21

---

Published in Transactions on Machine Learning Research (11/2022)
d 0.95
1- a.
1-a 0.85
a 0.8
0.005
11925
0.000
0.55
0.96
0.0055
0.67
0.890
0.50
0.94
0.525
0.000
0,830
0.50
0.70
0.770
0.75
0.65
0.6
0.50
0.70-
0.70
0.55
0.75
0.65
0.70
0.65
0.00
0.30
0.65
0.00
0.50
0.45
21
20
0.55
1- 0.5
0.45
0.4
0.663
0.50
11.30
0.00
0.50
11.45
0.55
0.50
0.43
0.40
0.00
0.43
0.40
0.30
0..00
0.35
0.25
0.35
a
0.25
0.35
0.45
11.01
Engittal coerage
0.,00
0.50
11.20
0..3
01300
0.25
0.20
0.30
0:15
1024
2098
0156
S192
16381 22708
05536
18062
-...
Figure 6: Evolution of the empirical expected coverage of bagging ensembles with respect ensemble size for various confidence levels. The rosults are obtained by training ratio estimators (NRE) the SLCP benchmark. A positive elfect is observed in terms of empirical expected coverage and ensemble size, i.e., larger ensemble size correlates with larger empirical expected coverage. This is unsurprising, because a larger ensemble is expected to capture more of the uncertainty that stems from the training procedure.
22


Reviews:
[{"review_id": "biekCJmrbL", "reviewer": "Reviewer_dket", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Overall I found the paper to be well-written. Most of the claims made by the authors are supported by the experiments. However, there are a few weaknesses that need to be addressed.\n\n1. The paper does not adequately consider – both in the discussions and the experiments – the significant amount of activity in the statistics literature on likelihood-free methods. In particular, there have been substantial methodological and theoretical advances in ABC and synthetic likelihood. For example, here is a short, very incomplete list: \n\n* Frazier, D. T., Nott, D. J., Drovandi, C. & Kohn, R. Bayesian Inference Using Synthetic Likelihood: Asymptotics and Adjustments. J Am Stat Assoc (2022). \n* Frazier, D. T., Robert, C. P. & Rousseau, J. Model misspecification in approximate Bayesian computation: consequences and diagnostics. Journal of the Royal Statistical Society: Series B (Statistical Methodology) 162, 2025–24 (2020).\n* Li, W. & Fearnhead, P. On the asymptotic efficiency of approximate Bayesian computation estimators. Biometrika 105, 285–299 (2018).\n* ​​Karabatsos, G. & Leisen, F. An approximate likelihood perspective on ABC methods. Statistics Surveys 12, 66–104 (2018).\n* Frazier, D. T., Martin, G. M., Robert, C. P. & Rousseau, J. Asymptotic properties of approximate Bayesian computation. Biometrika 105, 593–607 (2018).\n* Moral, P. D., Doucet, A. & Jasra, A. An adaptive sequential Monte Carlo method for approximate Bayesian computation. Statistics and Computing 22, 1009–1020 (2012). \n* Fearnhead, P. & Prangle, D. Constructing summary statistics for approximate Bayesian computation: semi‐automatic approximate Bayesian computation. 74, 419–474 (2012).\n* Blum, M. G. B. Approximate Bayesian computation: A nonparametric perspective. Journal of the American Statistical Association 105, (2010).\n  \nThe basic ABC method used in the comparisons is known to be very problematic. Far better alternatives such as MCMC-ABC and SMC-ABC are available. Moreover, techniques like regression adjustment can substantially improve inferences. \n\n2. As the authors point out at the top of p. 4, in addition to having well-calibrated credible sets, it is also crucial to consider the size of such sets to determine if any meaningful inferences are being drawn. The authors try to sidestep this question by placing it out of scope of their paper. However, I do not think this can be done: to make the comparisons meaningful, something must be said about whether these methods are, in fact, learning anything from the data! While attempting to estimate something like the expected information gain presents significant challenges, there are simpler metrics that would provide substantial insights. For example, using the results of the existing experiments, it is trivial to compute the volume of the credible regions of each method. \n\n3. The ensemble method used is not clearly described, to the degree that I’m unsure exactly what was done. It seems that the proposed ensembling technique may be very similar to BayesBag or various other bootstrapping methods:\n\n* Pompe, E. Introducing prior information in Weighted Likelihood Bootstrap with applications to model misspecification. arxiv.org arXiv:2103.14445v2 [stat.ME] (2021).\n* Huggins, J. H. & Miller, J. W. Robust Inference and Model Criticism Using Bagged Posteriors. arxiv.org arXiv:1912.07104 [stat.ME] (2019).\n* Lyddon, S. P., Holmes, C. C. & Walker, S. G. General Bayesian updating and the loss-likelihood bootstrap. Biometrika 106, 465–478 (2019).\n* Lyddon, S. P., Walker, S. G. & Holmes, C. C. Nonparametric learning from Bayesian models with randomized objective functions. In 32nd Conference on Neural Information Processing Systems (2018).\n  \n4. There are a few additional minor issues:\n\n* The abstract used numerous undefined acronyms\n* p. 1: “they are unfortunately not suitable for statistical inference​​.” This is false. They make statistical inference more challenging. \n* In the introduction it would be nice to list some applications outside of physics. \n* Matsubara et al. has been published. \n  \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper provides an empirical investigation of a variety of simulation-based (also called “likelihood-free”) statistical inference algorithms, which are used when (a) the mode likelihood cannot be computed but (b) data can be simulated from the model given the model parameters. Simulation-based inference methods are used in a range of scientific applications ranging from physics to ecology to population genetics. The paper is particularly concerned with the calibration properties of simulation-based algorithms: are the credible regions of the algorithms well-calibrated when the true parameter is drawn from the prior? Using a range of toy and realistic models, the authors show that all of the methods considered could significantly underestimate parameter uncertainty. While non-amortized methods were cheaper to run for a single problem, the authors argue that amortized methods are easier to validate in the kind of simulation-based framework they develop, where many datasets are generated from the model conditional on parameters sampled from the prior. ", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "1. Include comparisons to a more modern version ABC and synthetic likelihood \n2. Discuss how the results relate to existing theory for ABC and synthetic likelihood methods, and correct the claim that “advances in the field [of simulation-based inference] are mainly driven from a machine learning perspective.” \n3. Include results on the volume of the credible regions for each method.\n4. Clarify the ensembling method used and its connection to existing Bayesian flavored bootstrapping/bagging methods. \n5. Address all minor issues \n", "broader_impact_concerns": "I have no concerns ", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "cBFFgjI7PU", "reviewer": "Reviewer_zzma", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- This work is well motivated and presented. It’s easy to read and the conclusions are clearly presented and discussed. The details of experiment setting and code are provide, which is good for reproduction.\n- The summarised osbervations are well supported by the experiments. The authors do not over-claim their contributions.\n- The studied empirical coverage metric could be a useful tool to assess Bayesian estimators in additional to transitional metrics.\n\nWeaknesses:\n- The risk of excluding high probable regions of the parameter in inference is not a new problem. The proposed empirical coverage probably is the expected frequentist coverage probability as discussed by the authors below Equation 3. I would expect the authors to discuss how proposed method is related to and contribute to existing work along this ling of study in the literature later in the paper but didn’t find any.\n- While the over-confidence is different from accuracy to assess an estimator, they’re still correlated. As an estimator becomes more conservative, the size of the credible interval and the divergence of its posterior from the true distribution will become larger. I’m worried that comparing esitmators wrt coverage probability only may give a delusion especially when the credible interval can be adjusted post-hoc by a different confidence level. Providing other metrics, e.g., size of credible region / AUROC / divergence along with coverage seems important as it provide a more comprehensive look in the main figures.\n- In page 9, the authors discussed that using \\epsilon > 0 doesn’t always correspond to conservative approximations. Can they show any experiments to support it? Why not present the best found conservative estimators but using the naive identity function? I guess there is tradeoff between conservativeness and accuracy for \\epsilon, but that tradeoff exists when comparing other estimators too.\n\nThere are also a few minor concerns/comments\n- How are the hyper-parameters of each method chosen? Is the coverage sensitive to the choice of hyper-parameters?\n- How is ensemble bagging different from ensemble? Some more explanation would be appreciated.\n- It may be useful to make it clear when talking about the main conclusion/observations that the empirical studies are restricted to low dimensional parameter spaces. The conclusion may not extend to high dimensional spaces.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper studies empirically the coverage of credible intervals from Bayesian simulation-based approximate inference algorithms. Providing a sufficiently wide or conservative credible interval is more useful for scientific use cases than an over-confident estimator because the risk of excluding the truth value of the underlying parameter is much higher than including providing an conservative range. Based on this observation, the authors conduct empirical studies to estimate the over-confidence of recent simulation-based Bayesian inference methods on 5 synthetic and 2 real scientific problems. They observe that (1) all estimators can be over-confident for some problem and (2) amortised methods tend to provide more conservative estimators than non-amortised estimators (3) the empirical coverage of ensemble models is larger than individual models and (4) amortised methods are more simulation-efficient than non-amortised methods.\n\nThis paper provides an important perspective to evaluate the conservativeness of Bayesian inference methods for scientific discoveries that is orthogonal to the commonly used diagnostic metrics  such as the divergence of the posterior distribution. I think this makes a nice contribution to raise the awareness of practitioners for risks in applying ML estimators in this application domain. The empirical observation provides a good step towards developing conservative estimators with guarantees.\n", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "Listed above in the weaknesses.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "rgxNMCupvy", "reviewer": "Reviewer_YGWM", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n\n1. The paper delivers an important message that can influence significant amount of methodological development and that provides practitioners useful empirical information. It is important to address the faithfulness of posterior approximations and methods papers should do this better.\n\n2. The authors are very transparent in explaining how the empirical basis of the work required costly computation, which nicely justifies the need to publish the work to prevent others from needing to repeat similar effort to select a suitable algorithm for their specific case.\n\n3. The paper is very well written and is a pleasure to read. Difficult concepts are nicely illustrated with examples, the somewhat complex inference algorithms are described with brief paragraphs that introduce the main idea without trying to unnecessarily explain the details here.\n\n4. The technical development is sound; the chosen measures and the empirical setups are reasonable, the evaluated inference methods are a representative collection, and the benchmarks are diverse and interesting. I cannot pinpoint any specific weaknesses in the execution.\n\n**Weaknesses:**\n\n1. Too bold title.\n\n2. Previous work on evaluating posterior quality from this perspective in context of standard inference algorithms could be better integrated into the story.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper provides empirical evaluation of a broad range of simulation-based inference methods from the perspective of whether they are overconfident or conservative. The authors propose a technically sound framework for evaluating this and then run a large collection of evaluations to provide an overall summary. In general, the methods are found to be overconfident, and this observation is lifted as the main conclusion of the work as an alert for the practitioners and methods developers. The authors also indicate that ensembles are better in this respect, and that simulation-efficient amortized methods perhaps have higher potential for sensitive applications.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "The paper does not directly require any significant changes for publication (with the possible exception of the title), but I feel that addressing the following aspects would improve the paper by making the limitations and previous work more transparent and by making the value of the benchmark environment more visible for the community:\n\n1. I feel that the title is too much of a click-bait and should be changed. It is a good idea to explicitly write \"can be unfaithful\" in the title to deliver the message directly, but I see no reason for calling it a 'crisis' or even using the word 'beware'. Your results do not show a surprising finding that undermines previous work, but rather just quantify better a behavior most people already expected and even show that many of the methods work reasonably well even though they are not perfect. Hence, calling the current situation 'crisis' is not justified.\n\n2. You mention the parallel literature focusing on evaluation (of faithfulness) of standard inference algorithms only in Section 4. I would recommend mentioning this line of work for a highly related task already in Introduction. When I was reading the paper I was starting to get worried you are ignoring it completely. For example, Section 2.2 now implicitly suggests that looking at the posterior quality from the perspective of credible regions would be a novel idea, since you have not mentioned any previous approaches using the same principle in context of standard MCMC methods. You should avoid giving such impressions, especially when the value or your work does not depend at all on whether the specific measures are new or not. Finally, it would be good if you could more directly say whether you feel the issue of overconfident posterior estimates is more severe for simulation-based inference methods or whether practitioners should be aware of the same risks also for standard MCMC methods. This would naturally be done at the level of discussion, saying what kind of results others have found in past literature, not by actually running new experiments.\n\n3. All of the benchmarks are low-dimensional; it seems the largest has 5 parameters. I understand this choice for a simulator-based inference paper that focuses on scientific validation of theoretical models, but it would still be useful to say something about models with larger parameter space. Would you expect the situation to get even worse if having 10+ parameters, or would it be better to indicate that this would need proper evaluation? Also, it would be good to indicate the scope of your empirical evaluation already in Introduction, explaining that you evaluate the methods in context of problems of specific dimensionality and explaining why this is the case.\n\n4. One of your conclusions is that ensembling works well. I feel that you could emphasise the method a bit more already in the technical section 3.1.1. Now you have only a four-line summary of the idea, with no citations or motivation. If this is to be interpreted as a novel contribution of the work, I would change the writing here to explicitly say so and to provide a proper motivation and explanation for the method. If not, you should add references to previous works that have constructed such ensembles (in simulation-based inference or otherwise).\n\n5. I see you provided code in a Supplement and the large file size indicates you must include some simulation results there as well, but the supplement is not very well documented and the manuscript does not state what is provided there. I recommend making this explicit. If you are making available both the simulation results (for which you spent significant amount of computation) and the code for running the evaluation metrics for new methods, then you are giving the community an extremely valuable resource since those enable evaluating faithfulness of future methods in a nice quantifiable benchmark. That would definitely be worth advertising better.\n", "broader_impact_concerns": "The paper contributes by making limitations of core inference algorithms more transparent and hence has only positive broader impact; the results may help e.g. identifying scientific results or practices that are unreliable. This is communicated well throughout the paper and there is no need for addressing possible concerns in this respect.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors have already made many of the changes requested by the reviewers. I thank both the reviewers and the authors for their fruitful conversation during the review period.

I am following up on one reviewer who wishes to see a few minor changes:
- define the acronyms in the abstract
- update the final paragraph of section 2 to reflect the additional predictive accuracy experiments
- include results for both the deep ensemble and bagging in figures 4 and 5

With that I am happy to accept this manuscript and look forward to seeing it published.

Decision: Accept (poster)