Robust Neural Posterior Estimation and Statistical Model Criticism
Authors: Daniel Ward
Patrick Cannon
Mark Beaumont
Matteo Fasiolo
Sebastian M Schmon
Venue: NeurIPS 2022 Accept (2022)

Abstract: Computer simulations have proven a valuable tool for understanding complex phenomena across the sciences. However, the utility of simulators for modelling and forecasting purposes is often restricted by low data quality, as well as practical limits to model fidelity. In order to circumvent these difficulties, we argue that modellers must treat simulators as idealistic representations of the true data generating process, and consequently should thoughtfully consider the risk of model misspecification. In this work we revisit neural posterior estimation (NPE), a class of algorithms that enable black-box parameter inference in simulation models, and consider the implication of a simulation-to-reality gap. While recent works have demonstrated reliable performance of these methods, the analyses have been performed using synthetic data generated by the simulator model itself, and have therefore only addressed the well-specified case. In this paper, we find that the presence of misspecification, in contrast, leads to unreliable inference when NPE is used naïvely. As a remedy we argue that principled scientific inquiry with simulators should incorporate a model criticism component, to facilitate interpretable identification of misspecification and a robust inference component, to fit ‘wrong but useful’ models. We propose robust neural posterior estimation (RNPE), an extension of NPE to simultaneously achieve both these aims, through explicitly modelling the discrepancies between simulations and the observed data. We assess the approach on a range of artificially misspecified examples, and find RNPE performs well across the tasks, whereas naïvely using NPE leads to misleading and erratic posteriors.

Robust Neural Posterior Estimation and Statistical Model Criticism
Daniel Ward Patrick Cannon
Mark Beaumont Sebastian M. Schmon 2.4
Mattco Fasiolo
School of Mathematics, Bristol University, UK 'Improbable, UK
School of Biological Sciences, Bristol University, UK *Department Mathematical Sciences, Durham University, UK
Abstract
Computer simulations have proven valuable tool for understanding complex phenomena across the sciences. However, the utility of simulators for modelling and forecasting purposes often restricted by low data quality, as well as practical limits to model fidelity. In order to circumvent these difficulties, we argue that modellers must treat simulators as idealistic representations of the true data generating process, and consequently should thoughtfully consider the risk of model misspecification. In this work we revisit neural posterior estimation (NPE), a class of algorithms that enable black-box parameter inference in simulation models, and consider the implication of a simulation-to-realiic gap. While recent works have demonstrated reliable performance of these methods, the analyses have been performed using synthetic data generated by the simulator model itself, and have therefore only addressed the well-specified case. In this paper, we find that the presence of misspecification, in contrast, leads to unreliable inference when NPE is used naïvely. As remedy argue that principled scientific inquiry with simulators should incorporate a model criticism component, to facilitate interpretable identification of misspecification and robust inference component, to fit 'wrong but useful' models. We propose robust neural posterior estimation (RNPE), extension of NPE simultaneously achieve both these aims, through explicitly modelling the discrepancies between simulations and the observed data. We assess the approach find RNPE performs well across the tasks, whereas naively using NPE leads to misleading and erratic posteriors.
range of artificially misspecified examples, and
Introduction
Stochastic simulators have become ubiquitous modelling tool across the sciences and are regularly applied to some of the most complex and challenging problems of scientific interest, including climate change (see e.g. Randall et al., 2007), particle physics (e.g. Brehmer al., 2020), and the Covid-19 pandemic (e.g. Ferguson et al., 2020). Simulators implicitly define likelihood function p(z 0), where æ is the simulator output and 0 are the simulator parameters. Although running the simulator to sample from the model is straightforward, the inherent complexity of simulators often makes analytic calculation of the likelihood intractable. As result, classical inference techniques to find parameter posterior p(0) æ) such as Markov chain Monte Carlo (MCMC) (Metropolis et al., 1953) infeasible. To overcome this issue, large family of simulation based inference (SB1) methods have been developed that allow parameter inference to be performed on arbitrary black-box simulators (see Cranmer et al., 2020). Broadly, these approaches estimate a function that allows access to an
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

approximate posterior. This can be achieved by approximating the posterior directly (Papamakarios and Murray, 2016: Greenberg al., 2019; Lueckmann al., 2017), or indirectly, via approximating ratio (Hermans et al., 2020;
the likelihood (Papamakarios et al., 2019b) or ilkelihood-t-eevidene Thomas et al., 2022), from which the posterior can be sampled using MCMC. For estimating the posterior or likelihood, neural posterior estimation (NPE) and neural likelihood estimation (NLF) have shown to be powerful approaches (Lueckmann et al., 2021), which rely on neural network-bass conditional density estimators, such normalising flows, to approximate the likelihood or posterior (Papamakarios al., 2019a).
Denoising
Model Criticism
Simulations
0.43
Denoised
272
0.94
Observed
Posterior misspecificatiin probability
32
Posterior Inference
RNPE
g(fl<z)
NPE
True
21
Figure I: Overview of the robust neural posterior estimation (RNPE) framework. Through 'denoising' observed data, we can simultaneously perform model criticism to identify how the model is misspecified, and perform inference robust to model misspecification. See Gaussian example in section 4.1 for details of the experiment.
To define misspecification, it necessary make a clear distinction between the true data generating process and the simulator: we will write for observed data, following some unknown data generat- ing process with distribution denoted p", and we will write æ p(æ: () to denote samples from the simulator. In many applications of SBI, summary statistics are used to reduce the dimensionality of the problem. this is the case, we use x and yo denote the simulated and observed summary statistics, rather than the raw data, and treat the raw data an unobserved latent variable of the model. The simulator said be misspecified the true data generating process does not fall within the family of distributions defined by the simulator, i.e. p" & {p(æ 0); O E {). Simulators gener- ally form idcalised and simplistic representations of a more complex data generating process, and hence like all statistical models will be wrong to some extent. This misspecification often results discrepancy between simulated and observed data, sometimes termed "simulation to-reality gap" (e.g. Miglino et al., 1995). Despite this, the general approach in SBI is to learn a posterior approximation ( æ) (either directly, or indirectly), using simulated data, (x, 0) ~ p(æ: ())()) and to condition the observed data, q yo). This approach is only theoretically justified under the assumption that the simulator well-specified; relying on this approach for misspecified simulators leads
issues:
(i) As the posterior approximation g x) is learned using samples from the simulator, we can only expect it to form reasonable approximation of p(O x)) for regions well covered by simulations. For misspecified simulators, the observed data ye may be very unlikely in p(z) p(x 0 (O))))) in which case the approximation q( poor (Cannon et al., 2022). In some cases, yo may even fall outside the support of p(æ), which results in NPE attempting to extrapolate estimate the undefined posterior.
yo) often becomes
(ii) It has been demonstrated more generally that Bayesian inference is frequently problematic under misspecification. Even for simple models, misspecification can lead to posterior con- centration around "bad" models (Grünwald and Langford, 2007; Grünwald and Van Ommen, 2017) and poorly calibrated credible regions (Kleijn and van der Vaart, 2012: Syring and

---

Martin, 2019). These issues will persist even in the unlikely case that q(@ approximates p(@) x))
x) perfectly
One approach to improve robustness is to incorporate an error model to explicitly model potential discrepancies between the observed data and simulations. A simple method would be to directly incorporate the error model within the simulator itself, such as by applying additive noise to the output ensure that the observed data has reasonable marginal likelihood under the model (Cranmer et al., 2020). However, this approach has some limitations: i) it is challenging to interpret how the model misspecified an obvious choice is use posterior predictive checks, but poor predictive performance could result from failed inference, limitations of simulator, or both; ii) retraining of the approximate posterior would be required to investigate different error models, which can be computationally expensive; and iii) the eπror model itself is likely tractable, which can be utilised to improve inference.
Our contribution. We propose robust neural posterior estimation (RNPE), an SBI framework that incorporates an error model, p(y x)) to account for discrepancies between simulations and the observed data, whilst avoiding the limitations of naively incorporating it directly into the simulator. We explicitly learn to invert the eπror model for the observed data prior to inference of the parameter posterior, process we call 'denoising'. This approach yields the following advantages:
Efficient Model Criticism. Understanding a model's limitations is crucial for principled model development (Box, 1980; Gelman and Shalizi, 2013). By probabilistically inverting the error process for the observed data, and inferring any associated error model parameters, we can detect and interpret discrepancies between the simulated and observed data, allowing criticism of the simulator. This process can achieved in a manner decoupled from inference, which provides efficiency benefits, whilst avoiding confounding inference failures with problems arising due to limitations of the simulator.
Robust Inference.
Even after several rounds of model development and model criticism, the best available model will still frequently be misspecified to an appreciable degree. However, is often still desirable to use 'wrong but useful' models for parameter inference and predictions, and hence the inference process should be robust misspecification. By exploiting the amortisation' property of NPE, we can ensemble posterior estimates over set of 'denoised" observations, leading to posterior estimates robust to misspecification.
An overview of the RNPE framework can be seen in Fig. 1. We examine the practical value of this approach for model criticism and robust inference on examples in which we artificially introduce realistic levels of misspecification.
2 Method
2.1
Error Model
As outlined above, we can explicitly model discrepancies between the observed data and simulations by introducing error model p(y æ), such that the assumed generative model is
p00, x. 3) p(y x)p((= ())(
(1)
where we assume that the error is independent of the simulator parameters 0, and we treat æ as an unobserved latent variable of the model. Using the equivalent factorisation
p(6 0, x, y) p(0 x)p(< y)p(y),
we can marginalise over the latent æ and find an expression for the parameter posterior
p(0. y) = =r()) / p( "))p( x dæ,
(2)
p(0) y) = f p(az v)p(O æ)) dar Be-p(al) w)).
=
Equation (3), implies that if we had access to p(æ: y) and p(O x)) could sample from the parameter posterior by first sampling from the posterior over the latent variables x ~ p(z y), and
'Amortisation refers to the ability of conditional distribution approximation to condition on arbitrary instances of the conditioning variable, rather than being specialised to particular instance.
3

---

then sampling from p(0 æ). This approach reverses the data generating process from Equation (I), whilst propagating uncertainty throughout, first removing the error from the observed data by denoising, and then finding the associated simulator parameters given the denoised data. A Monte Carlo approximation of the expectation Equation (3) can also be used to estimate the posterior density p(0) y), i.e.
M
0 Øm); pO1 Σ
Sid B1...., HM, ~ p( y);
however, we note that in some scenarios this approximation could have high variance. In practice, p(æ y) and p(0 x) are unknown. We can approximate p(0 æ)) by training normalising flow q(0 æ)) on simulated data (x, 0) (a p))))) as is commonly done for NPE. To approximately sample from p( y), we speciff error model ply x), ii) train a normalising flow, ((x). fitted to samples from the prior predictive distribution of the simulator x ~ p(z), and iii) sample p(z y), along with any emror model parameters, using MCMC, as p(x y) ply x) x))(x). We denote samples from p(ær y) and ( x) as o and 0 avoid confusion with prior samples and simulations. Pseudo-code for the overall approach is given in Algorithm 1.
Algorithm 1: Robust neural posterior estimation (RNPE)
for i N do
Sample 0: ~ p(0)
Simulate x; ~ p( 0) end
Train NPE q(0 x) on (0),
Train g(z) on {x
Sample 2. p(a yo) ply a))(a),, m = I,..., M using MCMC
Sample On ~ (0 ãm), m I...., M
return [10m. Rmm) m=1- samples drawn approximately from p(0, x yo)
Under this framework, the standard SBI approach can be retrieved as the special case in which ply æ) p( y) 6(x - y). where 5 is the Dirac delta distribution, meaning that we revert to assuming no discrepancy between the simulator and data generating process.
The idealised nature of simulation models implies that frequently there will be some characteristics of the observed data that are well captured by simulations, and some aspects which there exists discrepancy. Using a set of summary statistics for inference, rather than the raw data naturally captures an interpretable, low dimensional and diverse set of characteristics of the simulated and observed data. Because of this, we find that in general, reducing the raw data to summary statistics is advantageous for model criticism, as the practitioner can choose summary statistics which align with their belief about important model characteristics, and easily interpret any discrepancies (see Section 5). The requirement for domain specific knowledge to develop summary statistics has been alleviated by recent work (e.g. Fearnhead and Prangle, 2012; Chan al., 2018; Chen al., 2020), which seek to automatically embed the raw data to summary statistics; however, these embeddings often lack tangible interpretation. Additionally, the embedding methods themselves may not be robust to misspecification. One approach is to use both hand-crafte and automatic summary statistics in tandem. Hereafter, however, generally perform model criticism and inference using hand-crafter summary statistics, such that x and yo refer to the simulated and observed summary statistics, not the raw data.
2.2 Spike and Slab
In practice, the true error model is unknown, and hence the error model should describe our prior belief over the discrepancy between the data generating process and simulations. Additionally, to facilitate model criticism, it should ideally allow easy assessment of which summary statistics are approximately well-specified and which are misspecified. Inspired by sparsity-idducing priors used in Bayesian regression literature (George and McCulloch, 1993), we use a "spike and slab" error

---

model on each summary statistic
x q(z),
(5)
2j ~ Bernoulli(p),
j == I,...,. D
(6)
Nxx 09),
if 2 0 if 2
Vj Tg,
1....,
Cauchy(zz,,7)
where a TRD. As the summary statistics have varying scales, we standardise them to have mean zero and variance one. The Bernoulli variables indicate whether the j-th summary statistic is approximately well-specified; we use the probability of p = 0.5 to express the belief that it is equally likely summary statistic will be misspecified or well-specified priori. When a summary statistic is well-specified (i.e., 2j = 0), we take the eπror distribution to be a tight Gaussian (a spike) 0.01 to enforce consistency between the simulator and the observed data for the j-th statistic. In contrast, in the misspecified case (i.e., 2j = 1), we take the error model to be the much wider and heavy tailed Cauchy distribution (a slab). Generally, misspecification to be relatively subtle, but some model inadequacies can lead to catastrophically misspecified summary statistics. We chose the Cauchy scale т = 0.25, to reflect this, it places half the mass within +0.25 standard deviations of Ty but the long tails accommodate summary statisties that are highly misspecified. The sparsity-inducing effect of this error model matches what we consider to be a common scenario, namely that a proportion of the summary statistics are jointly consistent with the simulator, whereas others may be incompatible. If we expect a priori more or fewer of the summary statistics to be misspecified, the prior misspecification probability can be adjusted accordingly. By marginalising over z, the spike and slab error model can also be written as
centred x, choosing
might expect
D
ply x)) II [(1 - p) plus m 2j 0) + p p(us Øj 1)]
and as such can be viewed an equally weighted mixture of the Gaussian spike, p(y; z5 = 0), and the Cauchy slab, p(y; 2z. 1), for each summary statistic. We note that error model hyperparameter tuning approaches could be considered, e.g. a reasonable heuristic would be to choose an eтror model that is broad enough that the denoised data 전 tend not to be outliers in g(x), compared to simulations However, we chose to keep the hyperparameters consistent across tasks, to avoid the risk of overfitting to the tasks and to demonstrate that neither strong prior knowledge on the error model, nor careful hyperparameter tuning is necessary to yield substantial improvements in performance.
A key advantage of the spike and slab error model is given by the latent variable z. Similar to posterior inclusion probabilities in Bayesian regression, the posterior frequency of being in the slab, Pr(2, = y). can be used as an indicator of the posterior misspecification probability for the j-th summary statistic. By comparing to the prior misspecification probability we can say that if Pr(2, = 1 3) > p. the model provides evidence of misspecification for the j-th summary statistic, and if Pr(z, = 1 y) < p. it provides evidence it is well-specified (Talbott, 201 1). For the purpose of generality, the latent variable z was not included when describing RNPF thus far. However, we can jointly infer the posterior $(xr, y) in step 5 of algorithm 1, by using an MCMC algorithm that supports sampling both continuous and discrete variables. We used mixed Hamiltonian Monte Carlo (IMC) (Zhou, 2020, 2022) from the NumPyro python package (Phan et al., 2019), which is an adaptation of IICC (Neal et al., 2011: I; Duane et al., 1987), for this purpose.
3
Related Work
3.1 Model Criticism in SB1
Posterior predictive checks have been widely used in SBI, compare the predictive samples the observed data (e.g. Durkan et al., 2020; Greenberg et al., 2019; Papamakarios et al., 2019b). Although this is form of model criticism, discovered, it may not be clear whether this is due to the failure of the inference procedure, or due to simulator misspecification. In general, it may be possible to identify the presence of misspecification using anomaly/novelty detection. One such approach was suggested by Schmitt et al. (2021) in the
note key limitation of this approach is that if discrepancy is
"This could for example be assessed by estimating highest density regions of qæx), using the density quantile approach from Hyndman (1996).
5

---

context of SBI. However, this approach does not facilitate interpreting misspecification in the sense presented here, and is not clear how to extend perform robust inference.
There are a few related methods that criticise a posterior estimate by assessing its calibration using simulated data (e.g. Hermans et al., 2021; Prangle et al., 2014; Talts al., 2018); however, these approaches criticise the inference procedure, not the simulator itself, and the associated results rely on the simulator being well-specified.
3.2 Robust Inference in SB1
Approximate Bayesian computation (ABC) is family of SBI methods characterised by their use of rejection sampling alongside a kernel function to compare the simulated and observed data (Tavaré et al., 1997: Pritchard et al., 1999; Beaumont et al., 2002). The simplest form of ABC samples candidate parameters from the prior distribution and accepts them with probability proportional to the similarity of the corresponding simulations the observed data. The use of kernel is often seen as a necessary evil, as enforcing an exact match between the observed and simulated data generally results in all samples being rejected. However, it has been noted that the kernel function can often be interpreted as specifying an error model distribution pyy x) (Wilkinson, 2013). This idea has been expanded using generalised Bayesian inference, in which the kernel is replaced with a loss function,
which can be chosen to use a robust measure of discrepancy such as the maximum mean discrepancy (MMD) Chérief-AAdellattee and Alquier. 2020; Frazier et al., 2020a; Fujisawa al., 2021; Park et al.. 2015; Schmon et al., 2020; Dyer et al., 2021).
Synthetic likelihood is an SBI method that uses a Gaussian approximation of the likelihood (Wood, 2010). Frazier and Drovandi (2021) investigated a robust synthetic likelihood algorithm, in which the variance components of the Gaussian likelihood are expanded based additional free parameters, inferred a nested MCMC sampling scheme. In the context of the paper, they interpret this approach as "likelihood adjustment": however, this could equivalently be reframed as the addition of an independent Gaussian error model, with a prior on the variances*.
In contrast to more traditional SBI methods, few solutions have been proposed for performing robust inference with neural SBI methods. Perhaps the most common approach is to augment the model with additional nuisance parameters to expand the range of values the simulator can reasonably produce (Cranmer et al., 2020). This again can often be interpreted as a method of incorporating an error model within the simulator. However, as previously discussed, the primary limitation of this approach is that it does not easily facilitate model criticism.
4 Experimental setup
On three misspecified tasks, we benchmark the performance of three methods for parameter inference: RNPE, üi) noisy neural posterior estimation (NNPE) in which we directly approximate p(0) | y) by incorporating the error model into the simulator, and iii) NPE, in which we assume no model error present. To reliably assess performance, we repeat the inference procedure with 1000 different observations and ground truth parameter pairs for each misspecified task, and calculate two metrics: i) the log probability of the true parameter vector, 0- and ii) the posterior coverage properties (see Section 4.2). Further metrics, and results for the well-specified case, can be found in Appendix A. We further show examples of RNPF on a single observation for each task to demonstrate how RNPF can be applied in practice for model criticism and robust posterior inference.
For all experiments, 20, 000 warm up steps. The MCMC chains were initialised using a random simulation, and žj 1 for j = 1,..., D. To build the approximation ((x), we used block neural autoregressive flows (De Cao et al., 2020), as we do not require the ability to sample directly from g(æ:). For the approximation of q( x). we used neural spline flows (Durkan al., 2019). For all tasks the hyperparameters were kept consistent; information on hyperparameter choices can be found in Appendix C. The code required to reproduce all the results from this manuscript is available at https://githun com/ danielward27/rupe
used N = 50, 000 simulations, with M = 100, 000 MCMC samples following
4.1 Tasks
Detailed descriptions of each task can be found in Appendix B.
'This results due the fact that the addition of two independent Gaussian variables yields another Gaussian random variable, with covariance equal to the sum of the individual covariance matrices.

---

Gaussian. A simple Gaussian example from Frazier et al. (2020b). The task involves predicting a single parameter μ. using the sample mean and variance of 100 i.i.d. samples drawn from N N(,, 1). where is the variance of the Gaussian distribution. To artificially produce misspecified observations, the 100 i.idd are instead drawn from (1,, 2). Despite simplicity, this task usefully demonstrates that issues with misspecification can arise even in the simplest of models.
Gaussian Linear. A 10-d linear Gaussian model from Lueckmann et al. (2021). The parameter is the mean vector of a Gaussian likelihood. Additive Gaussian noise is used to introduce misspecifica- tion. For this task, the spike and slab error model employed will be reasonably misspecified, as the noise is not sparse or heavy tailed.
SIR. A stochastic model of epidemic spread with time-varying infection rate. We attempt infer two parameters, the infection rate 3 and the recovery rate y. We computed the following summary statisties: 1-3) Mean, Median and Max the mean, median and maximum number of infections; 4) Max Day- the day of occurrence of the maximum number of infections, 5) Half Day the day at which half the total number of infections was reached, and 6) Autocor the mean autocorrelation (lag 1) of infections. To artificially produce misspecified observations, we chose to introduce a small degree of reporting delays, whereby weekend infection counts are reduced by 5%, which are subsequently recouped on the following Monday, Fig. 4d shows an example of a raw observation.
CS. A marked point process model of cancer and stromal (CS) cell development in 2D-spac, in which the locality of a cell relative to unobserved parent points determines whether it is a cancer or a stromal cell. Similar point process models can be found in Jones- -ood et al. (2019). There are three Poisson rate parameters, corresponding to the cell rate Ae. the parent rate Ap and the daughter rate Aa where aa controls the number of nearest cells to each parent which become cancerous. The following summary statistics were used: 1-2) N Cancer and N Stromal- - the number of cancer and stromal cells; 3-4) Mean Min Dist and Max Min Dist the mean and maximum distance from stromal cells to their nearest cancer cell. To artificially produce misspecified observations, cancer cells were removed if they fell close to a parent point, mimicking necrosis that often occurs in core regions of tumours. A plot of an example misspecified simulation and corresponding artificially produced observation can be found in Fig. 9 in the appendix.
4.2 Performance Metries
Log probability of 0". The mean posterior log probability of the true parameters over multiple observations is an extensively used performance metric (e.g. Papamakarios and Murray, 2016; Hermans al., 2020; Durkan al., 2020), and is closely related to the average KL divergence between the true and approximate posteriors (Lueckmann et al., 2021). We found that NPE would occasionally fail catastrophically, placing negligible posterior mass on the true parameters, skewing the mean estimates. Due to this, we instead use box plots to visualise the distribution over the preprbbbbblities of the true parameters, rather than reporting the mean.
Posterior Coverage. Given a confidence level a and a posterior approximation p(0 y). let HDRRpoly)) (1 a) represent its highest posterior density region, i.c. the smallest region that contains at least 100(1 3 of the mass of p(0 y). The expected posterior coverage is the frequency with which the true parameter value falls within this highest density region
Eg (yx((0))) [1(0- HDR (1-a)))))))) a)}]
where 1 is the indicator function. Posterior coverage is useful for assessing whether an inference procedure is likely to yield overconfident or conservative posterior estimates (Hermans et al., 2021). We empirically estimate this expectation, approximating the highest density regions using the density quantile approach outlined in Hyndman (1996), with 10,000 samples from the approximate posterior.
5
Results and Discussion
5.1 Overall Performance
The results shown in Fig. 2 show that both RNPE and NNPE perform better than NPE, both in terms of producing posteriors with more conservative coverage properties (Fig. 2a) and by tending to place more mass on 0" (Fig. 2b). It is important to note that having conservative posterior estimates is generally considered preferable to having overconfident posterior estimates, as the latter could lead to drawing of erroneous scientific conclusions. RNPF. slightly outperforms NNPF terms of the mass placed on 0". We hypothesise this increased performance is due to RNPE exploiting the fact that the
7

---

Empirical Coverage
(a)
Gaussian
Gaussian Linear
SIR
CS
0.5
RNPE
10
0.5
0.5
0.5
Confidence Level
NNPE
NPE
(b)
Gaussian
Gaussian Linear
SIR
Log Probubility 8
-20
Figure 2: Comparative performance of RNPE, NNPE and NPE the four misspecified tasks, using 1000 different observation and true parameter pairs. (a) The coverage of each approach at different confidence levels. A well-calibrated posterior follows the dotted line, with conservative posteriors lying above the dotted line, and overcontident posteriors falling below the dotted line. (b) The hog-robbability of the true parameters O in the approximate posteriors.
error model known and tractable, rather than naively relying on NPE to account for it. For all the tasks the error model is misspecified: these results highlight the importance of accounting for model misspecification, even if doing so may introduce misspecified error model.
5.2 Examples
Gaussian. An example for the Gaussian task can be seen in Fig. in which di corresponds the mean statistic, and 22 the variance. From observing the posterior misspecification probabilities and the denoised samples, the results clearly identify discrepancy in which the observed variance statistic is higher than the simulated variance statistics, allowing identification of source of misspecification As this is simple example, can analytically access the posterior (obtained using the true data generating model). RNPE produces a posterior close to the ground-truth, whereas NPE confidently places most of the posterior mass far from the ground-truti posterior.
Gaussian Linear. For the Gaussian Linear task, misspecification was introduced by applying additive Gaussian noise. Due the lack of structure (e.g. correlations between simulation output dimensions), and the lack of sparsity in the errors, the model generally cannot provide much evidence for or against misspecification, as shown by Fig. 3. Corresponding pair plots of the denoised data and posterior are shown in Figs. 10-11, 11, in the appendix.
Xs
X7
X4
X2
X6
Xo
X3
X8
SIR. In the SIR model, the observed data were artificially corrupted by mimicking reporting de- lays over the weekend. With this in mind, we can consider how the model criticism aspect of RNPE could facilitate identification of the issue. Fig. 4a shows that most summary statistics are more likely to be well-specified than was be- lieved a priorí, with the clear exception the Autocor summary statistic, which is inferred to be misspecified. Fig. 4b shows density plot of the denoised data for the Autocor summary statistic (for full pair plot, see Fig. 12, in the appendix), showing that the observed autocorrelation much lower than would be expected under the model. This may prompt the practitioner to investigate the
X9
0.0 Posterior Misspscification Probability
0.5
1.0
Figure 3: Posterior misspecification probabilities
for an example of the Gaussian Linear task.

---

y). We show an example of this in day-to-day changes in infections, in which case the discontinuities introduced by the reporting delays could be identified. Fig. shows the posteriors for RNPE and NPE. RNPE yields posterior with far lower variance than NPE, demonstrating that it does not always achieve robustness simply by producing broader posterior estimates.
(a)
(b)
Median
Simulations
Mean
Denoised
Density
Max
Observed
Half Day
Max Day
Autocor
0.0
1.0
0,9900
0.9925
0.9950
0.9975 1.0000
Posterior Misspecification Probability
Autocor
(c)
(d)
Prior NPE
RNPE
NPE
0.4
True
Infections
RNPE
Observed
0.2
0.0
Days
0.0
0.2
0.4
B
Figure 4: The results for example of the SIR task. (a) The inferred posterior misspecification probabilities for each summary statistic. (b) Density plot for the denoised samples the most misspecified summary statistic (autocorrelation). (c) The posteriors for both NPE and RNPE and; (d) The associated posterior predictive distributions for NPE and RNPE. Note the (artificially introduced) reporting delays are visible for the raw observed data.
CS. In the CS model, the observed data corrupted by removing cells in the core regions of tumours, mimicking necrosis (see Fig. 9 in the appendix). This misspecification primarily impacts the Cancer summary statistic (the number of cancer cells). From Figs. 5a.5b, we can see that the results provide evidence that the N Cancer statistic is misspecified, with the observed data having fewer cancer cells than would be expected if it was generated from the simulator. Once again, alongside domain knowledge, this information points the practitioner to the source of misspecification (necrosis), and hence the model could be adjusted appropriately. For this example, NPE produced an overconfident posterior with little mass on the true parameters, whereas RNPE does not exhibit this issue (Fig. 5c).
5.3 Indeterminacy
In some cases, there can be indeterminacy when inferring misspecification, in which it is impossible to determine which summary statistic is misspecified, despite clear evidence of misspecification. This occurs, for example, when a model can only be consistent with one observed summary statistic, e.g. at the expense of being able to recreate another summary statistic, e.g. Z2 (or vice versa). RNPE allows these trade-off's be identified, as the statistics will show small frequency of being jointly well-specified, Pr(21 = 0, 22 = 0 y). alongside reasonable probabilities of either statistic being well-specified, Pr(z1 1, 22 = 0 y) and Pr(21 O, 22 Fig. 8 in the Appendix.

---

a
(c)
Max Min Dist Mean Min Dist
Prior RNPE
NPE True
Stromal N Cancer
Ap
0.0
0.5
1.0
Posterior misspecification probability
(b)
Ad
Simulations Denoised Observed
Density
dc
200
400
N Cancer
Ap
Ad
Figure 5: The results for an example of the CS task. (a) The inferred posterior misspecification probabilities for each summary statistic. (b) Density plot for the denoised samples of the N Cancer summary statistic. (c) pair plot of the posteriors for both and RNPF.
6 Conclusions
We described RNPE, an approach that can be applied to any black- box simulator to simultancously perform model criticism and robust inference. RNPE is philosophically similar to existing robust SBI approaches for ABC and synthetic likelihood (Wilkinson, 2013; Frazier and Drovandi, 2021). However, this work introduces three novel contributions: we apply the idea of using error models for robust inference to neural SBI method, which has advantages in simulation efficiency, flexibilityy and scaling to high dimensional problems (Lueckmann al., 2021), ii) we facilitate model criticism decoupled from inference, which avoids confounding of inference failures and model misspecification. and üii) we use spike-and-slab error model to obtain explicit misspecification probability diagnostics for summary statistics, to facilitate model criticism. Our work is not without limitations. Firstly, RNPE is more computationally costly than NPE, requiring an additional MCMC step and fitting of q(x). Secondly, only evaluated performance of single choice of error model limited number of tasks. would be beneficial the future to have a larger set of realistic misspecified tasks for assessing performance, similar to the work by Lueckmann al. (2021) for the well-specified case. Finally, we restricted our focus to NPE; in future work it would be interesting to consider other SBI algorithms and to compare results with existing methods for robust inference.
We demonstrated with four examples that RNPE consistently outperforms under misspecification, tending to put more posterior mass the true parameters whilst also exhibiting more conservative coverage properties. The tasks were designed to mimic levels of misspecification that could be realistically encountered, and hence the poor performance of NPE calls into question its use in practice, particularly when the level of misspecification could be significant. The substantial difference in performance highlights, suggested by Box (1980), that should not have blind faith that a model sufficiently accurate, and therefore must devise methods for performing model criticism and robust inference. With further research and wider adoption of these principles in SBI, we believe simulators will only become more powerful tools for scientific discovery.
for parameter inference
10

---

References
Atkeson, A., Kopecky, K., and Zha, T. (2020). Estimating and forecasting disease scenarios for COVID-19 with an SIR model. Technical report, National Bureau of Economic Research.
Beaumont, M. A., Zhang, W., and Balding, D. J. (2002). Approximate Bayesian computation in population genetics. Genetics, 162(4):2025-2035
Bingham, E.,, Chen, J. P., Jankowiak, M., Obermeyer, F., Pradhan, N., Karaletsos, T., Singh, R., Szer- lip. P., Horsfall, P., and Goodman, N. D. (2019). Pyro: Deep universal probabilistic programming. The Journal of Machine Learning Research, 20(1):973-977.
Box, G. E. (1980). Sampling and Bayes' inference in scientific modelling and robustness. Journal of the Royal Statistical Society: Series (General), 143(4):383-44)
Bradbury, J., Frostig, R., Hawkins, P., Johnson, M. J., Leary, C., Maclaurin, D., Necula, G., Paszke,
A., VanderPlas, J., Wanderman-Min Milne, S., and Zhang, Q. (2018). JAX: composable transformations of Python+NumPy programs.
Brehmer, J., Kling, F., Espejo, I., and Cranmer, K. (2020). Madminer: Machine learning-basee inference for particle physics. Computing and Software for Big Science, 4(1):1-25.
Cannon, P., Ward, D., and Schmon, S. M. (2022). Investigating the impact of model misspecification in neural simulation-base inference. arXiv preprint arXiv:2209.01845.
Chan, J., Perrone, V., Spence. J., Jenkins, P., Mathieson, S., and Song, Y. (2018). A likelihood-free inference framework for population genetic data using exchangeable neural networks. Advances in neural information processing systems, 31.
Chen, Y., Zhang, D., Gutmann, M., Courville, A., and Zhu, Z. (2020). Neural approximate sufficient statistics for implicit models. arXiv preprint arXiv:2010.10079
Chérief-Abdellate B.-E. and Alquier, P. (2020). MMD-Bayes: Robust Bayesian estimation via maximum mean discrepancy. Symposium on Advances Approximate Bayesian Inference, pages 1-21. PMLR.
Cranmer, K., Brehmer, J., and Louppe, G. (2020). The frontier of simulation-based inference. Proceedings of the National Academy of Sciences, 117(48):30555-30062
De Cao, N., Aziz, W., and Titov, I. (2020). Block neural autoregressive flow. In Uncertuinty in artificial intelligence, pages 1263-1273. PMLR.
Duane, S., Kennedy, A. D., Pendleton, B. J., and Roweth, D. (1987). Hybrid Monte Carlo. Physics letlers B, 196(2):216-222.
Durkan, C., Bekasov, A., Murray, I., and Papamakarios, G. (2019). Neural spline flows. Advances in neural information processing systems, 32.
Durkan, C., Murray, I., and Papamakarios, G. (2020). On contrastive learning for likelihood-ffree
inference. In International Conference on Machine Learming, pages 2771-2781. PMLR.
Dyer, J., Cannon, P., and Schmon, S. M. (2021). Approximate bayesian computation with path signatures. arXiv preprint 10..v:2106.1255
Fearnhead, P. and Prangle, D. (2012). Constructing summary statistics for approximate Bayesian computation: semi-automatic approximate Bayesian computation. Journal of the Royal Statistical Society: Series B (Statistical Methodology). 74(3):419
Ferguson, N. M., Laydon, D., Nedjati-Gilane G., Imai, N., Ainslie, K., Baguelin, M., Bhatia, S.,
Boonyasiri, A., Cucunubá, 7., Cuomo--annenburg. G., et al. (2020). Impact of non-pharmaceutical interventions (NPIs) to reduce COVID-19 mortality and healthcare demand.
Frazier, D. T. and Drovandi, C. (2021). Robust approximate Bayesian inference with synthetic likelihood. Journal of Computational and Graphical Statistics, 30(4):958-976.

---

Frazier, D. T., Drovandi, C., and Loaiza- Maya, R. (2020a). Robust approximate Bayesian computation: An adjustment approach. arXiv preprint 1xXiv:0008.40099
Frazier, D. T., Robert, C. P., and Rousseau, J. (2020b). Model misspecification approximate
Bayesian computation: consequences and diagnostics. Journal of the Royal Statistical Society: Series (Statistical Methodology), 82(2):421-444.
Friedman, J. (2004). On multivariate goodness-of-iit and two-sample testing. Technical report, Citeseer.
Fujisawa, M., Teshima, T., Sato, I., and Sugiyama, M. (2021). 7-abe: Outlier-robust approximate bayesian computation based on a robust divergence estimator. In International Conference on Artificial Intelligence and Statistics, pages 1783-1791. PMLR.
Gelman, A. and Shalizi, C. R. (2013). Philosophy and the practice of Bayesian statistics. British Journal of Mathematical and Statistical Psychology, 66(1):8-38.
George, E. I. and McCulloch, R. E. (1993). Variable selection via Gibbs sampling. Journal of the American Statistical Association, 88(423):881-88)
Greenberg, D., Nonnenmacher, M., and Macke, J. (2019). Automatic posterior transformation for likelihood-free inference. In International Conference on Machine Learning, pages 2404-2414. PMLR.
Grünwald, and Langford. (2007). Suboptimal behavior of Bayes and MDL in classification under misspecification. Machine Learning. 66(2):119-149,
Grünwald, P. and Van Ommen, T. (2017). Inconsistency of Bayesian inference for misspecified linear models, and a proposal for repairing Bayesian Analysis, 12(4): 1109-113..
Hermans, J., Begy, V., and Louppe, G. (2020). Likelihood-free meme with amortized approximate ratio estimators. In International Conference on Machine Learning, pages 2239-4248.. PMLR.
Hermans, J., Delaumoy, A., Rozet, F., Wehenkel, A., and Louppe, G. (2021). Averting crisis in simulation-based inference. arXiv preprint arXiv:2110.0658//
Hyndman, R. J. (1996). Computing and graphing highest density regions. The American Statistician, 10(2):120-126.
Jones-Todd, C. M., Caie, P., Illian, J. B., Stevenson, B. C., Savage, A., Harrison, D. J., and Bown, L. (2019). Identifying prognostic structural features in tissue sections of colon cancer patients using point pattern analysis. Statistics in medicine, 384))144211144
Kidger, P. and Garcia, C. (2021). Equinox: neural networks in JAX via callable pytrees and filtered transformations. arXiv preprint arXiv:2111.00244
Kleijn, J. and van der Vaart, A. W. (2012). The Bernstein-von-Mises theorem under misspecifica- tion. Electronic Journal of Statisties, 6:354-381.
Lam, S. K., Pitrou, A., and Seibert, S. (2015). Numba: A LLVV--base python JIT compiler. In Proceedings of the Second Workshop on the L.VM Compiler Infrastructure in HPC, pages 1-6.
Lopez-Paz, D. and Oquab, M. (2016). Revisiting classifier two-sample tests. arXiv preprint 1rXiv:1610.06555.
Lueckmann, J.-M., Boelts, Greenberg, D., Goncalves, P., and Macke, (2021). Benchmarking iimulation-based inference. In International Conference on Artificial Intelligence and Statistics, pages 343-351 PMLR.
Lueckmann, J.-M., Goncalves, P. J., Bassetto, G., Öcal, K., Nonnenmacher, M., and Macke, H. (2017). Flexible statistical inference for mechanistic models of neural dynamics. Advances in neural information processing systems, 30.
Metropolis, N., Rosenbluth, W., Rosenbluth, N., Teller, H., and Teller, (1953). Equation of state calculations by fast computing machines. The journal of chemical physics, 1186)):087-1092
12

---

Miglino, O., Lund, H. H., and Nolf,, S. (1995). Evolving mobile robots in simulated and real environments. Artificial life, 2(4):417-434.
Neal, R. M. et al. (2011). MCMC using Hamiltonian dynamics. Handbook of markov chain monte curlo, 2(11):2.
Papamakarios, G. and Murray, I. (2016). Fast e-free inference of simulation models with Bayesian conditional density estimation. Advances in neural information processing systems, 29.
Papamakarios, G., Nalisnick, E., Rezende, D. J., Mohamed, S., and Lakshminarayanan, B. (2019a). Normalizing flows for probabilistic modeling and inference. arXiv preprint arXiv:
Papamakarios, G., Sterratt, D., and Murray, I. (2019b). Sequential neural likelihood: Fast likelihood- free inference with autoregressive flows. The 22nd International Conference on Artificial Intelligence and Statistics, pages 837-848. PMLR.
Park, M., Jitkrittum, W., and Sejdinovic, D. (2015). K2-ABC: approximate Bayesian computation with kernel embeddings. arXiv preprint arXiv: 1502.02558.
Perla, J., Sargent, T. J., Stachurski, J., and Rackauckas, C. (2022). Quantitative economics with julia: Modeling shocks in COVID-19 with stochastic differential equations.
Phan, D., Pradhan, N., and Jankowiak, M. (2019). Composable effects for flexible and accelerated probabilistic programming in NumPyro. arXiv preprint 191..11911...54...
Prangle. D., Blum, M. G., Popovic, G., and Sisson, S. (2014). Diagnostic tools for approximate Bayesian computation using the coverage property. Australian & New Zealand Journal of Statistics, 56(4))309-33) 3299.
Pritchard, J. K., Seielstad, M. T., Perez-Lezaun, A., and Feldman, M. W. (1999). Population growth of human y chromosomes: study of y chromosome microsatellites. Molecular biology and evolution, 16(12):1791-1798
Rackauckas, C. and Nie, Q. (2017). DifferentiallEuuations, jl-a performant and feature-rich ecosystem for solving differential equations in julia. Journal of Open Research Software, 5(1).
Randall, D. A., Wood, R. A., Bony, S., Colman, R., Fichefet, T., Fyfe, J., Kattsov, V., Pitman, A., Shukla, J., Srinivasan, J., et al. (2007). Climate models and their evaluation. In Climate change 2007: The physical science basis. Contribution of Working Group the Fourth Assessment Report of the IPCC (FAR), pages 589-662. Cambridge University Press.
Schmitt, M., Bürkner, P.-C., Köthe, U., and Radev, S. (2021). BayesFlow can reliably detect model misspecification and posterior enrors in amortized Bayesian inference. arXiv preprint arXiv:2 088866
Schmon, S. M., Cannon, P. W., and Knoblauch, J. (2020). Generalized posteriors in approximate Bayesian computation. arXiv preprint arXiv:201 1.006444.
Syring, N. and Martin, R. (2019). Calibrating general posterior credible regions. Biometrika. 106(2):479-486,
Talbott, W. (2016). Bayesian Epistemology. In Zalta, E. N., editor, The Stanford Encyclopedia of Philosophy. Metaphysics Research Lab, Stanford University, Winter 2016 edition.
Talts, S., Betancourt, M., Simpson, D., Vehtari, A.. and Gelman, A. (2018). Validating Bayesian inference algorithms with simulation- calibration. arXiv preprint arXiv:1804.06788
Tavaré, S., Balding, D. J., Griffiths, C., and Donnelly, P. (1997). Inferring coalescence times from DNA sequence data. Genetics, 145(2):505-518.
Thomas, O., Dutta, R., Corander, J., Kaski, S., and Gutmann, M. U. (2022). Likelihood-free inference by ratio estimation. Bayesian Analysis, 17(1):1-11.
13

---

Wilkinson, R. D. (2013). Approximate Bayesian computation (ABC) gives exact results under the assumption model error. Statistical applications in genetics and molecular biology, 12(2):129- 141.
Wood, S. N. (2010). Statistical inference for noisy nonlinear ecological dynamic systems. Nature, 466(7310): 102-10001111-104 102-1104.
Zhou, G. (2020). Mixed Hamiltonian Monte Carlo for mixed discrete and continuous variables.
Advances in Neural Information Processing Systems, 13:1709--77144
Zhou, G. (2022). Metropolis augmented Hamiltonian Monte Carlo, arXiv preprint 4rX1w:2201.0844.
14

---

Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Section 6.
(c) Did you discuss any potential negative societal impacts of your work? [No] We see robust inference approaches allowing for better informed decision making, which should generally have positive societal consequences.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you including theoretical results...
(a) Did you state the full of assumptions all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes] We used a box plot (Fig. ??) visualise the distribution of results over seeds.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
4. If you using existing assets (e.g., code, data, models) or curating/releasing assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] NumPyro Python package (Section 2.2).
(b) Did you mention the license of the assets? [No]
(c) Did you include any new assets either in the supplemental material or as URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
15


Reviews:
[{"review_id": "_fJOwxBCOu6", "reviewer": "Reviewer_yiXg", "summary": "This study develops a new method for simulation-based inference (RNPE), capable of diagnosing model misspecification and reducing its impact on posterior inference. The empirical results on three tasks show that RNPE performs more reliable inference than a previous method (neural posterior estimation, NPE) according to a few metrics -- log probability of true parameters under the inferred posterior, and posterior coverage. In addition, RNPE provides a misspecification probability for each summary statistic on each task, information that NPE does not provide.\n", "questions": "As written above, the manuscript would benefit if the authors would include a task with misspecification across several summary statistics.\n", "limitations": "The authors have adequately addressed the limitations of their work.\n", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Originality\n\nSimulation-based inference has the potential to accelerate the process of scientific discovery across domains of science. One of the main limitations of this set of tools is the problem of model misspecification, a problem which has rarely been tackled. The questions addressed by this study are thus timely and relevant.\n\n\n### Quality\n\nThe paper is technically sound, claims are backed by empirical evidence, and the contributions are appropriately put in the context of previous work. However, I have one point that I believe would be crucial for the authors to tackle:\n\n-RNPE was tested in tasks for which the misspecification impacted only one summary statistic. Although this could be a realistic scenario in some cases, for many cases, the misspecification is across several summary statistics. In such cases, and if the summary statistics are not correlated with each other, then the extension to misspecification across several summary statistics seems trivial. However, in many realistic cases, there could be some residual correlation between summary statistics, and there one might expect the feedback from RNPE regarding misspecification probabilities to be less precise: there could be some indeterminacy in terms of misspecification, e.g. with a trade-off between misspecifications of different summary statistics. I believe the study would benefit from discussing such cases and including at least one task tackling this.\n\n\n### Clarity\n\nThe manuscript is clearly written, providing enough information to understand the technical contribution and empirical results. I just have one point that I would appreciate the authors to clarify:\n\n-in Figure 3b, the density for the denoised samples is shifted towards the right, without (seemingly) any mass around the observation. I might be misunderstanding something, but wouldn't one expect some non-negligible mass around the observation? If so, is it hidden in the plot by the green lines, and could one make it more visible?\n\n\n\n### Significance\n\nThe technical development and empirical results will be of interest to the ML community. In addition, the technical development could quite effortlessly be extended to other simulation-based inference algorithms, a possibility that will certainly entice other researchers on this topic.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "YvQr5PtgBNJ", "reviewer": "Reviewer_HK3N", "summary": "Simulation-based inference (SBI) enables Bayesian parameter inference in complex scientific simulators for which the evaluation of the likelihood is usually intractable. In most recent works on SBI this is achieved by conditional density estimation with neural networks trained on data simulated from the model. However, SBI generally assumes that the model is well-specified, and up to now it is not clear how SBI methods behave in the misspecified case, e.g., when the data distribution induced by the simulator (and the prior) does not contain the experimentally observed data. \nThis paper proposes a new SBI approach, RNPE, to enable inference even in the misspecified scenario, and to additionally obtain insights into the nature of misspecification present in the model. The goal is to obtain the posterior over the model parameters given the experimentally observed data, $p(\\theta | y)$, given only (potentially misspecified) simulations from the model $(\\theta, x)$. RNPE (robust NPE) extends the established SBI method Neural Posterior Estimation (NPE) to a two-step procedure: First, one performs neural density estimation on simulated data to learn $p(x)$, and assumes a noise model of the mismatch between the observed data $y$ and the simulated data $x$ , $p(y | x)$, to then obtain “denoised” posterior samples from $p(x|y)$ via MCMC. Second, one performs NPE on simulated data to obtain an amortized posterior estimator $q(\\theta | x)$ . Finally, one can obtain the posterior via a Monte Carlo approximation by repeatedly sampling denoised $\\tilde{x}$ from the first step, and obtaining corresponding posterior samples from $q(\\theta | \\tilde{x})$. \nRNPE is compared to NPE on three misspecified SBI example problems and shows consistently better performance on the evaluated metrics. In addition, the authors demonstrate that the inferred noise model from the first step can by useful to diagnose whether specific summary statistics are misspecified, which in turn can serve as a tool for model criticisms. ", "questions": "3. **Questions:** \nAs explained in detail above, I think it is essential to evaluate RNPE performance on a fully tractable example like the Gaussian task, in order to answer questions like: \n    - Are RNPE and NPE empirically equivalent in the well-specified case?\n    - How does slowly increasing the amount of misspecification change metrics like posterior mean error, variance error, dispersion, C2ST between the analytical and the NPE and RNPE posteriors?\n    - How do the posterior predictive distributions of the true, NPE and RNPE posteriors look like in the misspecified case?\n    - How does the Monte Carlo estimate in RNPE affect the RNPE vs the true posterior?\n    \n    Evaluation of additional metrics is especially important as in my impression, the expected coverage metric can be misleading. For example, in Figure 2b SIR it seems clear that NPE is massively overconfident, suggestion that it is too small posterior variance. But what is actually happening is that the NPE posterior is highly biased to other regions in the parameters space and actually has a very large variance. This behaviour would be immediately evident from a simulation-based calibration visualization of the ranks (Talts et al. 2018). I see that it might be challenging to perform SBC over $y$, but there might alternative metrics like the local coverage test (Zhao et al.). Alternatively, one could obtain reference posteriors for the SIR (Lueckmann et al. 2021) and then calculate metric between the posterior directly to evaluate RNPE vs NPE performance. \n    \n    The results of the spike-and-slap error model are promising, however, I am wondering whether you investigated other error model, and if so, how did they compare?\n    \n    The denoised data distributions shown in Figure 4b and in the appendix tend to have quite funky shapes. Are there intuitive explanations why this is the case, e.g., in 4b, what is the reason for the bimodal shape, opposed to the unimodal shape of the simulated data? That makes me wonder, how was the training of the flow and successful approximation of $p(x)$ evaluated? \n    \n    In the context of Figure 4c you point out that in contrast NPE, RNPE puts more posterior mass around the underlying ground-truth parameters. Here, I think it is important to note that in general posterior do not have be centered on ground-truth parameters, so in principle NPE could be correct here as well. One way to tell them apart would be checking the posterior predictive samples—how do they look like?", "limitations": "Limitations of RNPE to data and parameter space dimensionality of the SBI problem should be discussed in more detail (see above). Overall, I think this is an important and promising contribution that requires some additional evaluation and discussion of limitations. ", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "- *Originality:* \nThe paper introduces a new SBI method, RNPE which extends the established SBI method neural posterior estimation (NPE) to handle scenarios where the assumed model is misspecified. This is a new direction of research in the field of SBI and RNPE clearly differs from previously proposed approaches in this direction. Most of the previous work is adequately cited, except for the work on neural ratio estimation for SBI (NRE, Hermans et al. 2019, Durkan et al. 2019, Thomas et al. 2020), which should be seen as the third arm of neural network-based SBI methods (next to NPE, and NLE). Additionally, I think it should be considered citing the original papers on HMC alongside the mixed HMC paper, e.g., Neal (1995), MacKay (2003), or Neal (2011).\n- *Quality*: \nThe idea of RNPE to assume a noise model and to learn $p(x)$ via density estimation to then obtain denoised data, followed by NPE, seems quite original, complete and overall technically sound to me. The required derivations are straight forward to presented clearly in the paper. I have two critique points regarding the evaluation:\n    - first, I think it is essential to fully study and evaluate RNPE on a tractable example like the proposed Gaussian task (the Gaussian task is ideal because one can gradually increase the misspecification from well-specified (equal variance) to strongly misspecified (variance, say 25), and because one has easy access to the ground-truth analytical posterior). \n    In particular, it would good to first, empirically demonstrate that RNPE and NPE are equivalent in the well-specified case, and second, it would be good to show how the Monte Carlo (MC) approximation in the last step affects the results. For example, it would good to see (in the misspecified case) how as a function of the number of MC samples, the RNPE posterior samples approach the true posterior.\n    - second, I think it is essential to use more evaluation metrics for all three tasks. On the Gaussian task with access to reference posteriors one could additionally show the error in posterior mean, in posterior variance, and the actual dispersion between true and approximated posteriors. For all three task one could show additional calibration metrics, as I believe the expected posterior coverage can be misleading in some scenarios (see questions below). One possibility would be simulation-based calibration (SBC, Talts et al. 2018), although I acknowledge that it might be computationally infeasible because the entire RNPE procedure would need to repeated ~100 for different $y$. Alternatively, local coverage tests around the observed $y$ could be performed using Zhao et al. 2021 ([https://proceedings.mlr.press/v161/zhao21b.html](https://proceedings.mlr.press/v161/zhao21b.html)). \n    Finally, I think it would be instructive to additionally compare posterior predictive samples of the different approaches, e.g., by sampling parameters from the posterior and then passing them through simulator and the noise model and compare “simulated” and “noised” predictive samples.\n    \n    The evaluation on three example tasks is informative. However, all three tasks are rather low-dimensional (or designed to be) in the data and the parameter domain. I am wondering how RNPE scales to higher-dimensional problems. NPE was shown to infer plausible posteriors in up to 30-dimensional parameters spaces (Goncalves et al. 2020, eLife) and large data spaces in the image domain using embedding nets (see NPE results in the GATSBI paper by Ramesh et al. 2021), but I fear that this will be challenging for RNPE for several reasons. RNPE requires a Monte Carlo approximation from NPE, which might become computational expensive in large parameter spaces. RNPE requires learning $p(x)$, which will quickly become challenging as $x$ is not reduced to low-dimensional summary statistics. \n    For the first problem I think it is feasible to design another evaluation task with more parameters, e.g., a high-dimensional Gaussian task, and to show Monte Carlo approximation behaves. For the second problem I see that it is challenging to extend RNPE to be used with embedding networks. But even more importantly, I think it is essential that the reader is informed about these limitations, which does not seem to be the case in the current version of the paper. \n    \n- *Clarity*: \nThe paper is clearly written and well organized. Limitations should be stated more clearly (see above). Nitpick: the numbers in 227 and 228 are formatted inconsistently.\n- *Significance:* \nThis is a significant contribution. RNPE appears conceptually solid and shows promising results. Once evaluated thoroughly, I believe it has the potential to be used both by SBI practitioners (depending on the availability of RNPE in an open-source package, e.g., $\\texttt{sbi}$ Tejero-Cantero et al. 2020), and for future research on new SBI methods.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "SSSdiqOkL7o", "reviewer": "Reviewer_veSL", "summary": "The paper addresses the issue of model misspecification in simulation-based inference (sbi). Simulation-based inference refers to the task of inferring parameters of interest $\\theta$ based on an observation $y_o$ and a simulator. The simulator is based on scientific knowledge and allows to sample synthetic observations $x \\sim p(x|\\theta)$. Sbi assumes that those simulations match the real data generation process. However, in practice, this simulator is imperfect and there is a gap between simulation and reality. This work aims to resolve this gap by introducing an error model $y \\sim p(y|x)$ explicitly modeling the gap and proposing a strategy to perform robust inference using both the simulator and the error model. There are two contributions presented in this work:\n* Perform robust inference with a neural sbi method by leveraging the error model\n* get explicit misspecification posterior probabilities from the error model.", "questions": "As explained in the strengths and weaknesses section, my main questions would be:\n* How does combining posterior misspecification with the simulator helps to perform model criticism? \n* What advantages does it have over the naive approach of including the error model in the simulator for robust sbi?", "limitations": "I do not see any potential negative societal impact of this work.", "rating": 3, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 1, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "* (+) The paper is overall well written\n* (+) Model misspecification is a hot topic in simulation-based inference\n* (+) Experiments are convincing\n\n* (-) My main concerns about this work are about the problem statement. The authors proposed to use two models for the inference: the simulator $p(x|\\theta)$ and the error model $p(y|x)$ which is assumed to be known. In my opinion, if the error model were to be known a priori, it could be included in the simulator, hence viewing $x$ as a latent variable and $y$ as the simulated observable. In this case, classical simulation-based inference algorithms could be used. The big challenge with model misspecification is that, in practice, part of those misspecifications are unknown. One can then leverage real observations or assume specific types of misspecification without having a proper distribution and leverage Generalized Bayesian inference to be robust to those types of misspecification. Note that the authors do say something about this at l212-l216:\n> In contrast to more traditional SBI methods, few solutions have been proposed for performing robust inference with neural SBI methods.  Perhaps the most common approach is to augment the model with additional nuisance parameters (e.g. additive noise), to expand the range of values the simulator can reasonably produce (Cranmer et al., 2020). This again can generally be interpreted as a naïve method of incorporating an error model and does not facilitate model criticism.\n>\nHowever, it is not clear to me how combining the two components provides additional value regarding the two contributions presented in the paper.\n   - Contribution 1: perform robust inference with a neural sbi method. What advantages does it have over the naive approach of including the error model in the simulator?\n   - Contribution 2: get explicit posterior misspecification probability. Why do we need to combine the error model with the simulator? Couldn't the error model be used alone for this task?\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "-TESMfdRVt-", "reviewer": "Reviewer_3EMZ", "summary": "The authors proposed an extension of a class of simulation-based inference using neural networks called \"robust neural posterior estimation\". They intorduced an additional inference layer to a simulation-based inference method \"neural posterior estimation\" by combining it with an \"error model\" that is a generative model of data point from a simulated sample. In this work, the error model was specified by a spike-and-slab distribution, which in turn provides an indicator \"posterior misspecification probability\" for degrees of misspecification of each specified data summary statistics. Their experiments compared \"robust neural posterior estimation\" and \"neural posterior estimation\".\n", "questions": "- To me personally, there seems several sentences whose meanings were not fully clear and accurate throughout the paper in general. For example, on the sentence '... probabilistically “denoising” the observation, thus producing samples consistent with both the error model and simulator' in the page 3, it wasn't clear what \"denosing\" means here and what \"consistent with both the error model and simulator\" means exactly. For other example, the sentence 'Despite this, the general approach in SBI is ... to query it with the observed data, q(θ | x = yo)' seemed too coarse description of  the procedure; what would you mean by \"query with\" for example? I understand some of intensions behind are to make explanation intuiteve but it would be helpful to refine inaccurate explanations so that they sounds more accurate and rigorous.\n- It would be helpful to add an exact literature that coined the term \"neural posterior estimation\" if there is any.\n- It would be helpful to add an exact literature that used the term “simulation-to-reality gap” in the page 2.\n- The explanation of the algorithm procedure below (3) does not read well. For example, it may be helpful to write them as a certain type of well-organised lists.", "limitations": "There would be no negative societal impact. The clear limitation is that there was no comparison with other existing methodologies apart from neural posterior estimation. The significance of the proposal would have to be justified clearly by either or both extensive experimental evaluation or comprehensive theoretical analysis.", "rating": 3, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 1, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "- If I understand correctly, a core of the proposed idea is to introduce an error model $p(y \\mid x)$, a model that generates $y$ from a simulated sample $x \\sim \\text{Simulator}_{\\theta}$. Would this approach can be equivalent with defining a new simulator by plugging the original simulator into some condition probability distribution: $p(y \\mid x = \\operatorname{Simulator}_\\theta )$? In that case, other general methods in SBI or ABC would become available to learn all the parameters of the new simulator?\n- In this work, a spike-and-slab distribution is used for the error model $p(y \\mid x)$. The choise of the distribution seems to implicitly specify what type of misspecification we assume to happen because the model $p(y \\mid x)$ describe how datum $y$ is generated from a simulation $x$. In spike-and-slab case, it seems like at first glance that an implicitly assumed type of misspefication is less frequent outlier. The performance of robust neural posterior estimation seems to be dependent of the choice of the error model to a large degree. The spike-and-slab would be the only choice that posterior misspecification probability can be computed? Would it be possible to provide other  available choices and examine impacts by them?\n- Given that above perspectives, if it is possible to apply other other general methods in SBI or ABC to learn the entire parameter of the simulator, there would be vairous other robust inference methods available such as one based on maximum mean descrepancy. If this is the case, it would be helpful to see or commnet on how efficient the proposed procedure is than others. In this respect, the reviewer was personally wondering if the broad name \"robust\" neural posterior estimation is throughly accurate/appropriate to represents the exact content of the proposal, given that there seems to exist many possible approaches to \"robust\" inference.\n- The experiments seem to compare \"robust neural posterior estimation\" with \"neural posterior estimation\" only. It would be strongly helpful to compare it with other existing methodologies such as ones cited in Section 3.2 and to clarify when and how the robust neural posterior estimation has the advantage to others. The performances of robust neural posterior estimation and neural posterior estimation would also be dependent of the choice of neural networks to be used, where therefore it may also be helpful to justify that the used models were reasonable somewhere in the main paper.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept