Misspecification-robust Sequential Neural Likelihood for Simulation-based Inference
Authors: Ryan P. Kelly
David J Nott
David Tyler Frazier
David J Warne
Christopher Drovandi
Venue: Accepted by TMLR ()

Abstract: Simulation-based inference techniques are indispensable for parameter estimation of mechanistic and simulable models with intractable likelihoods. While traditional statistical approaches like approximate Bayesian computation and Bayesian synthetic likelihood have been studied under well-specified and misspecified settings, they often suffer from inefficiencies due to wasted model simulations. Neural approaches, such as sequential neural likelihood (SNL) avoid this wastage by utilising all model simulations to train a neural surrogate for the likelihood function. However, the performance of SNL under model misspecification is unreliable and can result in overconfident posteriors centred around an inaccurate parameter estimate. In this paper, we propose a novel SNL method, which through the incorporation of additional adjustment parameters, is robust to model misspecification and capable of identifying features of the data that the model is not able to recover. We demonstrate the efficacy of our approach through several illustrative examples, where our method gives more accurate point estimates and uncertainty quantification than SNL.

Published in Transactions on Machine Learning Research (06/2024)
Misspecificttion-rbuust Sequential Neural Likelihood for
Simulation-based Inference
Ryan P. Kelly
rel.kelly@hdr quw.ar..uuuuuu........ edu.ou
School of Mathematicnl Sciences
Centre for Data Science
Queensland University of Technology
David J. Nott
slandj@rus.edde edu..g
Institute of Operations Research and Analytics
National University of Singopore
David T. Frazier
daviddfyacieer@mmnascccc edu
Department of Econometrics and Business Statistics
Monash University
David J. Warne
wavid.warneeQut....
School of Mathematicul Sciences
Centre for Data Science
Queensland University of Technology
Christopher Drovandi
c.drovandi@qut.c edu.
School of Mathematical Sciences
Centre for Dala Science
Queensland University of Technology
Reviewed on OpenReview: https: // openreview. net/ forum? id= tb0YJwTheY
Abstract
Simulation-based inference techniques are indispensable for paramcter estimation of mechanis- lie and simulable models with intractable likelihoods. While traditional statistical approaches like approximate Bayesian computation and Bayesian synthetic likelihood have been studied under well-specified and misspecified settings, they often suffer from inefficiencies due to wasted model simulations. Neural approaches, such as soquential neural likelihood (SNL) avoid this wastage by utilising all model simulations to train neural surrogate for the likelihood function. However, the performance of SNL under model misspecification is unreliable and can result in overconfident posteriors centred around inaccurate parameter estimate. In this paper, we propose novel SNL method, which through the incorporation of additional adjustment parameters, robust to model misspecification and capable of identifying features of the data that the model is not able to recover. We demonstrate the efficacy of approach through several illustrative examples, where our method gives more accurate point estimates and uncertainty quantification than SNL.
1 Introduction
Statistical inference for complex models can be challenging when the likelihood function is infeasible to evaluate numerous times. However, if the model is computationally inexpensive to simulate given parameter values, possible perform approximate parameter estimation by so-called simulation-base inference (SBI) techniques (c.g. Cranmer et al, (2020)). The difficulty of obtaining reliable inferences in the SBI setting is exacerbated when the model misspecified (e.g. Cannon et al. (0022 Frazier et. al. (020bb

---

Published in Transactions on Machine Learning Research (06/2024)
SBI methods are widely employed across numerous scientific disciplines, such cosmology (Hermans et al. ecology (Hauenstein et al. 2019), mathematical biology (Wang et al. 2024), neuroscience (Conlavreux et al. 2023 Gonçalves et al. 2020 population genetics (Beaumont) 2010) and in epidemiology to model the spread of infectious agents such as S. pneumoniae (Numminen et al. 2013) and COVID-19 (Ferguson et al.) Warne et al. 2020p. Model building is an iteralive process involving fitting Lentative models, model criticism and model expansion: Blei (2014) have named this process the "Box Loop" after pioneering work by Box (1976). We refer to Gelman et al. (2020) as a guide to Bayesian model building. Although methods for statistical model criticism are well-developed, there is less work on reliably refining models for SBI methods. One reason is that SBI methods are susceptible to model misspecification, which may be especially prevalent in the early stages of modelling when complexity is progressively introduced. A misspecified model may lead to misleading posterior distributions, and the available tools for model criticism may also be unreliable [Schmitt et al. 2023a). Ensuring that misspecification is detected and its negative impacts are mitigated is important if we are to iterate and refine our model reliably. However, this is often overlookod in SBI methods.
2021
2020:
Posterior predictive checks (PPCs) are frequently used for model evaluation and provide tool to check for model misspecification (Gabry et al. 2019). For SBI, if a model is well-specified, should be able to generate data that resembles the observed data. However, as noted in Schmitt et al. (2023a), PPCs rely on the fidelity of the posterior and can only serve as an indirect measure of misspecification. Similarly, scoring rules are another way to assess how well a probabilistic model matches the observed data (Gneiting & Raftery, 2007
In this paper, we focus on the type of misspecification where the model is unable to recover the observed summary statistics as the sample size diverges. This form of misspecification is referred to as incompatibility
by Marin et al. (2014). Statistical approaches for SBI, such approximate Bayesian computation (ABC,
Sisson et al. (2018) and Bayesian synthetic likelihood (BSL, Price et al. (2018) have been well studied,
both empirically (e.g. Drovandi & Frazier (2022)) and theoretically (e.g. Li & Fearnhead (2018). Frazier
ct al. 20118) David T. Frazier & Kohn (2023)). ). Wilkinson (2013) reframes the approximate ABC postcrior
88 an exact result for a different distribution that incorporates an assumption of model error (i.e. model misspecification. Ratmann et al. (2009) proposes an ABC method that augments the likelihood with unknown error terms, allowing model criticism by detecting summary statisties that require high tolerancos. Frazier et al. (2020a) incorporate adjustment parameters, inspired by robust BSL (RBSL), in an ABC context. These approaches often rely on a low-dimensional summarisation of the data to manage computational costs. ABC aims to minimise the distance between observed and simulated summaries, whereas BSL constructs a Gaussian approximation of the model summary to form an approximate likelihood. In the case of model misspecification, there may be additional motivation to replace the entire dataset with summaries, as the resulting model can then be trained to capture the broad features of the data that may be of most interest; see, e.g., Lewis et al. (2021) for further discussion.
Neural approaches, such as SNL and neural posterior estimation (NPE), have been shown to exhibit poor empirical performance under model misspecification (e.g. BBon et al. (2023): Cannon et al. (2022 [ccmiit et al. (2 ; Ward et al. (2022)). Thus, there is a critical need to develop these neural approaches so they are robust to model misspecification. |Ward et al. (2022) develop robust NPE (RNPE), a method to make
NPE robust to model misspecification and useful for model criticism. Cannon et al. (2022) develop robust
SBI methods by using machine learning techniques to handle out-of-distribution (OOD) data. Cranmer et al.
(2020) advise incorporating additional noise directly into the simulator if model misspecification is suspected.
We develop a robust version of SNL, inspired by the mean adjustment approach for BSL FFazieer & Drovandi 2021). Unlike Ward et al. 2022) who consider NPE, we focus on neural likelihood estimation, which is useful for problems where the likeldhood is easier to emulate than the posterior. Our method is the first sequential neural approach that simultaneously detects and corrects for model misspecification. By shifting incompatible summary statistics using adjustment parameters, our method matches quite closely the posterior obtained when only considering compatible summary statistics. This is demonstrated in Figure D on an example discussed further in Section 4 where our method is shown to closely match the posterior density obtained using the compatible summary statistic. We further demonstrate the reliable performance of our approach on several illustrative examples.
2

---

Published in Transactions on Machine Learning Research (06/2024)
0.4
20
True Assumed
RSNL
SNL
0.3
RNPE
Density
Density
True Posterior
0.2
10
0.1
0.0
0
10
-
5
10
0
2
3
0
(b)
(a)
Posterior Prior
Posterior
0.4
0.75
Prior
Density
Density
0.50
0.2
0.25
0.0
0.00
-10
-10
-5
5
10
-10
10
-5
0
5
10
71
7/2
(c)
(d)
Figure Posterior plots for contaminated normal model (soc Section for details). The top left plot (a) shows the univariale densily of samples generated from the true (solid) and assumed (dashed) DGPs. The top right plot (b) shows the estimated univariate SNL (dashed), RSNL (solid), RNPE (dashed) and true (dash-dotted) posterior densities for 0. The true parameter value is shown vertical dashed line. Plots (c) and (d) show the estimated marginal posterior (solid) and prior (dashed) densities for the components of the mdjustment parameters of the RSNL, method.
2 Background
Let y = (yi...., nT denote the observed data and define (()) the true unknown distribution of The observed data is assumed to generated from a class of parametric models, (P(m), e Rdo.. The posterior density of interest is given by
( b) x g(v ()).
(1)
where gv 0) is the likelihood function and x (0) is the prior distribution. In this paper, we are interested in models for which g(y 0) is analytically or computationally intractable, but from which we can easily simulate pseudo-data x for any 0 E θ where is de dimensional.
2.1 Simulation-basee Inference
The traditional statistical approach to conducting inference on 0 in this setting is to use ABC methods. Using the assumed DGP, these methods search for values of that produce pseudo-data x which "close enough" to y, and then retain these values build an approximation to the posterior. The comparison is generally carried out using summaries of the data to ensure the problem computationally feasible. Let S: R Rdd, denote the vector summary statistic mapping used in the analysis, where > do, and 2 d.
3

---

Published in Transactions on Machine Learning Research (06/2024)
Two prominent statistical approaches for SBI are ABC and BSL. ABC approximates the likelihood for the summaries via the following:
9e (S 0) 1
0)dx.
where p(S(y)) measures the discrepancy bet ween observed and simulated summaries and K₂(+) is a kernel that allocates higher weight to smaller p. The bandwidth of the kernel, e, often referred to as the tolerance in the ABC literature. The above integral is intractable, but can be estimated unbiasedly by drawing m 2 1 mock datasets #1.... Fm ~ p(n) and computing
(S 0) = Σ K((o()())
1=1
It is common to set m and choose the indicator kernel function,
Kg (.(p((()),S(z)) = ((p(S(g), <).
Using arguments from the exact-approximate literature (Andrieu & Roberts 2009), unbiasedly estimating the ABC likelihood leads Bayesian algorithm that samples from the approximate posterior proportional to (S(y) 0) n (0). As is evident from the above integral estimator, ABC non-paramettrcally estimates the summary statistic likelihood. Unfortumately, this non-parametric approximation causes ABC to exhibit the "curse of dimensionality", mcaning that the probability of a proposed parameter being accepted decreases
dramatically as the dimension of the summary statistics increases (Barber et al. 2015 Csilléry el. al. 2010
In contrast, BSL uses a parametric estimator. The most common BSL approach approximates g.((- ( using Gaussian:
gA (S 0) N (O) E())
where uu0) = [x) 0) and 20) = Var S(x) 0) denote the mean and variance of the model summary statistic at 0. In almost all practical cases (0) and Σ(0) are unknown, but we can replace these quantitics with those estinated from m independent model simulations, using for example the sample mean and variance:
m
(m)
S(x*),
Em (0) (S(x*) Mm (0)) (S(x*) )- um (0))
and where each simulated data set x², i = l...., m. is generated i..d. from (()) The synthetic likelihood is then approximated as
(A 0) = N (S(y): (0), (0))
Unlike ABC, (S()) 0) is not an unbiased estimator of (S()))) 0). David T. Frazier & Kohn 2023
demonstrate that if the summary statistics are sub-Gaussian, then the choice of m is immaterial so long as diverges as diverges. Empirical evidence supporting the insensitivity to m is presented in Price et al. (0018), indicating that as long as m is sufficiently large, the variance of the plug-in syuthetic likelihood estimator remains small enough to prevent any negative impact on MCMC mixing. The Gaussian assumption can be limiting: however, neural likelihoods provide more flexible alternative to BSL while retaining the advantages of parametric approximation.
Unfortunately, both ABC and BSL are inefficient in terms of the number of model simulations required to produce posterior approximations. In particular, most algorithms for ABC and BSL are wasteful in the sense that they use relatively large number of model simulations associated with rejected parameter proposals. In contrast, methods have been developed in machine learning that utilise all model simulations to learn either the likelihood (c.g. |Papamakarios et al. (2019)), postcrior (c.g. Greenberg ct al. (2019); Papamakarios & Murray (2016)) or likelihood ratio (e.g. Durkan et al. (2020) : Hermans et al. (2020): Thomas et al. (2022)).
We consider the SNL method of Paaamakarios et al. (2019 in more detail in Section
4

---

Published in Transactions on Machine Learning Research (06/2024)
Neural SBI methods have seen rapid advancements, with various approaches approximating the likelihood [Boelts et al. 2022 Wiqvist et al. 2021). These methods include diffusion models for approximating the score of the likelihood (Simons et al. 2023), energy-based models for surrogating the likelihood (Glaser et al.
2023 and a "neural conditional exponential family" trained via score matching (Pacchiardi & Dutta 2022 Bon el al. (2022) present method for refining approximate posterior samples to minimise bias and enhance uncortainty quantification by optimising a transform of the approximate posterior, which maximises scoring rule.
One way to categorise neural SBI methods is by differentiating between amortised and sequential sampling schemes. These methods differ in their proposal distribution for 0. Amortised methods estimate the neural density for any a within the support of the prior predictive distribution. This allows the trained flow to approximate the posterior for any observed statistic, making it efficient when analysing multiple datasets. However, this requires using the prior as the proposal distribution. When the prior and posterior differ significantly, there will be few training samples of close to y, resulting in the trained flow potentially being less accurale in the vicinity of the observed statistic.
2.2 SBI and Model Misspecification
In the standard Bayesian inference framework, we denote the model parameter value that minimises the Bullback-Leibler (KL) divergence between the assumed model distribution and the true distribution as Og- This can be interpreted cither the true distribution pscudo-true parameter value, depending whether the true distribution does or does not belong to the assumed parametric family. When we apply summary statistic mapping, we need to redefine the usual notion of model misspecification, i.e., no value of 0 E O such that pa P(""), as it is still possible for p(n) to generate summary statistics that match the observed statistic even if the model is incorrect (Frazier et al. 2020b). We define b(0) [[S(x) 0 and bo [y)) as the expected values of the summarr statistic with respect to the probability measures p(n) and p(a) respectively. The meaningful notion of misspecification in SBI is when no O satisfies b(0) = bo, implying there is no parameter value for which the expected simulated and observed summaries match. This is the definition of incompatibility proposed in Marin et al. (2014).
The behaviour of ABC and BSL under incompatibility is now well understood. In the context of ABC, we consider the model to be misspecified if
c" = inf p(b(O), bo) > 0, AEe
for some metric P, and the corresponding pseudo-true parameter is defined
0. = arg inf p(b(0), bo)-
Frazier et al. (2020b) show, under various conditions, the ABC posterior concentrates onto Og for large sample sizes, providing an inherent robustness to model misspecification. However, they also demonstrate that the asymptotic shape of the ABC posterior non-Gaussian and credible intervals lack valid frequentist. coverage; i.e. confidence sets do not have the correct level under p(²))
In the context of BSL, Frazier et al. 2021 show that when the model is incompatible, i.c. b(0) + bo AA E e, the KT. divergence between the true data generating distribution and the Gaussian distribution associated with the synthetic likelihood diverges as n diverges. In BSL, we say that the model incompatible if
lim inf (b(0) - bo) (nE(O)) (b(0) bo} >
n- Bee
We define
(, = -a ga (S 0) /80.
The behaviour of BSL under misspecification depends on the number of roots of M,) = If there is a single solution, and under various assumptions, the BSL posterior will concentrate onto the pseudo-true parameter Ao with an asymptotic Gaussian shape, and the BST. posterior mean satisfies Bernstein von-Mises
5

---

Published in Transactions on Machine Learning Research (06/2024)
result. However, if there are multiple solutions to M, 0, then the BSL posterior will asymptotically exhibit multiple modes that do not concentrate on Oo- The number of solutions to Mn = 0 for a given problem is not known priori and is very difficult to explore.
In addition to the theoretical issues faced by BSL under misspecification, there are also computational challenges. Frazier& & Drovandi (2021) point out that under incompatibility, since the observed summary lies in the tail of the estimated synthetic likelihood for any value of 0, the Monte Carlo estimate of the likelihood suffers from high variance. Consequently, significantly large value of is needed to enable the MCMC chain to mix and avoid getting stuck, which is computationally demanding.
Due to the undesirable properties of BSL under misspeciiicatioon, Frazier & Drovandi 2021 propose RBSL as a way to identify incompatible statistics and make inferences more robust simultaneously RBSL is a model expansion that introduces auxiliary variables, represented by the vector = (71...., 7a) T. These variables shift the mcans (RBSL-M) or inflate the variances (RBSL-VV in the Gaussian approximation, ensuring that the extended model is compatible by absorbing any misspecification. This approach guarantees that the observed summary does not fall far into the tails of the expanded model. However, the expanded model is now overparameterised since the dimension of the combined vector (0, T) T is larger than the dimension of the summary statistics.
To regularise the model, Frazier & Drovandi 2021 impose a prior distribution on that favours compatibility. However, each component of I's prior has a heavy tail, allowing it to absorb the misspecification for a subset of the summary statistics. This method identifies the statisties the model is incompatible with while mitigating their influence on the inference. Frazier & Drovandì (2021) demonstrate that under compatibility, the posterior for r is the same its prior, so that incompatibility can be detected by departures from the prior.
The mean adjusted synthetic likelihood is denoted
N ((S)): =())
(2)
where (0) = diag (ia)( is the vector of estimated standard deviations of the model summary statistics, and o denotes the Hadamard delement-by-eeeeennf product. The role of (0) is to ensure that we can treat each component of I as the number of standard deviations that we are shifting the corresponding model summary statistic.
Frazier & Drovandi [2021) suggest using prior where O and I are independentt, with the prior density for r being Laplace prior with scale X for each 7r. The prior is chosen because it is peaked at zero but has a moderately heavy tail. Sampling the joint posterior can be done using a component-wise wise MCMC algorithm that iteratively updates using the conditionals 0 I and I | S, The update for I holds the m model simulationn fixed and uses a slice sampler, resulting an acceptance rate of one without requiring tuning a proposal distribution. Frazier & Drovandi (2021) find empirically that sampling over the joint space (0, T does not slow down mixing the 9-marginal space. In fact, in cases of misspecification, the mixing is substantially improved as the observed value of the summaries no longer falls in the tail of the Gaussian distribution.
Recent developments in SBI have focused on detecting and addressing model misspecification for both neural posterior estimation Ward et al. 2022) and for amortised and sequential inference (Schmitt et al., 2023a). Schmitt et al. (2023a) employs a maximum mcan discrepancy (MMD) estimator to detect "simulation gap" between observed and simulated data, while Ward et al. (002) detects and corrects for model misspecification by introducing an error model π (S(y) S(x)). Noise is added directly to the summary statistics during training in Bernaerts et al. (0023). Huang al. (2023 noted that incompatibility is based on the choice of summary statistics, if the summary statistics are learnt via NN, training this network with a regularisod loss function that penalises statistics with a mismatch between the observed and simulated values will lead to robust inference. Glöckler et al. (2023), again focused on the case of learnt (via NN) summaries, propose a scheme for robustness against adversarial attacks (i.c. small worst-casc perturbations) on the observed data. SSchmitt al.) (2023c) introduce a meta- uncertancyyy framework that blends real and simulated data quantify uncertainty in posterior model probabilities, applicable to SBI with potential model misspecification.
6

---

Published in Transactions on Machine Learning Research (06/2024)
Generalised Bayesian inference (GBI) is alternative class of methods suggested to handle model misspecifi cation better than standard Bayesian methods (Knoblauch et al. 2022). Instead of targeting the standard Bayesian posterior, the GBI framework targets a generalised posterior,
π T y) xc (0) exp(-w (0000--ww,,))))) 0),,
where (yy 0) is some loss function and w is a tuning parameter that needs to be calibrated appropriately
Bissiri et al. 2016). Various approaches have applied GBI to misspecified models with intractable likelihoods
Chérief-Al Abdelatif & Alquier 2020 Matsubara et. al. 2022 Pacchiardi & Dutta 2021 Schmon et al. 2021
Gao et al. (2023) extends GBI to the amortised SBI setting, using a regression neural network to approximate
the loss function, achieving favourable results for misspecified examples. Dellaporta et al. (0022)
employ
similar ideas to GBI for an MMD posterior bootstrap.
3 Robust Sequential Neural Likelihood
SNL is within the class of SBI methods that utilise a neural conditional density estimator (NCDE). An NCDE is a particular class of neural network, gø, paramcterised by Φ, which learns a conditional probability density from a set of paired data points. This is appealing for SBI since we have access to pairs of (0, x)) but lack a tractable conditional probability density in either direction. The idea is to train 96 on D {(0, ),,) and use it as a surrogate for the unavailable density of interest. NCDEs have been employed as surrogate densities
for the likelihood (Papamakarios et al., 2019) and posterior (Papamakarios & Murray
2016
Greenberg et al.)
2019) or both simultaneously (Radey et al. 2023 Schmitt el 2003b Wiqvist et al. 2021
Most commonly,
a normalizing flow is used as the NCDE, and we do so here.
Normalising flows are a useful class of neural network for density estimation. Normalising flows convert simple base distribution {u)), e.g. a standard normal, to complex target distribution, T(n), e.g. the likelihood, through sequence of L diffeomorphic transformations (bijective, differentiable functions with a differentiable inverse), T = Tr. ① ..................... O Tí. The density of n (u), n E IR, where ~ u=)( is,
x (nn = T()) det JT()))
(3)
where Jr is the Jacobian of T. Autoregressive flows, such as neural spline flow used here, are one class of normalising flow that ensure that the Jacobian is a triangular matrix, allowing fast computation of the determinant in Equation We refer to Papamakarios et al. (2021) for more details. Normalising flows are also useful for data generation, although this has been of lesser importance for SBI methods.
Sequential approaches aim to update the proposal distribution so that more training datasets are generated closer to S(y), resulting in a more accurate approximation of T(0 | S(y) for a given simulation budget. In this approach, R training rounds are performed, with the proposal distribution for the current round given by the approximate posterior from the previous round. As in Papamakarios et al. (2019), the first round, r y= O, proposes 0 ~~ r0), then for subsequent rounds 1, is trained on all generated (0, D. The choice of amortised sequential methods depends application. For instance, in epidemiology transmission models, we typically have a single set of summary statistics (describing the entire population of interest), relatively uninformative priors, and a computationally costly simulation function. In such cases, a sequential approach is more appropriate.
1, a normalising flow, (S(x)())) 0)
the
Neural-based methods can efficiently sample the approximate posterior using MCMC methods. The evaluation of the normalising flow density is designed to be fast. Since we are using the trained flow as a surrogate function, no simulations are needed during MCMC sampling. With aulomatic differentiation, one can efficiently find the gradient of an NCDE and use it in effective MCMC sampler like the No- U-Turn sampler (NUTS) (Hoffman & Gelman 2014).
Unlike ABC and BSL, it is currently unclear to the authors how one can formally define pseudo-true values for the SNL posterior in general settings. Furthermore, the authors are unaware of any theoretical work that discusses the asymptotic properties of SNL under correct or misspecified models. The complication in defining where one would expect the SNL posterior to concentrate in the asymptotic regime is a direct consequence of the way the SNL estimator is obtained, and, by extension, how the SNL posterior is produced.
7

---

Published in Transactions on Machine Learning Research (06/2024)
In SNL, the NCDE is trained via the generation of simulated summary statistics from the assumed model, which means that the SNT. estimator only learns about the likelihood of the simulated summaries. Hence, if the distribution of the simulated summaries differs markedly from that of the observed summaries, there is no reason to suspect that the SNL posterior will concentrate on a meaningful point in the parameter space.
Recent research has found that neural SBI methods behave poorly under model misspecification (Bon et al. 2023: Cannon et al., 2022 Schmitt et al., 2023a Ward et al., 2022), prompting the development of more robust approaches. We propose robust SNL (RSNL), sequential method that adapts to model misspecification by incorporating an approach similar to that of Frazier & Drovandi (2021). Our approach adjusts the observed summary based on auxiliary adjustment parameters, allowing it to shift to region of higher surrogate density when the summary falls in the tail. RSNL evaluates the adjusted surrogate likelihood as (5(S()) r 0) estimating the approximate joint posterior,
π T S(y)) x (5( - -r ()))(())
(4)
where we set (0) and (T) independently of each other.
The choice of prior, x (T)), is crucial for RSNL. Drawing inspiration from RBSL-M, we impose a Laplace prior distribution on I to promote shrinkage. The components of T are sot to be independent, (T()) = II= (*x). We find that the standard Laplace(O, 1) prior works well for low to moderate degrees of misspecification. However, lacks sufficient support for high degrees of misspecification, leading to undetected misspecification and prior-data. conflict (Evans & Jang 2011]. To address this, we employ data-driven prior that, similar to weakly informative prior, provides some regularisation without imposing strong assumptions for each component:
7 (T)) = Laplace = (1)) = 2A exp (-
77 A
(5)
where (1)) is the i-th standardised observed summary. Large observed standardised summarics indicate a high degree of misspecification, and including this in the prior helps the expanded model detect it. We set m0(%)) ~ Laplace(0, 1) for the initial round and update म, (1) in each round by recomputing S(y). This approach detects highly misspecified summaries while introducing minimal noise for well-specified summaries. Setting 7 adjusts the scale, with larger values allowing larger adjustments. We illustrate the effect of differing values of in Appendix G. Setting т involves balancing trade-off: it must be small enough to minimise noise introduced by the adjustment paramcters, and thereby the accuracy of inference for the model parameters, yet sulficiently large to delect misspecification more readily than lower values, which leads to more stable posteriors for the adjustment parameters. We note for the primary objectives of detecting model misspecification and giving robust inference, a broad range of r values are suitable. But we promote T == 0.3 as a robust choice across different scenarios. The adjustment parameters can map a misspecified summary to the mean of sinulated summaries, regardless of its position in the tail. We consider our proposed prior against fixed variance prior in Appendix E, which illustrates that the fixed variance prior is prone to introducing excessive noise
Standardising the simulated and observed summaries is necessary to account for varying scales, and it is done after generating additional model simulations at each training round. Since all generated parameters are used to train the flow, standardisation is computed using the entire dataset D. Standardisation serves two purposes: 1) facilitating the training of the flow, and 2) ensuring that the adjustment parameters are on a similar scale to the summarics. It important to note that standardisation is performed unconditionally, using the sample mean and sample standard deviation calculated from all simulated summary statistics in the training set. For complex DGPs, unconditioned standardisation may not adequately address heteroscedasticity non-lincar dependencies across the parameter space, leading to potential biases in inference. We discuss possible extensions in Section
Algorithm D outlines the complete process for sampling the RSNL approximate posterior. The primary distinction between SNL and RSNL lies in the MCMC sampling, which now targets the adjusted postcrior as a marginal of the augmented joint posterior for and r. As a result, RSNL can be easily used as a substitute for SNT.. This comes at the additional computational cost. of targeting a joint posterior of dimension
8

---

Published in Transactions on Machine Learning Research (06/2024)
de + d, rather than of dimension d. Nonetheless, this increase in computational cost is generally marginal compared to the expense of running simulations for neural network training, aspect not ad adversely affected by RSNL. Since RSNL, like SNL, can efficiently evaluate both the neural likelihood and the gradient of the approximate posterior, we employ NUTS for MCMC sampling. This is in contrast to Ward et al. (2022). who use mixed Hamiltonian Monte Carlo, an MCMC algorithm for inference on both continuous and discrete variables, due their use of spike-and-slab prior.
Once obtain samples from the joint posterior, can examine the 0 and I posterior samples separately. The 0 samples can be used for Bayesian inference functions of 0 relevant to the specific application. In contrast, the r approximate posterior samples can aid in model criticism.
RSNL can be employed for model criticism similarly to RBSL. When the assumed and actual DGP are incompatible, RSNL expected to behave like RBSL, resulting in a discrepancy between the prior and posterior distributions for the components of T. Visual inspection should sufficient to detect such a discrepancy, but researchers can also use any statistical distance function for assessment (e.g. total variation distance). This differs from the approach in RNPE, which assumes spike and slab error model and uses the posterior frequency of being in the slab an indicator of model misspecification.
Algorithm Robust MCMC SNL
Input:
The observed summaries, Syy) the prior distributions (0) and TT();; the number of training rounds, R; the assumed data generating process, p(*); the number of simulated datasets from p(n) generated per round, m; the neural density estimator family, (s((()) O).
Output: MCMC samples (fo...., 0m) and (To.....) Imm from the RSNL posterior.
Set D= {}, 20. (S()) 00
for to R do
Update (I') when r 0
4: for i = 0 to - do
5: Sample 0²²) ((r) ar (n [ ( ([)))(0)); using MCMC or directly when r =
Simulate x. (r) p(n)
0
7: Compute summarics S(x
Add (<x(")))))) into D
end for
10: Standardise D and S(y)
11: Train (5(1+((((x)) 0) on D
12: end for
13: Sample ( -I!! (S r ()) (T)
14: return (o(m) r
0:m
4 Examples and Results
In this section, we demonstrate the capabilities of RSNT, on five benchmark misspecified problems of increasing complexity and compare the results obtained SNL and RNPE. The same hyperparameters were applied all tasks, detailed in Appendix C. Further details and results for some examples can be found in Appendix D.
We brcak down the overall computational cost of the algorithm into three main components: running the simulations, training the normalising flow, and MCMC sampling, The breakdown of computational times for each example shown in Appendix B. Often, for complex problems, the simulator might be the most costly component, and no difference expected in the time to run simulations between RSNL and SNL. Likewise, no difference expected in training the normalising flow. The difference between RSNL and SNL lies in the MCMC computation.
9

---

Published in Transactions on Machine Learning Research (06/2024)
As might expect, targeting higher-dimeenional posterior with the NUTS algorithm can lead to increased computation times, particularly when adaptive settings during the warmup stage result in increased computation per sample. RSNL targets a joint posterior of dimension dg + d instead of dg, where d is the number of summarics and hence the number of adjustment parameters. When there is a large number of summaries, such as the toad movement model, this leads to a noticeable increase MCMC computational time. Still, the computational time remains feasible for the summary statistic dimension range where SNL is typically considered.
The expected behaviour of scaling is similar to MCMC more broadly, dependent the problem and geometry of the posterior. For example, in the contaminated simple likelihood complex postcrior (SLCP) example in Section 4.3 RSNL actually has faster MCMC time than SNL, likely due to improved posterior geometry. Specifically, the NUTS algorithm may increase the amount of computation in cases of poor geometry, such as the SNL posterior shown in Figure
The expected coverage probability, a widely used metric in the SBI literature (Hermnn el. al. 2022) measures the frequency at which the true parameter lies within the HDR. The HDR is the smallest volume region containing 100(1-a)% of the density mass in the approximate posterior, where 1-a represents the credibility level. The HDR is estimated following the approach of Hyndman (1996). Conservative posteriors are considered more scientifically reliable (Hermans et al., 2022) as falsely rejecting plausible parameters is generally worse than failing to reject implausible parameters. To calculate the empirical coverage, we generate
C = 200 observed summaries S(ya) ~ S(P(")) at o,, where o,, represents the "true" data generating parameter and i = L,.... C. When O, is contained in the parameter space of our assumed models, we take t,, ~ (0)). Using samples from the approximate postcrior obtained using RSNL, we use kernel density estimation to give p(Oo.; S(y:)). The empirical coverage is calculated using:
1 Σ Pr(Oo HDR(1 - aa) ~ C
1(0,,t e HI (1- a)).
(6)
The mean posterior log density at the true (or pseudo-true) parameters is another metric used in the SBI literature (Lueckmann et al.] 2021). We consider the approximate posterior log density at oo, for each S(y;). These results are attained using the same runs used to calculate the empirical coverage. As in Ward et al. (2022), we find that non-robust SBI mothods can occasionally fail catastrophically. Consequently, we also present boxplots to offer a more comprehensive depiction of the results. The whiskers are the upper and lower quartiles, with flier points (outliers beyond 1.5 times the interquartile range) not. being displayed.
In Figure 2 we illustrate the coverage and log-density at Og of SNL, RSNL and RNPE across four simulation examples. Figure 2 illustrates the performance of SNL and RSNL, with RSNL exhibiting more conservative coverage and higher density around Pp- While RSNT, is better calibrated than SNL on these misspecified examples, RSNL can still be overconfident, as observed in the contaminated normal and contaminated SLCP examples. We find that RNPE coverage is similar to RSNL. This further reassures that the differing coverage is artefact of the misspecified examples, in which case we do not have any guarantees of accurade frequentist. coverage. There is also a high degrec of underconfidence on two of the benchmark tasks, but this is preferable to the misleading overconfidence displayed by SNL in misspecified models. The boxplots reveal that RSNL consistently delivers substantial support around Gg At the same time, SNL has lower median log density and is highly unreliable, often placing negligible density around to-
4.1 Contaminated Normal
Here we consider the contaminated normal example from Frazier & Drovandi (2021 to assess how SNL and RSNL perform under model misspecification. In this example, the DGP assumed to follow:
y/ = 0 + G. q ~x ((O,,)),
where == I,..., n. However, the actual DGP follows:
(0+ El,bs
61 ~ No0,, 1), with probability w,
yi
0 1 62 62 NN(, of), with probability 1 w.
10

---

Published in Transactions on Machine Learning Research (06/2024)
1c12
Log density
Log density
Log density
Log density
-3000
-250
-15
RSNL SNL RNPE
RSNL SNL RNPE
RSNL SNL RNPE
RSNL SNL RNPE
Empirical coverage
Empirical coverage
Empirical coverage
Empirical coverage
RSNL
RSNL
RSNL
SNL RNPE
SNL RNPE
SNL RNPE
RSNL SNL RNPE
Credibility level
Credibility level
Credibility level
Credibility level
(a) Contaminated normal
(b) Misspecified MA(1)
Contaminated SLCP
(d) SIR
Figure 2: Comparison of SNL, RSNL and RNPE performance four misspecified examples. The top presents the approximate posterior log density boxplots at the true (or pscudo-true) parameters. The bottom row displays the empirical coverage across various credibility levels for SNL (dashed), RSNL (solid) and RNPE (dashed). A well-calibrated posterior closely aligns with the diagonal (dotted) line. Conservative posteriors are situated in the upper triangle, while overconfident ones are the lower triangle.
The sufficient statistic for under the assumed DGP is the sample mean, Si(y) - En For demon- stration purposes, let us also include the sample variance, S2(y) = - Ei= (ii - Si())) When a, & 1, unable to replicate the sample variance under the assumed model. We use the prior, A~ (N 10?) and 100. The actual DGP is set to 0 1, incompatible. Since S1(y) is sufficient, so is S(y), and one might still be optimistic that useful inference will result. We thus want our robust amgorithm to concentrate the posterior around the sample mean. Under the assumed DGP we have that b(0) = (0, 1)T, for all E R. Since bo = [1.0, 2.25]T, the sample variance is incompatible. We thus have infeeR [|b(0) - boll > meaning our model is misspecified. We include additional results for the contaminated normal, included Appendix D, due to the ease of obtaining an analytical true posterior. We also consider this cxample with no summarisation (i.e. using 100 draws directly) to see how RSNL scales as we increase the number of summaries, with results found in Appendix F.
0.8 and a, 2.5, and hence the sample variance is
For the contaminated normal example, Figure demonstrates that RSNL yields reliable inference, with high posterior density around the true paramcter value, # = In contrast, SNL provides unreliable inference, offering minimal support near the true value. The posteriors for the components of T are also shown. The prior and posterior for ?1 (linked to the compatible summary statistic) are nearly identical, aligning with RBSL behaviour. For 72 (associated with the incompatible statistie), misspecification is detected the posterior exhibits high density away from 0. A visual inspection suffices for modellers to identify the misspecified summary and offers guidance for adjusting the model. Further, we observed that in the well-specified case, the effect of the adjustment parameters on the coverage is minimal (sce Appendix D).
4.2 Misspecified MA(1)
We follow the misspecified moving average (MA) of order 1 example in Frazier & Drovandi 2021)) where the assumed DGP is an MA(1) model, Yt = + Ow,-1, -1 < 0 < and DGP is a stochastic volatility model of the form:
1.d.
~ NN 1). However, the true
Vt exp 2 We:
+ k2; + dus
where 0 < k6, < 1, and u4 U 0.90 and 0. 0.36. The data is summarised using the autocovariance function, ()(z) + Σi -,
ldd. NOO, 1). We generate the observed data using the parameters w = - -0.76,
11

---

Published in Transactions on Machine Learning Research (06/2024)
where T is the number of observations and {0, 1} is the lag. We use the prior 0 ~~ 11) and set T = 100.
We note that = 0 is a meaningful point on which to conduct inference, and one we would hope our posteriors would concentrate towards as the sample size increases. To see why, it is enough to note that under the true DGP for this experiment the observed data displays no autocorrelation the levels of the series. such, one would hope that the posterior for the parameter 0, which captures the level of autocorrelation in the first lag of the series, would place significant posterior mass around 6 0 and signify there is no meaningful autocorrelation in the data.
Figure 3 shows that while RSNL and RNPE methods both place significant amounts of posterior around the point = O, SNL appears to be concentrating onto a completely different point in the parameter space. The point around which most the SNL mass is present clearly at odds with the observed data: the SNL postcrior suggests that a moderate level of autocorrclation required fit the data, when in fact there no autocorrelation in the observed data.
As expected, 7i (corresponding to the incompatible statistic) has significant posterior density away from 0 seen in Figure Also, the postcrior for "/2 (corrosponding to the compatible statistic) closely resembles the prior. The computational price of making inferences robust for the misspecified MA(1) model is minimal, with RSNL taking around 20 minutes to run and SNL, taking around 10 minutes.
RSNL
Posterior
0.4
Posterior Prior
SNL
Prior
Density
Density
Density
RNPE
0.5
0.2
2-
0.0 10
0.0
10
10
0
10
71
Y/2
Figure 3: Posterior plots for the misspecified MA(1) model. The leftmost plot shows the estimated SNL (dashed), RSNL (solid) and RNPE (dashed) posterior densitics for 6. The true parameter value is shown as a vertical dashed line. The right two plots show the estimated marginal posterior (solid) and prior (dashed) densities for the components of T.
We may be concerned with the under-confidence of RSNL. (and RNPE) on the misspecified MA(1) example as illustrated in the coverage plot in Figure 2 But from the posterior plots in Figure 3 we see that, for the misspecified MA(1) benchmark example, SNL is not only over-confident but it is over-confident for point in the parameter space where the actual summaries and observed summaries are very different. In contrast, RSNL is under-confident, in that its posteriors are inflated relative to the standard level of frequentist. coverage, however, it is under-confident for the right point: the values over which the RSNL posterior are centred deliver simulated summarics that are as close as possible to the observed summaries in the Euclidean norm. We also note that, in general when models are misspecified, Bayesian inference does not deliver valid froquentist coverage (Kleijn & van der Vaart. 2012).
4.3 Contaminated SLCP
The simple likelihood complex posterior (SLCP) model devised in Papamakarios et al. (2019) is a popular example in the SBI literature. The assumed DGP is a bivariate normal distribution with the mean vector, #e = (2), and covariance matrix:
s
pS182
Σg
P8182
where st +02. 82 82 and p p= tanh(Os). This results in a nonlinear mapping from (01, Bg. 0,, 0s)) e R yj 2R22 for 1,....5. 5. The posterior is "complex', having multiple modes due to squaring and
12

---

Published in Transactions on Machine Learning Research (06/2024)
vertical cutoffs from the uniform prior that define in more detail later. Hence, the likelihood is expected to be easier to emulate than the posterior, making it suitable for SNL. approach. Four draws are generated from this bivariate distribution giving the likelihood, g 0) = _ My p4, Eg) for = (y1. V2. Vs, Y4). No summarisation is done, and the observed data is used in place the summary statistic. We generate the observed data at parameter values, 0 = (0.7, 2.9, - 1.0, 0.9, 0.6) and place an independent 4(-3,33 prior on each component of 0.
To impose misspecification on this illustrative example, draw contaminated 5-th observation, ys and use the observed data y (y1, 322 Ys. y/. ys). Contamination is introduced by applying the (stochastic) misspecification transform considered in Cannon et al. 2022). The assumed DGP is incompatible with this contaminated observation, and ideally, the approximate posterior would ignore the influence of this observation. Due to the stochastic transform, there is small chance that the contaminated draw is compatible with the assumed DGP. However, the results presented here consider a specific example, where the observed contaminated draw is ys (23.41, 117..90) which is very unlikely under the assumed DGP.
æs + 100zs, where 25 ~ ~No,, Σo), and
2B ~ O'), 10012).
We thus want inference to only use information from the four draws from the true DGP. The aim is to closely resemble the SNL posterior, where the observed data is the four non-contaminated draws. Figure ! shows the estimated posterior densities for SNT, (for both compatible and incompatible summaries) and RSNL for the contaminated SLCP example. When including the contaminated 5-th draw, SNL produces a nonsensical posterior with little useful information. Similarly, RNPE cannot correct the contaminated draw and does not give useful inference. Conversely, the RSNL posterior has reasonable density around the true parameters and has identified the separate modes.
The first eight compatible statistics are shown in Figure The prior and posteriors reasonably match each other. In contrast, the observed data from the contaminated draw is recognised as being incompatible and has significant density away from 0, as evident in Figure + Again, there is no significant computational burden induced to estimate the adjustment parameters, with total computational time of around 6 hours to run RSNL and around hours for SNL..
13

---

Published in Transactions on Machine Learning Research (06/2024)
RSNL
SNL-incompatible
SNL-compatiblc
RNPE
6
6
3
0
3
6
0
-3
3
0
3
0
-3
3
0
-3
-3
0
3
-3
-3
0
3
-3
-3
0
3
01
02
θ
O4
O5
Figure 4: Univariate and bivariate density plots of the estimated posterior for on the SLCP example. Plots on the diagonal are the univariate posterior densities obtained by RSNL (solid), RNPE (dash-dotted), SNL (dashed) the contaminated SLCP example, and for SNL without the contaminated draw (dotted). The bivariate posterior distributions for contaminated SLCP are visualised as contour plots when applying RSNL. (solid, lower triangle off- off- diagonal) and SNL (dashed, upper triangle off-diagonal). The true parameter values are visualised as a vertical dashod line for the marginal plots and the x symbol in the bivariate plots.
Figure shows that while RSNL, is reasonably well-calibrated, RNPE does not give reliable inference. This is because RNPE could not correct for the high model misspocification of the contaminated draw. Additionally, approaches that emulate the likelihood rather than the posterior typically give better inference for this example [Lueckmann et al. 2021
14

---

Published in Transactions on Machine Learning Research (06/2024)
Density
0.4
Posterior Prior
Density
0.4
Posterior Prior
Density
0.4
Posterior Prior
Density
0.4
Posterior
Prior
0.2
0.2
0.2
0.2
0.0
0.0
0.0
0.0
-10
-5
5
10
10
-5
0
5
10
-10
-5
0
10
-10
-5
0
5
10
71
72
73
Y4
Density
0.4
Posterior Prior
Density
0.4
Posterior Prior
Density
0.50
Posterior Prior
Density
00.4
Posterior Prior
0.2
0.2
0.25
0.2
0.0
0.0
0.00
0.0
-5
5
10
10
-5
0
5
10
-10
-5
0
5
10
-10
-5
5
10
75
76
77
Y88
Figure 5: Estimated marginal posterior (solid) and prior (dashed) for components of associated with the non-contaminated draws of the contaminated SLCP example.
5.0
Posterior Prior
Posterior Prior
Density
Density
20
2.5
10-
0.1
0.0
0
5
-10
10
79
Figure 6: Estimated marginal posterior (solid) and prior (dashed) for components of I associated with the contaminated draw of the contaminated SLCP example.
4.4 Misspecified SIR
We follow the misspecified museeptibe-iiffeetd-rroooo models are a simple compartmental model for the spread of infectious diseases. The standard STR model has an infection rate, B, and recovery rate, 7). The population consists of three states: susceptible (S), infected (I) and recovered (R). The standard SIR model defined by:
(SIR) model described in Ward et al. (2022). SIR
dS dt = 3ST, dt BST -
dI
dRR.
n/,
n/.
dt
The assumed DGP is a stochastic extension of the deterministic SIR model with time- varevngrying infection rate, B. This allows the SIR model to better capture heterogeneous disease outcomes, virus mutations and mitigation strategies (Spannaus et al. 2022). In addition to the equations in 7 the assumed SIR model is also parameterised by a time-varying effective reproduction number, Ret = n
dR.t = v (1 Ret di + a RodW,, -Ret
where is the reversion to Ret, o is the volatility and W, Brownian motion. We in Ward al. (2022), and use priors, 7) 2 (., and B U(n 0.5). We use 8 = 0.15 and y 0.1 for all observed data. The assumed DGP is run for 365 days, and only the daily number of infected individuals is considered. The initial number of infected individuals is 100. The infected counts are scaled by 100,000 to represent larger population. A visualisation of the observed DGP is shown in Appendix D.
0.5 and a = 0.5
The true DGP has a reporting lag in recorded infections. Weekend days have the number of recorded infections roduced by 5%, being recorded on Monday, which sees an increase of 10%. Six summary statistics are considered: mean, median, max, max day (day of max infections), half day (day when half of the total infections was reached), and the autocorrelation of lag T.
15

---

Published in Transactions on Machine Learning Research (06/2024)
The model is misspecified, the assumed SIR model cannot replicate the observed autocorrelation summary. We want our robust algorithm to detect misspecification in the autocorrelation summary and deliver useful inferences. We consider a specific example where the observed autocorrelation is 0.9957. Under the assumed DGP, the simulated autocorrelation is tightly centred around 0.9997. Despite the secmingly minor difference between the observed and simulated summaries, the observed summary lies lar the lails post standardisation.
As shown in Figure 7 RSNL produces useful inference with high density around the parameters. that the misspecification has been detected for
Inspecting the adjustment parameters in Figure g the autocorrelation summary statistic and adjusted accordingly. Consequently, the modeller further refine the simulation model to better capture the observed autocorrelation. This refinement could be achieved by recognising that the assumed DGP fails to account for the reporting lag evident in the observed data. RNPE behaves similarly to RSNL and concentrates around the true gentaaeenn parameters. SNL, however, focuses overconfidently an inconsequential region of the parameter space.
As the main computational cost in this example is run the SIR simulations, there is negligible impact from the addition of adjustment parameters. In the example considered in Figures 7 and SNL and RSNL both took approximately 24 hours.
0.3
0.3
0.3
0.0.2
00 0.2-
00..22
0.1
0.1
0.1
0.050
0.175
0.050
0.175
0.050
0.175
7
7)
77
(a) RSNL
(b) RNPE
(c) SNL
Figure 7: Depiction of univariate and bivariate density plots of the estimated posterior for parameters 3 and y in the SIR model. The bivariate posterior distributions, visualised as contour plots, are presented for RSNL (left plot), RNPE (middle plot) and SNL (right plot). Corresponding univariate density plots are displayed on the sides of each plot. The true parameter values are marked with a x in the bivariate plots.
16

---

Published in Transactions on Machine Learning Research (06/2024)
0.5-
Posterior
1.2
Posterior
0.5
Posterior
Density
0.4
Prior
Density
1.0
Prior
Density
0.4
Prior
0.3-
0.8
0.3
0.6
0.2
0.2
0.4
0.1-
0.2
0.1
0.0
0.0
0.0
10
5
0
5
10
10
-5
5
10
10
5
0
5
10
71
Y/
0.6
0.8
Posteriór Prior
Posteriór
100
Posterior
0.5
Prior
80
Prior
Density
0.6
Density
0.4
Density
60-
0.4
0.3
0.2
40
0.2
0.1
20
0.0
0.0
0
10
-5
0
5
10
10
5
0
5
10
10
5
0
5
10
7/4
Y/s
%
Figure 8: Estimated marginal posterior (solid) and prior (dashed) for components of obtained using RSNL for the STR model.
4.5
Toad movement model
We consider here the animal movement model by Marchand et al. 2017 to simulate the dispersal of Fowler's toads (Anaryrus fowleri). This is individual-based model that encapsulates two main behaviours that have been observed in amphibians: high site fidelity and small possibility of long- dintdiit movement. The assumed behaviour is for each toad to act independently, stay at a refuge site during the day and move to forage at night. After foraging, the toad either stays in its current location or returns to a previous refuge site, A toad returning to previous refuge site occurs with constant probability Po- We concentrate on "model 2" in Marchand al. (2017), which models the behaviour of returning its nearest previous refuge. This specific model was chosen there is evidence of model misspecification, allowing us to assess method misspecified example with real data.
The dispersal distance is modelled using Lévy alpha-stable distribution, parameterised by a stability factor atoad and scale factor 5. This distribution was chosen for its heavy tails, which allow for occasional long-distance movement while still being symmetric around zero. Although the Lévy alpha-stable distribution lacks a closed form, is straightforwant We assume the following uniform prior distributions for the model parameters: Qroad [11, 2), U(20, 70), and Pa ~ 0.9).
simulate. Thus the model is governed by three parameters: O [arondd 5, 5.po]
The Marchand et al. (017 GPS data was collected from 66 individual toads, with the daytime location (i.e. while resting refuge) being recorded. The number of recorded days varied across toads, with maximum of 63 days. The two-dimensional GPS data converted to a one-dimensional movement component, resulting in an observed (63 x 66) matrix. The observed matrix was summarised using four sets of displacement vectors with time lags of 1, 2, 4 and days. For cach lag, the number of absolute displacements less than 10m, the median of the absolute displacements greater than 10m, and the log difference of the 0, 0.1,..., 1 of the absolute displacements greater than 10m are calculated, resulting in total of 48 summary statistics (12 for each time lag).
In addition to SNL and RNPE, we evaluate our method against RBSL, as we are assessing performance on the real observed data, and the ground truth is unavailable for direct comparison. We examined plots for RSNL, SNL, RNPE and RBSL using real and simulated data from the toad example. In Figure g the estimated model parameter posteriors are conditioned on real observed summary statistics, with RBSL the baseline for comparison with RSNL. The marginal plots reveal that RSNL closely resembles RBSL, while
17

---

Published in Transactions on Machine Learning Research (06/2024)
RSNL
SNL
RBSL
RNPE
2.0
rood
1.5
1.0
70
60 45
20
0.90
0.65
0.40
1.0
1.5 Groad
2.0
20
45
70
0.40
0.65
0.90
Figure Univariate and bivariate density plots of the estimated postcrior for applied to the real data for the toad movement example. Plots the diagonal the univariate posterior densities obtained by RSNL (solid), SNL (dashed), RBSL (dotted) and RNPE (dash-dotted). The bivariate posterior distributions for the toad movement example are visualised as contour plots when applying RSNL. (solid, lower triangle off-diagonal) and SNL (dashed, upper triangle off-diagonal).
SNL differs, showing minimal density around the RSNL and RBSL maximum a posteriori estimates. RNPE also gives similar marginal posteriors RSNT. and RBSL. The 95% credible intervals for each parameter are displayed in Table 1 However, when considering simulated data (sce Appendix D), SNL yiclds similar inferences to the robust methods. This suggests that SNL's differing results in the real data scenario arise from incompatible summary statistics.
The most incompatible summary statistics were identified as the number of returns with lag of and the first quantile differences for lags 4 and 8. These were determined via MCMC output and visual inspection. We depict the posteriors for the adjustment parameters corresponding to these incompatible summnaries Figure I0 alongside the first three posteriors for compatible summarics, which are expected to closely match their priors. This example illustrates the advantage of carefully selecting summary statisties that hold intrinsic meaning for domain experts. For example, the insight that the current model cannot adequately capture the
18

---

Published in Transactions on Machine Learning Research (06/2024)
low number of observed toad returns- partieulaly while also fitting other movement behaviours- has direct meaning to the domain expert modeller, enabling them to refine the model accordingly.
0.5
0.4
Posterior Prior
Posterior Prior
0.4
Posterior
Density
0.3
Density
0.4
Density
0.3
Prior
0.3
0.2
0.2
0.2
0.1
0.1
0.1
0.0
0.0
0.0
10
5
5
10
10
5
0
5
10
10
5
0
5
10
71
127
Y39
(a) Adjustment parameters corresponding with the most incompatible summary statistics.
0.5
0.5
Posterior
Posterior Prior
0.4-
Posterior Prior
0.4
Prior
0.4
Density
Density
Density
0.3
0.3
0.3
0.2
0.2
0.2
0.1
0.1
0.1-
0.0
0.0
0.0
10
5
5
10
10
5
10
10
5
5
10
72
%/3
7/4
(b) First three adjustment parameters corresponding with compatible summary statistics.
Figure 10: Estimated marginal posterior (solid) and prior (dashed) for selected components of T obtained
using RSNL for the toad movement model.
In Figure 11 we present distributions for the log distance travelled by non-returning toads derived from the (pre-summarisee)) observation matrix. The simulated distances closely align with, and are tightly distributed around, the observed distances. Differences between the simulated and observed log distances appear most noticcable at lags 4 and 8 for shorter distances, aligning with the identified incompatible summarics. Figure 12 displays boxplots for the number of returns across the four lags. As confirmed by the first incompatible summary, the model has difficulies replicating the observed number of toad returns while accurately capturing other sunmary statistics. Overall, the posterior predictive distributions largely agree with observed data, with discrepancies coinciding with the incompatible summaries identified.
For the toad movement model, as the focus is on the performance of the methods on real data (with no ground-truth parameter), we instead consider the posterior predictive distribution. To compare predictive performance across methods, we computed the MMD between the observed summary statistic and samples from the posterior predictive, as presented in Table I Implementation details for the MMD can be found Appendix D. We highlight that RSNL has a lower discrepancy than SNL and similar results RNPE. While RBSL is the best-performing method in this example, it requires orders of magnitude more model simulations. We also note that the most incompatible summary statistics identified via the adjustment parameters (see results in Appendix D) agree with those found in Frazier & Drovandi (002))
19

---

Published in Transactions on Machine Learning Research (06/2024)
Lag
Lag 2
0.6-
0.6
Density
Density
0.4
0.4
0.2
0.2
0.0
0.0
0
5
10
5
10
Log distance
Log distance
Lag 4
Lag 8
0.6
0.6
Density
Density
0.4
0.4
0.2
0.2
0.0
0.0
10
10
Log distance
Log distance
Figure 11: The RSNT posterior predictive distributions for the log distance of toads who moved to refuge site, applied across four lags. The thick black line is the observed distribution.
Observed
Number returned
300
200
100
0
2
4
8
Lag
Figure 12: The RSNL posterior predictive for the number of toads who returned to the same refuge site across four lags.
20

---

Published in Transactions on Machine Learning Research (06/2024)
Table 1: Estimated 95% credible intervals and the MMD discrepancy between the observed summary statistic and samples from the posterior predictive for RSNL, SNL, RNPE and RBSL-M on the toad movement model with real data.
Method Number simulations
Otoad (2.5%
97.5%)
7 (2.5% 97.5%)
Po (2.5% 97.5%
MMD
RSNL 10000
(1.28 1.77)
(35.21 -77.4)
(0.61 0.76)
0.006
SNL 10000
(1.63 1.95)
(44.61 53336)
(0.63 0.74)
0.015
RNPE 10000
(1.28 1.84)
(31.45 48.688
(0.60 0.80)
0.007
RBSL-M
25000000
(1.35 1.80)
(35.67 47.48)
(0.59 0.73)
0.001
5
Discussion
In this work, we introduced RSNL, a robust neural SBI method that detects and mitigates the impact of model misspecification, making it the first method of its kind that uses a surrogate likelihood or sequential sampling. RSNL demonstrated robustness to model misspecification and efficient inference on several illustrative examples.
RSNL provides useful inference with significantly fewer simulation calls than ABC and BSL methods. For instance, only 10,000 model simulations were needed for the RSNL posterior in the contaminated normal model, while RBSL in Frazier & Drovandi (2021) required millions of simulations. A more comprehensive comparison, such as the benchmarks in Lueckmann et. al. (2021), could assess the robustness of ABC, BSL, and neural SBI methods to model misspecification and evaluate their performance across different numbers of simulations. Ideally, such a benehmark would include challenging real-world data applications, showcasing the utility of these methods for scientific purposes.
In this paper, we consider summaries that are carefully specified by the modeller, as opposed to neural
networks or algorithms that learn the summarics (e.g. via an autocncoder (Albert et al. 2022 ). While learnt summaries can be valuable, including addressing model misspecification (Huang et al. 2023 interpretable summary statistics chosen by domain experts to be meaningful to them are often crucial to model development. We see model fitting under misspecification as having two main functions. The first is to minimise harm from fitting a misspecified model, where inaccurate uncertainty quantification will lead the domain expert extract misleading insights into the phenomena of interest. The second is to enable more meaningful model criticism allowing better model to be developed. For the first function, the purpose of the model must be laken into account, and this can take the form of deciding which of a set of interprelable summaries it is important to match in the application. For the second function, knowing which of set of interpretable summary statistics cannot be matched is insightful to experts for the purpose of model refinement and improvement. needs to be clarified how current methods for obtaining learnt summary statistics allow the modeller to refine the current. model further. However, our adjustment parameter approach could also be applied when the summaries are learnt.
Our relevant form of model misspecification, incompatibility, could be interpreted an issue of OOD data. OOD data refers to data drawn from a different distribution than the one used to train the neural network. It is an important issue in the ML community to detect the presence of OOD data (Hendrycks & Gimpel 2017 Yang et al. 2022). Normalising flows struggle with OOD data [Kricchenko et al. (020). Mechanically, this is exactly what happens when we train surrogate normalising flow using simulated data from the misspecified model and evaluate it on the observed data from reality. Favourable results across numerous neural SBI methods have been achieved in Cannon et al. (2022) using OOD detection methods based on ensemble posteriors (Lakshminarayanan et al. 2017) and sharpness-are -waeee minimisation (Foret et al.) 2021 Combining these OOD methods with adjustment parameters could enhance their benefits. Another strategy to counteract the effects of OOD data that has been considered in a non-SBI context involves employing a physics-basee correction to the latent distribution of the conditional normalising flow that learns the posterior (Siahkoohi et al. 2021 2023
We highlight one example where the model misspecification is not incompatible summaries but rather an inappropriate choice of summary statistics. Consider the contaminated normal described in Section 4 but
21

---

Published in Transactions on Machine Learning Research (06/2024)
with only the sample mean. The sample mean is sufficient and compatible summary statistic, and we can match it with the assumed univariate normal distribut ion despite it being generated from two normal distributions with different standard deviations. So, we can have compatible summaries where the assumed DGP misrepresents reality. However, in such instances, we can expect SBI algorithms to be "well-behaved" in that they will produce meaningful inference on the unknown model parameters (Frazier el al. 20206 David T. Frazier& & Kohn 2023). It is difficult to derive the same theoretical backing for neural SBI methods as has been done for ABC and synthetic likelihood; however, given the expressive power of normalising flows PPapamakaiios et al., 2021), we may expect similar results to hold when the observed data is "in-distributii of the simulated data. This stands in contrast to the case of incompatibility, where it has already been observed that various SBI methods, such as synthetic likelihood (Frazier et al. 2021), and neural methods, such as SNL (Cannon et al. 2022), can give nonsensical results under incompatibility. In this described instance, looking at more detailed features of the full data would have revealed the deficiencies of the assumed DGP. In general, the modeller can use the posterior predictive distribution to generate the full data and probe for any aspects the assumed model is unable to explain (e.g. the log-distance plots for the load movement. model in Figure []]. This highlights the importance of judiciously selecting summary statistics when building models in SBI. Ideally, the summaries would be selected to capture key aspects of the data (Lewis et al. 2021), and there is a broad literature on choosing appropriate summaries (Prangle) 2018). Our proposed method would assist in selecting summaries, as it ensures more reliable inference and provides diagnostics when the summaries are incompatible.
RBSL-M accounts for the different summary scales and the fact that these scales could be d-dependent by adjusting the mean using pu00) + (0) I, where a (f) is vector of estimated standard deviations of the model summarics at O. RBSL estimates these standard deviations from the model simulations generated based on O. Analogously, we could consider a similar approach in RSNL and define the larget
π T0, r S(y)) x (S(y)) - (0) o o O) (T)
The question then becomes, how do we estimate (0) in the context of RSNL? In the MCMC phase, we do not want to generate more model simulations as this would be costly. If we believed that the standard deviation of the model summaries had little dependence on 0, we could set x00) = a = a0) where O is some reasonable point estimate of the parameter. Another approach would, for each @ proposed in the MCMC, estimate (0) using surrogate model simulations generated using the fitted normalising flow. This would be much faster than actual model simulations but could still slow down the MCMC phase substantially. Instead of using a normalising flow, we could train a mixture density network (Bishop, 1994) to emulate the likelihood, which would then lead to an analytical expression for (0). A multivariate mixture density network could replace the flow completely, or the multivariate flow for the joint summary could be retained and a series of univariate mixture density networks applied to each summary statistic for the sole purpose of emulating (0). We plan to investigate these options in future rescarch.
The introuuction of adjustment parameters in RSNL might raise concerns about introducing noise into the estimated posterior. However, our empirical findings indicate that the impact of this noise is negligible, particularly when using our chosen prior. This observation aligns with the RBSL results presented in Frazier & Drovandi (0221). Furthermore, Hermans et all (2022) noted that SBI methods, including SNL, often produce overconfident posterior approximations. Thus, it is unlikely that the minor noise introduced by adjustment parameters would lead to excessively conservative posterior estimates. Recent work has proposed solutions to neural SBI overconfident posterior approximations [Delannoy et al. 2022: 2023: Falkiewicz et al. 2023). It would be interesting to see if these methods could be combined with our proposed method to have both correctly calibrated postcriors and robustness to model misspecification, however this is beyond the scope here.
Evaluating misspecified summaries can be done by comparing the prior and posterior densities for each component of T. In our examples, we used visual inspection for this purpose. However, for cases with large number of summaries, this method may become cunbersome. Instead, an automated approach could be implemented to streamline the process and efficiently identify misspecified summaries. While the posteriors of the adjustment parameters can be used to diagnose misspecification, RSNL lacks many of the diagnostic tools available to amortised methods (c.g Talts et al. (2018): Hermans et al. (2022)) due to its sequential sampling scheme.
22

---

Published in Transactions on Machine Learning Research (06/2024)
Our proposed prior may not be suitable for two scenarios, even if the normalising flow learns the likelihood perfectly. First, consider where summary statistic is incompat iblle and, after standardisation, the adjustment parameter variance is set small. This could occur when the distribution of the model summary statistic is multi-modal at a specific parameter value. However, in such multi-modal sccnarios, flow-based noural likelihood methods generally exhibit poor performance (Glaser et al., 20230, 80 we might not want to apply RSNL, this socnario regardless. In this case, RSNL, will behave similarly to SNL for that particular summary. Second, a summary is correctly specified but lies in the tails. This is unlikely by definition of being in the tails, and the effect of this would be for the summary in the tails to be corrected as if it was misspecified. If concerns arise, researchers can visualise the summary statistic plots generated at reasonable model parameter point or examine posterior predictive distributions. If necessary, an alternative prior can be employed. Finally, RSNL could diagnose and shift incompatible summaries as expected, but there might be broader issues that prevent useful insight into the scientific questions of the phenomena of interest (e.g. poor quality of observed data). We would not expect an inference method be able to handle this, but RSNL can still be useful within broader workflow for model criticism and further model refinement.
The choice of π (I) was found to be crucial in practice. Our prior choice was based on the dual requirements to minimisc noise introduced by the adjustment paramcters if the summaries are compatiblc and to be capable of shifting the summary significant distance from the origin if they are incompatible. The horseshoe prior is an appropriate choice for these requirements. Further work could consider how to implement this robustly in a NUTS sampler. Another approach the spike-and-slab prior as in Ward et al. (2022). This type of prior is a mixture of two distributions: one that encourages shrinkage (the spike) and another that allows for a wider range of values (the slab). Further research is needed to determine the most appropriate prior choice for RSNL and similar methods, which could involve comparative study of different prior choices and their effects on the robustness and efficiency of the resulting inference.
Modellers constructing complex DGPs for real-world data should address model misspecification. The machine learning and statistics community must develop tools for practitioners to conduct neural SBI mothods without producing misleading results under model misspecification. We hope that our proposed method contributes to the growing interest in addressing model misspecification in SBI.
Acknowledgments
The authors express their gratitude to the anonymous reviewers for providing useful comments and the TMLR editorial team for their guidance. RPK was supported by a PhD Research Training Program scholarship from the Australian Government and a QUT Centre for Data Science top-up scholarship. CD and DTF were supported by Australian Research Council funding schemes FT210100260 and DE200101070, respectively. DJN acknowledges the support from the Singapore Ministry of Education Academic Research Fund Tier grant. RPK, DJW, and CD thank the Centre for Data Science at QUT for its support. The eResearch Office at QUT provided computational resources.
References
Carlo Albert, Simone Ulzoga, Firat Ozdemir, Fernando Porez-Cruz, and Antonictta Mira. Learning summary statistics for Bayesian inference with autoencoders. SciPost Physics Core, 5(3):043, 2022.
Christophe Andrieu and Gareth O. Roberts. The margin-aarginal approach for efficient Monte Carlo computations. The Annals of Statistics, (7 2009. URL http:://ww. wwtgssttrrrg//teeeeeeeeeeeee atg/stable/ 30243645
Stuart Barber, Jochen Voss, and Mark Webster. The rate of convergence for approximale Bayesian computation. Electronic Journal of Statistics, 9(1):80- 105, 2015. doi: 10.1214/15-EJS983 URL httpp::/ddoi.rgg//
1214/15-EJ9988
Mark A. Beaumont. Approximate Bayesian computation in evolution and ecology. An- nual Review of Ecology, Evolution, and Systematics, (11)3379-4406 2010. ISSN 1543-
592X. doi: 10.1146/amurev-ccolsyy 100.0y--0022--1111 amnue
102209-144621. 144621.
URL https://áx: www.dooo.oooo.o. orgg.or//10..146..
23

---

Published in Transactions on Machine Learning Research (06/2024)
Yves Bernaerts, Michael Deistler, Pedro J Goncalves, Jonas Beck, Marcel Stimberg, Federico Scala, Andreas S Tolias, Jakob H Macke, Dmitry Kobak, and Philipp Berens. Combined statistical-mechaniss modeling links ion channel genes to physiology of cortical neuron types. bioRriv, pp. 2023 03, 2023.
Christopher M. Bishop. Mixture density networks. Technical Report, Aston University, 1994.
Pier Giovanni Bissiri, Chris C Holmes, and Stephen G Walker. A general framework for updating belief distributions. Journal of the Royal Statistical Sociely Series B: Statistionl Methodology, 78(5):1103, 2016.
David M. Blei. Build, compute, critique, repeat: Data analysis with latent variable models. Annual
Review of Statistics and Its Application, 1(Volume 1, 2014):203 232, 2014. ISSN 2326-831X. doi: https:
//doi.org/10. 1t////000111///mm// /ata///rrsstaasstiis-22222 // journals/10. 1100..1466, 1146/ amaraeessstaaaaeieee-22---------
115637. URL https://www
Jan Boelts, Jan-Matthis Lueckmann, Richard Gao, and Jakob H Macke. Flexible and efficient simulation-baser inference for models of decision-making. eLife, 11:e77220, 2022.
Joshua J Bon, David J Warne, David J Nott, and Christopher Drovandi. Bayesian score calibration for approximate models. arXie preprint or arXiv:2211.052 0533577, 2022.
Joshua J Bon, Adam Bretherton, Katic Buchhorn, Susanna Cramb, Christopher Drovandi, Conor Hassan, Adrianne L Jenner, Helen J Mayfield, James M McGree, Kerrie Mengersen, et al. Being Bayesian in the 2020s: opportunities and challenges in the practice of modern applied Bayesian statistics. Philosophical Transactions of the Royal Society A, (81(2247):2022015555 2023.
George EP Box. Science and statistics. Journal of the American Statistical Association, 71(356):791 799, 1976.
James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milnn and Qiao Zhang, JAX: composable transformations of Python NumPy programs, 2018. Python package version 0.3.13, URL httpps://github jom/google/jax
Patrick Cannon, Daniel Ward, and Sebastian M. Schmon. Investigating the impact of model misspecification in neural simulation-based inference, 2022. URL httpp::/aaxxi..orr/bbbstt 01855 arXiv preprint 22l::2299008445.
Badr-Eddine Chérief-Abdellane and Pierre Alquier. MMD-Bayes: robust Bayesian estimation via maximum mean discrepancy. In Symposium on Advances in Approzimate Bayesian Inference, pp. 1-21. PMLR, 2020.
Basile Confavreux, Poornima Ramesh, Pedro Goncalves, Jakob H. Macke, and Tim P. Vogels. Meta-learning familics of plasticity rules in recurrent spiking networks using simulation-based inference. In Thirty-seventi Conference on Neural Information Processing Systems, 2023. URL Stttpss///pperrvviee.eeeefooo FLFasCFJNo
Kyle Cranmer, Johann Brehmer, and Gilles Louppe. The frontier of simulation-based inference. Proceedings of the National Academy of Sciences, 117(48):30055 30062, 2020.
Katalin Csilléry, Michael G.B. Blum, Oscar E. Gaggiolti, and Olivier François. Approximate Bayesian computation (ABC) in practico. Trends in Ecology & Enolution, 2507):410-418, 2010. ISSN 0169-5347. doi: http://o.ooorg/00001/.....e.... pi1/800699534710000622
URL http://www..cceeeeeeeeeccc..o//a
Christopher Drovandi David T. Frazier, David J. Nott and Robert Kohn. Bayesian inference using synthetic likelihood: asymptotics and adjustments. Journal of the American Statistical Association, 118(544): 1821-28331 2023. doi: 10.1080/01621459..00 2022 2086132. URL https://doi. http://....0rr010100010000111111
2086132
24

---

Published in Transactions on Machine Learning Research (06/2024)
Arnaud Delaunoy, Joeri Hermans, François Rozet, Antoine Wehenkel, and Gilles Louppe. Towards reliable simulation-based inference with balanced neural ratio estimation. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL
https://openreviors net/forumiid 72mMM444
Arnaud Delaunoy, Benjamin Kurt Miller, Patrick Forré, Christoph Weniger, and Gilles Louppe. Balancing simulation-based inference for conservative posterions. ar preprint or 100::8344.0978, 2023.
Charita Dellaporta, Jeremias Knoblauch, Theodoros Damoulas, and François-Xavits Briol. Robust Bayesian inference for simulator-based models via the MMD posterior bootstrap. In International Conference Artificial Intelligence and Statistics, pp. 943 970. PMLR, 2022.
Christopher Drovandi and David T Frazier. A comparison of likelihood frree methods with and without summary statistics. Statistics and Computing, 32(3):1-23, 2022.
Conor Durkan, Artur Bekasov, Jain Murray, and George Papamakarios. Neural spline flows In Advances in Newral Information Processing Systems 32, 2019. URL httpp:://rrocesdings newrips...ppp.. / pdf
Conor Durkan, lain Murray, and George Papamakarios. On contrastive learning for likelihood-free inference.
In The 37-th International Conference on Machine Learning, pp. 2771-2781. PMT.R, 13-18 Jul 2020. URL
https://proceedingg. mlr, htgp://ddenn.........ese//////aa
html
Michael Evans and Gun Ho Jang. Weak informativity and the information in one prior relative to another. Statistical Science, 26(3):423 439, 2011. doi: 10.1214/11-STSS0 URL attps://doi.orr/100 1214/
11-ST$357
Macicj Falkiewicz, Naoya Takeishi, Imahn Shekhzadeh, Antoine Wehenkel, Arnaud Delaunoy, Gilles Louppe, and Alexandros Kalousis. Calibrating neural simulation-base inference with differentiable coverage probability. In Thirty-scventh Conference on Newral Information Processing Systems, 2023. URL https: //openrevies ew
Neil M Ferguson, Daniel Laydon, Gemma Nedjati-Gilan Natsuko Imai, Kylic Ainslic, Marc Baguelin, Sangeeta Bhatia, Adhiratha Boonyasiri, Zulma Cucunubá, Gina Hhooo--Danenburgg et al. Impact of non-pharnaceutical interventions (NPIs) to reduce COVTD-19 mortality and healthcare demand. Technical Report, Imperial College COVID-19 Response Team London, 2020. DOL 10.25561/77482.
Pierre Foret, Ariel Kleiner, Hosscin Mobahi, and Behnam Neyshabur. Sharpncss-aware minimization for efficiently improving generalization. In International Conference on Learning Representations, 2021. URL
D Frazier, G M Martin, C P Robert, and Rousseau. Asymptotic properties of approximate Bayesian computation. Biometrika, 105(3):593) 607, 06 2018. ISSN 0006-3444. doi: 10.1093 10.1093/biomet// /sy0077 URL
https://doi.org org/ 10. 10w110099/ooee /00110/////ommtt///////
David T. Frazier and Christopher Drovandi. Robust approximate Bayesian inference with synthetic likelihood. Journal of Computational and Graphical Statistics, 30(4):058 2021. doi: http//0066660000000 2021. 1875839. URL
httg//00.10001006116000 18771
David T Frazier, Christopher Drovandi, and Ruben Loyiza Robust approxinate Bayesian computation: an adjustment approach. arXiv preprint arXiv:2008.040 000:2000004099, 2020a.
David T. Frazier, Christian Robert, and Judith Rousseau. Model misspecification in approximate
Bayesian computation: consequences and diagnostics. Journal of the Royal Statistical Society: Series B
(Statistical Methodology),, 82(2):421 444, 2020b. doi: https://doo http:///doooor/000111111111 1235. URL https:
//rss. www.elliiaar...lll..ooomm abs////0.aaaa/....../ /11payyoco//o///a//////11111//11
1111/rssb. 12356
David T Frazier, Christopher Drovandi, and David Nott. Synthetic likelihood in misspecified models: consequences and corrections. Xiu preprint arXiv:2104 04444304 2021.
25

---

Published in Transactions on Machine Learning Research (06/2024)
Jonah Gabry, Daniel Simpson, Aki Vchtari, Michael Betancourt, and Andrew Gelman. Visualization in Bayesian workflow. Journal of the Royal Stalistical Society Series A: Statistics in Sociely, (82(())389) :3:8--99000 402, 01 2019. ISSN 0964-1998. doi: 10.1111/r9sa. 12378. URL https://doi. otg/10..r1111100.
http::///o...org/...11111//1//// 11tp///0..11///aa..r 12378
Richard Gao, Michael Deistler, and Jakob H. Macke. Generalized Bayesian inference for scientific simulators via amortized cost. estimation. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL. htttt:://pperreiieeeeeee http://wwww....eeeorrrmomaaaaaaa
Andrew Gelman, Aki Vehtari, Daniel Simpson, Charles C Margossian, Bob Carpenter, Yuling Yao, Lauren Kennedy, Jonah Gabry, Panl-Christian Bürkner, and Martin Modrák. Bayesian workflow. arXiv preprint arXiv:2011.01888, 2020.
Pierre Glaser, Michael Arbel, Arnaud Doucct, and Arthur Gretton. Maximum likelihood learning of energy-based models for simulation-based inference, 2023. URL htttps://openreviexx. ittper/rriieeeettt/urrrriii
gL68u5UuWa
Manuel Glöckler, Michael Deistler, and Jakob H. Macke. Adversarial robustness of amortized Bayesian inference. In Proceedings of the 40th International Conference on Machine Learning, ICML '23. JML.R.org, 2023.
Tilmann Gneiting and Adrian E Raftery. Strictly proper scoring rules, prediction, and estimation. Journal of the American statistical Association, 102(477):359 359-378, 2007.
Pedro J Gonçalves, Jan-Matthis Lueckmann, Michael Deistler, Marcel Nonnenmacher, Kaan Öcal, Giacomo Bassetto, Chaitanya Chintaluri, William F Podlaski, Sara A Haddad, Tim P Vogels, al. Training deep neural density estimators to identify mechanistic models of neural dynamics. Elife, 9cc56261, 2020.
David Greenberg, Marcel Nonnenmacher, and Jakob Macke. Automatic posterior transformation for likelihood- free inference. In The 36-th International Conference on Machine Learning, pp. 2404 2414. PMLR, 09-15 Jun 2019. URL
mlr, pttpeee//8./gggeeerrrl...aa.
Arthur Gretton, Karsten M. Borgwardt, Malte J. Rasch, Bernhard Schölkopf, and Alexander Smola. A kernel two-sample test. Journal of Machine Learning Research, 13(25):723 773, 2012. URL httt::/jtmmm org/
sapers/v13/grettontlt html
Severin Hauenstein, Julien Fattebert, Martin U. Grüebler, Beat Naef-Daenzer, Guy Pe'Er, and Florian Hartig. Calibrating an individual-based movement model to predict functional connectivity for little owls. Ecological Applications, 29 (3)4470,, 2019. ISSN 1051-0761. doi: 10.1002/eap.1873. URL https:
//dx.doi. org/ 10. 1002/ea0aaap eap. 1873
Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and out-of-distribution examples in neural networks. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.nef. 2017. URL httpp:///penreevir.. net/ forum? Hkg4419x1
Joeri Hermans, Volodimir Begy, and Gilles Louppe. Likelihood-free MCMC with amortized approximate ratio estimators. In The 37-th International Conference on Machine Learning, pp. 1239-4248 PMLR, 13-18 Jul 2020. URL htttps://proceede/ee http:s//ppoeelngggg....ssssss/ss
mlr.
html
Joeri Hermans, Nilanjan Banik, Christoph Weniger, Gianfranco Bertone, and Gilles Louppe. Towards constraining warm dark matter with stellar streams through neural simulation-based inference. Monthly Notices of the Royal Astronomical Society, (07(9))1999-201 08 2021. ISSN 0035-8711. doi: 10.1093/ mnras/stab2181. URL htttps://doi.
Joeri Hermans, Arnaud Delaunoy, François Rozet, Antoine Wehenkel, Volodimir Begy, and Gilles Louppe. A crisis in simulation-basod inference? Beware, your posterior approximations can be unfaithful. Transactions on Machine Learning Research, 2022. ISSN 2835-8856. URL https://openrevie:: http:://pppmoorrrrrreeffoortttt net/formm?id
LHAbHkt6Aq
26

---

Published in Transactions on Machine Learning Research (06/2024)
Matthew Hoffman and Andrew Gelman. The No-U-Turn Sampler: adaptively setting path lengths in Hamiltonian Monte Carlo. Journal of Machine Learning Research, 15(1):1593-160 1623, 2014. ISSN 1532-4435.
Daolang Huang, Ayush Bharti, Amauri H Souza, Luigi Acerbi, and Samuel Kaski. Learning robust statistics for simulation-based inference under model misspecification. In Thirly-seventh Conference Newral Information Processing Systems, 2023. URL hnttps://operreveee wttp://www.eeeeooorrrsssststaa.
John D Hunter. Matplotlib: 2D graphics environment. Computing in science & engineering, 9(03):90-955 2007.
Rob J. Hyndman. Computing and graphing highest density regions. The American Statistician, 50(2): 120 126, 1996. doi: 1080/00031305.1 1996. 10474359
19444394 URL http:p////ww..aaaaooooeaeeeeeee/
Ryan P. Kelly. Implementing Bayesian synthetic likelihood within the engine for likelihood-free inference, 2022. URL https::/ qut edu. au / Master of Philosophy Thesis. Queensland University of Technology.
Patrick Kidger. On neural differential equations. PhD thesis, University of Oxford, 2021.
Diederik P. Kingma and Jimmy Ba. Adam: method for stochastic optimization. In Yoshua Bengio and Yann LeCun (eds.), The 3rd International Conference Learning Representations, ICLR, 2015. URL http://arxiv org/ abs/ 1412.11220 6980
Polina Kirichenko, Pavel Izmailov, and Andrew Gordon Wilson. Why normalizing flows fail detect out-of-distribution data. In Advances in Newral Information Processing Systems 34, 2020.
B.J.K. Kleijn and A.W. van der Vaart. The Bernstein- Von-Mises theorem under misspecification.. Electromic Journal of Statistics, 6(none):354 381, 2012. doi: 10.1214/12-EJSS0 URL https:://ai.oor////1 1214/
12-EJS675
Jeremias Knoblauch, Jack Jewson, and Theodoros Damoulas. An optimization view on Bayes' rule: reviewing and generalizing variational inference. Journal of Machine Learning Research, 23(13)):1-109, 2022.
Ravin Kumar, Colin Carroll, Ari Hartikainen, and Osvaldo Martin. ArviZ a unified library for exploratory analysis of Bayesian models in Python. Journal of Open Source Software, 4(33):1143, 2019. doi: 10.21105/ joss.01143. URL https://aoi.
21100/ 01143
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In Advances in Neural Information Processing Systems 31, 2017.
John R. Lewis, Steven N. MacFachern, and Yoonkyung Lee. Bayesian restricted likelihood methods: conditioning on insufficient statistics in Bayesian regression (with discussion). Bayesian Analysis, 16(4): 1393 2854, 2021. doi: 10. 10.1214/11-BA257. URL 1ott:: 1214/21-BA1257
Wentao Li and Paul Fearnhead. On the asymptotic efficiency of approximate Bayesian computation estimators. Biometrika, 105(2):285-299, 01 2018. ISSN 0006-3444. doi: 10.1093/bbie biomet/asx0778. //doi //doi.org/// 10. 109////immm///sx077
URL https
J Lintusaari, H Vuollekoski, A Kangasrääsiö, K Skytén, M Järvenpää, P Marttinen, M Gutmann, A Vehtari, J Corander, and S Kaski. ELFI: Engine for likelihood-free inference. Journal of Machine Learning Research, 19(1), 2018. doi: 3005///2291125.333
URL https::/ts//t http:///w..oi....1.......... 0tt.:/10..5555522911120 155555//322111119911111111111111
David Lopez-Paz and Maxime Oquab. Revisiting classifier two-sample tests. In International Conference on Learning Representations, 2017. URL httpes//gpppeeeeeerrrr/o/oo/ttdp
Jan-Matthis Lueckmann, Jan Boclts, David Greenberg, Podro Goncalves, and Jakob Macke. Benchmarking simulation-based inference. In The 24-th International Conference on Artificial Intelligence and Statistics, pp. 343 351. PMLR, 13-15 2021. URT [tt.hppppp/ppppeeiiiig.g.prrr/ee
html
27

---

Published in Transactions on Machine Learning Research (06/2024)
Philippe Marchand, Morgan Boenke, and David M. Green. A stochastic movement model reproduces patterns of site fidelity and long-distance dispersal in population of Fowler's toads (Anaæyruu fowleri). Ecological Modelling, 360:63 69, 2017. ISSN 0304-3800. doi: 00.100//j.eolmmdde2217...66
URL
https://dss //tp doi. wwg/oo..//... 1016/j ecolmodel. 201d .0.. .6.025
Jean-Michel Marin, Natesh S Pillai, Christian P Robert, and Judith Rousseau. Relevant statistics for Bayesian model choice. Journal of the Royal Slalistical Sociely: Series B (Slalistical Melhodology), 76(5):833 859, 2014.
Takuo Matsubara, Jeremias Knoblauch, Xavver Briol, and Chris J Oates. Robust generalised Bayesian inference for intractable likelihoods. Journal of the Royal Statistical Society Series B: Statistical Methodology, 84(3))997-1122, 2022.
Elina Nummiinen, Lu Cheng, Mats Gyllenberg, and Jukka Corander. Estimating the transmission dynamics of Streptococcus pneumonine from strain prevalence data. Biometrics, (3) 757-777777 2013. doi: https:/ dbi..rr/11.1111///mmi http:/00.111//bom.1120 12040. URL https://onliiielirrray wwwp://wwwwl..ccooo///////////// http://bb//00011//ooorro htt1//10m....100000 12040
Lorenzo Pacchiardi and Ritabrata Dutta. Generalizod Bayesian likelihood-free inference using scoring rules
estimators. aiv preprinl arXiv:2104.0388 2021.
Lorenzo Pacchiardi and Ritabrata Dutta. Score matched neural exponential families for likelihood-fiie inference. Journal of Machine Learning Research, 23, 2022. ISSN 1532-4435.
George Papamakarios and Tain Murray. Fast e-free inference simulation models with Bayesian conditional
density estimation. In Advances in Newral Information Processing Systems 29, 2016. URL https: pdf
//proceedings htt.a//.....pp.eeeeeoo.///////a1
George Papamakarios, David Sterratt, and lain Murray. Sequential neural likelihood: fast likelihood-free inference with autoregressive flows. In The 22nd International Conference on Artificial Intelligence and Statistics, pp. 937-848 PMLR, 2019. URL httpps:/pproceeeiing. mor.esm.rrrpppp... ptt.ppppeeaa/////ppaaaaaaaaraaaa html
George Papamakarios, Eric Nalisnick, Danilo Jimenez Rezende, Shakir Mohamed, and Balaji Lakshmi- narayanan. Normalizing flows for probabilistic modeling and inference. Journal of Machine Learning Research, 22()7), 2021. ISSN 1532-4435.
Mijung Park, Wittawat Jitkrittum, and Dino Sejdinovic. K2-ABC: approximate Bayesian computation with kernel embeddings. In Arthur Gretton and Christian C. Robert (eds.), Proceedings of the 19th International Conference on Artificial Intelligence and Statistics, volume 51 of Proceedings of Machine Learning Research, pp. 398-9077 407, Cadiz, Spain, 09-11 11 May 2016. PMLR. URL httph: httpps://proceedings. mll. /ttpoeeeiiii......pree////////// html
Du Phan, Neeraj Pradhan, and Martin Jankowiak. Composable effects for flexible and accelerated probabilistic programming NumPyro. In Program Transformations for ML Workshop at NeurIPS 2019, 2019. URL https://openreviev... http:///pppaaeeeeeeeeee////tttap
Dennis Prangle. Summary statistics. In Handbook of Approrimate Bayesion Computation, pp. 125-152.
Chapman and Hall/CRC, 2018.
Leah Price, C. C. Drovandi, A. Lee, and D. J. Nott. Bayesian synthetic likelihood. Journal of Computational and Graphical Statistics, 27(1):1 11, 2018. ISSN 1061-8600. doi: 10.1080 000/8/00061600.211.....00. 1302888 URL https://ddt wwt.://....r/11..10/////110 2017 1302882
Stefan T. Radev, Marvin Schmilt,, Valentin Pratz, Umberto Picchini, Ullrich Köthe, and Paul-Christian Bürkner. Jana: jointly amortized neural approximation of complex Bayesian models. In Robin Evans and Πya Shpitser (ods.), Proceedings of the Thirty-Ninth Conference on Uncertainty Artificial Intelligence, volume 216 of Proceedings of Machine Learning Research, pp. 1693-1706. PMLR, 31 Jul-04 Aug 2023. URL httpp:://proceediieee mlr. wrw.s//resseeeeeeeeeeee raw.ev23rrr html
28

---

Published in Transactions on Machine Learning Research (06/2024)
Oliver Ratmann, Christophe Andrieu, Carsten Wiuf, and Sylvia Richardson. Model criticism based on likelihood-free inference, with an application to protein network evolution. Proceedings of the National Academy of Sciences, 106(26):10576 1772::11577-10581,, 2009.
M. Schmitt, P.-C. Bürkner, U. Köthe, and S. T. Radev. Detecting model misspecification in amortized Baycsian inference with neural networks. In 45th German Conference on Pattern Recognition (GCPR), 2023a.
M. Schmitt, D. Habermann, P.-C. Bürkner, U. Koethe, and S. T. Radev. Leveraging self-consistency for data-efficient amortized Bayesian inference. In NeurIPS UniReps: The First. Workshop on Unifying Representations in Newral Models, 2023b.
Marvin Schmitt, Stefan T. Radev, and Paul-Chriss Chui-Chrsstian Bürkner. Meta-uncertainty in Bayesian model comparison. In Francisco Ruiz, Jennifer Dy, and Jan-Willem van de Mcent (cds.), Proceedings of The 26th International Conference on Artificial Intelligence and Statistics, volume 206 of Proceedings of Machine Learning Research, pp. 11-29. PMIRR 25 27 Apr 2023c. URL https://proceddings mlr press/v206/ schmitt23a.htmr
Sebastian M Schmon, Patrick W Cannon, and Jeremias Knoblauch. Generalized posteriors in approximate Bayesian computation. In Thind Symposium on Advances in Approzimate Bayesian Inference, 2021. URL
https://openreviee: http::/wwww..tt.ooo//////drtaaaa
Ali Siahkoohi, Gabrio Rizzuti, Mathias Louboutin, Philipp Witte, and Felix Herrmann. Preconditioned training of normalizing flows for variational inference in inverse problems. In Third Symposium on Advances Approzimate Bayesian Inference, 2021. URL https://openrevvie: http::/www....ee..oooouoaaaaaaaa sMaNQQQ
Ali Siahkoohi, Gabrio Rizzuti, Rafael Orozco, and Felix J Herrmann. Reliable amortized variational inference with physics-based latent distribution correction. Geophysics, R297 2023.
Jack Simons, Louis Sharrock, Song Liu, and Mark Beaumont. Neural score estimation: likelihood-frre inference with conditional score based diffusion models. In Pifth Symposium on Advances in Approcimate Bayesian Inference, 2023. URL https:// net wtt.://ooo..mm/.///////.......a
Scott A Sisson, Yanan Fan, and Mark Beaumont, Handbook of approrimate Bayesian computation. CRC Press, 2018.
Adam Spannaus, Theodore Papamarkou, Samantha Erwin, and J. Blair Christian. Inferring the spread of COVID-19: the role of time-varying reporting rate epidemiological modelling. Scientific Reports, (1):10761, 06 2022. ISSN 2322. doi: 10.1038/s41598, 022- 14979-0. URL 641598-022-14979-1
Sean Talts, Michael Betancourt, Daniel Simpson, Aki Vehtari, and Andrew Gelman. Validating Bayesian inference algorithms with simulation-bused calibration. arXiv preprint vrXiv:1804.06788 2018.
Alvaro Tejero-Cantero, Jan Boelts, Michael Deistler, Jan-Matthis Lueckmann, Conor Durkan, Pedro J. Gonçalves, David S. Greenberg, and Jakob H. Macke. sbi: A toolkit for simulation based inference. Journal of Open Source Software, 5(52):2505, 2020. doi: 02500005/ URL https://ddoi.rg/11 21105/joss.0255
Owen Thomas, Ritabrata Dutta, Jukka Corander, Samuel Kaski, and Michael U Gulmann. Likelihood-free inference by ratio estimation. Bayesian Analysis, 17(1):1-31, 2022.
Aki Vehtari, Andrew Gelman, Daniel Simpson, Bob Carpenter, and Paul-Christian Bürkner. Rank- normalization, folding, and localization: an improved R for assossing convergence of MCMC (with discussion). Bayesian Analysis, 16(2):667 718, 2021.
Xiaoyu Wang, Adrianne L Jenner, Robert Salomone, David J Warne, and Christopher Drovandi. Calibration of agent based models for monophasic and biphasic tumour growth using approximate Bayesian computation. Journal of Mathematical Biology, 88(3)28, 2024.
29

---

Published in Transactions on Machine Learning Research (06/2024)
Daniel Ward. flowjan, 2023. Python package version 7.0.0, [Online; accessed 17-Jamuay--0233, URL htt.://w.cccmmmmeeellld////////p
Daniel Ward, Patrick Cannon, Mark Beaumont, Matteo Fasiolo, and Sebastian M Schmon. Robust. neural posterior estimation and statistical model criticism. In Advances in Newral Information Processing Systems 35, 2022.
David J. Warne, Authony Ebert, Christopher Drovandi, Wenbiao Hu, Antonietta Mira, and Kerrie Mengersen. Hindsight is 2020 vision: characterisation of the global response the COVID-19 pandemic. BMC Public Health, 20(1):1868, 2020. ISSN 1471-2458, doi: 02.118//112889-0000097777 URL https://doi.org//
Richard David Wilkinson. Approximate Bayesian computation (ABC) gives exact results under the assumption of model error. Statistical Applications in Genetics and Molecular Biology, 12(2):129 141, 2013. doi: doi:10.1515/sset 1ww.115///ambb20333.... 2013-0010. URL
1515/....1.1...... 0010 2013-0010
Samuel Wigvist, Jes Frellsen, and Umberto Piechini. Sequential neural posterior and likelihood approximation, 2021. URL attps://arzxi..rg/abbb///// 06522 arXiv preprint arXir:2102.06522.
Jingkang Yang, Pengyun Wang, Dejian Zou, Zitang Zhou, Kunyuan Ding, Wenxuan Peng, Haoqi Wang, Guangyao Chen, Bo Li, Yiyou Sun, Xuefeng Du, Kaiyang Zhou, Wayne Zhang, Dan Hendrycks, Yixuan Li, and Ziwei Liu. OpenOOD: benchmarking generalized out-of-distributtin detection. In Thirty-siath Conference Neurol Information Processing Systems Datasets and Benchmarks Track, 2022. URL https://opearevieu net wtt.:otountdddg1eeeo 1itkttj___ss tskUt
Appendix
A
Implementation Details
The code reproduce the results has been included as supplementary material. All simulations and inference were executed on high-performance computer, each rumning using single Intel Xeon core with SGB of RAM. We used approximately 11,800 hours of CPU time across all experiment runs included in this paper. The robust sequential neural likelihood (RSNL) inference algorithm was implemented using the JAX (Bradbury ct al. 2018) and NumPyro (Phan et al.) 2019 librarics due to the speed of these libraries for MCMC sampling. Plotting was done using the matplotlin library (Hunter 2007), and MCMC diagnostics and visualisations were done using the ArviZ library (Kuar et al,, 2011).. The SIR model is implemented in JAX using the diffrax library (Kidger] 2021). RNPE results were obtained from implementing the example using the publicly available code at hattps://githii www.co///aaaa datpe/warn2///////// rnpe
RBSL results were obtained using the
ELFT software package implementation (Lintusaari et al. 2018
Kelly 2022
30

---

Published in Transactions on Machine Learning Research (06/2024)
B
Computation Times
Example
Method
MCMC dimension
Simulation
Flow
MCMC
Overall
Contaminated
normal
SNL
1
557
1931
2489
(standard)
RSNL
3
1
523
1222
1746
Contaminated normal (no
SNL
1
1
3634
2513
6148
summarisation)
RSNL
101
1
3524
43308
46833
Misspecified MA(1)
SNL
1
1
490
989
1480
RSNL.
3
1
441
1098
1540
Contaminated SLCP
SNL
5
1
1748
35153
36902
RSNL
15
1699
21358
23058
Misspecified SIR
SNL
2
65305
1429
1839
68573
RSNL
8
64987
1238
21233
87458
Toad Movement
SNL
3
52
1226
1216
2493
RSNT.
51
110
4150
23331
27591
Breakdown the computational time (rounded to the nearest second) for RSNL and SNL. Simulation time is the time spent rumning simulations. Flow time is the time training the neural network. MCMC the time spent sampling the posterior via MCMC. Times are aggregated across all rounds, and the MCMC time includes the final MCMC run used get the approximate posterior samples. Times are shown for the five experiments in the paper and the contaminated normal example without summarisation (more details in Appendix F).
Table 2:
C Hyperparameters
Normalising Flow
We utilise conditional neural spline flow (Durkan al. 2019) for (s((()) 0): implemented in the flowjax package (Ward] 2023). The flow design follows the choices in the sbi package Tejero-Cantert et al. 2020 We use 10 bins over the interval (-5, 5] for the rational quadratic spline transformer. The transformer function defaults to the identity function outside of this range, an important detail for the considered misspocified models, as the observed summary often appears in the tails. The conditioner consists of five coupling layers, each using a multilayer perceptron of two layers with 50 hidden units. The flow is trained using the Adam optimiser (Kingma & Bay 2015) with a learning rate of x 10-4 and a batch size of 256. Flow training is stopped when either the validation loss, calculated on 10% of the samples, has not improved over 20 epochs or when the limit of 500 epochs is reached.
MCMC
We use the no-U-turn sampler with four individual MCMC chains. To assess chain convergence, we ensure that the nork-normalised R of Vehtari et al. (2021) falls within the range (1.0, 1.05), that the effective sample size (ESS) reasonably large, and we inspect the autocorrelation and trace plots for each example. In the initial round of our RSNL algorithm, the chains are initialised with random draw from the prior. For each subsequent round, they are initialised using random sample from the current approximate posterior. We run each chain for 3,500 iterations, discarding the first 1,000 for burn-in. The resulting 10,000 combined samples from the four MCMC chains are thinned by a factor of 10. Model simulations are then run at the 1,000 sampled model parameter values. We use thinning to ensure that polentially expensive model simulations are run using relatively independent parameter values, leveraging that running MCMC with the learnt normalising flow is typically faster than rumning model simulations. The number of training rounds is set 10, resulting in 10,000 model simulations. After R rounds, we use qR (Sy)) 0) run 10,000 MCMC iterations targeting the approximate posterior.
31

---

Published in Transactions on Machine Learning Research (06/2024)
1.00
C2ST
0.75
0.50
RSNL
SNL
Figure 13: Boxplots illustrating the C2ST scores for RSNL and SNL on the contaminated normal example.
Additional Information and Results for Examples
Contaminated Normal
We consider more results on the contaminatod normal example as has an analytical posterior in the well-specified case. We can thus use draws from the true posterior for the classifier 2-sample test. (C2ST), popular performance metric in SBI.
C2ST. The C2ST trains binary classifier to determine if two samples come from the same distribution Lopez-Paz & Oquab 2017). In SBI, C2ST is used to differentiate between samples from the true and approximate posteriors, with the score being the classification accuracy of the classifier on held-out data Lueckmaan et al. ).21). the true and estimated posteriors are indistinguishabll, the classification score will converge to 0.5, while poor posterior approximate will be close in sbibm (Lueckmann et al. 2021 J. The results depicted in Figure 13 which displays boxplots of the C2ST score for 200 seeds for RSNL and SNL. As evidenced by the consistently low C2ST score, RSNL performs well, implying that the classifier finds distinguishing samples from the approximate and true posterior challenging. Conversely, SNL displays poor performance, with a median C2ST score close to 1.
We use the C2ST as implemented
Well-specifted example. Modellers might worry that in a well-specified model, the adjustment parameters in RSNL could introduce noise, thus reducing the accuracy of uncertainty quantification. We address this concern by briefly comparing the performance of RSNL and SNL using well-specified Gaussian example (i.c., the contaminated normal example without contaminated draws). The comparative results for RSNL and SNL are displayed in Figure 14 The empirical coverage and the log density of the approximate posterior at 0o are notably similar for RSNL. and SNL. These findings reassure that RSNL does not adversely impact inference on well-specified examples.
Empirical coverage
2
RSNL SNL
Log density
Credibility level
RSNL SNL
Figure 14: The left. plot shows the empirical coverage across various credibility levels for SNL (dashed) and RSNL (solid) on a well-specified Gaussian example. A well-calibrated posterior closely aligns with the diagonal (dotted) line. Conservative posteriors are situated in the upper triangle, while overconfident ones are in the lower triangle. The right plot shows the approximate posterior log density boxplots at the true parameter values.
Posterior predictive checks. Figure 15 illustrates this concept through the PPCs for both RSNL and SNL the contaminated normal example. For SNL, the posterior prodictive distribution has negligible support around both observed summaries, with no indication of what summary is misspecified. For RSNL, however, the correctly specified summary has good support,, and the misspecified summary has negligible support.
32

---

Published in Transactions on Machine Learning Research (06/2024)
So, in this example, PPCs only correctly identified the misspecified summary when the inference is robust to model misspecification. In more complicated setting, with many summaries, PPCs with non-robust method (such as SNL) are unlikely to be sufficient for modeller dealing with model misspecification. If there is model misspecification, the modeller must have some indication of what is misspecified to refinc the model. Further, the model can be correctly misspecified and still have bad predictive performance for reasons unrelated the model, such as poor inference.
2-
Density
Density
0
0.5
1.0
1.5
0.5
1.0
1.5
2.0
Sample mean
Sample variance
(a) RSNL.
4-
3
2
Density
Density
2
1
0
%
0
1.5 Sample mean
2.0
0.5
1.0
1.5 Sample variance
2.0
(b) SNL
Figure 15: Posterior predictive checks on the contaminated normal example for RSNL and SNL. The observed summaries are shown as a vertical dashed line.
Misspecified SIR
Figure 16 shows a representative visualisation of the full data for the true DGP of the misspocified SIR example. The spikes, corresponding to increased observed infections on Mondays, lead to the autocorrelation summary statistic being unable to be recovered from the assumed DGP that does not contain these spikes.
33

---

Published in Transactions on Machine Learning Research (06/2024)
70000
Number of infected
35000
167
365
Days
Figure 16: An example simulation from the true SIR model, using parameters B = 0.15 and ry = 0.1.
Toad Movement Model
We outline the implementation details of the MMD Gretton et al. 2012 used for the toad example,
included in the main text. The MMD serves as a measure of distance between two probability distributions and is particularly robust against outliers. Due to this robustness, MMD has been used for robustness to model misspecification in SBI (Park et al.) 2016 Huang et al. 2023 In our case, we limit the MMD calculation to samples from the posterior predictive and the Dirac mcasure centred on the observed summary statistics, leading a simplified expression:
MMD 1 12 ij=1
Σ
((S(z;),
Σ ((S())),S()))
(8)
where 1 = 1000 denotes the number of samples from the posterior predictive distribution, K exp( -1 z' 2211//22 is the radial basis function kernel, and 3 is determined via the median heuristic, B = /median/ 2, with the median being of the Euclidean distances between pairs of simulated summaries.
We also include the estimated posterior for RSNL, SNL and RBSL for the toad movement model with simulated data. Here, SNL gives similar inferences to RSNL and RBSL, and all three methods have reasonable support around the true parameter value.
34

---

Published in Transactions on Machine Learning Research (06/2024)
RSNL
SNL
RBSL
Grood
55
10 45
35
0.7
0.6
0.5
1.6
1.8 (road
2.0
35
45
55
0.5
0.6
0.7
Po
Figure 17: Univariate and bivariate density plots of the estimated postcrior for 0 applied to simulated data for the toad movement example. Plots on the diagonal the univariate posterior densities obtained by RSNL (solid), SNL (dashed) and RBSL (dotted). The bivariate posterior distributions for the toad movement example are visualised contour plots when applying RSNL (solid, lower triangle off-diagonal) and SNL (dashed, upper triangle off-diagonal). The true parameter values are visualised as a vertical dashed line for the marginal plots and the x symbol in the bivariate plots.
E Comparison of Laplace prior with proposed data-driven prior
In Figure 18 we showcase the impact prior selection using the contaminated normal example with a sample variance 4. The data-driven prior employed for RSNL yiclds a tighter 90% credible interval of (0.81, 1.17). In contrast, despite being centred on the correct value and allowing for adjustment of misspocified summaries, the Laplace(O, 0.5) prior results in substantially wider 90% credible interval, spanning (-4.53, 6.24).
35

---

Published in Transactions on Machine Learning Research (06/2024)
3.5
3.0
2.5
Density
2.0
RSNL Data-driven prior
RSNL Laplace(O, 0.5)
1.5
1.0
0.5
-2
0
Figure 18: Estimated univariate posterior density for the contaminated normal using RSNL with our rocommended data-driven prior (solid) and fixed scale Laplace prior (dotted).
F Additional Results for RSNL on the Contaminated Normal Example with No Summarisation
This section evaluates the scalability of RSNL in scenarios involving an increased number of adjustment parameters. To evaluate this, we considered the contaminated normal example without summarisation. That is, we perform inference on the one hundrod samples directly. The postcrior plots presented here for model parameter 0 and the first adjustment summary are primarily confirm that the increase the number of adjustment parameters does not adversely affect inference quality. Comprehensive details on computational time, which is the focal point of this examination, are available in Appendix B.
2.5
Posterior
0.4
Prior
2.0
Density
Density
0.3
0.2
1.0
0.5
0.1
0.0
0.0
-5
0
V.
Figure 19: The left plot shows the RSNL approximate posterior for on the contaminated normal example with no summarisation. The true parameter value is shown as a vertical dashed line. The right plot shows the prior (dashed) and estimated marginal posterior (solid) densities for the first (of the 100) adjustment parameter, 71-
G Laplace Prior Hyperparameter
Figure 20 displays the approximate posterior for the contaminated normal model under varying levels of model inisspecification, with posteriors for the model parameter 0 in the left. column and posteriors for 72 (i.e. the adjustment parameter that corresponds with the incompatible summary) in the right column. We define the degree of misspecification by sctting the "observed" sample variance to 2.0, 7.0, and 12.0 for mild, moderate, and severe conditions, respectively. We compute and visualise the approximate posteriors for ..05,..., in increments of 0.05, with 0 (i.e. SNL.) omitted for better visualisation. From the 0
36

---

Published in Transactions on Machine Learning Research (06/2024)
Mild
Moderate
Severe
1.0
1.0
1.0
True Posterior
True Posterior
True Posterior
Density
Density
Density
0.5
2
0.5
0.5
0.3
0.3
0.3
0.0
0.0
1
0
(a) Posterior distributions for 0.
Mild
Moderate
Severe
1.0
1.0
1.0
4
30
Density
Density
Density
0.5 I-
10-
20
0.5
0.3
0.3
10
0.3
0.0
0.0
01
0.0
0.9
1.0
0.95
1.00
1.05
72
722
(b) Posterior distributions for adjustment parameter 72-
Figure 20: RSNL approximate posteriors for the contaminated normal under mild, moderate, and severe model misspecification.
posteriors, we can see all values 7 lead to useful inference on 6, in that they centre on 6g- However, the hyperparameter does have the effect of influencing the sharpness of the posterior. Ideally, should be kept. small to avoid introducing excessive noise through the adjustment parameters. However, the behaviour of small values be peculiar, as evident in the posterior for ?2 under mild misspecification. Surprisingly, the most unusual-looking behaviour for 72 is observed in the case of mild misspecification for low values of т. The reason may be that under severe model misspecification, it is much easier to identify misspecification, and the data-driven prior allows for this correction despite the greater distance required to be shifted. But for mild model misspecification, parameters may be split between not adjusting and adjusting. If more knowledge of the nature of the model misspecification known, the modeller may be able to use this to select values of T. But promote the choice of r 0.3 a robust choice when the level of model misspecification is unknown. Finally, although have only presented results here for the contaminated normal example, similar outcomes observed across the other examples in this paper.
we set the adjustment parameter with a smaller variance, the adjustment
37


Reviews:
[{"review_id": "9ZIzMRTZPy", "reviewer": "Reviewer_uWfC", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "While the technical novelty is somewhat limited, I think it is interesting to see how the RBSL approach performs when used in concert with normalizing flows instead of Gaussian approximations. This is, to the best of my knowledge, the first work to explore this.\n\nI think something interesting to see relates to the choice for the prior over the adjustment parameters and the method's performance. How sensitive is the method to this choice? At some point it will break, as there is a balance in modeling the \"OOD summary statistics\" in the tail of the flow or by moving the adjustment parameters away from zero. When does the method lose its robustness capabilities? How gradually does this happen as we change the sharpness of the prior over the adjustment parameters?\n\nThe discussion briefly touches on scenarios where methods as RBSL or the one proposed here may fail to detect misspecification or perform as expected (e.g. when there is model misspecification but the summary statistics are not OOD). The Gaussian assumption in BSL simplifies the analysis of scenarios where the method will work well or fail. While the introduction of neural density estimators complicates the analysis, I'd be interested in hearing the authors' thoughts about this, possibly under certain simplifying assumptions. For instance, assuming the flow perfectly learns the likelihood, when can we expect the proposed method to not work as expected? Shedding light on scenarios where the method is expected to fail / work would be quite useful for practitioners willing to use this approach.\n\nThe discussion states \"RSNL provides useful inference with significantly fewer simulation calls than ABC and BSL methods.\" This is true with existing techniques. However, with BSL methods, I'm assuming that if you learn an amortized Gaussian parameterization instead of estimating the Gaussian parameters through observations, BSL's sample efficiency could drastically improve. (In short, this would be equivalent to setting the normalizing flow in NL to a Gaussian with learnable mean and variance in an amortized way.) While to the best of my understanding such a method was not implemented nor tested yet in the literature, it is a very natural thing to consider, which may decrease the relevance of this claim. \n\nFinal comment, at least one sentence in this work is an exact copy from the RBSL paper. For instance \"[...] provided that m is chosen large enough so that the plug-in synthetic likelihood estimator has a small enough variance to ensure that MCMC mixing is not adversely affected\" (close to the end of page 4). This is not affecting my judgement towards the paper, but should be rephrased.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes a method to handle misspecification in the context of simulation based inference with neural likelihood (NL) methods (i.e. methods that use a density estimator, in this case a normalizing flow, to estimate the intractable likelihood). The main idea behind the approach involves the use of auxiliary variables that increase the robustnes of NL methods against misspecification. The paper builds heavily on the Robust Bayesian synthetic likelihood (RBSL) approach, adapting it from the use of Gaussian likelihoods to normalizing flows.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See above. I believe addressing the questions above (prior, expected behavior under ideal training of the flow) could improve the paper considerably.", "broader_impact_concerns": "No", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Bmfn6YqbAB", "reviewer": "Reviewer_odrb", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- Addresses a relevant issue, and proposes a method that is widely applicable\n- The method is sound and grounded in existing literature (which seems appropriately cited)\n- A range of different experiments is included, providing convincing evidence of RSNL's merits; promising performance on experiments with misspecified summaries\n\nWeaknesses:\n- While the authors discuss that they found the choice of prior to be crucial, experiments are reported for a single choice of prior/hyperparameter setting (apart from Appendix E, which compares RSNL with the proposed data-driven prior against a Laplace prior on the contaminated Normal example)\n- Some experiments reveal a high degree of underconfidence – while this is preferable to overconfidence, ideally, resulting posteriors would be well-calibrated\n\nAs discussed by the authors, there is ample opportunity for follow-up work (e.g., searching for methods that yield well-calibrated posteriors, exploring parameter-dependent summary scales, or additional benchmarking as a function of simulation budget). I agree with the authors that these are interesting aspects that can be regarded out-of-scope for this submission.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The manuscript proposes a simulation-based inference (SBI) method that aims to be robust to model misspecification: SBI allows parameter estimation for models with intractable likelihoods. As SBI methods have been shown to yield unreliable results if the model is misspecified, development of new methods that can both detect and mitigate the impact of misspecification is highly desirable.\n\nRecent work towards this goal is reviewed, and a new approach proposed: This work adapts the model expansion approach for Bayesian Synthetic Likelihood (BSL) proposed by Frazier & Drovandi (2021) to Sequential Neural Likelihood (Papamakarios et al., 2019). The resulting method, robust Sequential Neural Likelihood (RSNL), uses auxiliary parameters to account for misspecified summary statistics. Inspecting deviation of posterior distribution over adjustment parameters from the (data-driven) prior allows detection of incompatible summary statistics.\n\nEmpirical results show that the proposed algorithm can mitigate misspecification. More specifically, the authors find that RSNL compares favorably against SNL on five examples (Contaminated Normal, Misspecified MA(1), Contaminated SLCP, Misspecified SIR, and Toad Movement) when considering empirical coverage and the mean log probability of true parameters. Comparisons to robust Neural Posterior Estimation (RNPE; Ward et al., 2022) and BSL are included as well.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Proposed changes to strengthen this work:\n- Given that the choice of prior is highlighted as a very sensitive one, I'd highly appreciate a more thorough investigation of this issue. A first step might be to include results showing performance for choices other than τ=0.3 – to get a better understanding how sensitive results to different scales are, and to make the choice taken more transparent.\n- It should be clearly stated how many observations/true parameters were used to calculate the mean posterior log probability, and what the whiskers in the box plot represent.\n- I'd propose to improve overall consistency between figures by using a common font size, font family, and color code.\n\nMinor issues:\n- Page 1: Blei et al. (2017) and Box (1976) are cited but do not appear in the references.\n- Page 6/7: Something seems to have been mixed up at the page transition, the sentence reads \"[...] applicable to SBI with potential model misspecification Generalised Bayesian inference (GBI) is an alternative class [...]\"\n- Page 22: The sentence \"First, if a summary is incompatible but becomes very close to 0 after standardisation, this seems unlikely but may occur when simulated summaries generate a complex multi-modal distribution at a specific parameter value.\" should be revised.", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ChTFVe78Ek", "reviewer": "Reviewer_erZ8", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Essentially this work transposes the work on BSL done in Frazier-Drovandi (2021) into the SNL realm. It is very well written, convincing and (importantly) useful, in ways that I am going to describe. \n\nBut first a couple of weak points:\n- in light of the existence of the background work of Frazier-Drovandi (2021), this contribution is only incremental. In fact, the way misspecificaton is detected, as mentioned above, is taken from Frazier-Drovandi (2021).\n- Table 2 in appendix B is not clear. See the \"Requested changes\" below.\n\nThe strengths of the paper are:\n- as already mentioned, the paper is clear and well written;\n- RSNL works, and if applied to a model without misspecifications, it behaves just as SNL (meaning that it is not detrimental to use RSNL), at least on the considered examples.\n- Multiple examples are provided.\n- It is useful, as it gives a clear way on how to detect misspecifications.\n- It reinforces the notion that misspecification in SBI is understudied and deserves more attention.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The work proposes a model misspecification tool for \"simulation based inference\" (SBI), in the context of Bayesian inference via MCMC (Metropolis-Hastings). SBI implies that, in absence of a readily-available likelihood function, it is only possible to resort to simulations from an assumed data generating model. The authors then replace the true but unavailable likelihood function, with a normalising flow trained on many pairs of parameter draws and corresponding model simulations. They follow the \"sequential Neural likelihood\" (SNL) approach of Papamakarios et al 2019. However the authors take a specific route, in that they address the problem of 1) \"robustify\" SNL, for when there is a mismatch between the \"assumed data-generating model\" and the unknown actual data generating model. And then they consider: 2) given their robustified SNL (RSNL), they propose a way to detect which features of the model are misspecified. \n\nIn fact, it is assumed that inference is not conducted conditionally to each and every observed data point, but conditionally to summary statistics of the data. Part (2) in the goal above essentially gives a way to visually determine which summary statistics are likely to be corresponding to features of the model that do not quite match the data (and as such, which parts of the model are misspecified).\n\nThe strategy to robustify SNL and then identify misspecification is taken from Frazier-Drovandi (2021), as declared by the authors, however in Frazier-Drovandi the focus is on robustifying BSL (Bayesian synthetic likelihood). There (and also here) the MCMC is carried on an augmented space, where the target is the posterior on the space theta x Gamma, where Gamma is a vector with the same length as summary statistics. The authors observe wether the marginal posteriors for the entries in Gamma look different from the corresponding priors. If this is the case, the given component of Gamma (ie the corresponding summary) is misspecified.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Entries marked with an asterisk (*) are \"critical\".\n\n- (*) the computational efforts in table 2 (appendix B) are not clear. Please clarify what the headings of the table mean. In particular, 1) what does it mean \"Flow\" and \"MCMC\" in this context? Is it the number of seconds to train the flow and the number of seconds spent in running the mcmc respectively? But most importantly, what does it mean the \"Overall\" heading? Because \"Overall\" is not the sum of \"MCMC\" and \"Flow\" so what does it mean? \n- (*) again in table 2: How is it possible that the number of seconds spent running MCMC is sometimes larger for SNL that for RSNL? RSNL should be more complex to run, due to the extra computations implied by using $\\Gamma$. Please clarify.\n- (*) Table 2: sometimes there is a factor 10 of difference between RSNL and SNL in terms of seconds for the MCMC column, where RSNL is much more expensive (eg for the Misspecified SIR and Toad movement). I do not think you discuss this that much in the paper . Please comment.\n- page 2: spell out \"RBSL\" the first time it is mentioned\n- page 5: the entire paragraph \"Wilkinson 2013 reframes (...) in an ABC context\", is a repetition of the same paragraph from page 2, so this repetition should be eliminated.\n- page 5: notation-wise using $\\theta_0$ for the pseudo-true parameter is a bit misleading, since you used $P_0$ to denote the true data-generating model, while $\\theta_0$ may not be the true parameter.\n- end of page 6: in the last line \"misspecification\" should be followed by a full stop.\n- page 8 just after eq (5), you mentioned that the standardised summaries would be discussed later, but I failed to find such a discussion or explicit calculations for their construction. Please let me know in case I missed this and otherwise integrate.\n- page 8: in the fourth paragraph you mention \"conditional standardisation\". What is this? Please clarify.\n- in Algorithm 1 step 3: change $r \\neq 0$ to $r>0$.\n- (*) in Algorithm 1 step 7: I don't understand why it seems that no summaries of simulated data are taken. After all, the normalising flow is trained on (I believe) summary statistics, as it gets evaluates on S(x). So why it is never explicit in the notation that training data includes summaries rather than raw data?\n- section 4.1 on page 10: I guess it is missing the specification that the ground truth for theta is theta=1.\n- section 4.1 on page 10: what is the value of b0 here?\n- page 22 lines 5-6: \"a summary is correctly specified but lies in the tail\". Under which circumstances can this happen?", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper has already undergone a revision that incorporated suggestions for improvement by the reviewers, with no outstanding requested changes remaining. Therefore, the paper can be accepted as is.
Decision: Accept (poster)