Title: Learning Robust Statistics for Simulation-based Inference under Model Misspecification
Venue: NeurIPS 2023
Accepted: yes
Keywords: Simulation-based inference

model misspecification

likelihood-free inference

approximate Bayesian computation

neural posterior estimation
Abstract: Simulation-based inference (SBI) methods such as approximate Bayesian computation (ABC),  synthetic likelihood, and neural posterior estimation (NPE) rely on simulating statistics to infer parameters of intractable likelihood models. However, such methods are known to yield untrustworthy and misleading inference outcomes under model misspecification, thus hindering their widespread applicability. In this work, we propose the first general approach to handle model misspecification that works across different classes of SBI methods. Leveraging the fact that the choice of statistics determines the degree of misspecification in SBI, we introduce a regularized loss function that penalizes those statistics that increase the mismatch between the data and the model. Taking NPE and ABC as use cases, we demonstrate the superior performance of our method on high-dimensional time-series models that are artificially misspecified. We also apply our method to real data from the field of radio propagation where the model is known to be misspecified. We show empirically that the method yields robust inference in misspecified scenarios, whilst still being accurate when the model is well-specified.

Learning Robust Statistics for Simulation-based Inference under Model Misspecification

Abstract

Simulation-based inference (SBI) methods such as approximate Bayesian computation (ABC), synthetic likelihood, and neural posterior estimation (NPE) rely on simulating statistics to infer parameters of intractable likelihood models. However, such methods are known to yield untrustworthy and misleading inference outcomes under model misspecification, thus hindering their widespread applicability. In this work, we propose the first general approach to handle model misspecification that works across different classes of SBI methods. Leveraging the fact that the choice of statistics determines the degree of misspecification in SBI, we introduce a regularized loss function that penalizes those statistics that increase the mismatch between the data and the model. Taking NPE and ABC as use cases, we demonstrate the superior performance of our method on high-dimensional time-series models that are artificially misspecified. We also apply our method to real data from the field of radio propagation where the model is known to be misspecified. We show empirically that the method yields robust inference in misspecified scenarios, whilst still being accurate when the model is well-specified.

1 Introduction

Bayesian inference traditionally entails characterizing the posterior distribution of parameters assuming that the observed data came from the chosen model family [7]. However, in practice, the true data-generating process rarely lies within the family of distributions defined by the model, leading to model misspecification . This can be caused by, among other things, measurement errors or contamination in the observed data that are not included in the model, or when the model fails to capture the true nature of the physical phenomenon under study. Misspecified scenarios are especially likely for simulator-based models, where the goal is to describe some complex real-world phenomenon. Such simulators, also known as implicit generative models [25], have become prevalent in many domains of science and engineering such as genetics [71], ecology [3], astrophysics [40], economics [29], telecommunications [8], cognitive science [79], and agent-based modeling [87]. The growing field of simulation-based inference (SBI) [21, 51] tackles inference for such intractable likelihood models, where the approach relies on forward simulations from the model, instead of likelihood evaluations, to estimate the posterior distribution.

Traditional SBI methods include approximate Bayesian computation (ABC) [4, 51, 76], synthetic likelihood [67, 85] and minimum distance estimation [15]. More recently, the use of neural networks

2000

0

Statistic 1

800

600

400

200

0

Statistic 2

NPE

Simulated

Observed

200

0

Statistic 1

0

1000

2000

3000

Statistic 2

Our method

Simulated

Observed

2.5

5.0

7.5

1

0

10

20

2

True

Ours

NPE

as conditional density estimators [72] has fuelled the development of new SBI methods that target either the likelihood (neural likelihood estimation) [52, 61, 75, 83], the likelihood-to-evidence ratio (neural ratio estimation) [22, 23, 28, 30, 39, 46, 77], or the posterior (neural posterior estimation) [38, 41, 53, 58, 59, 70, 75, 83]. A common feature of all these SBI methods is the choice of summary statistics of the data, which readily impacts their performance [14]. The summary statistics are either manually handcrafted by domain experts, or learned automatically from simulated data using neural networks [1, 17, 18, 26, 31, 47, 84].

The unreliability of Bayesian posteriors under misspecification is exacerbated for SBI methods as the inference relies on simulations from the misspecified model. Moreover, for SBI methods using conditional density estimators [5, 13, 41, 61, 69], the resulting posteriors can be wildly inaccurate and may even go outside the prior range [9, 16, 35]; see [27] for an instance of this problem in the study of molecules. This behavior is exemplified in Figure 1 for the posterior-targeting neural SBI method called neural posterior estimation (NPE). We see that when the model is misspecified, the NPE posterior goes outside the prior range. This is because the model is unable to match the observed statistic for any value of the parameters, as shown in Figure 1(left), and hence, NPE is forced to generalize outside its training distribution, yielding a posterior that is outside the prior range.

We argue that the choice of statistics is crucial to the notion of misspecification in SBI. Suppose we wish to fit a Gaussian model to data from some skewed distribution. If we choose, for instance, the sample mean and the skewness as statistics, then the Gaussian model would fail to match these statistics for any choice of parameters, and we would end up in the same situation as in Figure 1 where the SBI method is forced to generalize outside its training distribution. However, choosing the sample mean and sample variance as statistics instead may solve this issue, as the model may be able to replicate the chosen statistics for some value of parameters, even though the Gaussian model itself is misspecified for any skewed dataset. This problem of identifying a low-dimensional statistic of a high-dimensional dataset for which the model of interest is correctly specified is termed the data selection problem [82]. We refer to the statistics that solve the data selection problem as being robust .

Contributions. In this paper, we learn robust statistics using neural networks to solve the data selection problem. To that end, we introduce a regularized loss function that balances between learning statistics that are informative about the parameters, and penalizing those choices of statistics or features of the data that the model is unable to replicate. By doing so, the observed statistic does not go outside the set of statistics that the model can simulate, see Figure 1(middle), thereby yielding a posterior around the true parameter even under misspecification (Figure 1(right)). As our method relies on learning appropriate statistics and not on the subsequent inference procedure, it is applicable to all the statistics-based SBI methods, unlike other robust methods in the literature [24, 48, 81]. We substantiate our claim in Section 4 through extensive numerical studies on NPE and ABC - two fundamentally different SBI methods. Additionally in Section 5, we apply our method to a real case of model misspecification in the field of radio propagation [8].

Related works. Model misspecification has been studied in the context of different SBI methods such as ABC [9, 33, 35, 36, 74], synthetic likelihood [32, 34, 57], minimum distance estimation [24], and neural estimators [48, 81]. It was first analyzed for ABC methods such as the regressionadjustment ABC [5] in [35], and a robust approach was proposed in [33]. In [9], the authors proposed to handle model misspecification in ABC by incorporating the domain expert while selecting the statistics. Generalized Bayesian inference [12, 43, 49], which is a popular framework for designing methods that are robust to misspecification, has also been linked to ABC [74] and synthetic likelihood [57]. In [24], the authors propose a robust SBI method based on Bayesian nonparametric learning and the posterior bootstrap. More recently, robustness to model misspecification has been tackled for neural likelihood [48] and posterior estimators [81]. We use the latter method, called robust neural posterior estimation (RNPE) [81], as a baseline for comparing our proposed approach. Our method differs from these previous robust approaches in that it is applicable across distinct SBI methods.

2 Preliminaries

Simulation-based inference (SBI). Consider a simulator { P θ : θ ∈ Θ } , which is a family of distributions parameterized by θ on data space X ⊆ R d . We assume that P θ is intractable, but sampling independent and identically distributed (iid) data x 1: n = { x (1) , . . . , x ( n ) } ∼ P θ is straightforward for x ( i ) ∈ X , 1 ≤ i ≤ n . Given iid observed data y 1: n ∼ Q , y ( i ) ∈ X , from some true data-generating process Q and a prior distribution of θ , we are interested in estimating a θ ⋆ ∈ Θ such that P θ ⋆ = Q , or alternatively, in characterizing the posterior distribution p ( θ | y 1: n ) . Since X is a high-dimensional space in most practical cases, it is common practice to project both the simulated and the observed data onto a low-dimensional space of summary statistics S via a mapping η : X n → S , such that s = η ( x 1: n ) is the vector of simulated statistics, and s obs = η ( y 1: n ) the observed statistic. In the following, we assume that this deterministic map η is not known a priori , but needs to be learned using a neural network, henceforth called the summary network η ψ , with trainable parameters ψ . We now introduce two different paradigms for learning the statistics.

Learning statistics in neural posterior estimation (NPE) framework. NPEs are a class of SBI methods which are becoming popular in astrophysics [40] and nuclear fusion [37, 80], among other fields. They map each statistic vector s to an estimate of the posterior p ( θ | s ) using conditional density estimators such as normalizing flows [41, 53, 59, 69]. The posterior is assumed to be a member of a family of distributions q ν , parameterized by ν . NPEs map the statistics to distribution parameters ν via an inference network h ϕ , where ϕ constitutes the weights and biases of h , such that q h ϕ ( s ) ( θ ) ≈ p ( θ | s ) . The inference network is trained on dataset { ( θ i , s i ) } i m =1 , simulated from the model using prior samples { θ i } i m =1 ∼ p ( θ ) , by minimising the loss L ( ϕ ) = -E p ( θ, s ) [log q h ϕ ( s ) ( θ )] . Once trained, the posterior estimate is then obtained by simply passing the observed statistic s obs through the trained network. Hence, NPEs enjoy the benefit of amortization, wherein inference on new observed dataset is straightforward after a computationally expensive training phase. In cases where the statistics are not known a priori , the NPE framework allows for joint training of both the summary and the inference networks by minimizing the loss function

on the training dataset { ( θ i , x 1: n,i ) } i m =1 [69]. Even though NPEs are flexible and efficient SBI methods, they are known to perform poorly when the model is misspecified [16, 81]. This is because the observed statistic s obs becomes an out-of-distribution sample under misspecification (see Figure 1 for an example), forcing the inference network in NPE to generalize outside its training distribution.

Learning statistics using autoencoders. Unlike NPEs, statistics for other SBI methods are learned prior to carrying out the inference procedure. This is achieved, for instance, by training an autoencoder with the reconstruction loss [1]

where ψ and ψ d are the parameters of the encoder η and the decoder ˜ η , respectively. The trained encoder η ψ is then taken as the summarizing function in the SBI method. In this paper, we will use the encoder η ψ to perform inference in an ABC framework, which we now recall.

Approximate Bayesian computation (ABC). ABC is arguably the most popular SBI method, and is widely used in many research domains [64]. ABC relies on computing the distance between the simulated and the observed statistics to obtain samples from the approximate posterior of a simulator. Given a discrepancy ρ ( · , · ) and a tolerance threshold δ , the basic rejection-ABC method involves repeating the following algorithm: (i) sample θ ′ ∼ p ( θ ) , (ii) simulate x ′ 1: n ∼ P θ ′ , (iii) compute s ′ = η ( x ′ 1: n ) , and (iv) if ρ ( s ′ , s obs ) < δ , accept θ ′ , until a sample { θ ′ i } n δ i =1 is obtained from the approximate posterior. Conditional density estimators have also been used in ABC to improve the posterior approximation of the rejection-ABC method [5, 11, 13, 14]. These so-called regression adjustment ABC methods involve fitting a function g between the accepted parameters and statistics pairs { ( θ ′ i , s ′ i ) } n δ i =1 as θ ′ i = g ( s ′ i ) + ω i , 1 ≤ i ≤ n δ , where { ω i } n δ i =1 are the residuals. Once fitted, the accepted parameter samples are then adjusted as ˜ θ ′ i = ˆ g ( s obs ) + ˆ ω i , where ˆ g ( s ) is the estimate of E [ θ | s ] and ˆ ω i is the empirical residual. Despite the popularity of the regression adjustment ABC methods, they have also been shown to behave wildly under misspecification [9, 35], similar to NPEs.

3 Methodology

Model misspecification in SBI. Bayesian inference assumes that there exists a θ ⋆ ∈ Θ such that P θ ⋆ = Q . Under model misspecification, P θ = Q for any θ ∈ Θ , which can lead to unreliable predictive distributions. However, this definition does not apply for models whose inference is carried out in light of certain summary statistics. For instance, a Gaussian model is clearly misspecified for a bimodal data sample. However, if inference is based on, say, the sample mean and the sample variance of the data, the model may still match the observed statistics accurately. Hence, even when a simulator is misspecified with respect to the true data-generating process, it may still be well-specified with respect to the observed statistic. The choice of statistics therefore dictates the notion of misspecification in SBI. We propose an analogous definition for misspecification in SBI:

̸

̸

Definition 3.1 ( Misspecification of model-summarizer pair in SBI ) . Let η # P n θ be a pushforward of the probability measure P n θ = P θ ×··· × P θ on X n under η : X n →S , meaning that for A ⊂ S , η # P n θ ( A ) = P n θ ( η -1 ( A )) = P θ ( η -1 ( A ) 1 ) ×··· × P θ ( η -1 ( A ) n ) . Then, { η # P n θ : θ ∈ Θ } is a set of distributions on the statistics space S induced by the model for a given summarizer η , and η # Q n is the distribution of the statistics from the true data-generating process. Then, the model-summarizer pair is misspecified if, ∀ θ ∈ Θ , η # Q n = η # P n θ .

It is trivial to see that if the model is well-specified in the data space X , it will also be well-specified in the statistic space S for any choice of η . The converse, however, is not true, as previously mentioned. Using this definition, we quantify the level of misspecification of a model-summarizer pair as follows:

Definition 3.2 ( Misspecification margin in SBI ) . Let D be a distance defined on the set of all Borel probability measures on S . Then, for a given η , we define the misspecification margin ε η in SBI as

Note that the margin is equal to zero for the well-specified case. Moreover, the larger the margin, the bigger the mismatch between the simulated and the observed statistics w.r.t η . Our aim is to learn an η such that P θ is represented well whilst the model-summarizer pair is no longer misspecified as per Definition 3.1, or alternatively, has zero margin as per Definition 3.2. We formulate this as a constrained optimization problem in Equation (6).

Learning robust statistics for SBI. Suppose we have the flexibility to choose the summarizer η from a family of functions parameterized by ψ . Then, a potential approach for tackling model misspecification in SBI is to select the summarizer η that minimizes the misspecification margin ε η . However, that involves computing the infimum for all possible choices of ψ . To avoid that, we minimize an upper bound on the margin instead, given as

Note that Equation (4) is valid for any choice of p ( θ ) , and we call ε upper η ψ the margin upper bound .

Minimizing the margin upper bound alone would lead to η being a constant function, which when used for inference, would yield the prior distribution. Hence, there is a trade-off between choosing an η that is informative about the parameters and an η that minimizes the margin upper bound. We

propose to navigate this trade-off by including the margin upper bound as a regularization term in the loss function for learning η . To that end, let L ( ω, ψ ) be the loss function used to learn η ψ (along with additional parameters ω ) which, for instance, can be either L NPE from Equation (1) or L AE from Equation (2). Then, our proposed loss with robust statistics (RS) is written as

where λ ≥ 0 is the weight we place on the regularization term relative to the standard loss L . Minimizing the loss function in Equation (5) corresponds to solving the Lagrangian relaxation of the optimization problem:

with λ ≥ 0 being the Lagrangian multiplier fixed to a constant value.

Estimating the margin upper bound. In order to implement the proposed loss from Equation (5), we need to estimate the margin upper bound using the training dataset { ( θ i , x 1: n,i ) } i m =1 ∼ p ( θ, x 1: n ) sampled from the prior and the model. However, we only have one sample each from the distributions η ψ # P n θ i , i = 1 , . . . , m , and one sample of the observed statistic from η ψ # Q n . Taking D to be the Euclidean distance ∥·∥ , we can estimate the margin upper bound as 1 m ∑ m i =1 ∥ η ψ ( x 1: n,i ) -η ψ ( y 1: n ) ∥ . Although easy to compute, we found this choice of D to yield overly conservative posteriors even in the well-specified case, see Appendix B.3 for the results. This is because the Euclidean distance is large even if only a handful of simulated statistics - corresponding to different parameter values in the training data - are far from the observed statistic. As a result, the regularizer dominates the loss function while the standard loss term L related to θ is not minimized, leading to underconfident posteriors. Hence, we need a distance D that is robust to outliers and can be computed between the set { η ψ ( x 1: n,i ) } i m =1 and η ψ ( y 1: n ) , instead of computing the distance point-wise.

To that end, we take D to be the maximum mean discrepancy (MMD) [42], which is a notion of distance between probability distributions or datasets. MMD has a number of attractive properties that make it suitable for our method: (i) it can be estimated efficiently using samples [15], (ii) it is robust against a few outliers (unlike the KL divergence), and (iii) it can be computed for unequal sizes of datasets. The MMD has been used in a number of frameworks such as ABC [8, 10, 50, 54, 62], minimum distance estimation [2, 15, 19, 56], generalized Bayesian inference [20, 57], and Bayesian nonparametric learning [24]. Similar to our method, the MMD has previously been used as a regularization term to train Bayesian neural networks [66], and in the NPE framework to detect model misspecification [73]. While we include the mmD-based regularizer to ensure that the observed statistic is not an out-of-distribution sample in the summary space, the regularizer in [73] involves computing the MMD between the simulated statistics and samples from a standard Gaussian, thus ensuring that the learned statistics are jointly Gaussian. They then conduct a goodness-of-fit test [42] to detect if the model is misspecified. Their method is complementary to ours, such that our method can be used once misspecification has been detected using [73].

Maximum mean discrepancy (MMD) as D . The MMD is a kernel-based distance between probability distributions, computed by mapping the distributions to a function space called the reproducing kernel Hilbert space (RKHS). Let H k be the RKHS associated with the symmetric and positive definite function k : S × S → R , called a reproducing kernel [6], defined on the space of statistics S such that f ( s ) = ⟨ f, k ( · , s ) ⟩ H k ∀ f ∈ H k (called the reproducing property). We denote the norm and inner product of H k by ∥ · ∥ H k and ⟨· , ·⟩ H k , respectively. Any distribution P can be mapped to H k via its kernel-mean embedding µ k, P = ∫ S k ( · , s ) P ( d s ) . The MMD between two arbitrary probability measures P and Q on S is defined as the distance between their embeddings in H k , i.e., MMD k ( P , Q ) = ∥ µ k, P -µ k, Q ∥ H k [55]. Using the reproducing property, we can express the squared-MMD as

which is computationally convenient as the expectations of the kernel can be estimated using iid samples from P and Q . Given samples of simulated statistics { η ψ ( x 1: n,i ) } l i =1 and the observed

statistic η ψ ( y 1: n ) , we estimate the squared-MMD between them using the V-statistic estimator [42]:

Note that the last term in Equation (7) is always constant for the estimator above as we only have one data-point of the observed statistic, hence we disregard it. Equation (8) therefore corresponds to estimating the MMD between the distribution of the simulated statistics for a given ψ and a Dirac measure on η ψ ( y 1: n ) . The computational cost of estimating the squared-MMD is O ( l 2 ) and its rate of convergence is O ( l -1 2 ) . The NPE loss with robust statistics (NPE-RS) can then be estimated as

Similarly, the autoencoder loss with robust statistics (AE-RS) reads

We use a subset of the training dataset of size l < m to compute the MMD instead of m to avoid incurring additional computational cost in case m is large. The MMD-based regularizer can also be used as a score in a classification task to detect model misspecification, as shown in Appendix A.

Role of the regularizer λ . The regularizer λ penalizes learning those statistics for which the observed statistic η ψ ( y 1: n ) is far from the set of statistics that the model can simulate given a prior distribution. When λ tends to zero, maximization of the likelihood dictates learning of both ϕ and ψ in Equation (9), and our method converges to the NPE method. On the other hand, the regularization term is minimized when the summary network outputs the same statistics for both simulated and observed data, i.e., when D is zero. In this case, the inference network can only rely on information from the prior p ( θ ) . As a result, for large values of λ , we expect the regularization term to dominate the loss and the resulting posterior to converge to the prior distribution. Similar argument holds for the autoencoder loss in Equation (10). We empirically observe this behavior in Section 4. Hence, λ encodes the trade-off between efficiency and robustness of our inference method. Choosing λ can be cast as a hyperparameter selection problem, for which we can leverage additional data (if available) as a validation dataset, or use post-hoc qualitative analysis of the posterior predictive distribution.

4 Numerical experiments

We apply our method of learning robust statistics to two different SBI frameworks - NPE [41] and ABC [5] (see Appendix B.6 for results on applying our method to neural likelihood estimator [52]). We also compare the performance of our method against RNPE [81], by using the output of the trained summary network in NPE as statistics for RNPE. Experiments are conducted on synthetic data from two time-series models, namely the Ricker model from population ecology [67, 85] and the Ornstein-Uhlenbeck process [18]. Real data experiment on a radio propagation model is presented in Section 5. Analysis of the computational cost of our method and results on a 10-dimensional Gaussian model is presented in Appendix B.4 and B.5, respectively. The code to reproduce our experiments is available at https://github.com/huangdaolang/robust-sbi .

Ricker model simulates the evolution of population size N t over the course of time t as N t +1 = exp( θ 1 ) N t exp( -N t + e t ) , t = 1 , . . . , T , where exp( θ 1 ) is the growth rate parameter, e t are zeromean iid Gaussian noise terms with variance σ 2 e , and N 0 = 1 . The observations x t are assumed to be Poisson random variables such that x t ∼ Poiss ( θ 2 N t ) . For simplicity, we fix σ 2 e = 0 . 09 and estimate θ = [ θ 1 , θ 2 ] ⊤ using the prior distribution U ([2 , 8] × [0 , 20]) , and T = 100 time-steps.

Ornstein-Uhlenbeck process (OUP) is a stochastic differential equation model widely used in financial mathematics and evolutionary biology. The OU process x t is defined as:

0.0

2.5

5.0

7.5

10.0

12.5

15.0

RMSE

Ricker (NPE)

Method

NPE

RNPE

NPE-RS (ours)

0

1

2

3

4

5

OUP (NPE)

0

2

4

6

8

10

12

Ricker (ABC)

Method

ABC

ABC-RS (ours)

0

2

4

6

8

OUP (ABC)

0%

10%

20%

/epsilon1

0.00

0.25

0.50

0.75

1.00

1.25

MMD

0%

10%

20%

/epsilon1

0.0

0.2

0.4

0.6

0.8

1.0

1.2

0%

10%

20%

/epsilon1

0.0

0.2

0.4

0.6

0.8

1.0

1.2

0%

10%

20%

/epsilon1

0.00

0.25

0.50

0.75

1.00

1.25

where T = 25 , ∆ t = 0 . 2 , x 0 = 10 , and w ∼ N (0 , ∆ t ) . A uniform prior U ([0 , 2] × [ -2 , 2]) is placed on the parameters θ = [ θ 1 , θ 2 ] ⊤ .

Contamination model. We test for robustness relative to the ϵ -contamination model [65], similar to [24]. For both the Ricker model and the OUP, the observed data comes from the distribution Q = (1 -ϵ ) P θ true + ϵ P θ c , where a large proportion of the data comes from the model with θ true , while ϵ proportion of the data comes from the distribution P θ c . Hence, ϵ ∈ [0 , 1] denotes the level of misspecification within both the models, with ϵ = 0 resulting in the well-specified case. We take θ true = [0 . 5 , 1 . 0] ⊤ , θ c = [ -0 . 5 , 1 . 0] ⊤ for OUP, and θ true = [4 , 10] ⊤ , θ c = [4 , 100] ⊤ for the Ricker model.

Implementation details. We take m = 1000 samples for the training data and n = 100 realizations of both the observed and simulated data for each θ . Weset λ using an additional dataset simulated from the models using θ true . For the MMD, we take the kernel to be the exponentiated-quadratic kernel k ( s , s ′ ) = exp( -∥ s -s ′ ∥ 2 2 /β 2 ) , and set its lengthscale β using the median heuristic β = √ med / 2 [42], where med denotes the median of the set of squared two-norm distances ∥ s i -s j ∥ 2 2 for all pairs of distinct data points in { s i } i m =1 . We use l = 200 samples of the simulated statistics to estimate the MMD. For the Ricker model, the summary network η ψ is composed of 1D convolutional layers, whereas for the OUP, η ψ is a combination of bidirectional long short-term memory (LSTM) recurrent modules and 1D convolutional layers. The dimension of the statistic space is set to four for both the models. We take q to be a conditional normalizing flow for all the three NPE methods. We take the tolerance δ in ABC to be the top 5% of samples that yield the smallest distance.

Performance metrics. We evaluate the accuracy of the posterior, as well as the posterior predictive distribution of the SBI methods. For the posterior distribution, we compute the root mean squared error (RMSE) as (1 /N ∑ N i =1 ( θ i -θ true ) 2 ) 1 2 where { θ i } N i =1 are posterior samples. For the predictive accuracy, we compute the MMD between the observed data and samples from the posterior predictive distribution of each method. As the models simulate high-dimensional data, we use the kernel specialized for time-series from [8]. The lengthscale of this kernel is set to β = 1 for all misspecification levels to facilitate fair comparison.

Results. Figure 2 presents the results for both the Ricker model and the OUP across 100 runs. We observe that both NPE and ABC with our robust statistics (RS) outperform their counterparts, including RNPE, under misspecification ( ϵ = 10% and 20% ). Moreover, our performance is similar

2

4

6

8

1

0

5

10

15

20

2

true

NPE

RNPE

NPE-RS

(ours)

(a) Well-specified

(

ϵ

= 0)

2

4

6

8

1

0

5

10

15

20

25

2

(b) Misspecified

(

ϵ

= 10%)

2

4

6

8

1

0

5

10

15

20

25

2

(c) Misspecified

(

ϵ

= 20%)

to NPE in the well-specified case ( ϵ = 0% ). An instance of the NPE posteriors is shown in Figure 3 for the Ricker model. Our NPE-RS posterior is very similar to the NPE posterior for the well-specified case, whereas the RNPE posterior is underconfident, as noted in [81]. Although RNPE is more robust than NPE under misspecification, its posterior still drifts away from θ true , while NPE-RS posterior stays around θ true even for ϵ = 20% . This is because our method has the flexibility to choose appropriate statistics based on misspecification level, while RNPE is bound to a fixed choice of pre-selected statistics. Posterior plots for ABC and OUP are given in Appendix B.2.

Prior misspecification. So far our discussion has pertained to the case of likelihood misspecification. However, in Bayesian inference, the prior is also a part of the model. Hence, a potential form of misspecification is prior misspecification , in which the prior gives low or even zero probability to the 'true' data generating parameters. We investigate the issue of prior misspecification using the Ricker model as an example. To that end, we modify the prior distribution of θ 2 by setting it to LogNormal(0 . 5 , 1 . 0) while keeping the prior of θ 1 unchanged. We use θ true = [4 , 25] ⊤ to generate the observed data, as θ 2 = 25 has very low density under the lognormal prior, and the probability that a value of θ 2 ≥ 25 will be sampled is 0.00372. The result in Figure 4 shows that NPE and RNPE yield posteriors that do not include the true value of θ 2 , whereas our NPE-RS posterior is still around θ true . This highlights the effectiveness of our method to handle cases of prior misspecification as well.

2

4

6

8

1

0

20

40

60

80

2

true

NPE

RNPE

NPE-RS

(ours)

Varying regularizer λ . As mentioned in Section 3, our method converges to NPE as λ tends to zero, and to the prior distribution as λ becomes high. To investigate this effect, we vary λ and measure the distance between the NPE-RS and the NPE posteriors, and between the NPE-RS posterior and the prior. We use the MMD from Equation (7) to measure this distance on the Ricker model in the misspecified case ( ϵ = 20% ). We see in Figure 5(left) that the MMD values between the prior and the NPE-RS posteriors go close to zero for λ = 10 3 , indicating that our method essentially returns the prior for high values of λ . On the other hand, the MMD values between NPE and NPE-RS posteriors in Figure 5(middle) are smallest for λ = 0 . 01 and λ = 0 . 1 . For non-extreme values of λ (i.e. λ = 1 or 10), we observe maximum difference between NPE-RS and NPE. This is because the NPE posteriors tend to move out of the prior support under misspecification, whereas the NPE-RS posteriors remain around θ true , as shown in Figure 3. Finally, the NPE-RS posteriors are also close to the NPE posteriors for small values of λ in the well-specified case, as shown in Figure 5(right).

0.01

0.1

1.0

10.0

100.0

1000.0

0.0

0.5

1.0

1.5

MMD

Dist. to prior (misspecified)

0.01

0.1

1.0

10.0

100.0

1000.0

0.0

0.5

1.0

1.5

MMD

Dist. to NPE posterior (misspecified)

0.01

0.1

1.0

10.0

100.0

1000.0

0.0

0.5

1.0

1.5

MMD

Dist. to NPE posterior (well-specified)

5 Application to real data: A radio propagation example

Our final experiment involves a stochastic radio channel model, namely the Turin model [45, 63, 78], which is used to simulate radio propagation phenomena in order to test and design wireless communication systems. The model is driven by an underlying point process which is unobservable, thereby rendering its likelihood function intractable. The model simulates high-dimensional complexvalued time-series data, and has four parameters. The parameters govern the starting point, the slope of the decay, the rate of the point-process, and the noise floor of the time-series, as shown in Figure 6.

We attempt to fit this model to a real dataset from a measurement campaign [44] for which the model is known to be misspecified [8], on account of the data samples being non-iid. The averaged power of the data (square of the absolute value) as a function of time is shown by the black curve in Figure 6. The data dimension is 801 and we have n = 100 realizations. We take the power of the complex data in decibels * as input to the summary network, which consists of 1D convolutional layers, and we set uniform priors for all the four parameters. Descriptions of the model, the data, and the prior ranges are provided in Appendix C.

We fit the model to the real data using NPE, RNPE and our NPE-RS methods. As there is no notion of ground truth in this case, we plot the resulting posterior predictive distributions in Figure 6. Note that the multiple peaks present in the data are not replicated by the Turin model

-80

startingpoint

Real data

NPE

-90

RNPE

NPE-RS (ours)

[qp]

-100

slope

Power

-110

-120

-130

noiseflooro

0

50

100

150

200

Time Delay [ns]

for any method, which is due to the model being misspecified for this dataset. Despite that, our NPE-RS method appears to fit the data well, while NPE performs the worst. Moreover, our method is better than RNPE at matching the starting point of the data, the slope, and the noise floor, with RNPE underestimating all three aspects on average. The MMD between the observed data and the posterior predictive distribution of NPE, RNPE and NPE-RS is 0.11, 0.09, and 0.03, respectively. Hence, NPE-RS provides a reasonable fit of the model even under misspecification.

6 Conclusion

We proposed a simple and elegant solution for tackling misspecification of simulator-based models. As our method relies on learning robust statistics and not on the subsequent inference procedure, it is applicable to any SBI method that utilizes summary statistics. Apart from achieving robust inference under misspecified scenarios, the method performs reasonably well even when the model is well-specified. The proposed method only has one hyperparameter that encodes the trade-off between efficiency and robustness, which can be selected like other neural network hyperparameters, for instance, via a validation set.

Limitations and future work. A limitation of our method is the increased computational complexity due to the cost of estimating the MMD, which can be alleviated using the sample-efficient MMDestimator from [10] or quasi-Monte Carlo points [56]. Moreover, as our method utilizes the observed statistic during the training procedure, the corresponding NPE is not amortized anymore a limitation we share with RNPE. Thus, working on robust NPE methods which are still amortized (to some extent) is an interesting direction for future research. An obvious extension of the work could be to investigate if the ideas translate to likelihood-free model selection methods [68, 76] which can suffer from similar problems as SBI if all (or some) of the candidate models are misspecified.

References

Albert, C., Ulzega, S., Ozdemir, F., Perez-Cruz, F., and Mira, A. (2022). Learning Summary Statistics for Bayesian Inference with Autoencoders. SciPost Phys. Core , 5:043.

Alquier, P. and Gerber, M. (2021). Universal robust regression via maximum mean discrepancy. arXiv:2006.00840 .

Beaumont, M. A. (2010). Approximate Bayesian computation in evolution and ecology. Annual Review of Ecology, Evolution, and Systematics , 41(1):379-406.

Beaumont, M. A. (2019). Approximate Bayesian computation. Annual Review of Statistics and Its Application , 6(1):379-403.

Beaumont, M. A., Zhang, W., and Balding, D. J. (2002). Approximate Bayesian computation in population genetics. Genetics , 162(4):2025-2035.

Berlinet, A. and Thomas-Agnan, C. (2004). Reproducing Kernel Hilbert Spaces in Probability and Statistics . Springer Science+Business Media, New York.

Bernardo, S. (2000). Bayesian Theory . John Wiley & Sons.

Bharti, A., Briol, F.-X., and Pedersen, T. (2022a). A general method for calibrating stochastic radio channel models with kernels. IEEE Transactions on Antennas and Propagation , 70(6):39864001.

Bharti, A., Filstroff, L., and Kaski, S. (2022b). Approximate Bayesian computation with domain expert in the loop. In International Conference on Machine Learning , volume 162, pages 18931905.

Bharti, A., Naslidnyk, M., Key, O., Kaski, S., and Briol, F.-X. (2023). Optimally-weighted estimators of the maximum mean discrepancy for likelihood-free inference. arXiv:2301.11674 .

Bi, J., Shen, W., and Zhu, W. (2021). Random forest adjustment for approximate Bayesian computation. Journal of Computational and Graphical Statistics , 0(0):1-10.

Bissiri, P. G., Holmes, C. C., and Walker, S. G. (2016). A general framework for updating belief distributions. Journal of the Royal Statistical Society. Series B (Statistical Methodology) , 78(5):1103-1130.

Blum, M. G. B. and Franc ¸ois, O. (2010). Non-linear regression models for approximate Bayesian computation. Statistics and Computing , 20(1):63-73.

Blum, M. G. B., Nunes, M. A., Prangle, D., and Sisson, S. A. (2013). A comparative review of dimension reduction methods in approximate Bayesian computation. Statistical Science , 28(2).

Briol, F.-X., Barp, A., Duncan, A. B., and Girolami, M. (2019). Statistical inference for generative models with maximum mean discrepancy. arXiv:1906.05944 .

Cannon, P., Ward, D., and Schmon, S. M. (2022). Investigating the impact of model misspecification in neural simulation-based inference. arXiv preprint arXiv:2209.01845 .

Chan, J., Perrone, V., Spence, J., Jenkins, P., Mathieson, S., and Song, Y. (2018). A likelihoodfree inference framework for population genetic data using exchangeable neural networks. In Advances in Neural Information Processing Systems , volume 31.

Chen, Y., Zhang, D., Gutmann, M. U., Courville, A., and Zhu, Z. (2021). Neural approximate sufficient statistics for implicit models. In International Conference on Learning Representations .

Ch´ erief-Abdellatif, B.-E. and Alquier, P. (2020). MMD-Bayes: Robust Bayesian estimation via maximum mean discrepancy. In Proceesings of the 2nd Symposium on Advances in Approximate Bayesian Inference , pages 1-21.

Ch´ erief-Abdellatif, B.-E. and Alquier, P. (2021). Finite sample properties of parametric MMD estimation: robustness to misspecification and dependence. arXiv:1912.05737 .

Cranmer, K., Brehmer, J., and Louppe, G. (2020). The frontier of simulation-based inference. Proceedings of the National Academy of Sciences , 117(48):30055-30062.

Dax, M., Green, S. R., Gair, J., Deistler, M., Sch¨ olkopf, B., and Macke, J. H. (2022). Group equivariant neural posterior estimation. In International Conference on Learning Representations .

Delaunoy, A., Hermans, J., Rozet, F., Wehenkel, A., and Louppe, G. (2022). Towards reliable simulation-based inference with balanced neural ratio estimation. arXiv preprint arXiv:2208.13624 .

Dellaporta, C., Knoblauch, J., Damoulas, T., and Briol, F.-X. (2022). Robust bayesian inference for simulator-based models via the mmd posterior bootstrap. In International Conference on Artificial Intelligence and Statistics , volume 151, pages 943-970.

Diggle, P. J. and Gratton, R. J. (1984). Monte carlo methods of inference for implicit statistical models. Journal of the Royal Statistical Society: Series B (Methodological) , 46(2):193-212.

Dinev, T. and Gutmann, M. U. (2018). Dynamic likelihood-free inference via ratio estimation (DIRE). arXiv preprint arXiv:1810.09899 .

Dingeldein, L., Cossio, P., and Covino, R. (2023). Simulation-based inference of singlemolecule force spectroscopy. Machine Learning: Science and Technology , 4(2):025009.

Durkan, C., Murray, I., and Papamakarios, G. (2020). On contrastive learning for likelihood-free inference. In International Conference on Machine Learning , volume 119, pages 2771-2781.

Dyer, J., Cannon, P., Farmer, J. D., and Schmon, S. (2022a). Black-box bayesian inference for economic agent-based models. arXiv preprint arXiv:2202.00625 .

Dyer, J., Cannon, P. W., and Schmon, S. M. (2022b). Amortised likelihood-free inference for expensive time-series simulators with signatured ratio estimation. In International Conference on Artificial Intelligence and Statistics , volume 151, pages 11131-11144.

Fearnhead, P. and Prangle, D. (2012). Constructing summary statistics for approximate Bayesian computation: semi-automatic approximate bayesian computation. Journal of the Royal Statistical Society. Series B (Statistical Methodology) , 74(3):419-474.

Frazier, D. T. and Drovandi, C. (2021). Robust approximate bayesian inference with synthetic likelihood. Journal of Computational and Graphical Statistics , 30(4):958-976.

Frazier, D. T., Drovandi, C., and Loaiza-Maya, R. (2020a). Robust approximate bayesian computation: An adjustment approach. arXiv preprint arXiv:2008.04099 .

Frazier, D. T., Drovandi, C., and Nott, D. J. (2021). Synthetic likelihood in misspecified models: Consequences and corrections. arXiv preprint arXiv:2104.03436 .

Frazier, D. T., Robert, C. P., and Rousseau, J. (2020b). Model misspecification in approximate bayesian computation: consequences and diagnostics. Journal of the Royal Statistical Society: Series B (Statistical Methodology) , 82(2):421-444.

Fujisawa, M., Teshima, T., Sato, I., and Sugiyama, M. (2021). γ -abc: Outlier-robust approximate bayesian computation based on a robust divergence estimator. In International Conference on Artificial Intelligence and Statistics , volume 130, pages 1783-1791.

Furia, C. S. and Churchill, R. M. (2022). Normalizing flows for likelihood-free inference with fusion simulations. Plasma Physics and Controlled Fusion , 64(10):104003.

Geffner, T., Papamakarios, G., and Mnih, A. (2022). Score modeling for simulation-based inference. arXiv preprint arXiv:2209.14249 .

Gl¨ ockler, M., Deistler, M., and Macke, J. H. (2022). Variational methods for simulation-based inference. In International Conference on Learning Representations .

Green, S. R. and Gair, J. (2021). Complete parameter inference for GW150914 using deep learning. Machine Learning: Science and Technology , 2(3):03LT01.

Greenberg, D., Nonnenmacher, M., and Macke, J. (2019). Automatic posterior transformation for likelihood-free inference. In International Conference on Machine Learning , volume 97, pages 2404-2414.

Gretton, A., Borgwardt, K., Rasch, M. J., and Scholkopf, B. (2012). A kernel two-sample test. Journal of Machine Learning Research , 13:723-773.

Gr¨ unwald, P. (2012). The Safe Bayesian. In Lecture Notes in Computer Science , pages 169-183. Springer Berlin Heidelberg.

Gustafson, C., Bolin, D., and Tufvesson, F. (2016). Modeling the polarimetric mm-wave propagation channel using censored measurements. In 2016 Global Communications Conference . IEEE.

Haneda, K., J¨ arvel¨ ainen, J., Karttunen, A., Kyr¨ o, M., and Putkonen, J. (2015). A statistical spatio-temporal radio channel model for large indoor environments at 60 and 70 ghz. IEEE Transactions on Antennas and Propagation , 63(6):2694-2704.

Hermans, J., Begy, V., and Louppe, G. (2020). Likelihood-free MCMC with amortized approximate ratio estimators. In International Conference on Machine Learning , volume 119, pages 4239-4248.

Jiang, B., yu Wu, T., Zheng, C., and Wong, W. (2017). Learning summary statistic for approximate Bayesian computation via deep neural network. Statistica Sinica , page 1595-1618.

Kelly, R. P., Nott, D. J., Frazier, D. T., Warne, D. J., and Drovandi, C. (2023). Misspecificationrobust sequential neural likelihood. arXiv preprint arXiv:2301.13368 .

Knoblauch, J., Jewson, J., and Damoulas, T. (2019). Generalized variational inference: Three arguments for deriving new posteriors. arXiv:1904.02063 .

Legramanti, S., Durante, D., and Alquier, P. (2022). Concentration and robustness of discrepancy-based ABC via Rademacher complexity. arXiv:2206.06991 .

Lintusaari, J., Gutmann, M. U., Dutta, R., Kaski, S., and Corander, J. (2017). Fundamentals and recent developments in approximate Bayesian computation. Systematic Biology , 66:66-82.

Lueckmann, J.-M., Bassetto, G., Karaletsos, T., and Macke, J. H. (2019). Likelihood-free inference with emulator networks. In Proceedings of The 1st Symposium on Advances in Approximate Bayesian Inference , volume 96, pages 32-53.

Lueckmann, J.-M., Goncalves, P. J., Bassetto, G., ¨ Ocal, K., Nonnenmacher, M., and Macke, J. H. (2017). Flexible statistical inference for mechanistic models of neural dynamics. In Advances in Neural Information Processing Systems , volume 30.

Mitrovic, J., Sejdinovic, D., and Teh, Y .-W. (2016). DR-ABC: Approximate Bayesian computation with kernel-based distribution regression. In Proceedings of the International Conference on Machine Learning , pages 1482-1491.

Muandet, K., Fukumizu, K., Sriperumbudur, B., and Sch¨ olkopf, B. (2017). Kernel mean embedding of distributions: A review and beyond. Foundations and Trends® in Machine Learning , 10(1-2):1-141.

Niu, Z., Meier, J., and Briol, F.-X. (2021). Discrepancy-based inference for intractable generative models using quasi-Monte Carlo. arXiv:2106.11561 .

Pacchiardi, L. and Dutta, R. (2021). Generalized bayesian likelihood-free inference using scoring rules estimators. arXiv preprint arXiv:2104.03889 .

Pacchiardi, L. and Dutta, R. (2022). Likelihood-free inference with generative neural networks via scoring rule minimization. arXiv preprint arXiv:2205.15784 .

Papamakarios, G. and Murray, I. (2016). Fast ϵ -free inference of simulation models with bayesian conditional density estimation. In International Conference on Neural Information Processing Systems , page 1036-1044.

Papamakarios, G., Pavlakou, T., and Murray, I. (2017). Masked autoregressive flow for density estimation. Advances in neural information processing systems , 30.

Papamakarios, G., Sterratt, D., and Murray, I. (2019). Sequential neural likelihood: Fast likelihood-free inference with autoregressive flows. In International Conference on Artificial Intelligence and Statistics , volume 89, pages 837-848.

Park, M., Jitkrittum, W., and Sejdinovic, D. (2015). K2-ABC: approximate Bayesian computation with kernel embeddings. International Conference on Artificial Intelligence and Statistics , 51:398-407.

Pedersen, T. (2019). Stochastic multipath model for the in-room radio channel based on room electromagnetics. IEEE Transactions on Antennas and Propagation , 67(4):2591-2603.

Pesonen, H., Simola, U., K¨ ohn-Luque, A., Vuollekoski, H., Lai, X., Frigessi, A., Kaski, S., Frazier, D. T., Maneesoonthorn, W., Martin, G. M., and Corander, J. (2022). ABC of the future. International Statistical Review .

Peter J. Huber, E. M. R. (2009). Robust Statistics . WILEY.

Pomponi, J., Scardapane, S., and Uncini, A. (2021). Bayesian neural networks with maximum mean discrepancy regularization. Neurocomputing , 453:428-437.

Price, L. F., Drovandi, C. C., Lee, A., and Nott, D. J. (2018). Bayesian synthetic likelihood. Journal of Computational and Graphical Statistics , 27(1):1-11.

Pudlo, P., Marin, J.-M., Estoup, A., Cornuet, J.-M., Gautier, M., and Robert, C. P. (2015). Reliable ABC model choice via random forests. Bioinformatics , 32(6):859-866.

Radev, S. T., Mertens, U. K., Voss, A., Ardizzone, L., and K¨ othe, U. (2022). Bayesflow: Learning complex stochastic models with invertible neural networks. IEEE Transactions on Neural Networks and Learning Systems , 33(4):1452-1466.

Ramesh, P., Lueckmann, J.-M., Boelts, J., Tejero-Cantero, ´ A., Greenberg, D. S., Goncalves, P. J., and Macke, J. H. (2022). GATSBI: Generative adversarial training for simulation-based inference. In International Conference on Learning Representations .

Riesselman, A. J., Ingraham, J. B., and Marks, D. S. (2018). Deep generative models of genetic variation capture the effects of mutations. Nature Methods , 15(10):816-822.

Rothfuss, J., Ferreira, F., Walther, S., and Ulrich, M. (2019). Conditional density estimation with neural networks: Best practices and benchmarks. arXiv preprint arXiv:1903.00954 .

Schmitt, M., B¨ urkner, P.-C., K¨ othe, U., and Radev, S. T. (2021). Detecting model misspecification in amortized bayesian inference with neural networks. arXiv e-prints .

Schmon, S. M., Cannon, P. W., and Knoblauch, J. (2020). Generalized posteriors in approximate bayesian computation. arXiv preprint arXiv:2011.08644 .

Sharrock, L., Simons, J., Liu, S., and Beaumont, M. (2022). Sequential neural score estimation: Likelihood-free inference with conditional score based diffusion models. arXiv preprint arXiv:2210.04872 .

Sisson, S. A. (2018). Handbook of Approximate Bayesian Computation . Chapman and Hall/CRC.

Thomas, O., Dutta, R., Corander, J., Kaski, S., and Gutmann, M. U. (2022). Likelihood-free inference by ratio estimation. Bayesian Analysis , 17(1).

Turin, G. L., Clapp, F. D., Johnston, T. L., Fine, S. B., and Lavry, D. (1972). A statistical model of urban multipath propagation. IEEE Transactions on Vehicular Technology , 21(1):1-9.

van Opheusden, B., Acerbi, L., and Ma, W. J. (2020). Unbiased and efficient log-likelihood estimation with inverse binomial sampling. PLoS Computational Biology , 16(12):e1008483.

Vasist, M., Rozet, F., Absil, O., Molli` ere, P., Nasedkin, E., and Louppe, G. (2023). Neural posterior estimation for exoplanetary atmospheric retrieval. Astronomy & Astrophysics , 672:A147.

Ward, D., Cannon, P., Beaumont, M., Fasiolo, M., and Schmon, S. M. (2022). Robust neural posterior estimation and statistical model criticism. In Advances in Neural Information Processing Systems .

Weinstein, E. N. and Miller, J. W. (2023). Bayesian data selection. Journal of Machine Learning Research , 24(23):1-72.

Wiqvist, S., Frellsen, J., and Picchini, U. (2021). Sequential neural posterior and likelihood approximation. arXiv preprint arXiv:2102.06522 .

Wiqvist, S., Mattei, P.-A., Picchini, U., and Frellsen, J. (2019). Partially exchangeable networks and architectures for learning summary statistics in approximate Bayesian computation. In International Conference on Machine Learning , volume 97, pages 6798-6807.

Wood, S. N. (2010). Statistical inference for noisy nonlinear ecological dynamic systems. Nature , 466(7310):1102-1104.

Zaheer, M., Kottur, S., Ravanbakhsh, S., Poczos, B., Salakhutdinov, R. R., and Smola, A. J. (2017). Deep sets. Advances in neural information processing systems , 30.

Zbair, M., Qaffou, A., and Hilal, K. (2022). Approximate bayesian estimation of parameters of an agent-based model in epidemiology. In Lecture Notes in Networks and Systems , pages 302-314. Springer International Publishing.

Supplementary Materials

The appendix is organized as follows:

In Appendix A, we present the results on detecting model misspecification.

In Appendix B, we provide further implementation details and results for the numerical experiment in Section 4.

-Appendix B.1: Implementation details

-Appendix B.2: Additional posterior plots

-Appendix B.3: Results for D being the Euclidean distance

-Appendix B.4: Computational cost analysis

-Appendix B.5: Adversarial training on Gaussian linear model

-Appendix B.6: Experiment with neural likelihood estimator

In Appendix C, we provide details of the radio propagation experiment of Section 5.

A Detecting misspecification of simulators

Considering that existing SBI methods can yield unreliable results under misspecification and that real-world simulators are probably not able to fully replicate observed data in most cases, detecting whether the simulator is misspecified becomes necessary for generating confidence in the results given by these methods. As misspecification can lead to observed statistics or features falling outside the distribution of training statistics, detecting for it essentially boils down to a class of out-of-distribution detection problems known as novelty detection , where the aim is to detect if the test sample s obs come from the training distribution induced by { s i } i m =1 . This two-label classification problem can potentially be solved by adapting any of the numerous novelty detection methods from the literature. We propose the following two simple novelty detection techniques for detecting misspecification:

Distance-based approach. We assign a score to the observed statistic based on the value of the margin upper bound, as introduced in the main text. We use the MMD as the choice of distance D , and estimate the MMD between the set of simulated statistics { s i } i m =1 and the observed statistic s obs . This MMD-based score can be used in a classification method to detect misspecification.

Density-based approach. In this method, the training samples { s i } i m =1 are used to fit a generative model q , and the log-likelihood of the observed statistics under q are used as the classification score. We use a Gaussian mixture model (GMM) with k components as q , having the distribution

where ξ i , µ i , and Σ i are the weight, the mean and the covariance matrix associated with the i th component, and φ denotes the Gaussian pdf. The score ln q ( s obs ) can then be used to classify it as either being from in or out of the training distribution.

Experimental set-up. We test the performance of the proposed detection methods on the Ricker model and the OUP with the same contamination model as given in the main text. For each of these simulators, we first train the NPE method on m = 1000 training data points, and fit a GMM with k = 2 components to them. We then generate 1000 test datasets or points, half of them from the well-specified model and the other half from the misspecified model, and compute their score. The area under the receiver operating characteristic (AUROC) is used as the performance metric.

Baseline. We construct a baseline for comparing performance of the proposed detection methods. The baseline is based on the insight that under model misspecification, the NPE posterior moves away from the true parameter value (even going outside the prior range). Therefore, we take the root mean squared error (RMSE), defined as (1 /N ∑ N i =1 ( θ i -θ true ) 2 ) 1 2 where { θ i } N i =1 are posterior samples, as the classification score.

0.01

0.05

0.1

0.15

0.2

Misspecification level

0.4

0.5

0.6

0.7

0.8

0.9

1.0

AUROC

Ricker

RMSE

GMM

MMD

0.01

0.05

0.1

0.15

0.2

Misspecification level

0.4

0.5

0.6

0.7

0.8

0.9

1.0

OUP

Results. The AUROC of the classifiers for different levels of misspecification ( ϵ in the main text) is shown in Fig. 7 for both the models. The proposed GMM-based detection method performs the best, followed by the MMD-based method. The RMSE-based baseline performs the worst at the classification task. We conclude that it is possible to detect model misspecification in the space of summary statistics using simple to use novelty detection methods.

B Additional details and results of the numerical experiments

B.1 Implementation details

We implement our NPE-RS models based on publicly available implementations from https: //github.com/mackelab/sbi . We use the NPE-C model [41] with Masked Autoregressive Flow (MAF) [60] as the backbone inference network, and adopt the default configuration with 50 hidden units and 5 transforms for MAF. The batch size is set to 50, and we maintain a fixed learning rate of 5 × 10 -4 . The implementation for RNPE is sourced directly from the original repository at https://github.com/danielward27/rnpe .

Regarding the summary network in NPE tasks, for the Ricker model, we employ three 1D convolutional layers with 4 hidden channels, and we set the kernel size to 3. For the OUP model, we combine three 1D convolutional layers with one bidirectional LSTM layer. The convolutional layers have 8 hidden channels and a kernel size equal to 3, while the LSTM layer has 2 hidden dimensions. We pass the data separately through the convolutional layers and the LSTM layer and then concatenate the resulting representations to obtain our summary statistics. For the Turin model in Section 5, we utilize five 1D convolutional layers with hidden units set to [8, 16, 32, 64, 8], and the kernel size is set to 3. Across all three summary networks, we employ the mean operation as our aggregator to ensure permutation invariance among realizations.

In ABC tasks, we incorporate autoencoders as our summary network. For the Ricker model, the encoder consists of three 1D convolutional layers with 4 hidden channels, where the kernel size is set to 3. The decoder comprises of three 1D transposed convolutional layers with the same settings as the encoder's convolutional layers, allowing for data reconstruction. For the OUP model, we adopt a similar summary network as the one used for the Ricker model but with a smaller stride.

In NPE tasks, we use 1000 samples for the training data, along with 100 realizations of both observed and simulated data for each value of θ . We also use 1000 samples for training the autoencoders. For ABC, we use 4000 samples from the prior and accept n δ = 200 samples giving a tolerance rate of 5%. We take ρ to be Euclidean distance in the rejection ABC and normalize the statistics by the median absolute deviation before computing the distance to account for the difference in their magnitude.

B.2 Additional posterior plots

We now present examples of the remaining posterior plots, apart from the one shown in the main text. The posterior plots for OUP using the NPE-based methods is shown in Figure 8. The observations

are similar to the Ricker model example in the main text: we see that our NPE-RS method yields similar posterior as NPE in the well-specified case, whereas RNPE posteriors are underconfident. When the model is misspecified, NPE posterior goes far from the true parameter value. The NPE-Rs posteriors, however, are still around θ true , demonstrating robustness to misspecification.

0

1

2

1

2

1

0

1

2

2

true

NPE

RNPE

Ours

(a) Well-specified

(

ϵ

= 0)

0

1

2

1

2

1

0

1

2

2

(b) Misspecified

(

ϵ

= 10%)

1

0

1

2

3

1

2

1

0

1

2

2

(c) Misspecified

(

ϵ

= 20%)

Similar behavior is observed in the ABC case for both the Ricker model and OUP in Figure 9 and Figure 10, respectively. The ABC posteriors go outside the prior range under misspecification, while ABC with our robust statistics yields posteriors closer to θ true . In Table 1, we report the sample mean and standard deviations for the results shown in Figure 2 of the main text.

2

4

6

8

1

0

5

10

15

20

2

true

ABC

Ours

(a) Well-specified

(

ϵ

= 0)

2

4

6

8

1

0

5

10

15

20

25

2

(b) Misspecified

(

ϵ

= 10%)

2

4

6

8

1

0

5

10

15

20

25

2

(c) Misspecified

(

ϵ

= 20%)

B.3 Results for D being the Euclidean distance

We present results for D being the Euclidean distance in the well-specified case of the Ricker model in Figure 11(a). As mentioned in Section 3 of the main text, this choice leads to very underconfident posteriors. This is because the Euclidean distance is not a robust distance: it becomes large even if a few points are far from the observed statistic. As a result, using this as the regularization term penalizes most choices of summarizer η , and we learn statistics that are very concentrated around the observed statistic (orange dot). Although a good choice for being robust, Euclidean distance leads to statistics that are not informative about the model parameters, yielding posterior that is similar to the uniform prior. Hence, we used the MMD as the distance in the margin upper bound, which provides a better trade-off between robustness and efficiency (in terms of learning about model parameters).

0

1

2

1

2

1

0

1

2

2

true

ABC

Ours

(a) Well-specified

(

ϵ

= 0)

0

1

2

1

2

0

2

4

2

(b) Misspecified

(

ϵ

= 10%)

0

1

2

1

0

5

10

15

2

(c) Misspecified

(

ϵ

= 20%)

B.4 Computational cost analysis

We now present a quantitative analysis of the computational cost of training with and without our MMD regularization term. The results, presented in Table 2, are calculated on an Apple M1 Pro CPU. As expected, we observe a higher runtime for our method due to the computational cost of estimating the MMD from 200 samples of simulated data. The total runtime also depends on the number of batchsize N batch, hence, as N batch increases, the proportion of runtime used for estimating MMDreduces. As a result, we see that for large N batch, the increase in the computational cost of our method with robust statistics is not significant.

B.5 Adversarial training on Gaussian linear model

To verify the robustness of our method on higher dimensional parameter space, we run an experiment on the Gaussian linear model, where the data x ∈ R 10 is sampled from N ( θ, 0 . 1 · I 10 ) . Auniform prior U ([ -1 , 1]) 10 is placed on the parameters θ ∈ R 10 . We take θ true = [0 . 5 , ..., 0 . 5] ⊤ , θ c = [2 , ..., 2] ⊤ . To introduce contamination to the observed data, we employ the same approach as outlined in the main text of our paper. However, there is a slight divergence in our experimental setup. In this example, we employ adversarial training, meaning that the model is trained on observed data with a high degree of misspecification ( ϵ = 20% ), while we perform inference on data with a lower misspecification degree ( ϵ = 10% ). For the summary network, we utilize the DeepSet [86] architecture. The encoder comprises two linear layers, each with a width of 20 hidden units, paired with the ReLU activation function. The decoder is constructed with a single linear layer of 20 hidden units.

The results are shown in Table 3. NPE-RS outperforms NPE in terms of MMD between the posterior predictive distribution and the observed data, which highlights the effectiveness of our approach in

2

4

6

8

1

0

5

10

15

20

2

true

NPE

NPE-RS

(Euclidean)

(a) Posteriors

(b) Summary statistics

S

S

3

S1

S3

high-dimensional parameter spaces, even though the observed data was not used in the training of NPE-RS. This points towards the possibility that by employing adversarial training, we might achieve robustness against lower levels of misspecification whilst still being amortized.

B.6 Experiment with neural likelihood estimators

In this section, we explore the performance of our method when paired with Neural Likelihood Estimators (NLE). NLE are a class of methods that leverage neural density estimators to directly estimate likelihood functions, bridging the gap between simulators and statistical models.

For this experiment, we adopt NLE-A as proposed by [61]. The original implementation can be found at https://github.com/mackelab/sbi . Similar to our approach with ABC, we adapt our method to NLE by pre-emptively training an autoencoder with our regularization term to learn the summary statistics. We refer to our adapted method as NLE-RS. The configurations for our summary network and simulator are consistent with those described in Appendix B.1.

Figure 12 presents the posterior plots for the Ricker model using the NLE-based methods. Consistent with our observations in the previous experiments, NLE-RS still demonstrates robustness to misspecification, while the NLE posterior tends to deviate away from the true parameters.

2.5

5.0

7.5

1

0

10

20

30

2

True

Ours

NLE

C Details of the radio propagation experiment

In this section, we describe the data and the Turin model used in Section 5 of the main text.

Data and model description. Let B be the frequency bandwidth used to measure radio channel data at K equidistant points, leading to a frequency separation of ∆ f = B/ ( K -1) . The measured transfer function at k th point, Y k , is modelled as

where H k is the transfer function at the k th frequency, and W k is additive zero-mean complex circular symmetric Gaussian noise with variance σ 2 W . Taking the inverse Fourier transform, the time-domain signal y ( t ) can be obtained as

The Turin model defines the transfer function as H k = ∑ l α l exp( -j 2 π ∆ fkτ l ) , where τ l is the time-delay and α l is the complex gain of the l th component. The arrival time of the delays is modeled as one-dimensional homogeneous Poisson point processes, i.e., τ l ∼ PPP( R + , ν ) , with ν > 0 . The gains conditioned on the delays are modeled as iid zero-mean complex Gaussian random variables with conditional variance E [ | α l | 2 | τ l ] = G 0 exp( -τ l /T ) /ν . The parameters of the model are θ = [ G 0 , T, ν, σ 2 W ] ⊤ . The prior ranges used for the parameters are given in Table 4.

The radio channel data from [44] is collected in a small conference room of dimensions 3 × 4 × 3 m 3 , using a vector network analyzer. The measurement was performed with a bandwidth of B = 4 GHz, and K = 801 . Denote each complex-valued time-series by ˜ y ∈ R K , and the whole dataset by ˜ y 1: n , where n = 100 realizations. We take the input to the summary network to be y 1: n = 10 log 10 ( | ˜ y 1: n | 2 ) .

Scatter-plot of learned statistics. In Figure 13 and Figure 14, we show the scatter-plots of the learned statistics using the NPE and our NPE-RS method, respectively. We observe that the observed statistics (shown in orange) is often outside the set of simulated statistics (shown in blue) for the NPE method. Hence, the inference network is forced to generalize outside its training distribution, which leads to poor fit of the model, as shown in Section 5 of the main text. On the other hand, the observed statistic is always inside the set of simulated statistics (or the training distribution) for our method in Figure 14, which leads to robustness against model misspecification.

S

5

6

5

S1

S2

S3

S5

S6

S7

S

S1

S2

S3

S5

S6

S7

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a general approach to handling model misspecification for SBI. The paper introduces a regularized loss function that penalises mismatches between the observed data and the learned model. The paper focuses on NPE and ABC as the likelihood-free approaches.\n\nSTRENGTHS:\n* The radio propagation example is interesting, and shows what the potential strength of using the MMD regularizer. \n* The structure of the paper is clear.\n\nWEAKNESSES:\n* The novelty of the work is a concern in light of reference [72] equation (9), compared to equation (9) in this paper. Additionally, equation (10) appears similar to the InfoVAE (https://arxiv.org/abs/1706.02262), which also uses an auto-encoder with a MMD loss in the latent space. \n* A minor weakness is that the paper is difficult to follow. While the structure made sense, the descriptions of the approach and the motivation for it were challenging to understand. For example, it seemed implicit that this approach did not perform amortised NPE until it was highlighted this was the case in the conclusion. This is not in itself a criticism of the approach, but it was very hard to follow what consisted of the training data, and observations. For example, line 255 mentions using 1000 samples for the training data and 100 realisations of both the observed and simulated data for each $\\theta$. Under an amortised setting we would not have access to the observed $\\theta$. It is not clear what this setup is from the description. Further questions in the next section highlight some of these confusions that likely stem from descriptions of the approach that could be improved.\n* An additional minor weakness is only comparing to ABC and NPE. Is there any limitation on applying this approach to other estimators such as Neural Likelihood Estimators? Also for ABC, the $\\rho$ was only described in the supplementary materials as the Euclidean distance. After defining the MMD as the better regularizer, why would this not also be used/incorporated into the discrepancy?\n\nTypo: line 180, the $i$ should be subscript?\n\n* In Figure 5, it would be useful to ensure the three plots share the same y axis.\n\nQUESTIONS:\n* Could the authors provide more details about the step in Eq. (4) to go from infimum to upper bound? I found this challenging to understand.\n* The motivation behind defining a Q and a P that are different is a bit confusing. Why were these definitions highlighted? It seems that Q is equal to P when using the $\\theta_{true}$, but is it a requirement to know $\\theta_{true}$ prior to the experiments in order to simulate from Q?\n* The last sentence of section 5 highlights that the MMD is superior for the NPE-RS method. Is this surprising given it is built into the loss function? Are there other statistics that could be used here that are specific to the application domain?\n\nLIMITATIONS:\nThis is well captured by the paper. This is appreciated!", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a method to make neural posterior estimation robust to misspecification. The method relies on adding a regularizer to NPE loss function. The regularizer is implemented as the MMD between the embedding of the approximate (learned) posterior and the embedded observation. They apply their method to two low-dimensional benchmark tasks and a real-world example from radio propagation.\n\nSTRENGTHS:\n**Originality**: \nThe paper tackles an important issue: how to make neural simulation-based inference robust to misspecification. The method is novel and intuitive.\n\n**Quality**: \nThe method is applied to benchmark tasks and to a real-world problem (with iid data samples and high-dimensional data).\n\n**Clarity**: \nThe paper is well-written and easy-to-follow. Figures are intuitive and well-designed.\n\nWEAKNESSES:\nMy main concerns with the paper are that it (1) uses only two very low-dimensional benchmark tasks and (2) overstates it contributions.\n\n**Originality**:\n- the method is a very straightforward extension to Schmitt et al 2021 (which is only cited with a passing reference). The authors should clearly discuss how the works are related.\n\n**Quality**: \n- The authors repeatedly emphasize that their method performs on par with NPE for well-specified models. I do not think that their results support this claim though. NPE-RS performs significantly worse than NPE on well-specified data on one of the (only) two benchmark tasks. In order to make this claim, the authors should use significantly more benchmark tasks, ideally with varying parameter and data-dimensionality, and demonstrate that the behaviour shown in figure 2, left is a rare exception. In addition, the authors would have to show that NPE-RS converges (at least very closely) to the true posterior with many simulations (see below).\n\n- Convergence: The method proposed by the authors does no longer converge to the true posterior distribution for well-specified models, which further emphasizes that the method does not replace detection of misspecification. I would appreciate an investigation of how regularization strenght trades-off performance on well-specified data vs robustness to misspecification.\n\n- how is the hyperparameter chosen? The authors claim that the regularizer can be selected with a validation set (L353). It is unclear to me how this would work. What would be the loss function used to assess performance on the validation set? In L231 the authors also say that this might be done based on the posterior predictive distribution, but I think this is tricky because it (1) requires (potentially many) more simulations and (2) can easily lead to pathological cases where the posterior is off but the predictive distribution is good. Please elaborate on how the hyperparameter should be set in practice.\n\nQUESTIONS:\nThe authors claim that RNPE is not amortized. However, as far as I understand, RNPE does not require retraining for new data (yes, it requires MCMC, but this can be very fast, especially for low-d parameter space). Please clarify if I misunderstood this or clarify in your paper.\n\nLIMITATIONS:\nState more explicitly that the method does **not** converge to the true posterior.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper describes a method to perform simulation based inference, focusing on neural posterior estimation and approximate Bayesian computation (ABC), under model miss specification when performing inference using summary statistics of the data by using an MMD loss between the simulated and observed data summaries as a means to regulate miss specification.  Experiments are performed to empirically examine the efficacy of the method, and the method is compared to standard neural posterior estimation and ABC, as well as a method called robust neural posterior estimation.\n\nSTRENGTHS:\nThe paper is tackling an important issue, the method is interesting, and the description of the method and its motivation are made clear.\n\nWEAKNESSES:\nIt is not clear how this work differs from ref [72], Schmitt et. al, \"Detecting Model Misspecification in Amortized Bayesian Inference with Neural Networks\". Much clearer description of the differences is needed to understand the novelty of this work\n\nIt is not clear how to determine the amount of regularization, i.e. how to set the hyperparameter lambda. This seems to be a key missing piece of information on how to practically use this method.\n\nThe paper seems to indicate that the summarizer acts on sets of iid samples. But is it not clear why must the summarizer act on the set of x's, rather than summarizing each observation, and using the fact that the summaries will remain iid. This seems to come back as a constraint later in line 179, but does not seem well motivated, or practically how people use summary statistics for iid data. This could also significantly impact the ability to train a conditional normalizing flow due to the massive reduction in information by summarizing over a set of examples rather that summarizing per example.\n\nIn the experiments, it is not clear what summary statistic is used for RNPE? Summary statistics are not described in the RNPE paper, but rather attempting to model the p(theta|x), so how was this choice made? Does it affect the results?\n\nMoreover, in the experiments, why not compare to the method in ref [72]?\n\nIn terms of reference, there are also methods to learn robust summaries prior to inference, e.g. using pivots, such as in Louppe, et. al, \"Learning to pivot with adversarial networks\", and similar domain adaptation approaches, some specifically using MMD.\n\nQUESTIONS:\nHow does this work differ from ref [72] ? Can you provide a much clearer description of the differences is needed to understand the novelty of this work. Can you compare to this work?\n\nOn line 257, is is stated that lambda is set using simulations using theta_true. What does it mean that you set lambda using a data set with known theta_true? isn't this something that needs to be estimated? Doesn't this greatly reduce the challenge of inference, and this information is not realistically available? \n\nWhy does the summarizer act on on sets of iid data, rather than on each iid data example? Acting on each example individually seems to be much closer to practical usage. Do you have experiments in this setting? Would this significantly impact the computation driven by MMD, it seems this may make the method impractical?\n\nPlease provide more details on how RPNE was used in the experiments, and how the summary statistics were determined.\n\nLIMITATIONS:\nThe authors adequately discuss limitations", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper addresses the \"data selection problem\", i.e. identify a low-dimensional statistic (e.g. mean and variance for Gaussian distribution) of a high-dimensional dataset for which the model can replicate even when misspecified. The paper propose using the auto-encoding framework to automatically extract statistics and a penalty on mismatched statistics, i.e. statistics that the model is unable to replicate, as a general approach to handle model misspecification. Empirical results show robust inference in misspecified scenarios whilst still being accurate in well-specified scenarios.\n\nSTRENGTHS:\n1. The paper introduces the \"data selection problem\". Sorry, I am not a subject matter expert in the field of \"data selection\" thus I cannot discuss on the originality and significance of this work. However, I learnt about the data selection problem from the paper.\n\nWEAKNESSES:\n1. The ideas of auto-encoding, reducing mismatch and tuning the regularizer are not new, but they are perhaps new in the application to \"data selection\". Please discuss further.\n2. From the paper, since it is an auto-encoding framework on the model, the statistics are informative of the model. On the other hand, how to ensure the statistics are informative of the observations? Please discuss further.\n\nQUESTIONS:\n1. Since the extracted statistics are changing/ unknown beforehand, how will these statistics fit into downstream applications for downstream applications that were fixed beforehand? Perhaps the downstream tasks are retrained on the new robust statistics.\n2. In the experiment implementation, should the model be the misspecified case while the observations be the true case? Or both can have some degree of misspecification while we try to extract the true parameters?\n\nLIMITATIONS:\n1. Authors mentioned the limitation of using the observed statistic during the training procedure and recommended working on it as future work.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 1, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}