Title: Generalized Bayesian Inference for Scientific Simulators via Amortized Cost Estimation
Venue: NeurIPS 2023
Accepted: yes
Keywords: simulation-based inference

generalized bayesian inference

neural network

machine learning for science
Abstract: Simulation-based inference (SBI) enables amortized Bayesian inference for simulators with implicit likelihoods. But when we are primarily interested in the quality of predictive simulations, or when the model cannot exactly reproduce the observed data (i.e., is misspecified), targeting the Bayesian posterior may be overly restrictive. Generalized Bayesian Inference (GBI) aims to robustify inference for (misspecified) simulator models, replacing the likelihood-function with a cost function that evaluates the goodness of parameters relative to data. However, GBI methods generally require running multiple simulations to estimate the cost function at each parameter value during inference, making the approach computationally infeasible for even moderately complex simulators. Here, we propose amortized cost estimation (ACE) for GBI to address this challenge: We train a neural network to approximate the cost function, which we define as the expected distance between simulations produced by a parameter and observed data. The trained network can then be used with MCMC to infer GBI posteriors for any observation without running additional simulations. We show that, on several benchmark tasks, ACE accurately predicts cost and provides predictive simulations that are closer to synthetic observations than other SBI methods, especially for misspecified simulators. Finally, we apply ACE to infer parameters of the Hodgkin-Huxley model given real intracellular recordings from the Allen Cell Types Database. ACE identifies better data-matching parameters while being an order of magnitude more simulation-efficient than a standard SBI method. In summary, ACE combines the strengths of SBI methods and GBI to perform robust and simulation-amortized inference for scientific simulators.

1

Abstract

Simulation-based inference (SBI) enables amortized Bayesian inference for simulators with implicit likelihoods. But when we are primarily interested in the quality of predictive simulations, or when the model cannot exactly reproduce the observed data (i.e., is misspecified), targeting the Bayesian posterior may be overly restrictive. Generalized Bayesian Inference (GBI) aims to robustify inference for (misspecified) simulator models, replacing the likelihood-function with a cost function that evaluates the goodness of parameters relative to data. However, GBI methods generally require running multiple simulations to estimate the cost function at each parameter value during inference, making the approach computationally infeasible for even moderately complex simulators. Here, we propose amortized cost estimation (ACE) for GBI to address this challenge: We train a neural network to approximate the cost function, which we define as the expected distance between simulations produced by a parameter and observed data. The trained network can then be used with MCMC to infer GBI posteriors for any observation without running additional simulations. We show that, on several benchmark tasks, ACE accurately predicts cost and provides predictive simulations that are closer to synthetic observations than other SBI methods, especially for misspecified simulators. Finally, we apply ACE to infer parameters of the Hodgkin-Huxley model given real intracellular recordings from the Allen Cell Types Database. ACE identifies better data-matching parameters while being an order of magnitude more simulation-efficient than a standard SBI method. In summary, ACE combines the strengths of SBI methods and GBI to perform robust and simulation-amortized inference for scientific simulators.

1 Introduction

Mechanistic models expressed as computer simulators are used in a wide range of scientific domains, from astronomy, geophysics, to neurobiology. The parameters of the simulator, θ , encode mechanisms of interest, and simulating different parameter values produces different outputs, i.e., sim ( θ i ) → x i , where each model-simulation x i can be compared to experimentally observed data, x o . Using such simulators, we can quantitatively reason about the contribution of mechanisms behind experimental

measurements. But to do so, a key objective is often to find all those parameter values that can produce simulations consistent with observed data.

One fruitful approach towards this goal is simulation-based inference (SBI) [1], which makes it possible to perform Bayesian inference on such models by interpreting simulator outputs as samples from an implicit likelihood [2], x ∼ p ( x | θ ) . Standard Bayesian inference targets the parameter posterior distribution given observed data, i.e., p ( θ | x o ) = p ( x o | θ ) p ( θ ) p ( x o ) , where p ( θ ) captures prior knowledge and constraints over model parameters, and the likelihood function p ( x o | θ ) is evaluated as a function of θ for a fixed x o . SBI methods can differ in whether they aim to approximate the likelihood [3, 4, 5, 6] or the posterior directly [7, 8, 9], and can be amortized , i.e., do not require new simulations and retraining for new data [10, 11]. In the end, each method provides samples from the posterior, which are all, in theory, capable of producing simulations that are identical to the observation we condition on. Furthermore, by definition, the posterior probability of drawing a sample scales as the product of its prior probability and, critically, the likelihood that this sample can produce a simulation that is exactly equal to the observation.

However, targeting the exact posterior may be overly restrictive. In many inference scenarios, modelers are primarily interested in obtaining a diverse collection of parameter values that can explain the observed data. This desire is also reflected in the common usage of posterior predictive checks, where seeing predictive simulations that resemble the data closely (in some specific aspects) is used to gauge the success of the inference process. In particular, it is often clear that the scientific model is only a coarse approximation to the data-generating process, and in some cases even cannot generate data-matching simulations, i.e., is misspecified [12]. For example, in the life-sciences, it is not uncommon to use idealized, theoretically motivated models with few parameters, and it would be unrealistic to expect that they precisely capture observations of highly complex biological systems. In such cases, or in cases where the model is fully deterministic, it is nonsensical to use the probability of exactly reproducing the data. In contrast, it would still be useful to find parameter values that produce simulations which are 'close enough', or as close as possible to the data. Therefore, instead of sampling parameters according to how often they produce simulations that match the data exactly , many use cases call for sampling parameters according to how closely their corresponding simulations reproduce the observed data.

Generalized Bayesian Inference (GBI) [13] offers a principled way to do so by replacing the (log) likelihood function with a cost function that simply scores a parameter given an observation, such as the expected distance between x o and all possible simulations x produced by θ i (Fig. 1). Several recent works have leveraged this framework to perform inference for models with implicit or intractable likelihoods, especially to tackle model misspecification: Matsubara et al. [14] use a Stein Discrepancy as the cost function (which requires the evaluation of an unnormalized likelihood and multiple i.i.d. data samples), and CheriefAbdellatif et al. and Dellaporta et al. [15, 16] use simulator samples to estimate maximum mean discrepancy and directly optimize over this cost function via stochastic gradient descent (which

requires a differentiable simulator). More broadly, cost functions such as scoring rule estimators have been used to generalize approximate Bayesian computation (ABC) [17] and synthetic likelihood approaches [3, 18], where the Monte Carlo estimate requires (multiple) simulations from p ( x | θ ) .

Thus, existing GBI approaches for SBI either require many simulations to be run during MCMC sampling of the posterior (similar to classical ABC methods), or are limited to differentiable simulators. Moreover, performing inference for new observations requires re-running simulations, rendering such methods simulation-inefficient and expensive at inference-time, and ultimately impractical for scientific simulators with even moderate computational burden.

We here propose to perform GBI for scientific simulators with amortized cost estimation (ACE) , which inherits the flexibility of GBI but amortizes the overhead of simulations by training a neural

network to predict the cost function for any parameter-observation pair. We first outline the GBI formalism in Sec. 2, then introduce ACE in Sec. 3. In Sec. 4, we show that ACE provides GBI posterior predictive simulations that are close to synthetic observations for a variety of benchmark tasks, especially when the simulator is misspecified. We showcase its real-world applicability in Sec. 5: using experimental data from the Allen Cell Types Database, ACE successfully infers parameters of the Hodgkin-Huxley single-neuron simulator with superior predictive performance and an order of magnitude higher simulation efficiency compared to neural posterior estimation [10]. Finally, we discuss benefits and limitations of GBI and ACE, and related work (Sec. 6).

2 Background

To construct the GBI posterior, the likelihood, p ( x o | θ ) , is replaced by a 'generalized likelihood function', L ( θ ; x o ) , which does not need to be a probabilistic model of the data-generating process, as long as it can be evaluated for any pair of θ and x o . Following the convention in Bissiri et al. [13], we define L ( θ ; x o ) ≡ e -βℓ ( θ ; x o ) , where ℓ ( θ ; x o ) is a cost function that encodes the quality of θ relative to an observation x o , and β is a scalar inverse temperature hyperparameter that controls how much the posterior weighs the cost relative to the prior. Thus, the GBI posterior can be written as

̸

As noted previously [13], if we define ℓ ( θ ; x o ) ≡ -log p ( x o | θ ) (i.e., self-information) and β = 1 , then we recover the standard Bayesian posterior, and "tempered" or "power" posterior for β = 1 [19, 20]. The advantage of GBI is that, instead of adhering strictly to the (implicit) likelihood, the user is allowed to choose arbitrary cost functions to rate the goodness of θ relative to an observation x o , which is particularly useful when the simulator is misspecified. Previous works have referred to ℓ ( θ ; x o ) as risk function [21], loss function [13], or (proper) scoring rules when they satisfy certain properties [22, 18] (further discussed in Section 6.1). Here we adopt 'cost' to avoid overloading the terms 'loss' and 'score' in the deep learning context.

3 Amortized Cost Estimation for GBI

3.1 Estimating cost function with neural networks

In this work, we consider cost functions that can be written as an expectation over the likelihood:

Many popular cost functions and scoring rules can be written in this form, including the average mean-squared error (MSE) [13], the maximum mean discrepancy (MMD 2 ) [23], and the energy score (ES) [22] (details in Appendix A2). While ℓ ( θ ; x o ) can be estimated via Monte Carlo sampling, doing so in an SBI setting is simulation-inefficient and time-intensive, since the inference procedure must be repeated for every observation x o , and simulations must be run in real-time during MCMC sampling of the posterior. Furthermore, this does not take advantage of the structure in parameter-space or data-space around neighboring points that have been simulated.

We propose to overcome these limitations by training a regression neural network (NN) to learn ℓ ( θ ; x o ) . Our first insight is that cost functions of the form of Eq. 2 can be estimated from a dataset consisting of pairs of parameters and outputs-in particular, from a single simulation run per θ for MSE, and finitely many simulation runs for MMD 2 and ES. We leverage the well-known property that NN regression converges to the conditional expectation of the data labels given the data: If we compute the distances d ( x , x o ) between a single observation x o and every x in our dataset, then a neural network f ϕ ( · ) trained to predict the distances given parameters θ will denoise the noisy distance labels d ( x , x o ) and converge onto the desired cost f ϕ ( θ ) → ℓ ( θ ; x o ) = E p ( x | θ ) [ d ( x , x o )] , approximating the cost of any θ relative to x o (see Appendix A3.1 for formal statement and proof).

3.2 Amortizing over observations

As outlined above, a regression NN will converge onto the cost function ℓ ( θ ; x o ) for a particular observation x o . However, naively applying this procedure would require retraining of the network

for any new observation x o , which prevents application of this method in time-critical or highthroughput scenarios. We therefore propose to amortize cost estimation over a target distribution p ( x t ) . Specifically, a NN which receives as input a parameter θ and an independently sampled target datapoint x t will converge to ℓ ( θ ; x t ) for all x t on the support of the target distribution (Fig. 2a,b), enabling estimation of the cost function for any pair of ( θ , x t ) (see Appendix A3.2 for formal statement and proof).

Naturally, we use the already simulated x ∼ p ( x ) as target data during training, and therefore do not require further simulations. In order to have good accuracy on potentially misspecified observations, however, such datapoints should be within the support of the target distribution. Thus, in practice, we augment this target dataset with noisy simulations to broaden the support of p ( x t ) . Furthermore, if the set of observations (i.e., real data) is known upfront, they can also be appended to the target dataset during training. Lastly, to keep training efficient and avoid quadratic scaling in the number of simulations, we randomly subsample a small number of x t per θ in each training epoch (2 in our experiments), thus ensuring linear scaling as a function of simulation budget. Fig. 2a,b summarizes dataset construction and network training for ACE (details in Appendix A4.1).

3.3 Sampling from the generalized posterior

Given a trained cost estimation network f ϕ ( · , · ) , an observed datapoint x o , and a user-selected inverse temperature β , the generalized posterior probability (Eq. 1) can be computed up to proportionality for any θ : p ( θ | x o ) ∝ exp( -β · f ϕ ( θ , x o )) p ( θ ) and, thus, this term can be sampled with MCMC (Fig. 2c). The entire algorithm is summarized in Algorithm 1.

Algorithm 1: Generalized Bayesian Inference with Amortized Cost Estimation (ACE)

Inputs: prior p ( θ ) , simulator with implicit likelihood p ( x | θ ) , number of simulations N , feedforward NN f ϕ with parameters ϕ , NN learning rate η , distance function d ( · , · ) , noise level σ , number of noise-augmented samples S , inverse temperature β , number of target datapoints per θ N target , K observations x (1 ,...,K ) o .

Outputs: M samples from generalized posteriors given K observations.

Generate dataset:

sample prior and simulate: θ , x ←{ θ i ∼ p ( θ ) , x i ∼ p ( x | θ i ) } i :1 ...N add noise and concatenate: x target = [ x , x 1 ...S + ϵ , x o ] , ϵ ∼ N ( 0 , σ 2 I )

Training:

while not converged do

Sampling:

for k ∈ [1 , ..., K ] do

Draw M samples, with MCMC, from: exp( -β · f ϕ ( θ , x ( k ) o )) p ( θ )

3.4 Considerations for choosing the value of β

We note that the choice of value for β is an important decision, though this is an issue not only for ACE but GBI methods in general [13]. A good 'baseline" value for β is such that the average distance across a subset of the training data is scaled to be in the same range as the (log) prior probability, both of which can be computed on prior simulations. From there, increasing β sacrifices sample diversity for predictive distance, and as β approaches infinity, posterior samples converge onto the minimizer of the cost function. In practice, we recommend experimenting with a range of (roughly) log-spaced values since, as we show below, predictive sample quality tend to improve with increasing β .

a

trainingdataconstruction

C

inference

well-specified

misspecified

X

X

'noised

X

real

example1D

X

simulator

(0,x

ACE network

(x'x)p

θ

exp( -β-&(0,x))

b

networktraining

θ

d(x,x)

well-specified x。

MCMC

X

misspecified x。

θ

4 Benchmark experiments

4.1 Experiment setup

Tasks We first evaluated ACE on four benchmark tasks (modified from Lueckmann et al. [24]) with a variety of parameter- and data-dimensionality, as well as choice of distance measure: (1) Uniform 1D : 1D θ and x , the simulator implements an even polynomial with uniform noise likelihood, uniform prior (Fig. 2c); (2) 2 Moons : 2D θ and x , simulator produces a half-circle with constant mean radius and radially uniform noise of constant width, translated as a function of θ , uniform prior; (3) Linear Gaussian : 10D θ and x , Gaussian model with mean θ and fixed covariance, Gaussian prior; (4) Gaussian Mixture : 2D θ and x , simulator returns five i.i.d. samples from a mixture of two Gaussians, both with mean θ , and fixed covariances, one with broader covariance than the other, uniform prior.

For the first three tasks, we use the mean-squared error between simulation and observation as the distance function. For the Gaussian Mixture task, we use maximum mean discrepancy (MMD 2 ) to measure the statistical distance between two sets of five i.i.d. samples. Importantly, for each of the four tasks, we can compute the integral in Eq. 2 either analytically or accurately capture it with quadrature over x . Hence, we obtain the true cost ℓ ( θ ; x o ) and subsequently the 'ground-truth' GBI posterior (with Eq. 1), and use that to draw, for each value of β and x o , 5000 samples (GT-GBI, black in Fig. 3). See Appendix A4.2 for more detailed descriptions of tasks and distance functions.

Training data For each task, we simulate 10,000 pairs of ( θ , x ) and construct the target dataset as in Fig. 2a, with 100 additional noise-augmented targets and 20 synthetic observations-10 wellspecified and 10 misspecified-for a total of 10120 x t data points. Well-specified observations are additional prior predictive samples, while misspecified observations are created by moving prior predictive samples outside the boundaries defined by the minimum and maximum of 100,000 prior predictive simulations (e.g., by successively adding Gaussian noise). See Appendix A4.3 for details.

Test data To evaluate inference performance, we use ACE to sample approximate GBI posteriors conditioned on 40 different synthetic observations, 20 of which were included in the target dataset x t , and 10 additional well-specified and misspecified observations which were not included in the target dataset. We emphasize that including observations in the target data is not a case of test data leakage, but represents a real use case where some experimental data which one wants to perform inference on are already available, while the network should also be amortized for unseen observations measured after training. Nevertheless, we report in Fig. 3 results for 'unseen' observations, i.e., not in the target dataset. Results are almost identical for those that were in the target dataset (Appendix A1). We drew 5000 posterior samples per observation, for 3 different β values for each task.

Metrics We are primarily interested in two aspects of performance: approximate posterior predictive distance and cost estimation accuracy. First, as motivated above, we want to find parameter

posteriorpredictive distance

posteriorpredictive distance

cost estimation

GBIposterioraccuracy

well-specifiedxo

misspecifiedxo

accuracy

ACEvS.GT

Uniform 1D

predicted

MSE

0.1

0.5

C2ST

0.0

0.0

0.5

N/A

4

20

100

N/A

4

20

100

0.0

0.5

4

20

100

0.10

F

1

2Moons

MSE

1.0

predicted

0.2

C2ST

0.05

0.5

0.00

0.0

0.0

0.5

T

N/A

10

1001000

N/A

10

100

1000

0.0

0.2

10

100

1000

10

F

1

Linear

predicted

MSE

5

C2ST

20

5

0.5

0

0

T

0

T

T

1

N/A

1

10

100

N/A

1

10

100

0

5

10

1

10

100

0.10

predicted

0.50

MMD

C2ST

0.05

0.2

0.25

0.00

0.0

0.00

0.5

T

T

N/A

10

25

75

N/A

10

25

75

0.0

0.2

0.4

10

25

75

beta

beta

true cost

beta

-

GT-GBI

ABC

NLE

-o well-specified

ACE-GBI

NPE

NLE (tempered)

-= misspecified

configurations which produce simulations that are as close as possible to the observation, as measured by the task-specific distance function. Therefore, we simulate using each of the 5000 ACE GBI posterior samples, and compute the average distance between predictive simulations and the observation. Mean and standard deviation are shown for well-specified and misspecified observations separately below (Fig. 3, 1st and 2nd columns). Second, we want to confirm that ACE accurately approximates ℓ ( θ ; x o ) , which is a prerequisite for correctly inferring the GBI posterior. Therefore, we compare the ACE-predicted and true cost across 5000 samples from each GBI posterior, as well as the classifier 2-sample test (C2ST, [25, 24]) score between the ACE approximate and ground-truth GBI posterior (Fig. 3, 3rd and 4th columns). Note that cost estimation accuracy can be evaluated for parameter values sampled in any way (e.g., from the prior), but here we evaluate accuracy as samples become more concentrated around good parameter values, i.e., from GBI posteriors with increasing β . We expect that these tasks become increasingly challenging with higher values of β , since these settings require the cost estimation network to be highly accurate in tiny regions of parameter-space.

Other algorithms As a comparison against SBI methods that target the standard Bayesian posterior (but which nevertheless might produce good predictive samples), we also tested approximate Bayesian computation (ABC), neural posterior estimation (NPE, [7]), and neural likelihood estimation (NLE, [4, 26]) on the same tasks. NPE and NLE were trained on the same 10,000 simulations, and 5000 approximate posterior samples were obtained for each x o . We used the amortized (single-round) variants of both as a fair comparison against ACE. Additionally, to rule out the possibility that ACE is simply benefiting from the additional inverse temperature hyperparameter, we implemented a "tempered" version of NLE by scaling the NLE-approximated log-likelihood term (by the same β s) during MCMC sampling. For ABC, we used the 10,000 training samples as a reference set, from which 50 were drawn as posterior samples with probability scaling inversely with the distance between their corresponding simulation and the observation, i.e., ABC with acceptance kernel [27].

4.2 Benchmark results

Overall, we see that for well-specified x o (i.e., observations for which the simulator is well-specified), ACE obtains GBI posterior samples that achieve low average posterior predictive simulation distance across all four tasks, especially at high values of β (Fig. 3, 1st column). In comparison, ABC is worse for the Linear Gaussian task (which has a higher parameter dimensionality than all other tasks), whereas NPE, NLE, and tempered NLE achieve similarly low posterior predictive distances.

On misspecified observations, across all tasks and simulation-budgets (with the exception of Gaussian mixture on 10k simulations) we see that ACE achieves lower or equally low average posterior predictive simulation distance as neural SBI methods, even at moderate values of β (Fig. 3, 2nd column, Figs. A4, A5). This is in line with our intuition that ACE returns a valid and accurate cost even if the simulator is incapable of producing data anywhere near the observation, while Bayesian likelihood and posterior probabilities estimated by NLE and NPE are in these cases nonsensical [28, 29, 30, 31]. Furthermore, simply concentrating the approximate Bayesian posterior via tempering does not lead to more competitive performance than ACE on such misspecified observations, and is sometimes even detrimental (e.g., tempered NLE at high β on Uniform 1D and Linear Gaussian tasks). Therefore, we see that ACE can perform valid inference for a broad range of simulators, obtaining a distribution of posterior samples with predictive simulations close to observations, and is automatically robust against model-misspecification as a result of directly targeting the cost function.

For both well-specified and misspecified observations, ACE-GBI samples achieve posterior predictive distance very close to ground-truth (GT)-GBI samples, at all values of β (Fig. 3, 1st and 2nd column), suggesting that ACE is able to accurately predict the expected distance. Indeed, especially for low to moderate values of β , the ACE-predicted cost closely matches the true cost (Fig. 3, 3rd column, light blue for well-specified x o , Fig. A2 for misspecified). For higher values of β , ACE-predicted cost is still similar to true cost, although the error is, as expected, larger for very large β (Fig. 3, 3rd column, dark blue).

As a result, highly concentrated generalized posteriors are estimated with larger (relative) discrepancies, which is reflected in the classifier 2-sample score (C2ST) between ACE and GT GBI posteriors (Fig. 3, 4th column): ACE posterior samples are indistinguishable from GT samples at low β , even for the 10D Linear Gaussian task, but becomes less accurate with increasing β . Nevertheless, predictive simulation distance dramatically increases with β even when ACE is less accurate, suggesting that sampling to explicitly minimize a cost function which targets parameters with data-similar simulations is a productive goal. Relative performance results across algorithms are qualitatively similar when using a training simulation budget of 200 (Fig. A4) and 1000 (Fig. A5), but ABC required a sufficiently high simulation budget and performed poorly for 1000 training simulations or less.

5 Hodgkin-Huxley inference from Allen Cell Types Database recordings

Finally, we applied ACE to a commonly used scientific simulator and real data: we used a singlecompartment Hodgkin-Huxley (HH) simulator from neuroscience and aimed to infer eight parameters of the simulator given electrophysiological recordings from the Allen Cell Types Database [32, 33, 34]. While this simulator can generate a broad range of voltage traces, it is still a crude approximation to real neurons: it models only a subset of ion channels, it ignores the spatial structure of neurons, and it ignores many intracellular mechanisms [35]. It has been demonstrated that parameters of the HH-model given synthetic recordings can be efficiently estimated with standard NPE [10], but estimating parameters given experimental recordings has been challenging [36] and has required ad-hoc changes to the inference procedure (e.g., Bernaerts et al. [37] added noise to the summary statistics, and Gonçalves et al. [10] used a custom multi-round scheme with a particular choice of density estimator). We will demonstrate that ACE can successfully perform simulation-amortized inference given experimental recordings from the Allen Cell Types Database (Fig. 4a).

We trained NPE and ACE given 100K prior-sampled simulations (details in Appendix A4.4). After training, ACE accurately predicts the true cost of parameters given experimental observations (Fig. 4b). We then used slice sampling to draw samples from the GBI posterior for three different values of β = { 25 , 50 , 100 } and for ten observations from the Allen Cell Types Database. Interestingly, the marginal distributions between NPE and ACE posteriors are very similar, especially for rather low values of β (Fig. 4c, cornerplot in Appendix Fig. A7). The quality of posterior predictive samples, however, strongly differs between NPE and ACE: across the ten observations from the Allen

e

observations

b

2.5

C

predicted

d

ACE predictives (β=100)

0.5

9Na

80.050

3000

0.0

0.0

2.5

true cost

0

9K

15

40

-VT

90

e

0.4

predictive distance

NPEpredictives

0.3

0.2

0.09leak

0.6 0.00 noise 0.15

0.1

0.0

NPE ACE ACE ACE

2550100

0.0

9M

0.6 35 -Eleak 100

Cell Types database, only 35.6% of NPE posterior predictives produced more than five spikes (all observations have at least 12 spikes), whereas the ACE posterior predictives closely match the data, even for low values of β (Fig. 4d, samples for all observations and all β values in Figs. A8,A9,A10 and for NPE in Fig. A11. 66% ( β = 25 ), 87% ( β = 50 ), 96% ( β = 100 ) of samples have more than five spikes). Indeed, across all ten observations, the average posterior predictive distance of ACE was significantly smaller than that of NPE, and for large values of β the distance is even less than half (Fig. 4e). Finally, for rejection-ABC, only the top 35 samples (out of the full training budget of 100K simulations) had a distance that is less than the average posterior predictive distance achieved by ACE.

To investigate these differences between NPE and ACE, we also evaluated NPE posterior predictive performance on synthetic data (prior predictives) and found that it had an average predictive distance of 0.189, which roughly matches the performance of ACE on the experimental observations (0.174 for β =50). This suggest that, in line with previous results [37], NPE indeed struggles with experimental observations, for which the simulator is inevitably imperfect. We then trained NPE with 10 times more simulations (1M in total). With this increased simulation budget, NPE performed significantly better than with 100K simulations, but still produced poorer predictive samples than ACE trained with 100K simulations (for β = { 50 , 100 } ), although the marginals were similar between NPE (1M) and ACE (100K) (Fig. A6, samples for all observations in Appendix Fig. A12).

Overall, these results demonstrate that ACE can successfully be applied to real-world simulators on which vanilla NPE fails. On the Hodgkin-Huxley simulator, ACE generates samples with improved predictive accuracy despite an order of magnitude fewer simulations and despite the marginal distributions being similar to those of NPE.

6 Discussion

We presented ACE, a method to perform distance-aware inference for scientific simulators within the Generalized Bayesian Inference (GBI) framework. Contrary to 'standard' simulation-based inference (SBI), our method does not target the Bayesian posterior, but replaces the likelihood function with a cost function. For real-world simulators, doing so can provide practical advantages over standard Bayesian inference:

First, the likelihood function quantifies the probability that a parameter generates data which exactly matches the data. However, in cases where the model is a rather crude approximation to the real system being studied, scientists might well want to include parameters that can generate data that is sufficiently close (but not necessarily identical) in subsequent analyses. Our method makes this possible, and is advantageous over other GBI-based methods since it is amortized over observations and the inverse temperature β . Second, many simulators are formulated as noise-free models, and it can be hard to define appropriate stochastic extensions (e.g., [38]). In these cases, the likelihood function is ill-defined and, in practice, this setting would require 'standard' SBI methods, whose density estimators are generally built to model continuous distributions, to model discrete jumps in the posterior density. In contrast, our method can systematically and easily deal with noise-free simulators, and in such situations more closely resembles parameter-fitting algorithms. Lastly, standard Bayesian inference is challenging when the model is misspecified, and the performance of neural network-based SBI methods can suffer drastically in this scenario [30].

6.1 Related work

GBIfor Approximate Bayesian Computation Several studies have proposed methods that perform GBI on simulators with either an implicit (i.e., simulation-based) likelihood or an unnormalized likelihood. Wilkinson et al. [39] argued that rejection-ABC performs exact inference for a modified model (namely, one that appends an additive uniform error) instead of approximate inference for the original model. Furthermore, ABC with arbitrary probabilistic acceptance kernels can also be interpreted as having different error models, and Schmon et al. [17] integrate this view to introduce generalized posteriors for ABC, allowing the user to replace the hard-threshold kernel (i.e., ϵ -ball of acceptance) with an arbitrary loss function that measures the discrepancy between x and x o for MCMC-sampling of the approximate generalized posterior.

Other recent GBI methods require a differentiable simulator [16, 15] or build tractable cost functions that can be sampled with MCMC [14, 18], but this still requires running simulations at inference time (i.e., during MCMC) and does not amortize the cost of simulations and does not reuse already simulated datapoints.

Finally, Bayesian Optimization for Likelihood-free Inference (BOLFI, [40]) and error-guided LFIMCMC [41] are not cast as generalized Bayesian inference approaches, but are related to ACE. Similarly as in ACE, they train models (for BOLFI, a Gaussian process and, for error-guided LFIMCMC, a classifier) to estimate the discrepancy between observation and simulation. In BOLFI, the estimator is then used to iteratively select new locations at which to simulate. However, contrary to ACE, neither of these two methods amortizes the cost of simulations over observations.

Misspecification-aware SBI Several other methods have been proposed to overcome the problem of misspecification in SBI: For example, Bernaerts et al. [37] add noise to the summary statistics in the training data, Ward et al. [42] use MCMC to make the misspecified data well-specified, and Kelly et al. [43] introduce auxiliary variables to shift the (misspecified) observation towards being well-specified. All of these methods, however, maintain that the inference result should be an 'as close as possible' version of the posterior distribution. Contrary to that, our method does not aim to obtain the Bayesian posterior distribution (which, for misspecified models, can often be nonsensical or even undefined if the evidence is zero), but is specifically targeted towards parameter regions that are a specified distance from the observation. More broadly, recent advances in uncertainty quantification in deep neural networks, where standard mean-predicting regression networks are supplemented with a uncertainty- or variance-predicting network [44, 45], may serve to further connect loss-minimizing deep learning with (misspecification-aware) SBI.

6.2 Limitations

While our method amortizes the cost of simulations and of training, it still requires another method to sample from the posterior distribution. We used multi-chain slice-sampling [46] for efficiency, but any other MCMC algorithm, as well as variational inference, could also be employed [47, 48]. While sampling incurs an additional cost, this cost is generally small in comparison to potentially expensive simulations.

In addition, our method can perform inference for distance functions which can be written as expectations over the likelihood. As we demonstrated, this applies to many popular and widely used distances. Our method can, however, not be applied to arbitrary distance functions (e.g., the minimum distance between all simulator samples and the observation). While the distances we investigated here are certainly useful to practioners, they do not necessarily fulfill the criterion of being 'proper' scoring rules [22, 18]. Furthermore, we note that the cost functions considered here by default give rise to unnormalized generalized likelihoods. Therefore, depending on whether the user aims to approximate the generalized posterior given the normalized or unnormalized likelihood, different MCMCschemes should be used in conjunction with ACE (standard MCMC vs. doubly-intractable MCMC, e.g., the Exchange algorithm [49]).

Compared to 'standard' SBI, GBI introduces an additional hyperparameter to the inference procedure, the inverse temperature β . This hyperparameter has to be set by the user and its choice strongly affects inference behaviour: low values of β will include regions of parameter-space whose data do not necessarily match the observation closely, whereas high values of β constrain the parameters to only the best-fitting parameter values. While we provide heuristics for selecting β , we acknowledge the inconvenience of an additional hyperparameter. However, our method is amortized over β , which makes exploration of different β values possible, and which could simplify automated methods for setting β , similar to works where β is taken as the exponent of the likelihood function [50].

Finally, as with any method leveraging deep neural networks, including neural density estimator-based SBI methods (such as NPE), sensitivity to the number of training samples and the dimensionality of the task should always be considered. As we demonstrate above, increasing simulation budget improves the performance of any algorithm, and a reasonable number of training simulations yielded improved performance on a real-world neuroscience application, while the amortization property shifts the cost of simulations up front. In addition, we consider tasks up to 10 dimensions here, as most existing SBI methods have been benchmarked and shown to perform adequately on such tasks [24], though it remains to be seen how ACE can extend to higher dimensional parameter-space and data-space and whether embedding networks will be similarly helpful.

7 Conclusion

We presented a method that performs generalized Bayesian inference with amortized cost estimation. Our method produces good predictive samples on several benchmark tasks, especially in the case of misspecified observations, and we showed that it allows amortized parameter estimation of Hodgkin-Huxley models given experimental recordings from the Allen Cell Types Database.

References

Kyle Cranmer, Johann Brehmer, and Gilles Louppe. The frontier of simulation-based inference. Proceedings of the National Academy of Sciences , 117(48):30055-30062, 2020.

Peter J Diggle and Richard J Gratton. Monte carlo methods of inference for implicit statistical models. Journal of the Royal Statistical Society. Series B, Statistical methodology , 46(2):193227, 1984.

Simon N Wood. Statistical inference for noisy nonlinear ecological dynamic systems. Nature , 466(7310):1102-1104, 2010.

George Papamakarios, David Sterratt, and Iain Murray. Sequential neural likelihood: Fast likelihood-free inference with autoregressive flows. In The 22nd International Conference on Artificial Intelligence and Statistics , pages 837-848. PMLR, 2019.

Joeri Hermans, Volodimir Begy, and Gilles Louppe. Likelihood-free mcmc with amortized approximate ratio estimators. In International Conference on Machine Learning , pages 42394248. PMLR, 2020.

Owen Thomas, Ritabrata Dutta, Jukka Corander, Samuel Kaski, and Michael U Gutmann. Likelihood-free inference by ratio estimation. Bayesian Analysis , 17(1):1-31, 2022.

George Papamakarios and Iain Murray. Fast ε -free inference of simulation models with bayesian conditional density estimation. Advances in neural information processing systems , 29, 2016.

David Greenberg, Marcel Nonnenmacher, and Jakob Macke. Automatic posterior transformation for likelihood-free inference. In International Conference on Machine Learning , pages 24042414. PMLR, 2019.

Katalin Csilléry, Michael GB Blum, Oscar E Gaggiotti, and Olivier François. Approximate bayesian computation (abc) in practice. Trends in ecology & evolution , 25(7):410-418, 2010.

Pedro J Gonçalves, Jan-Matthis Lueckmann, Michael Deistler, Marcel Nonnenmacher, Kaan Öcal, Giacomo Bassetto, Chaitanya Chintaluri, William F Podlaski, Sara A Haddad, Tim P Vogels, et al. Training deep neural density estimators to identify mechanistic models of neural dynamics. Elife , 9:e56261, 2020.

Stefan T Radev, Ulf K Mertens, Andreas Voss, Lynton Ardizzone, and Ullrich Köthe. Bayesflow: Learning complex stochastic models with invertible neural networks. IEEE transactions on neural networks and learning systems , 2020.

Stephen G Walker. Bayesian inference with misspecified models. Journal of statistical planning and inference , 143(10):1621-1633, October 2013.

Pier Giovanni Bissiri, Chris C Holmes, and Stephen G Walker. A general framework for updating belief distributions. Journal of the royal statistical society. series b, statistical methodology , 78(5):1103, 2016.

Takuo Matsubara, Jeremias Knoblauch, François-Xavier Briol, Chris Oates, et al. Robust generalised bayesian inference for intractable likelihoods. arXiv preprint arXiv:2104.07359 , 2021.

Badr-Eddine Cherief-Abdellatif and Pierre Alquier. MMD-Bayes: Robust bayesian estimation via maximum mean discrepancy. In Cheng Zhang, Francisco Ruiz, Thang Bui, Adji Bousso Dieng, and Dawen Liang, editors, Proceedings of The 2nd Symposium on Advances in Approximate Bayesian Inference , volume 118 of Proceedings of Machine Learning Research , pages 1-21. PMLR, December 2020.

Charita Dellaporta, Jeremias Knoblauch, Theodoros Damoulas, and François-Xavier Briol. Robust bayesian inference for simulator-based models via the mmd posterior bootstrap. In International Conference on Artificial Intelligence and Statistics , pages 943-970. PMLR, 2022.

Sebastian M Schmon, Patrick W Cannon, and Jeremias Knoblauch. Generalized posteriors in approximate bayesian computation. arXiv preprint arXiv:2011.08644 , 2020.

Lorenzo Pacchiardi and Ritabrata Dutta. Generalized bayesian likelihood-free inference using scoring rules estimators. arXiv preprint arXiv:2104.03889 , 2021.

Enzo Marinari and Giorgio Parisi. Simulated tempering: A new monte carlo scheme. May 1992.

N Friel and A N Pettitt. Marginal likelihood estimation via power posteriors. Journal of the Royal Statistical Society. Series B, Statistical methodology , 70(3):589-607, 2008.

Wenxin Jiang and Martin A Tanner. Gibbs posterior for variable selection in High-Dimensional classification and data mining. Annals of statistics , 36(5):2207-2231, 2008.

Tilmann Gneiting and Adrian E Raftery. Strictly proper scoring rules, prediction, and estimation. Journal of the American statistical Association , 102(477):359-378, 2007.

Arthur Gretton, Karsten M Borgwardt, Malte J Rasch, Bernhard Schölkopf, and Alexander Smola. A kernel two-sample test. The Journal of Machine Learning Research , 13(1):723-773, 2012.

Jan-Matthis Lueckmann, Jan Boelts, David Greenberg, Pedro Goncalves, and Jakob Macke. Benchmarking simulation-based inference. In International Conference on Artificial Intelligence and Statistics , pages 343-351. PMLR, 2021.

David Lopez-Paz and Maxime Oquab. Revisiting classifier two-sample tests. arXiv preprint arXiv:1610.06545 , 2016.

Jan-Matthis Lueckmann, Giacomo Bassetto, Theofanis Karaletsos, and Jakob H Macke. Likelihood-free inference with emulator networks. In Symposium on Advances in Approximate Bayesian Inference , pages 32-53. PMLR, 2019.

Scott A Sisson and Yanan Fan. Likelihood-free markov chain monte carlo. arXiv preprint arXiv:1001.2058 , 2010.

Peter Grünwald and John Langford. Suboptimal behavior of bayes and mdl in classification under misspecification. Machine Learning , 66:119-149, 2007.

Peter Grünwald and Thijs Van Ommen. Inconsistency of bayesian inference for misspecified linear models, and a proposal for repairing it. Bayesian Analysis , 2017.

Patrick Cannon, Daniel Ward, and Sebastian M Schmon. Investigating the impact of model misspecification in neural simulation-based inference. arXiv preprint arXiv:2209.01845 , 2022.

Daniel Ward, Patrick Cannon, Mark Beaumont, Matteo Fasiolo, and Sebastian Schmon. Robust neural posterior estimation and statistical model criticism. Advances in Neural Information Processing Systems , 35:33845-33859, 2022.

Allen Institute for Brain Science. Allen cell types database. http://celltypes.brain-map.org/ , 2016.

Corinne Teeter, Ramakrishnan Iyer, Vilas Menon, Nathan Gouwens, David Feng, Jim Berg, Aaron Szafer, Nicholas Cain, Hongkui Zeng, Michael Hawrylycz, Christof Koch, and Stefan Mihalas. Generalized leaky integrate-and-fire models classify multiple neuron types. Nature communications , 9(1):709, February 2018.

Martin Pospischil, Maria Toledo-Rodriguez, Cyril Monier, Zuzanna Piwkowska, Thierry Bal, Yves Frégnac, Henry Markram, and Alain Destexhe. Minimal hodgkin-huxley type models for different classes of cortical and thalamic neurons. Biological cybernetics , 99:427-441, 2008.

Romain Brette. What is the most realistic single-compartment model of spike initiation? PLoS computational biology , 11(4):e1004114, 2015.

Nicholas Tolley, Pedro LC Rodrigues, Alexandre Gramfort, and Stephanie R Jones. Methods and considerations for estimating parameters in biophysically detailed neural models with simulation based inference. bioRxiv , pages 2023-04, 2023.

Yves Bernaerts, Michael Deistler, Pedro J Goncalves, Jonas Beck, Marcel Stimberg, Federico Scala, Andreas S Tolias, Jakob H Macke, Dmitry Kobak, and Philipp Berens. Combined statistical-mechanistic modeling links ion channel genes to physiology of cortical neuron types. bioRxiv , pages 2023-03, 2023.

Joshua H Goldwyn and Eric Shea-Brown. The what and where of adding channel noise to the hodgkin-huxley equations. PLoS computational biology , 7(11):e1002247, 2011.

Richard David Wilkinson. Approximate bayesian computation (ABC) gives exact results under the assumption of model error. Statistical applications in genetics and molecular biology , 12(2):129-141, May 2013.

Michael U Gutmann and Jukka Corander. Bayesian optimization for likelihood-free inference of simulator-based statistical models. Journal of Machine Learning Research , 2016.

Volodimir Begy and Erich Schikuta. Error-guided likelihood-free MCMC. arXiv , October 2020.

Daniel Ward, Patrick Cannon, Mark Beaumont, Matteo Fasiolo, and Sebastian M Schmon. Robust neural posterior estimation and statistical model criticism. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems , 2022.

Ryan P Kelly, David J Nott, David T Frazier, David J Warne, and Chris Drovandi. Misspecification-robust sequential neural likelihood. arXiv preprint arXiv:2301.13368 , 2023.

Ian Osband, Zheng Wen, Seyed Mohammad Asghari, Vikranth Dwaracherla, Morteza Ibrahimi, Xiuyuan Lu, and Benjamin Van Roy. Epistemic neural networks. July 2021.

Salem Lahlou, Moksh Jain, Hadi Nekoei, Victor I Butoi, Paul Bertin, Jarrid Rector-Brooks, Maksym Korablyov, and Yoshua Bengio. DEUP: Direct epistemic uncertainty prediction. October 2022.

Radford M Neal. Slice sampling. The annals of statistics , 31(3):705-767, 2003.

Samuel Wiqvist, Jes Frellsen, and Umberto Picchini. Sequential neural posterior and likelihood approximation. arXiv preprint arXiv:2102.06522 , 2021.

Manuel Glöckler, Michael Deistler, and Jakob H. Macke. Variational methods for simulationbased inference. In International Conference on Learning Representations , 2022.

Iain Murray, Zoubin Ghahramani, and David MacKay. MCMC for doubly-intractable distributions. June 2012.

Pei-Shien Wu and Ryan Martin. A comparison of learning rate selection methods in generalized bayesian inference. Bayesian Analysis , 18(1):105-132, 2023.

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems , 32, 2019.

Omry Yadan. Hydra - a framework for elegantly configuring complex applications. Github, 2019.

Scott A Sisson, Yanan Fan, and Mark M Tanaka. Sequential monte carlo without likelihoods. Proceedings of the National Academy of Sciences , 104(6):1760-1765, 2007.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770-778, 2016.

Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poczos, Russ R Salakhutdinov, and Alexander J Smola. Deep sets. Advances in neural information processing systems , 30, 2017.

Alvaro Tejero-Cantero, Jan Boelts, Michael Deistler, Jan-Matthis Lueckmann, Conor Durkan, Pedro J. Gonçalves, David S. Greenberg, and Jakob H. Macke. sbi: A toolkit for simulationbased inference. Journal of Open Source Software , 5(52):2505, 2020.

Conor Durkan, Artur Bekasov, Iain Murray, and George Papamakarios. Neural spline flows. Advances in neural information processing systems , 32, 2019.

Michael Deistler, Pedro J. Goncalves, and Jakob H. Macke. Truncated proposals for scalable and hassle-free simulation-based inference. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems , 2022.

Appendix

A1 Software and Data

We used PyTorch for all neural networks [51] and hydra to track all configurations [52]. Code to reproduce results is available at https://github.com/mackelab/neuralgbi .

A2 Common cost functions as expectations of the likelihood

As described in Sec. 3, our framework includes all cost functions which can be written as expectations over the likelihood: ℓ ( θ ; x o ) = E p ( x | θ ) [ d ( x , x o )] . Below, we demonstrate that the average meansquared error (MSE), the maximum mean discrepancy (MMD 2 ) [23], and the energy score [22] fall into this category.

Average Mean-squared error The average mean-squared error between samples from the likelihood p ( x | θ ) and an observation x o is defined as:

which is trivially of the form required for our framework.

Maximum mean discrepancy Throughout this paper, we use the maximum mean discrepancy squared (MMD 2 ) [23]. The MMD 2 between the likelihood p ( x | θ ) and the data distribution p ( x o ) is defined as:

Assume we are given K iid samples as observations x (1 ,...,K ) o . Thus, the MMD 2 becomes:

which can be written as a single expectation over p ( x | θ )

and follows the form required for our framework.

Energy score Following Gneiting et al. [22], the (negative, in order to fit our notation) energy score (ES) is defined as

This can straight-forwardly be written as a single expectation over the likelihood

and, thus, falls within our framework.

A3 Convergence proofs

The proofs closely follow standard proofs that regression converges to the conditional expectation.

A3.1 Proposition 1 and proof

Proposition 1. Let p ( θ , x ) be the joint distribution over parameters and data. Let ℓ ( θ ; x o ) be a cost function that can be written as ℓ ( θ ; x o ) = E p ( x | θ ) [ d ( x , x o )] = ∫ x d ( x , x o ) p ( x | θ ) d x and let f ϕ ( · ) be a function parameterized by ϕ . Then, the loss function L = E p ( θ , x ) [( f ϕ ( θ ) -d ( x , x o )) 2 ] is minimized if and only if, for all θ ∈ supp ( p ( θ )) , f ϕ ( θ ) = E p ( x | θ ) [ d ( x , x o )] .

Proof. We aim to prove that

for every function g ( θ ) . We begin by rearranging terms:

X = E θ, x ∼ p ( θ , x ) [( d ( x , x o ) -E x ′ ∼ p ( x | θ ) [ d ( x ′ , x o )])( E x ′ ∼ p ( x | θ ) [ d ( x ′ , x o )] -g ( θ ))] . By the law of iterated expectations, one can show that X = 0 : X = E θ ′ ∼ p ( θ ) E θ ∼ p ( θ ) E x ∼ p ( x | θ ) [( d ( x , x o ) -E x ′ ∼ p ( x | θ ) [ d ( x ′ , x o )])( E x ′ ∼ p ( x | θ ) [ d ( x ′ , x o )] -g ( θ ))] . The first term in the products reads a difference of the same term, and, thus, is zero.

A3.2 Proposition 2 and proof

Proposition 2. Let p ( θ , x ) be the joint distribution over parameters and data and let p ( x t ) be a distribution of target samples. Let ℓ ( θ ; x o ) be a cost function and f ϕ ( · ) a parameterized function as in proposition 1. Then, the loss function L = E p ( θ , x ) p ( x t ) [( f ϕ ( θ , x t ) -d ( x , x t )) 2 ] is minimized if and only if, for all θ ∈ supp ( p ( θ )) and all x t ∈ supp ( p ( x t )) we have f ϕ ( θ , x t ) = E p ( x | θ ) [ d ( x , x t )] .

Proof. We aim to prove that

for every function g ( θ , x t ) . We begin as in proposition 1:

Below, we prove that, for any x t , the optimal g ( θ , x t ) is the conditional expectation E x ′ ∼ p ( x | θ ) [ d ( x ′ , x t )] :

X = E θ, x ∼ p ( θ , x ) [( d ( x , x o ) -E x ′ ∼ p ( x | θ ) [ d ( x ′ , x o )])( E x ′ ∼ p ( x | θ ) [ d ( x ′ , x o )] -g ( θ , x t ))] , which, as above, is X = 0 (proof is identical to proposition 1). Thus, since E x ′ ∼ p ( x | θ ) [ d ( x ′ , x t )] -g ( θ , x t ) 2 ≥ 0 , we have:

E θ, x ∼ p ( θ , x ) [( d ( x , x t ) -g ( θ , x t )) 2 ] ≥ E θ, x ∼ p ( θ , x ) [( d ( x , x o ) -E x ′ ∼ p ( x | θ ) [ d ( x ′ , x o )]) 2 ] Because this inequality holds for any x t , the average over p ( x t ) will also be minimized if and only if g ( θ , x t )) matches the conditional expectation E x ′ ∼ p ( x | θ ) [ d ( x ′ , x o )] for any x t within the support of p ( x t ) .

A4 Further experimental details

A4.1 Details on training procedure

Dataset construction Wegenerate samples from the prior θ i ∼ p ( θ ) and corresponding simulations x i ∼ p ( x | θ i ) , to obtain parameter-simulation pairs, Θ , X = { θ i , x i } i =1 ...N . Next, a dataset of target data points, X target, is constructed by concatenating in the batch-dimension: 1) X , 2) a random subset of X augmented with Gaussian noise with a specified variance, i.e., x i + ϵ i , ϵ i ∼ N (0 , σ 2 I ) , and 3) any number of real experimental observations the user wishes to include, X real ; in total, N target = N + N noised + N real. Note that X target does not technically need to include any simulated or real data, only 'realistic' targets in the neighborhood of the observed data that one eventually wants to perform inference for. Practically, 1) has already been simulated, and 2) and 3) does not require additional simulations while improving performance through noise augmentation and having access to real data targets. Finally, a pairwise distance matrix D is computed with elements d i,j = d ( x i , x t ) , x i ∈ X, x t ∈ X target . D can either be pre-computed in full, or partially within the training loop.

Network optimization and convergence to GBI loss Given dataset Θ , X, X target , D , a fully connected feed-forward deep neural network with weights ϕ is trained to minimize the mean squared error loss: 1 N sim n targets ∑ N sim i =1 ∑ n target t =1 ( NN ϕ ( θ i , x t ) -d ( x i , x t )) 2 . In other words, for a parameter configuration θ i and a target data point x t , the network is trained to predict the distance d i,t between the corresponding single (stochastic) simulation x i and the target x t . In every training epoch, n target target simulations (usually 1-10, compared to N target > 10000 ) are randomly sub-sampled per θ i , drastically reducing training time and allowing the relevant d i,t to be computed on the fly.

Note that we do not wish to accurately predict the distance d ( x i , x t ) in data-space for individual simulations x i , but rather the loss ℓ ( θ i ; x t ) . Conveniently, with mean squared error as the objective function for network training, for a fixed pair of θ i , x t , the trained network predicts the mean distance, E p ( x | θ i ) [ d ( x , x t )] , precisely the loss function we target in Eq.2 (proof in Appendix).

A4.2 Description of benchmark tasks, distance and cost functions

A4.2.1 Uniform 1D

A one-dimensional task with uniform noise to illustrate how expected distance from observation can be used as a cost function for inference, especially in the case of model misspecification:

A4.2.2 Two Moons

A two-dimensional task with a posterior that exhibits both global (bimodality) and local (crescent shape) structure to illustrate how algorithms deal with multimodality:

Prior

Simulator

Dimensionality

Cost function

d ( x , x o ) : MSE; and p ( x | θ ) computed exactly. To obtain the true cost, Eq. 2 is numerically integrated over a 2D grid of d x with 500 equal bins in both dimensions, where x (1) ∈ [ - 1 . 2 , 0 . 4] , x (2) ∈ [ - 1 . 6 , 1 . 6] .

Posterior

Ground-truth GBI posterior samples are obtained via rejection sampling with the prior as proposal distribution.

References

[24, 8]

A4.2.3 Linear Gaussian

Inference of the mean of a 10-d Gaussian model, in which the covariance is fixed. The (conjugate) prior is Gaussian:

Prior

Simulator

Dimensionality

Cost function

d ( x , x o ) : MSE; and p ( x | θ ) computed exactly. The true cost (Eq. 2) can be computed analytically.

Posterior

Ground-truth GBI posterior samples are obtained following the procedure in Lueckmann et al. [24]: We first ran MCMC to generate 10k samples from the GBI posterior. We then trained a neural spline flow on these 10k samples and, finally, used the trained flow as proposal distribution for rejection sampling.

References

[24, 8]

A4.2.4 Gaussian Mixture

The single-trial version of this task is common in the ABC literature. It consists of inferring the common mean of a mixture of two two-dimensional Gaussian distributions, one with much broader covariance than the other. In this study, we used this task to infer the common mean of the two distributions from five i.i.d. simulations:

Prior

Simulator

Dimensionality

Cost function

d ( x , x o ) : MMD 2 . The true cost (Eq. 2) is obtained by integrating the distance function on a grid for every trial independently and multiplying over the trials.

Posterior

Ground-truth GBI posterior samples are obtained via rejection sampling. As proposal, we used a Normal distribution centered around the ground truth parameter and with variance 50 β .

References

[24, 53]

A4.3 Training, inference, and evaluation for benchmark tasks

ACE For the benchmark tasks, the cost estimation network is a residual network with 3 hidden layers of 64 units [54]. The training dataset was split 90:10 into training and validation sets, with n target = 2 and 5 x t randomly sampled each epoch for evaluating training and validation loss, respectively. We used a batchsize of 500, i.e., 500 θ , 2 x t , for 1000 cost targets per training batch. Networks usually converge within 500 epochs, with 100 epochs of non-decreasing validation loss as the convergence criterion. For inference, we ran multi-chain slice sampling with 100 chains to sample the potential function. For the Gaussian Mixture task with 5 i.d.d. samples per simulation/observation, we appended to the cost estimation network a fully connected, permutation-invariant embedding network [55] (2 layers, 100 units each), which preprocessed the i.d.d. datapoints into a single 20-dimensional vector (but was trained end-to-end).

Kernel ABC We compared our algorithm with a version of kernel ABC on the benchmark tasks. For kernel ABC, we accepted samples (from the fixed set of 10000 prior simulations) with probability exp( -β · d ( x , x o )) , where d ( · , · ) is the distance function. In many cases, and especially for large β , this yielded very few or no accepted samples. We, therefore, reduced β until at least 50 samples were accepted.

NPE We used the implementation in the sbi toolbox [56], with neural spline flow [57] as density estimator (five transformation layers) to approximate the posterior. For tasks with bounded priors (all except Linear Gaussian), we appended a sigmoidal transformation to the flow such that its support matches the support of the prior [58]. For the Gaussian Mixture task, the same permutation-invariant embedding net was used as for ACE. Posterior samples were directly obtained from the trained flow. All other hyperparameters were the same as in the sbi toolbox, version 0.19.2 [56].

NLE and tempered NLE We used the implementation in the sbi toolbox [56], with neural spline flow [57] as density estimator (five transformation layers) to approximate the likelihood. For the Gaussian Mixture task with 5 i.d.d. simulations per sample/observation, they were split up as 5 independent simulations with the same θ repeated 5 times, effectively having 5 times the training simulation budget. We ran multi-chain slice sampling with 100 chains to sample the potential function as the sum of log-prior probability and flow-approximated log-likelihood. Tempered NLE uses the same learned likelihood estimator, but the log-likelihood term in the potential function is scaled by β during MCMC sampling (same values as used for ACE). All other hyperparameters were the same as in the sbi toolbox, version 0.19.2 [56].

Noise augmentation For each task, we randomly subsampled 100 of all simulated datapoints ( x ), and added Gaussian noise with zero-mean and standard deviation equaling two times the standard deviation of all prior predictives, i.e., N ( 0 , (2 σ x ) 2 I ) . We additionally varied noise amplitude ( 0 , 2 σ x , 5 σ x ) and found little to no effect on benchmark performance (see Appendix A3).

Synthetic misspecified observations For the Uniform 1D, 2 Moons, and Linear Gaussian task, we generated an additional 100,000 prior predictive simulations and defined the prior predictive bound as the minimum and maximum (of each dimension) of those simulations and σ x as their standard deviation. Then, to create 20 misspecified observations, we generated 20 model-simulations and added Gaussian noise with zero-mean and standard deviation equaling 0.5 σ x (i.e., ϵ ∼ N ( 0 , (0 . 5 σ x ) 2 I ) ) to the 20 observations iteratively until they are outside of the prior predictive bounds in every dimension. For the Gaussian Mixture task, we replaced the second Gaussian above by N (12 . 5 × sign ( θ ) , 0 . 5 2 I ) , i.e. displacing the mean to the corner of the quadrant whose signs match θ , and slightly increasing the variance.

Metrics For the metrics in Fig. 3, in columns 1, 2, and 4 and all corresponding figures in the Appendix, results were aggregated across the 10 well-specified and misspecified samples separately, where marker and error bars represent mean and standard deviation over 10 observations. Columns 1 and 2 show average distance between observation and 5000 posterior predictive simulations from each algorithm (except ABC, for which there were 50 posterior samples). Column 3 shows, for each of the 10 well-specified observations and 3 β values, 3 random posterior samples for which we compare the ACE-estimated and ground-truth cost, i.e., 10 × 3 × 3 = 90 points are shown for each

task. Column 4 shows C2ST score between 5000 ground-truth GBI posterior samples and ACE GBI posterior samples.

A4.4 Training procedure for Hodgkin-Huxley model

For NPE, we used the implementation in the sbi toolbox [56]. As density estimator, we used a neural spline flow [57] with five transformation layers. We used a batchsize of 5,000. We appended a sigmoidal transformation to the flow such that its support matches the support of the prior [58]. All other hyperparameters were the same as in the sbi toolbox, version 0.19.2 [56].

For ACE, we used a residual neural network [54] with 7 layers and 100 hidden units each. We used 10% of all simulations as held-out validation set and stopped training when the validation set did not decrease for 100 epochs. We used a batchsize of 5000. For sampling from the generalized posterior, we ran multi-chain slice sampling with 100 chains. As distance function, we used mean-absoluteerror, where the distance in each summary statistic was z-scored with the standard deviation of the prior predictives. To generate x t , we used all simulations which generated between 5 and 40 spikes (since any reasonable experimental recording would fall within this range) and we appended 1000 augmented simulations to which we added Gaussian noise with two times the standard deviation of prior predictives that produce between 5 and 40 spikes. We did not use the experimental recordings from the Allen Cell Types database in x t .

A5 Hodgkin-Huxley model

We used the same model as Gonçalves et al. [10], which follows the model proposed in Pospischil et al. [34]. Briefly, the model contains four types of conductances (sodium, delayed-rectifier potassium, slow voltage-dependent potassium, and leak) and has a total of eight parameters that generate a time series which we reduce to seven summary statistics (spike count, mean resting potential, standard deviation of the resting potential, and the first four voltage moments mean, standard deviation, skew, curtosis, same as in Gonçalves et al. [10]).

A6 Supplementary figures

posteriorpredictivedistance

posteriorpredictivedistance

costestimation

GBIposterioraccuracy

well-specifiedxo

misspecifiedxo

accuracy

ACE vs.GT

0.4

Uniform 1D

1.0

MSE

0.10

predicted

C2ST

0.2

0.05

0.5

0.5

0.00

0.0

0.0

N/A

4

20

100

N/A

4

20

100

0.0

0.2

4

20

100

0.10

2 Moons

1.0

predicted

MSE

C2ST

0.05

0.5

0.2

0.5

0.00

0.0

0.0

N/A

10

100

1000

N/A

10

100

1000

0.0

0.2

10

100

1000

1

Linear

MSE

10

predicted

5

C2ST

2

5

0.5

0

1

0

0

N/A

1

10

100

N/A

1

10

100

0

5

1

10

100

Mixture

0.10

0.4

predicted

MMD

C2ST

0.2

0.05

0.2

0.00

0.0

0.0

0.5

N/A

10

25

75

N/A

10

25

75

0.0

0.2

10

25

75

beta

beta

true cost

beta

GT-GBI

ACE-GBI

ABC

NPE

NLE

Owell-specified

==misspecified

well-specified,unseen

well-specified, seen

misspecified, unseen

misspecified,seen

0.4

0.4

Uniform 1D

predicted

0.5

0.2

0.5

0.2

0.0

0.0

0.0

0.0

T

0.0

0.5

0.0

0.2

0.0

0.5

0.0

0.2

Moons

predicted

0.2

0.2

0.2

0.2

2

0.0

0.0

0.0

0.0

0.0

0.2

0.0

0.2

0.0

0.2

0.0

0.2

10

101

Linear

predicted

5

5

5

5

0

T

0

0

T

0

0

5

10

0

5

0

5

10

0

5

Mixture

predicted

0.50

0.50

0.2

0.2

0.25

0.25

0.00

T

T

0.0

T

0.00

T

T

0.0

T

0.0

0.2

0.4

0.0

0.2

0.0

0.2

0.4

0.0

0.2

true cost

true cost

true cost

true cost

uniform 1d

two moons

lineargaussian

gaussianmixture

well-specified )

oX

0.10

0.10

+

=2

0=5

0.1

0.05

2

0.05

g=0

no augm.

0.0

0.00

0

0.00

4

20

100

10

1001000

1

10

100

10

25

75

misspecified xo

0.5

0.2

10

0.2

0.0

T

1

0.0

厂

T

0

T

1

0.0

厂

T

1

4

20

100

10

100

1000

1

10

100

10

25

75

beta

beta

beta

beta

posteriorpredictivedistance

posteriorpredictivedistance

costestimation

GBIposterioraccuracy

well-specifiedxo

misspecifiedxo

accuracy

ACE vS.GT

Uniform 1D

predicted

0.4

MSE

0.2

C2ST

0.2

0.5

0.0

0

T

T

1

0.0

T

N/A

4

20

100

N/A

4

20

100

0.0

0.5

4

20

100

2 Moons

MSE

0.2

predicted

0.2

C2ST

0.5

0.0

0

0.0

T

厂

T

N/A

10

100

1000

N/A

10

100

1000

0.0

0.1

0.2

10

100

1000

Linear

MSE

5.0

predicted

5.0

C2ST

2.5

10

2.5

0.0

0

0.0

0.5

N/A

1

10

100

N/A

1

10

100

0.0

2.5

5.0

1

10

100

1.0

F

1.0

一

Mixture

predicted

0.4

MMD

C2ST

0.5

0.5

0.2

0.5

0.0

0.0

0.0

N/A

10

25

75

N/A

10

25

75

0.00

0.25

0.50

10

25

75

beta

beta

true cost

beta

GT-GBI

ACE-GBI

ABC

NPE

NLE

Owell-specified

==misspecified

posteriorpredictivedistance

posteriorpredictivedistance

costestimation

GBIposterioraccuracy

well-specifiedxo

misspecifiedxo

accuracy

ACE vS.GT

1

predicted

Uniform

MSE

0.1

0.5

C2ST

0.5

0.0

1

0

0.0

N/A

4

20

100

N/A

4

20

100

0.0

0.5

4

20

100

0.10

2 Moons

MSE

predicted

0.2

C2ST

0.05

0.1

0.5

0.00

0

0.0

N/A

10

100

1000

N/A

10

100

1000

0.0

0.2

10

100

1000

Linear

predicted

MSE

5

C2ST

10

2

0.5

0

1

0

0

N/A

1

10

100

N/A

1

10

100

0

5

1

10

100

Mixture

0.2

0.50

MMD

C2ST

0.1

0.25

0.25

0.5

0.0

0.00

0.00

N/A

10

25

75

N/A

10

25

75

0.00

0.25

0.50

10

25

75

beta

beta

true cost

beta

GT-GBI

ACE-GBI

ABC

NPE

NLE

Owell-specified

==misspecified

e

observations

C

d

predictivedistance

0.4

7

0.3

0.2

0.5

9Na

80.050

3000

b

NPE(1M sims)predictives

0.1

0.0

NPE ACE ACE ACE

2550100

0

9K

1540

）-V

90

0.09leak

0.6 0.00 noise 0.15

0.0

0.5

80

9Na

0.0001

15

9K

0.0001

0.6

gleak

0.0001

0.6

9M

50

3000

tmax

40

90

-VT

1

0.0001

0.15

noise

35

100

-Eleak

ACE (β = 25)

ACE (β = 50)

ACE (β = 100)

E厂

NPE (100K simulations)

[TIMI

T

M

NPE (1M simulations)

Peer reviews:
comments: SUMMARY:
The authors present a novel approach for SBI by utilizing neural networks (NN) to estimate a generalized cost in GBI. Their proposed method, ACE, demonstrates superior computational efficiency compared to previous approaches, without compromising competitive performance across various evaluation metrics. The authors conducted extensive benchmarking of their method under diverse experimental settings, ranging from 1D to 10D. Furthermore, they conducted experiments using real intracellular recordings, adding an additional dimension to their research.

STRENGTHS:
The paper is well written and maintains a smooth flow throughout. Furthermore, the contribution made by the authors appears to be a novel and valuable improvement upon existing ideas in the field. Moreover, the authors have thoughtfully acknowledged and addressed the limitations of their work, placing their findings within the broader context of the field. It is worth noting that their code repository is well-organized, adding to the overall credibility and reproducibility of their research.

WEAKNESSES:
1. To enhance clarity, it would be beneficial to improve the visual presentation of Figure 2. When referring to subfigure C, the arrangement could be adjusted to prevent the automatic focus on the third pane.
2. Propositions 1 and 2 appear somewhat forced, as their results do not seem entirely compelling or noteworthy. It may be worth revisiting these propositions. In the appendix the authors state, "The proofs closely follow standard proofs that regression converges to the conditional expectation." Thus highlighting the lack of importance of these propositions.
3. A minor typo on line 116, "parameter- or".

QUESTIONS:
nothing, great work!

LIMITATIONS:
The authors do address the limitations of their work.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-06-27

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the problem of simulation-based inference - which can is encountered in a wide range of scientific problems - where one is interested in performing Bayesian inference using simulators with implicit likelihoods. Scientific problems can have two unique properties - a) the predictive quality of the simulation is more important than the posterior b) the model can be misspecified. These can be handled though Generalized Bayesian Inference - which generalizes traditional Bayesian Inference by admitting generalized likelihood functions which are defined by some general loss / cost function defined on the parameters. A challenge with employing Generalized Bayesian Inference in the simulation-based case is that estimating the cost function can be computationally expensive. The authors propose learning an amortized neural network estimator for the cost function to reduce the overhead for enabling GBI in simulation-based inference. This amortized cost estimator can be combined with standard inference algorithms (MCMC) to sample from the Generalized Bayesian posterior. The overall proposed approach consists of 3 stages: 1) Collecting data 2) Training ACE 3) Sampling using ACE. The authors present several experiments on a variety of benchmark simulation-based inference tasks and present a case study with the Hodgkin-Huxley simulator.\n\nSTRENGTHS:\n* The problem of simulation-based inference is an important one for various scientific problems and misspecification is an often under-studied but important aspect in practice. The paper attempts to address this important issue relevant to various communities. \n* The proposed approach is novel within the context of the simulation based inference. (Although as I mention below, similar ideas have been employed is some related areas)\n* Even despite some caveats that I discuss below, the idea is quite neat, and relatively simple (which is a good thing!) Replacing the estimation of the cost with an amortized predictor seems like a natural extension to enable generalized posteriors in the simulation-based inference setting. The simpler algorithm also makes it easier to adopt by practitioners with little overhead. \n* The experiments cover a fairly wide variety of tasks used in prior work in the SBI setting. The case study on the HH model was a nice example of leveraging the method on realistic tasks. \n* The presentation in the paper was generally quite clear. The paper is well written and easy to follow.\n* I also appreciate the authors open-sourcing the code to aid reproducibility.\n\nWEAKNESSES:\n* A fundamental weakness I see with the method is that it relies on the neural estimator to generalize well from finitely many simulations. This might not be true in practice (certainly isn’t guaranteed). Put differently, an implicit assumption for the approach to work is that the ACE is trained with _enough_ data to be able to provide informative guidance to the sampler learned in the next stages. As with other work relying on NN estimators this is a fundamental failure mode. \n* On a similar note, the method appears to be practical only for relatively low-dimensional problems. The NN learning issues would crop up if the dimensionality of the problem is increased. \n* Similar ideas of learning amortized neural estimators have been recently explored in the uncertainty estimation literature [1,2]. In particular, the secondary predictor learned in [1] amortizes a similar quantity. \n\n[1] DEUP: Direct Epistemic Uncertainty Prediction. Lahlou et al., TMLR 2023.\n\n[2] Epistemic Neural Networks. Osband et al., 2023.\n\nQUESTIONS:\n* There are already some results with different simulation budgets - but a more careful study of the effect of the number of samples available to train the ACE might be quite useful. \n* Could you comment on the connections to the work on uncertainty estimation mentioned above? \n* Do the authors have any thoughts on extending the approach to higher dimensional settings?\n\nLIMITATIONS:\nThe authors already discuss several limitations in the paper including a) need for another method to sample from the posterior induced by ACE, b) accommodation of arbitrary distances, c) additional hyper parameters. \n\nIn addition to these I would also add access to sufficient samples for training ACE and being limited to low dimensional settings as limitations.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes amortized cost estimation (ACE) for generalized Bayesian inference (GBI) for SBI. The paper trains a neural network to approximate the cost function. The paper demonstrates results on baseline synthetic SBI examples followed by a real-world application using experimental data from the Allen Cell Types Database.\n\nSTRENGTHS:\n* The paper is written clearly and is structured well.\n* The idea appears novel. While, GBI was previously defined in [13], its application within the SBI literature is interesting and is therefore of value to the community.\n* Algorithm box makes the implementation of the algorithm clear to the reader.\n\nWEAKNESSES:\n* The main weakness of the paper seems to be the experimentation. While the figures are neatly presented, it is not that clear what the reader is supposed to conclude from the results:\n    * For example, it was not clearly defined what the difference between specified and misspecified samples means within Figure 3. It would be helpful to mathematically define these differences rather than describe them within the text. \n    * What is the definition of the GT-GBI? It first appears on line 173 and it is not clear what the true distance is and how it is obtained. This misunderstanding makes it further difficult to understand Figure 3.\n    * Another question is why are the first three rows only showing MSE and the last row showing MMD? This is confusing and makes one question why the Figure is not consistent with the metric. \n    * Additionally, for the MSE/MMD it seems that as $\\beta$ is increased, the MSE/MMD goes down, but the C2ST seems to go up. Why is this the case? What happens in the limit that $\\beta$ goes to infinity?\n\nQUESTIONS:\n* Compared to BGI, equation (2) seems to not include the KL divergence term. Can this approach really be called GBI if it is not being regularized by some prior? Why was the KL term not included?\n* Following on from the question regarding the prior, is the addition of the noise ($\\epsilon$) implicitly adding a prior? It might be the case that adding this noise is also enforcing some implicit distance metric between nearby parameters. I.e. does it assume that there is a local Euclidean distance metric?\n* Why does this approach require multi-chain slice-sampling? Is random-walk Metropolis-Hastings not sufficient?\n\nLIMITATIONS:\n* The idea behind the paper is interesting, but the experimental results are difficult to interpret. Further clarifications on the above would help increase the score.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a new technique - amortized cost estimation (ACE) - that, as stated on the can, amortizes a broad class of loss functions used in generalized Bayesian inference (GBI) in place of the (log) likelihood. \nAfter training on a moderate-to-large number of model simulations (10K-100K in the examples of the paper), the amortized loss can then be used in place of the real loss to perform GBI (e.g., via MCMC). Crucially, this step does not require further simulations from the model.\nThe authors show that the amortized loss generally matches the ground-truth loss, and show that their amortized method inherits the advantages of GBI, in that it is typically more robust to misspecification than standard and amortized Bayesian inference, as in neural likelihood estimation and neural posterior estimation\n\nSTRENGTHS:\n- **Quality:** The general polish and quality of the paper is high.\n- **Clarity:** The paper is well written and generally clear. The aim and methodology are very well explained and motivated, with some points that would benefit from further expansion.\n- **Significance:** The idea of the paper (amortizing GBI for simulator-based inference, with the additional goal of dealing with misspecification) is very timely and there are surely many applications.\n- **Originality:** The idea at the core is not particularly original - amortizing the loss in GBI seems a natural direction for the field at the moment - but clearly worth pursuing. Similarly, the execution seems fairly straightforward (which is not a bad thing).\n\n\n### Post-rebuttal:\n\nThanks for having addressed most of my comments. I appreciated the additional experiments with tempered NLE, with varying noise augmentation, and the recommendations for the choice of $\\beta$. The proposed heuristic seems both reasonable and practical. I understand that given the limited time, it was not feasible to conduct an additional set of experiments.\n\nOverall, I am satisfied with the changes and I increased my score accordingly from 6 to 7.\n\nWEAKNESSES:\nMost of my concerns have to do with the treatment of the hyperparameter $\\beta$ (inverse temperature), a key element of GBI.\n\nThe role of $\\beta$ could be explained and discussed more (it is mentioned in the **Limitations**).\n\nMany experiments show the results for a range of $\\beta$ (which also changes from experiment to experiment), and it is unclear how the practitioner should choose its value. For example, $\\beta \\rightarrow \\infty$, (generalized) Bayesian inference tends to maximum-a-posteriori (or minimum-generalized-loss) point estimation. The paper is ambiguous on what the user should be trying to achieve. I appreciate that some of these issues are not of ACE (the authors' method) but of GBI, but the paper would benefit from directly addressing these points.\n\nThe comparison between GBI and Bayesian inference is somewhat unfair in that Bayesian inference is not allowed an inverse temperature hyperparameter (i.e., likelihood tempering), but in principle it could be easily applied (at least for NLE). Is ACE+GBI truly better here, or is the advantage just given by the fact that ACE/GBI has an extra free hyperparameter? In some cases, standard Bayesian inference can do better with large $\\beta$. We know this can be the case for example with neural network posteriors (possibly a case of prior misspecification).\n- With NLE, just run MCMC with a scaling factor $\\beta$ on the amortized log likelihood. \n- Amortizing different temperatures with NPE would be more complex. Naively, one would need to retrain the network for different values of $\\beta$. One could also subtract the log prior from the NPE posterior to get the log likelihood, and then rescale it, but that might lead to instabilities.\n\nSimilarly to $\\beta$ (but less importantly), the paper could explore more the role of the noise $\\sigma$ added to $S$ observations. This parameter seems to be less relevant though - it is very nice that the authors use a fixed value throughout the paper, and this value can be determined from prior-predictive checks. Nonetheless, a bit of exploration / insight could help.\n\nFor the rest, the empirical evaluation is acceptable, but it would have been nice to see at least a couple of applications to real-world data (even another simple one), since this is arguably (also according to the authors) where GBI shines.\n\nQUESTIONS:\n- Can you expand on the role of $\\beta$, e.g. how you chose (and a practitioner should choose) reasonable ranges of the hyperparameter?\n- What about having an inverse temperature to the log likelihood of standard Bayesian inference (as per standard tempering, also seen helping in Bayesian deep learning applications)? Can you show that in your examples (for NLE, and possibly NPE)?\n- Consider adding a lesion study or analysis about the role of $\\sigma$ and $S$.\n- Not strictly necessary, but it would be nice to see the method at work on another (simple) example with real data.\n\nLIMITATIONS:\nThe authors address the limitations of their method (and I already asked above to expand on the role of $\\beta$).\nThe work has no particular potential negative social impact (at least not above any other general method for statistical inference).", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-20"}