Sensitivity-Aware Amortized Bayesian Inference
Authors: Lasse Elsemüller
Hans Olischläger
Marvin Schmitt
Paul-Christian Bürkner
Ullrich Koethe
Stefan T. Radev
Venue: Accepted by TMLR ()

Abstract: Sensitivity analyses reveal the influence of various modeling choices on the outcomes of statistical analyses. While theoretically appealing, they are overwhelmingly inefficient for complex Bayesian models. In this work, we propose sensitivity-aware amortized Bayesian inference (SA-ABI), a multifaceted approach to efficiently integrate sensitivity analyses into simulation-based inference with neural networks. First, we utilize weight sharing to encode the structural similarities between alternative likelihood and prior specifications in the training process with minimal computational overhead. Second, we leverage the rapid inference of neural networks to assess sensitivity to data perturbations and preprocessing steps. In contrast to most other Bayesian approaches, both steps circumvent the costly bottleneck of refitting the model for each choice of likelihood, prior, or data set. Finally, we propose to use deep ensembles to detect sensitivity arising from unreliable approximation (e.g., due to model misspecification). We demonstrate the effectiveness of our method in applied modeling problems, ranging from disease outbreak dynamics and global warming thresholds to human decision-making. Our results support sensitivity-aware inference as a default choice for amortized Bayesian workflows, automatically providing modelers with insights into otherwise hidden dimensions.

Published in Transactions on Machine Learning Research (08/2024)
Sensitivity-Aware Amortized Bayesian Inference
Lasse Elsemüller Heidelberg University
lasse, www..eeleeeeeeeeeemm gmail.com
Hans Olischläger Heidelberg University
Marvin Schmitt University of Stultgart
Paul-Christian Bürkner TU Dortmund University
Ullrich Köthe Heidelberg University
Stefan T. Radev Rensselaer Polytechnic Institute
stefan.radev9 @ gmail.com
Reviewed on
Abstract
Sensitivity analyses reveal the influence various modeling choices on the outcomes of statistical analyses. While theoretically appealing. they are overwhelmingly inefficient for complex Bayesian models. In this work, propose sensitivity-aware amortized Bayesian inference (SA-ABI), multifaceted approach to efficiently integrate sensitivity analyses into simulation- based inference with neural networks. First, we utilize weight sharing to encode the structural similarities between alternative likelihood and prior specifications in the training process with minimal computational overhead. Second, we leverage the rapid inference of neural networks to assess sensitivity to data perturbations and preprocessing steps. contrast to most other Bayesian approaches, both steps cireumvent the costly bottleneck of refitting the model for each choice likelihood, prior, or data Finally, propose use deep ensembles detect sensitivity arising from unreliable approx- imation (e.g., due to model misspecification). We demonstrate the effectiveness of our method in applied modeling problems, ranging from disease outbreak dynamics and global warming thresh- olds to human decision decision-making. Our results support sensitivity inference as default choice for amortized Bayesian workflows, automatically providing modelers with insights into otherwise hidden dimensions.
Introduction
Statistical inference aims extract meaningful insights from empirical data through series of analytical procedures. Acknowledging that each of these procedures involves myriad of implicit choices and assumptions, any single analysis hides an iceberg of uncertainty (Wagenmakers et al., 2022). We consider sensitivity analysis as formal approach to shed light on this very iceberg of uncertainty.
For instance, global warming forecasts can change depending on the assumed earth system model. In other words: Climate change analyses can sensitive to the underlying observation model (i.e., likelihood; see Experiment 2). Yet. the likelihood the only model component that can induce sensitivity. The prior assumptions, the approximation algorithm, and the specifics of the collected data contribute further uncertainty to the results (Bürkner al., 2022).

---

Published in Transactions on Machine Learning Research (08/2024)
Stage 1: Training
Stage 2: Inference
Extending the amortization scope
Amortized sensitivity analysis
PCC2, Cp)
User-specified contexts
Likelihood
Deep ensemble with M members
Simulator
Deep ensemble
c
Neural net m
Posterior 9mn
Neural net m
Posterior 4mw
Prior
Data
C₁
I,... M
712 1, M
simulations:
1,,,,
V context variations:
1,... V
learn N M postetions during training
efficienlly evaluate M posteriors during inference
Our proposed approach for sensitivity- amortized Bayesian inference (SA-ABD). Stage 1: During training, distribution p(Cz, Cr) over plausible likelihood and prior choices encoded via context variables Ct and Cp in deep ensemble of neural approximators. Stage 2: During inference, we cast costly model refits as near-instan neural network prediction task conditioned on user-specified context C. Our amortized neural approach unlocks fast large-scale sensitivity analyses of all components in Bayesian model: likelihood (CL), prior (Cp), data (Cp). and approximator (Ca). Experiment 3 uses V = 100 variations in prior and data alongside M = 20 deep ensemble members. The resulting amortized sensitivity analysis encompassing V M = 162 000 approximate posteriors would have been infeasible with existing methods.
Figure 1:
Classical sensitivity analyses rely on costly model refitting under each configuration and quickly become infeasible for both likelihood-based (e.g., MCMC; Neal, 201 and simulation-based inference (SBI, Cranmer et al., 2020).
Recently, SBI methods have been accelerated through amortized Bayesian inference (ABI; Radev et al., 2020; Gonçalves al., 2020; Avecilla et al., 2022), where neural networks learn probabilistic inference tasks and com- pensate for the training effort with rapid inference on many unseen data sets. As we demonstrate this paper, this directly enables large-scale assessment of data sensitivity. However, assessing likelihood, prior, and approxima- tor sensitivity remains extremely challenging in standard ABI applications, which may be costly to train even for single configuration. To address this gap, we investigate sensitivity-awon amortized Bayesian inference (SA-ABI) and demonstrate that it unlocks highly efficient and multifaceted sensitivity analyses in realistic ABI applications (cf. Figure 1). Our main contributions are:
We conceptualize sensitivity via implicit context variables and integrate established methods for sensitivity analysis into amortized Bayesian inference:
2. We investigate context- neural architecture to quantify likelihood and prior sensitivity at inference time with minimal computational overhead and no notable loss of accuracy;
3. We assess approximator sensitivity via deep ensembles and data sensitivity via the near-instant ABI inference;
We demonstrate the utility of SA-ABI for Bayesian parameter estimation as well as Bayesian model compar- ison in three real-world scenarios of scientific interest, investigating sensitivity under up to 162 000 configu- rations.
2 Background
What follows is brief overview of Bayesian parameter estimation, model comparison, amortized Bayesian inference, and learnable summary statisties, Readers familiar with these topics can safely jump directly to Section 3.
2.1 Bayesian Inference
Bayesian Parameter Estimation
In Bayesian parameter estimation, the key quantity is the posterior distribution,
p(x ()p(O)
p(0 0 x)
(1)
p(x)
2

---

Published in Transactions on Machine Learning Research (08/2024)
which combines the likelihood p(æ ()) with prior information about parameter distributions p(0), normalized by the (analytically intractable) marginal likelihood p(z).
Bayesian Model Comparison In many scientific applications, no single generative model can provide the ultimate explanation for a data set æ. Instead, a set of models M = M2:.M M,, is plausible. Bayesian model comparison aims find the "best" model within M. In (prior-predictive) Bayesian model comparison, a model's marginal likelihood p( Mj) now takes the central role:
p M2) = f p( (.M,) p(0) M2
(2)
Marginalizing the likelihood over the parameter space automatically encodes Occam's razor through a preference for models with limited prior predictive flexibility (MacKay, 2003). The posterior model probabilities of competing models can be computed as
pla p(y)MM))((M)
p(M, x))
(3)
Emp(æ MM
where p(M) is the prior distribution over the model space.
2.2 Simulation-Basen Inference
Both Bayesian parameter estimation and model comparison have traditionally been limited by the ability to efficiently evaluate a model's likelihood density p(æ 0). Likelihood-baaee methods (e.g., MCMC) assume that the distributional family of the likelihood is explicitly known and can be evaluated analytically or numerically for any pair (x, 0). Differently, simulation-bail approaches only require simulations from a simulation program G,
x = G E with
P(E 0). p(0).
(4)
with latent program states or "outsourced" noise a. A single execution of such a program corresponds to generating samples from the Bayesian joint model (0, x) ~ p(0, x), since the execution paths of the simulation program define an implicit likelihood (e.g., Cranmer et al., 2020; Diggle & Gratton, 1984; Marin et al., 2012)
pa 0) = 6(xz G(O, p(E ( @g
(5)
where d is the Dirac delta function. However, the above equation (5) is analytically intractable for any simulation program of practical interest, turning simulation-barn based inference into a computational challenge.
2.3 Amortized Bayesian Inference
Amortized Bayesian inference (ABI) leverages simulations from G' to solve Bayesian inference tasks with neural networks in real time after an initial training phase. To achieve this, neural networks learn to encode the relationship between simulated data and model states during training. As result, costly probabilistic inference is replaced with a neural network prediction task. For parameter estimation, generative neural networks act as conditional neural density approximators of the parameter posterior p(0) x)) (Greenberg et al., 2019; Radev et al., 2020). Model comparison, on the other hand, can be framed as a probabilistic classification problem which addressed with discriminative neural networks approximate posterior model probabilities p(M x=) (Pudlo et al., 2016; Radev et al., 2021a).
2.4 End-to-end Summary Statistics
Common neural density estimators hinge on fixed-length vector-valued inputs a requirement that is violated by widespread data formats such sets of i.i.d. observations time series. To this end, previous research explored ways to learn end-to-end summary statisties for flexible adaption to the individual data structure and inference task (Chan et al., 2018; Wiqvist et al., 2019; Radev et al., 2020; Chen et al., 2020; 2023). In a nutshell, a summary network ha compresses input data æ of variable size to a fixed-length vector of learned summary statistics h (x) by exploiting probabilistic symmetries in the data (c.g., Reddy& & Teh, 2020). The resulting embedding is fed to an inference network fo that approximates the posterior (e.g.. with conditional normalizing flow). The summary network he and the inference network fo are simultaneously trained end-to-end in order learn optimal summary statistics for the inference task.
innitttiiiinnnniiannt networks for exchangeable data; Bloem-
3

---

Published in Transactions on Machine Learning Research (08/2024)
3 Methods
3.1 Extending the Amortization Scope
Sensitivity Sources as Context Variables The PAD framework (Bürkner et al., 2022) defines Bayesian model as combination of the joint Probability distribution p(O, æ | MM, which can be factorized into likelihood p(æ M) and prior p(0) IM), the posterior Approximator, and the observed Data Lobs- Building on this decomposition, we define sensitivity to a component of a Bayesian model as the extent of change in inferential results induced by perturbations in any of these components (Bürkner al., 2022). We will refer the opposite of sensitivity as robustness: When an inference procedure is robust, it is not sensitive changes in model components.
To enable systematic and comprehensive investigations of sensitivity, we consider sources of perturbations context variables that implicitly shape inferential results (see Figure 1) and can be cheaply varied as conditioning variables (or hyperparameters) in our simulation-base approach. In contrast, standard Bayesian workflows treat context variables as fixed and thus cannot typically investigate their effect explicitly without re-doing the entire analysis.
In the following, we denote context variables as C₁ (likelihood), Cp (prior), CA (approximator). Cp (data), and refer to the entirety of context variables C' (Cz, Cp,, CA, CD). Accordingly, we refer to posteriors with explicitly encoded context variables p( x. C?) for parameter estimation and p(Mj x, C)) for model comparison.
Sensitivity-Aware Training Before discussing key facets of practical sensitivity analysis, we describe extensions to standard simulation- based training that allow these analyses to be performed efficiently. Specifically, we eliminate the necessity of retraining neural approximators for every aspect of a sensitivity analysis by incorporating the Training contexts Cr = (CL, Cp) into the network's amortization scope (Radev, 2021). Since we realize CA via deep en- sembles and Cp only during inference (see Section 3.2), neither of these contexts influences the training objective of single ensemble members.
For sensitivity- parameter estimation (PE), we incorporate the context Cr into the standard negative log- posterior objective via
_E (Φ. sb: CT) = E log 9s (0 has (x:), (x)) Cr)].
(6)
The expectation E is here taken a contextualized joint Bayesian model p(0, Cr), which produces tuples of training parameters and corresponding data (0, æ)) for the given context Cr.
Analogously, for sensitivity-aware Bayesian model comparison (BMC), we can target the approximate posterior model probability (M (x), Or) via the cross-entropy
BBCC (Ø) cb: Cr) E
Σ Im, log (ogg
(x))), (M), Cr)].
where the expectation E is taken with respect to a contextualized generative mixture of Bayesian models (MM)CCr) p(z My, Cr. producing tuples of model indices and associated simulated data (M, The indica-
tor function IM, denotes one-hot encoding for the true model index, i.e., IM, 1 if Mj is the true model.
To achieve the desired amortization over any set of context variables Or, we define prior distribution )(Cr)) over the domains of Cr and minimize the context-aware (CA) loss:
CC (Φ. w = E| wb: Cr)],
(8)
where the (outer) expectation runs over p(CT) and can be either _ LBCC, We believe that uniform distributions are reasonable choice of p(Cr) for sensitivity analyses and employ them in all experiments. Nevertheless, p(Cr) can be tailored to specific modeling needs, such as giving more weight approximating a preferred baseline setting. In practice, we approximate Eq. 8 using standard mini-batch gradient descent over a finite data set D (Cr, 0 x) for parameter estimation, or D = MCr ] for model comparison.
Our approach seamlessly generalizes to other strictly proper losses (Gneiting & Raftery, 2007) which can be used as training objectives for amortized inference (Pacchiardi & Dutta, 2021). For the sake of generality, we can introduce function S that quantifies the fidelity of conditional distribution ds E Q for predicting target quantity y y
4

---

Published in Transactions on Machine Learning Research (08/2024)
Table I: Overview of our taxonomy for sensitivity in Bayesian inference via context variables. The rightmost column conveys that our context-aware (CA) loss function _eA in Eq. 8 enables the amortization over both likelihood (C₁,) and prior (Cp) contexts during training.
Context
Sensitivity source example
Implementation
CCA required?
Cl
Likelihood
Structural model assumptions Expert knowledge
Multiple simulator configurations
Cp
Prior
Multiple prior configurations Deep ensemble
/
CA
Approximator
Simulation gaps
x
CD
Data
Influential observations
Multiple data configurations
x
(Gneiting & Raftery, 2007). Thus, S Qx R is function of some qe and y (e.g., 0 or M2) which can easily be written to incorporate arbitrary conditions for qΦ. such as summarized data (w)()) and context Cr, resulting in (3 (x), Cr). y). Ideally, we would like to treat the expected score as an optimization objective
C(d, ub: Cr) = (m.))) S(ap(y hys (z:),
which for strictly proper scoring functions S would guarantee ( (x),)), Cr) p" (y) z) under perfect conver- gence (Gneiting & Raftery, 2007; Pacchiardi & Dutta, 2021). However, we usually cannot directly access the analytic expectation over the unknown true data-gem gettaaennrattigg distribution p" (æ, y). Thus, to achieve tractable amortization, we use the model-implied distribution p(æ, y M) as a proxy for the (unknown) true data generating process p" (x; y) and optimize the former objective in expectation over the simulator outputs (i.e., one or more Bayesian probabilistic models).
With this expansion of the amortization scope, we achieve amortization across all sensitivity dimensions C (CL, Cp, Ca, CD) during inference. What that means in practice is that, at inference time, can simply "turn knob" on any of the sensitivity dimensions and obtain the resulting posterior in an instant. A natural question that immediately arises is whether the resulting sensitivity-aware posterior is less accurate than the corresponding fixed-contett posterior. Intuitively, the answer depends on the sampling diversity of the contextualized joint model p(æ, y, Cr) and the potential for reaping the benefits of weight sharing: If the associated likelihood and prior varia- tions instantiate generative models with wildly different behaviors, weight sharing may not advantageous, resulting in diminishing returns from amortized training over the context variables Cr. Fortunately, the set of plausible choices for given modeling problem typically Icads to similar generative patterns, that weight sharing much more effi- cient than separate approximation. Indeed, our experiments demonstrate this for several representative model families even under small simulation budgets. Nevertheless, if amortization over very different simulators is desired, we rec- ommend increasing the expressiveness of the neural approximator, the simulation budget, and the allotted training time. The following section describes sensitivity sources and actionable manipulation strategies for each sensitivity dimension (see Table for an overview).
3.2 Sources of Sensitivity
Likelihood and Prior Sensitivity Varying likelihood context variables are ubiquitous in simulation-basee inference. Structural decisions within the simulator(s) may constitute context variables (e.g., the underlying scientific model, see Experiment 2) or exogenous experimental factors, such as design matrices, indicator variables, or time scales. Typical examples for prior context might be as simple as the scales of prior distributions, or as complex as different experts eliciting discrete sets of qualitative (e.g., non-probabilistic) domain knowledge. Moreover, both likelihood and prior can be continuously tempered to strengthen weaken their influence. For example, power-scaling exponentiates densities with a parameter 7 > 0, resulting in p(0) for prior power scaling and p(y/ 0)) for likelihood power scaling (Kallioinen et al., 2021).
We make all known picces of likelihood and prior context explicit by incorporating C₁ and Cp into the generative model. This enables amortization over these context variables, which drastically increases the generalization space of the trained neural approximator. During inference, the specific set of likelihood and prior can simply be selected by passing the respective Ct, and Cp configurations. Amortization over the context space leverages structural similarities
5

---

Published in Transactions on Machine Learning Research (08/2024)
between context configurations via weight sharing. Compared to separate training, this minimizes the associated computational cost and is especially beneficial whenever only finite training data is available (see Experiment 2).
Approximator Sensitivity We define approximator sensitivity as the variability of inferential results due to the ap- proximation method employed. To isolate approximator sensitivity in ABI, it seems helpful to (i) distinguish between a closed world (i.e., simulations) and open world (i.c., empirical data) setting; and (ii) realize an approximator context CA via deep ensemble of M equally configured and independently trained neural networks ((f(")), sb(m) " Jm=1.
In the closed-world setting, ground truth values for the approximation targets y (i.e., 0 M2) are available. Thus, we can readily validate amortized neural approximators on thousands of simulated data sets from the model(s) under consideration. We propose to additionally measure performance variability between the ensemble members to detect approximator sensitivity due to finite training or suboptimal convergence. After validating the approximator in the closed world, we consider the open-world setting, where the true data-generating process is unknown.
As a simulation-based method, ABI assumes that simulations are a faithful representation of a system's real behavior (Dellaporta al., 2022). Hence, simulation gaps, where atypical data violate this assumption, threaten its credibility (Schmitt et al., 2023; Cannon et al., 2022). Simulation gaps can be considered cause an out-of-distribution (OOD) setting inference time: For example, Cannon al. (2022) observed that misspecifcationninndeee simulation gaps result in neural approximators exhibiting the typical OOD behavior of unstable predictions (Ji et al., 2022: Shamir et al., 2021; Liu et al., 2021). Therefore, we can leverage the proven OOD detection capabilities of deep ensembles (Lakshminarayanan et al., 2017; Fort et al., 2019: Yang et al., 2022) to detect simulation gaps in ABI. Specifically, we hypothesize that variability across the M ensemble members in the open world despite consistent performance in the closed world indicates a simulation gap. Concretely, we expect simulation gap to translate into high variability in the predictive distribution of the unknown targets y given empirical data Zobs.
iyy Dobs) = Ep ( (D) (1 hy (xrbs)))))
(10)
which is approximated by the deep ensemble (Lakshminarayanann al., 2017) and can be augmented with arbitrary context Cr-
When we train a deep ensemble with M members, we clearly need to repeat the training loop M times. Crucially though, we can simulate a single training set upfront and then re-use the simulated training data for all ensemble members. This not only reduces the stochastic dependence by keeping the training data constant but also drastically reduces the computational cost in most realistic tasks where simulations are expensive. Finally, our ensemble approach can be easily extended to combine information from all ensemble members for potentially more accurate inference (e.g., via simulation-based stacking. Yao et al., 2023) or investigate hyperparameter sensitivity via hyperparameter ensembles (Wenzel et al., 2020). Hyperparameters that are particularly relevant in SBI include the architecture of the summary network (e.g.. inductive bias induced by the architecture, number of learned summary statistics), the choice of inference network (c.g., architecture, number of trainable weights), and common hyperparameters that are ubiquitous in deep learning (e.g., learning rate, regularization).
Data Sensitivity Statistical inference relies on finite samples to draw conclusions about populations. As such, any analysis influenced by the properties of this partieular sample. For instance, analysis outcomes might radically change under different preprocessing choices, such as handling extreme or missing data points, even if these choices only affect a small subset (Simmons al., 2011; 1; Broderick et al., 2023).
There are two straightforward strategies to assess this data sensitivity: (i) To assess disproportionally large influence of single data points (also known as influential observations), a context Cp of alternative data manifestations can be realized via small perturbations of the empirical data set, for instance via bootstrapping or leave-one-out folds; (ii) to analyze the effect of specific preprocessing decisions, we can generate data set variations for all combinations of reasonable decisions, which in turn constitutes Cp. For example, Kristanto al. (2024) identify 17 debatable prepro- cessing steps with 102 choice options in graph- based fMRI analysis, resulting in hundreds of potential manifestations of the final data set.
I Although Bayesian neural networks offer appealing uncertainty quantification properties, we focus deep ensembles for their practical implementation advantages akshminaraaaana al., 2017; Wilson Izmailov, 2020).

---

Published in Transactions on Machine Learning Research (08/2024)
Both strategies require large amount of model refits, which is computationally infeasible for MCMC non- amortized simulation-baser approximators. In contrast, ABI methods can amortize across data sets of variable sizes (Radev et al., 2020), enabling rapid inference on large number of data set variations.
3.3 Evaluating Sensitivity
Quantitative Sensitivity We can easily quantify sensitivity via a divergence metric D between target probability densities (Kallioinen et al., 2021; Roos et 2015): For an acceptable upper bound & based on domain knowledge, model is robust if
D (p(((() z, C.) p(g(() x C2)] < 0,
(11)
for arbitrary context realizations C₁ and Cj. where g is pushforward variable (e.g.. predicted quantities) or projection of the full target posterior onto subset y S y.
Measures from the family of F-divergences offer principled metrics for ID (Csiszár, 1964; Ali & Silvey, 1966; Liese & Vajda, 2006). In Bayesian model comparison, the model posterior containing the probabilities for each M, follows a discrete categorical distribution. Thus, obtaining F-divergences, such as the KI. divergence, is straightforward (see Experiment 3, Figure 5b). In Bayesian parameter estimation, the posterior typically not available as a closed-form density but as random draws. Thus, we prefer a probability integral metric, such as the maximum mean discrepancy (MMD; Gretton al., 2012), which can be efficiently estimated from posterior samples (see Bischoff al., 2024, for recent discussion of other suitable choices).
Qualitative Sensitivity Although quantitative sensitivity patterns provide detailed insights, sensitivity analysis is of- ten ultimately interested in qualitative robustness, i.e., invariance of analytical conclusions to the context C' (Kallioinen et al., 2021; Bürkner et al., 2022). For instance, an analyst might ask whether two choices of context variables C and C2 contain a certain parameter value within a specified highest density interval (HDI), or lead to the selection of the same model M,. Making decisions based on posterior distribution can be formalized via a decision function L A which maps distributions p E P (or their approximations) to possible qualitative conclusions or actions for given problem A. Formally, qualitative robustness is expressed with the indicator function
R(C, Cj) =
{
if Lp(yl C.)) (p( x. C2))
(12)
otherwise
which yields 1 if conclusion is invariant to the choice of arbitrary context realizations C2 Cj, and 0 otherwise. Note that our definition trivially generalizes to more than two choices of context variables C.
4 Related Work
Extending the amortization scope Wu et al. (2020) proposed variational inference (VT) algorithm that amortizes over a family of probabilistic generative models. This meta-amortized VI approach learns transferable representations and generalizes to unseen distributions within the amortized family. The dependence on an analytically tractable likelihood function makes this approach inapplicable to iomulation-based inference, while sequential approaches that enable likelihood-free VI (Wiqvist et al., 2021; Glöckler et al., 2022) lack the amortization properties essential for sensitivity analysis (see Table 2). Schröder Macke (2023) perform amortized inference on a set of models by combining model comparison and parameter estimation into a single mixture generative model. On a related note, Dax et al. (2021) avoid training separate neural approximators for gravitational-wan parameter estimation by conditioning detector-noise characteristics. Our SA-ABI method integrates ideas from amortization scope extension in unified framework, enabling amortization over any plausible prior, likelihood, and data configurations while also assessing approximator sensitivity.
Likelihood and prior sensitivity The posterior distribution clearly depends on the likelihood and prior, and a large body of research has studied the sensitivity to both likelihood and prior (for an overview, see Insua & Ruggeri, 2012: Depaoli et al., 2020). In amo-tiort Bayesian inference, several approaches aim to avoid costly model refits by estimating the effects of local likelihood or prior perturbations on a given posterior, for example, via the infinitesimal jackknife (Giordano et al., 2018; 2019) or Pareto-smoothed importance sampling (Kallioinen al., 2021). However,
7

---

Published in Transactions on Machine Learning Research (08/2024)
Table 2: Comparison the suitability of posterior approximation methods for sensitivity analysis in Bayesian infer- ence.
VI
MAVI
SNVI
MCMC
IJ
IS
ABI
SA-ABI
Can handle intractable likelihoods Amortized likelihood & prior sensitivity
X
X
X
X
X
V
X
X
X
/
V
X
Amortized data sensitivity
X
X
X
X
X
Variational Inference; MAVI: Meta-Amortized Variational Inference; SNVI: Sequential Neural Variational Inference; MCMC:
Markov Chain Monte Carlo; IJ: Infinitesimal Jackknife; IS: Importance Sampling; ABI: Amortized Bayesian Inference; SA-ABI: Sensitivity-Aware ABI (ours).
these approaches require an analytically tractable likelihood function and rather expensive refits to evaluate data sen- sitivity. SA-ABI, in contrast, allows for direct assessment of posteriors under different CL and Cp contexts while eliminating likelihood tractability restrictions and the computational burden of refits (see Table 2). Our approach al- lows sensitivity analyses under drastic perturbations, which is not possible via established methods that rely on MCMC and importance sampling (e.g., when the scaling factor y approaches zero; Kallioinen et al., 2021). In Experiment 3, we demonstrate how our method enables prior sensitivity analyses up to a scaling factor of y = 0.1.
Approximator sensitivity Recent work has employed deep ensembles for posterior approximation (Balabanov et al., 2023; Tiulpin & Blaschko, 2022) or, within the scope of simulation- based inference, for improving estimation perfor- mance (Modi al., 2023; Cannon al., 2022), but not for quantifying the sensitivity induced by the approximation procedure. Schmitt et al. (2023) developed a method to detect simulation gaps in amortized Bayesian parameter esti- mation via the resulting sensitivity in both parameter estimation and model comparison based on the variability of ensemble mem- bers. Beyond the identification of simulation gaps, SA-ABI has the crucial advantage of directly assessing the real-life impact of a simulation gap in terms of unreliable approximation.
detection. We adopt a similar perspective simulation gaps but focus on quantifying
Data sensitivity The fact that analyses are sensitive to the input sample (i.e., data sensitivity) has wide-ranging im- plications across the sciences. First, the immoderate influence of single data points is closely related traditional notions of robustness and simulation-based solutions thereof (Huang al., 2023; Ward et al., 2022). Framed as hostile scenario, adversarial attacks intend to exploit data sensitivity (Goodfellow et al., 2015; Biggio et al., 2012: Baruch et al., 2019), and adversarial robustness tries to prevent this (see Glöckler et al., 2023, for an ABI applica- tion). Second, the sensitivity to different preprocessing choices is directly linked to the reproducibility crisis in the empirical sciences (OSC, 2015; Wicherts et al., 2016). To render this sensitivity tangible, Steegen et al. (2016) intro- duced the multiverse analysis, which repeats an analysis across all alternatively processed data sets. The concept of holistic analysis across plausible data configurations has been continually extended but is typically restricted by the computational feasibility of large-scale refits (Hall al., 2022: Liu al., 2020) or specific estimation procedures (Broderick et al., 2023). In summary, our sensitivity-awant method unlocks (i) near-instant analyses of data sensitivity and ad adversarial susceptibility; and (ii) rapid multiverse analyses across a wide space of data processing decisions.
5 Experiments
In the following, we demonstrate the utility of our SA-ABI approach on applied, real-data modeling problems of COVID-19 outbreak dynamics (Experiment 1; prior sensitivity), climate modeling (Experiment 2; prior and like- lihood sensitivity), and human decision-making (Experiment 3; prior, approximator, and data sensitivity). In each experiment, we first ensure the trustworthiness of SA-ABI by benchmarking it against standard ABI as the state-of- the-art approach for amortized inference on simulation models. Afterward, we use our validated approach to obtain insights into sensitivity-indueed uncertainties that would have been hardly feasible with existing methods.
All implementations use the BayesFlow library for amortized Bayesian workflows (Radev et al., 2023b). Details for all experiments, such as model setup, network architecture and training, and additional results are available in the Supplementary Material.
8

---

Published in Transactions on Machine Learning Research (08/2024)
Table Experiment 1: Benchmarking approximation quality and time between standard ABI and SA-ABI (ours).
Simulation budget
MAE J
ECE &
PC t
Time by # of priors
Method
(#: SD)
(# SD)
(# SD)
1
1000
ABI
5.63 + 0.07
0.009 = 0.0001
0.27 = 0.05
2min
6min
1 1876min
2 096
SA-ABI
5.69 + 0.06
0.005 = 0.001
0.28 + 0.01
2min
2min
22min
ABI
5.42 0.04
0.008
0.001
0.38
0.006
6min
17min
5557min
2 16
SA-ABI
5.53 = 0.05
0.006 = 0.002
0.35 = 0.005
6min
6min
26min
1 65 536
ABI
5.37 + 0.005
0.01 - 0.001
0.40 - 0.007
21min
62min
20 721min
SA-ABI
5.44 - 0.006
0.009 = 0.001
0.39 = 0.01
21min
21min
41min
Note. SD Standard Deviation. MAE Mean Absolute Error. ECE Expected Calibration Error. PC Posterior Contraction.
Metrics are evaluated on the prior scaling setting 7 = 1.0 with N = 1 000 held-out data sets and averaged over ensembles of size
M 2 for each method. Thus, SDs reflect the within-ensemble variability. Total times for training and inference for M 1
reported (extrapolated for 1 000 prior sensitivity evaluations).
0.2
Transmission rate (A)
2.6
Number of cases
Reported cases
A ECDF
1700
24
1000
28
25
2.0
75
16.6
125
15.2
155
Fractional rank statistic
Noise dispersion (v)
Days since pandemic onset
(a) Marginal calibration over Cp.
(b) Bivariate posteriors.
(c) Posterior predictives.
0.5
7= 1.0
2.0
Figure 2: Experiment 1. (a) All parameters obtained by SA-ABI are well-calibrated over the full context space Cp. (b) The bivariate posterior of best recoverable parameters À and v indicates substantial sensitivity in terms of uncertainty reduction for vb, but (c) the posterior predictive distribution appears robust (overlapping median prediction lines and 90% Cls).
5.1 Experiment 1: COVID-19 Outbreak Dynamics
We set the stage with straightforward example: Modeling the very early stage of disease outbreak via SIR model (Dehning et al., 2020). We demonstrate that obtaining amortized prior sensitivity insights does not compromise the approximation performance compared to standard ABI with the same fixed simulation budget. We adopt the simulation model from Radev et al. (2021b) and use comparable neural network architecture specialized for time-series data. During training, we use power-scaling, that is, clement-wise exponentiation p(0)7, to amortize over different priors. We sample the scaling factors in log space, ~ exp(U (log(0.5) (0.5), log(2.0))). to ensure equal amounts of widening and shrinking. Thus, the prior context Cp comprises a vector scaling powers for each of the model parameters.
expp(((log(0))))
We first benchmark our SA-ABI approach against individual ABI instances trained solely on the tested baseline setting 7 1.0. This allows to determine the performance trade-off for the amortization scope expansion over Cp. Table 3 shows mostly comparable performance of our SA-ABI approach and individual ABI instances with only little trade- offs across all simulation budget settings, despite SA-ABI spending only fraction of the simulation budget on the tested settings. The small variability within the deep ensembles further indicates approximator robustness in the simulated setting. Lastly, Table 3 highlights the time advantage of our method even for sensitivity analyses that only consider Cp.
Figure 2 shows the prior sensitivity results of our SA-ABI approach for the medium simulation budget of 2 16 384 simulations: Complementing the low calibration error in the benchmark setting without prior scaling, also

---

Published in Transactions on Machine Learning Research (08/2024)
Table 4: Experiment 2: Benchmarking approximation quality and
time between standard ABI and SA-ABI (ours) in a limited data set- ting.
Method
MAE 4 (=: SD)
ECE 4 (= SD)
PC
Time
(#: SD)
ABI
4.2 + 1.1
0.08 + 0.07
0.980 + 0.004
313min
SA-ABI
3.8 + 1.3
0.04 + 0.04
0.982 + 0.015
67min
Note. SD Standard Deviation. MAE Mean Absolute Error. ECE Ex- pected Calibration Error. PC= Posterior Contraction. Metrics are averaged over test data from all emission scenarios climate model settings, resulting in 18 combinations with a total of N 2916 held-out data sets. Thus, SDs reflect the variability of individual results per method and metric. Total times for training and inference reported. All networks use the uninfor- mative prior context.
observe excellent calibration over the full prior space Cp in Figure 2a. The bivariate posteriors for the two parameters with the best recovery (i.e., transmission rate and noise dispersion) unveil that prior sensitivity only affects the noise dispersion (see Figure 2b). Despite prior sensitivity in terms of parameter recoverability, model-based predictive performance is robust to prior scaling (see Figure 2c).
5.2 Experiment 2: Climate Trajectory Forecasting
In this experiment, we study whether model-based global warming forecasts are sensitive the underlying climate model, emission scenario, and prior specification. Climate models estimate the solutions of differential equations for the fluid dynamics and thermodynamics of atmosphere, ocean, ice, and land masses. Since single forecasts can heavily depend on initial conditions, assumed emission scenarios, and the chosen climate model, modern global warming estimates build on multitude of simulated trajectories (Riahi al., 2017: Zelinka et al., 2020; Joshi al., 2011).
However, trajectories simulated from climate models typically start in pre-industrial times, tioned any information since 1850, and are only available in a limited number. In their pioneering work, Diff- enbaugh & Barnes (2023) combine neural networks trained on simulated trajectories with recent observational data to predict global warming trends and forecast when critical thresholds are reached. Here, we demonstrate the utility of our approach for efficiently assessing the sensitivity of model-based predictions in terms of qualitatively different assumptions regarding the underlying models, emission scenarios, and prior distributions.
explicitly condi-
Given high-dimensional spatial observation dataset of surface temperatures (see Figure 3b, right), we are inter- ested in temperature development predictions of different climate models under different future emission scenarios, specifically the time until global mean surface temperature threshold is exceeded. Framed as Bayesian parameter estimation task, we model the -othesh-hee 0 and explicitly incorporate the climate model and an emission sce- nario (i.e.. SSP1-3) in a likelihood context Cr. Furthermore, we encode a weakly informative prior ~ U(-40, 41) that encompasses the full range of values present in the training data vs. an informative Gaussian prior N (10, 10). truncated to positive values based on the IPCC sixth assessment report (Lee et al., 2021), in a prior context Cp (see Figure 3c. During training, the neural approximator learns to infer f from simulated observations with the correspond- ing context. For each training example, extract the ground truth from later stages of the simulated trajectory (see Figure 3b). At inference time, the network processes unseen real observation Wots from the year 2023 with context that specifies an emission scenario, a climate model, and a prior configuration. The output is the contextualized approximate posterior p0 Labs C.).
We reproduce the results of Diffennaugh & Barnes (2023) without sacrificing predictive accuracy (see Supplementary Material) and reveal sensitivity to the climate model. Further, Table highlights the advantages of our joint training method utilizing information from all context configurations via weight sharing, which is especially relevant in the present limited data setting.
10

---

Published in Transactions on Machine Learning Research (08/2024)
Simulation 1 30 years
Simmiation Gn years
Observation 2023 LA years
SSP1-2.6
ACCESS-ESMI4
CanESMS
Sustainable developmeat
MIROC-ES2L
UKBSMI-01 0-L1
SSP2-4.5
ACCESS-ESML-3
CNRM-ESMM-1
Temperature,
[C]]
Middle of
CasESM5
(b) Simulation snapshots (left, center) and the empirical ob- servation (right).
GLSS-E2-1-G
PSSL-CMAA-LR
MIRCC-ES2L,
SSP3-7.0
ACCESS ESM1-5
CESM2
Regional rivalty
SMS
informative Informative prior
GLSS-EZ ASS-E2-1-G
IPSL-CM6ALLR
MIRCC-ES2L
UXESM1-0-LL
2015
2020
2025
2030
2035
2040
2045
50
year of C threshold
0 [yeears]
(a) Sensitivity- aware global warming forecasts.
(c) Prior distributions in Cp.
Figure 3: Experiment 2. (a) Global warming forecasts sensitive to the assumed climate model (rows) but not the emission scenario (SSP; groups of rows) the prior (dotted: weakly informative prior; solid: informative prior). (b) Two examples of simulated observations from the climate model ACCESS-ESMI- (SSP-3) with known time- to-threshold (training data) and the current empirical observation that we use for the forecasts. (c) Prior predictive distributions of the weakly informative prior and the informative prior constituting the prior context Cp.
Finally, study the sensitivity of climate forecasts for reach the 1.5% global warming threshold? As depicted in Figure 3a, the answer to this question is sensitive to the underlying climate model but robust to the assumed emission scenario and informativeness of the prior. This finding is in accordance with Hawkins & Sutton (2009) who argue that the delayed effects of emission scenarios are unlikely to show on a short time scale such as the 1 1.5%C average warming threshold.
question of societal impact: When will we
5.3 Experiment 3: Hierarchical Models of Decision-Making
This experiment extends our method to Bayesian model comparison of complex hicrarchical models with analytically intractable likelihoods. The drift-diffusion model (DDM) is a popular stochastic model of decision-making widely used in cognitive science and neuroscience (Ratcliff et al., 2016). Elsemüller et al. (2023) recently compared four hierarchical models with ABI to test two proposed improvements of the DDM (referred as Mı): First, allowing model parameters to vary between experimental trials (M3 and M4) and second, allowing for evidence accumulation "jumps" via additional parameter (M2 and Ma), which renders the likelihood function intractable.
Elsemüller et al. (2023) found clear evidence for inter-trial variabilities but unclear results concerning the utility of a. In this experiment, we examine the sensitivity of these results to (i) the prior (via inference under 81 power-scaling perturbations of the hierarchical prior on a), (ii) the approximator (via an ensemble of 20 equally configured neural networks), and (iii) the data (via 100 bootstrap samples). Thus, our comprehensive sensitivity analysis based on 162 000 posterior model probabilities that are challenging to recover even once using existing methods. We now use the flexibility of our simulation- based approach investigate the effects of shrinking and widening the hierarchical prior up factor of 10 and thus sample the Cp scaling factors during training from 7 exp(U((og((...) (0.1), log( 10..))). The complexity of amortizing over the prior space is balanced by two aspects: While the scaling only affects the hyperpriors of the additional a parameter in M2 and Mg. scaling up to a factor of 10 leads to much more extreme variations than the usual perturbations in likelihood-based settings of up to factor of (Kallioinen et al., 2021).
As before, we benchmark our SA-ABI approach against individual ABI instances trained solely on the tested baseline setting 7 1.0, that is, without varying Cp during training. Despite amortizing over a wide Cp range, we observe little trade-offs in Table 5, with all ensemble members of both ABI and SA-ABI exhibiting near-perfect performance
11

---

Published in Transactions on Machine Learning Research (08/2024)
Table 5: Experiment Benchmarking approximation quality and time between stan-
dard ABI and SA-ABI (ours) in a model comparison setting.
Method
MAE +
ECE + (+: SD)
Accuracy t (+: SD)
Time by priors +
SD)
1000
ABI
0.012 + 0.01
0.005 + 0.002
0.99 + 0.01
66min 66 349min
SA-ABI
0.017 0.01
0.01 + 0.002
0.985 + 0.01
66min
415min
Note. SD =
Standard Deviation. MAE = Mean Absolute Error. ECE = Expected
Calibration Error. Metrics are evaluated on the prior scaling setting 7 1.0 with
N 8 000 held-out data sets (2 000 per model) and averaged over ensembles of size
M 20 for each method. Thus, SDs reflect the within-ensemble variability. Total
times for training and inference for M 1 reported (extrapolated for 1 000 prior sensitivity evaluations).
Approximator
Approximator & Prior Approximator & Data
Approximator, Prior & Data
Posterior model probability
0.8
0.6
0.2
0.0
M1
M2
M3
M₁
M1
M2
Ms
M₁
M1
M2
M3
M₁
M1
M2
M3
M₁
Figure 4: Experiment 3. Our sensitivity- aware posterior model probabilities indicate substantial approximator sensi- tivity but robustness to additional prior scaling and data perturbations. Dots represent the original results by Elsemüller
et al. (2023).
simulated data. Strikingly, Figure reveals highly inconsistent predictions of the ensemble members on the empirical data (also for ABI, see Supplementary Material). 2 This stark discrepancy implies the presence of a simulation gap. Indeed, contrasting the empirical data with the typical set of model simulations (Nalisnick et al., 2019; Morningstar et al., 2021) in Figure 5a flags the empirical data as ouitt--distiiuution for the deep ensemble. Further, Figure 4 shows comparatively low sensitivity against perturbing the hierarchical prior medians under all perturbations close to the results by Elsemüller al. (2023). Viewing deep ensemble predictions as approximate Bayesian model averaging (Wilson & Izmailov, 2020), we can conclude that the original results hold, but with substantial OOD uncertainty due to the simulation gap. A closer inspection of prior sensitivity in Figure 5b reveals (i) quantitative sensitivity to wide specifications of the hierarchical location Haa which increases under narrow specifications of the hierarchical scale o., and (ii) qualitative sensitivity to settings μa, concerning the model with the highest probability, M3.
the empirical data, with the
6 Conclusion
We proposed SA-ABI, an approach for large-scale sensitivity analyses with a keen emphasis on managing uncertainty in critical, high-impact scenarios. By leveraging amortized inference, our method causes minimal computational over- head during inference and can be directly integrated into software toolkits for amortized Bayesian workflows (such as Radev et al., 2023b; Tejero-Cantero al., 2020). Future work should investigate more efficient approaches quantify approximator sensitivity, with specific attention Bayesian neural networks (Izmailov al., 2021). Ad-
Recall that the approximation targets Bayesian model comparison (categorical) posterior model probahilities, not posterior distributions over parameters. Thus, the variability shown Figure directly reflects sensitivity caused by perturhing the respective model component(s).
12

---

Published in Transactions on Machine Learning Research (08/2024)
Avg.
(MM (x) Avg. KL divergence
Simulations Observed data
0.38
0.51
0.63
0.75
1.54
1.78 2.02 2.26
10.0
Scaling of da
5.6
3.2
1.8
1.0
0.6
0.3
0.2
0.1
100 150
200
230
Negative Log Density
Scaling of ja
Scaling of Hox
(a) Approximator sensitivity
(b) Prior scnsitivity.
Figure Experiment 3. (a) The learned summary statistics the observed data are out-of-distribution (OOD) relative to the typical set of summary statistics for the model simulations (both distributions marginalized over Cp and CA). (b) The posteriors are quantitatively sensitive power-scaling of the prior location Pas indexed by the forMs Ms (left) well the KL divergence between the original results by Elsemüller et al. (2023) vs. scaled model posteriors (right). Notable qualitative sensitivity is present mainly due to different Pax values.
ensemble-averee averrged probability for
ditionally, whereas arbitrary data and approximator configurations can be explored at any time, likelihood and prior configurations have to be integrated into the training process. We believe that transfer learning (Bengio et al., 2009; Zhuang al., 2021) a promising tool resolve this constraint, unlocking further flexibility on all sensitivity facets. We extend neural Bayesian inference (parameter estimation and model comparison) by amortizing over families of probabilistic models, as characterized by context variables C. This drastically expands the amortization scope of the employed neural upproximators and constitutes a major leap towards foundation models for probabilistic (Bayesian) inference. Follow-up research in this direction might further increase the probabilistic model space during the training stage to facilitate near-universal amortized inference with pre-trained neural networks.
Acknowledgments
We thank the anonymous reviewers and the action editor for improving the manuscript with their constructive and thoughtful feedback. Additionally, we thank Noah S. Diffenbaugh and Elizabeth A. Barnes for helping us build upon their pioneering work in Experiment LE was supported by grant from the Deutsche Forschungsgemeinschaatf (DFG, German Research Foundation; GRK 2277) to the research training group Statistical Modeling in Psychology (SMiP) and the Google Cloud Research Credits program with the award GCP19980904, HO was supported by the state of Baden-Wall Württemberg through bwHPC. MS was supported by the Cyber Valley Research Fund (grant number: CyVy- RF-2021-16) and the DFG under Germany's 's Excellence Strategy EXC-2075 390740016 (the Stuttgart Cluster of Excellence SimTech). UK was supported by the Informatics for Life initiative funded by the Klaus Tschira Foundation and the DFG under Germany's ' Excellence Strategy EXC-2181 390900948 (the Heidelberg Cluster of Excellence STRUCTURES).
References
Syed Mumtaz Ali and Samuel D Silvey, A general class of coefficients of divergence of one distribution from another. Journal of the Royal Statistical Society: Series B (Methodological), 28(1):131-142, 1966.
Grace Avecilla, Julie Chuong, Fangfei Li, Gavin Sherlock, David Gresham, and Yoav Ram. Neural networks enable efficient and accurate simulation-bass inference of evolutionary parameters from adaptation dynamics. PLoS Biology, 2005)e30 2022.
13

---

Published in Transactions on Machine Learning Research (08/2024)
Oleksandr Balabanov, Bernhard Mehlig, and Hampus Linander. Bayesian posterior approximation with stochastic ensembles. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2023. doi: 10.1 0t0p9//cwpP522292222...13131 URL
2023. 200.:2277222222...12 01317.
Gilad Baruch, Moran Baruch, and Yoav Goldberg.
little enough: Circumventing defenses for distributed learning. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Asso- ciates, Inc., 2019. URL 4clc59141046ccm e
neurips ctcpips.cccppppe ottpr///rrrrrerrrrrrrer////eer/p www.ff/ppppee//1111.11 pdf.
Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston, Curriculum learning. In Proceedings of the 26th annual international conference on machine learning, pp. 41 41-48, 2009.
Michael Betancourt. Calibrating model- bbased inferences and decisions. arXiv preprint arXiv: 1803,0830, 2018.
Jeff Bezanson, Alan Edelman, Stefan Karpinski, and Viral Shah. Julia: fresh approach to numerical computing. SIAM review, 59(1):65-98, 2017.
Battista Biggio, Blaine Nelson, and Pavel Laskov. Poisoning attacks against support vector machines. In Proceedings of the 29th International Coference on International Conference on Machine Learning, ICML' 12, pp. 1467-1474, Madison, WI, USA, 2012. Omnipress.
Sebastian Bischoff, Alana Darcher, Michael Deistler, Richard Gao, Franziska Gerken, Manuel Gloeckler, Lisa Haxel, Jaivardhan Kapoor, Janne K Lappalainen, Jakob H Macke, al. A practical guide statistical distances for evaluating generative models in science. arXiv preprint 22X3::2403.12636 2024.
Benjamin Rloem-Reddy and Yee Whye Teh. Probabilistic symmetries and invariant neural networks. J. Mach. Learn. Res., 21:90-1, 2020.
Seth Blumberg, Sebastian Funk, and Juliet RC Pulliam. Detecting differential transmissibilities that affect the size of self-limited outbreaks. PLoS puthogens, (0010)e1104455 2014.
Udo Boehm, Jeffrey Annis, Michael Frank, Guy E. Hawkins, Andrew Heathcote, David Kellen, Angelos- Krypotos, Veronika Lerche, Gordon Logan, Thomas J. Palmeri, Don van Ravenzwaaij, Mathieu Servant, Henrik Singmann, Jeffrey J. Starns, Andreas Voss, Thomas Wiecki, Dora Matzke, and Eric-Jan Wagenmakers. Estimat- ing across-trial variability parameters of the diffusion decision model: Expert advice and recommendations. Journal of Mathematical Psychology, 87:46-75, 2018.
Alexander Braumann, Jonas Krampe, Jens-Peter Kreiss, and Efstathios Paparoditis. Estimation of the distribution of the individual reproduction number: The case of the covid-19 pandemic. arXiv preprint arXiv:2101.07919, 2021.
Tamara Broderick, Ryan Giordano, and Rachael Meager. An automatic finite-saplle robustness metric: when can dropping little data make a big difference? arXiv preprint arXiv:2011.14999 2023.
Peul-Christian Bürkner, Maximilian Scholz, and Stefan Radev. Some models are useful, but how do we know which ones? towards a unified bayesian model taxonomy. arXiv preprint (tt.::20090024330 2022.
Patrick Cannon, Daniel Ward, and Sebastian M Schmon. Investigating the impact of model misspecification in neural simulation-based inference. arXiv preprint arXiv:2209.01855 2022.
Jeffrey Chan, Valerio Perrone, Jeffrey Spence, Paul Jenkins, Sara Mathieson, and Yun Song. A likelihood- infer- ence framework for population genetic data using exchangeable neural networks. Advances neural information processing systems, 31, 2018.
Yanzhi Chen, Dinghuai Zhang, Michael Gutmann, Aaron Courville, and Zhanxing Zhu. Neural approximate sufficient statistics for implicit models. arXiv preprint arXiv:2010. 10079, 2020.
Yanzhi Chen, Michael U Gutmann, and Adrian Weller. Is learning summary statistics necessary for likelihood-free inference? In International Conference Machine Learning, pp. 4529 4544. PMLR, 2023.
14

---

Published in Transactions on Machine Learning Research (08/2024)
Kyle Cranmer, Johann Brehmer, and Gilles Louppe. The frontier of simulation- based inference. Proceedings of the National Academy of Sciences, 117(48):30055330066 2020.
Imre Csiszár. Eine informmtiiosstheortisshhe ungleichung und ihre anwendung auf beweis der ergodizitact von markoff- schen ketten. Magyer Tud. Akad. Mat. Kutato Int. Koezl., 8:85-108, 1964.
Maximilian Dax, Stephen R Green, Jonathan Gair, Jakob H Macke, Alessandra Buonanno, and Bernhard Schölkopf. Real-time gravitational wave science with neural posterior estimation. Physical review letters, 127(24):241 103, 2021.
Jonas Dehning, Johannes Zierenberg, F Paul Spitzner, Michael Wibral, Joao Pinheiro Neto, Michael Wilezek, and Viola Priesemann. Inferring change points in the spread of covid-19 reveals the effectiveness of interventions. Science, 36966600 2020.
Charita Dellaporta, Jeremias Knoblauch, Theodoros Damoulas, and François- Xavier Briol. Robust bayesian inference for simulator-based models via the mmd posterior bootstrap. In International Conference on Artificial Intelligence and Statistics, pp. 943-970. PMLR, 2022.
Sarah Depaoli, Sonja D. Winter, and Marieke Visser. The importance of prior sensitivity analysis in bayesian statistics: Demonstrations using an interactive shiny app. Frontiers in Psychology, 11, 2020. doi: 10.3889//psye2222006004555 URL hwwph:tt////e/ //oii. org/10. 300.00338822F2f33333 2020. 608045.
Noah S. Diffenbaugh and Elizabeth A. Barnes. Data-driven predictions of the time remaining until critical global warming thresholds are reached. Proceedings of the National Academy of Sciences, 12066)x2207188120, February 2023. doi: 10.1073/pnas.220718333 1220183120. URL http:///www.assscc/////////11000 2207183120.
Peter Diggle and Richard Gratton. Monte carlo methods of inference for implicit statistical models. Journal of the Royal Statistical Society Series B: Statistical Methodology, 46(2):193-212 1984.
Lasse Elsemüller, Martin Schnuerch, Paul-Christian Bürkner, and Stefan T Radev. A deep learning method for com- paring bayesian hierarchical models. arXiv preprint arXi::200.11883,, 2023.
Stanislav Fort, Huiyi Hu, and Balaji Lakshminarayanan. Deep ensembles: A loss landscape perspective. arXiv preprint arXiv:1912.02757.000 2019.
Andrew Gelman, Jessica Hwang, and Aki Vehtari. Understanding predictive information criteria for bayesian models. Statistics and computing, 24(6))997-1016, 2014.
Ryan Giordano, Tamara Broderick, and Michael I. Jordan. Covariances, robustness and variational bayes. Journal of Machine Learning Research, (9()::191-2009) 2018.
Ryan Giordano, William Stephenson, Runjing Liu, Michael Jordan, and Tamara Broderick. A swiss army infinitesimal jackknife. In Kamalika Chaudhuri and Masashi Sugiyama (eds.), Proceedings of the Twenty-Second International Conference on Artificial Intelligence and Statistics, volume 89 of Proceedings of Machine Learning Research, pp. 1139-1147, PMLR, 2019.
Manuel Glöckler, Michael Deistler, and Jakob H Macke. Variational methods for simulation-based inference. arXiv preprint arXiv:2203.08 /76. 2022.
Manuel Glöckler, Michael Deistler, and Jakob Macke. Adversarial robustness of amortized bayesian inference. arXiv preprint arXiv:2305. 14984, 2023.
Tilmann Gneiting and Adrian E Raftery. Strictly proper scoring rules, prediction, and estimation. Journal of the American statistical Association, 102(477):359-378 2007.
Pedro Gonçalves, Jan-Matthis Lueckmann, Michael Deistler, Marcel Nonnenmacher, Kaan Öcal, Giacomo Bassetto, Chaitanya Chintaluri, William F Podlaski, Sara A Haddad, Tim Vogels, et al. Training deep neural density estimators identify mechanistic models of neural dynamics. Elife, 9:e56261, 2020.
15

---

Published in Transactions on Machine Learning Research (08/2024)
Ian Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In Inter- national Conference on Learning Representations, 2015. URl.tttp://rxiivee httt:::///aaai..oorr//////11111/ 6572.
David Greenberg, Marcel Nonnenmacher, and Jakob Macke. Automatic posterior transformation for likelihood free inference. In International Conference on Machine Learning, pp. 1404-24115 PMLR, 2019.
A Gretton, K. Borgwardt, Malte Rasch, Bernhard Schölkopf, and AJ Smola. A kernel two-sample test. The Journal of Machine Learning Research, 13:723-773, 03 2012.
Brian D. Hall, Yang Liu, Yvonne Jansen, Pierre Dragicevic, Fanny Chevalier, and Matthew Kay. A survey of tasks and visualizations in multiverse analysis reports. Computer Graphics Forum, 41(1):402-4P) 2022, doi: 10.1111/
egf. 14443. URL https: : /tt/ org /10. 11.01111 1111 111/cgf..14444
Ed Hawkins and Rowan Sutton. The Potential to Narrow Uncertainty in Regional Climate Predictions. Bulletin of the American Meteorological Society, 90(8):0095-110 August 2009. ISSN 0003-0007, 1520-0477. doi: 10.11752009AMS2207.1 URL https: /oooorooraaaa ametsoc. orr.oo/////o.... httgg//eww/oourallssss /90/ 8/ 2009bams2607, xml.
Daolang Huang, Ayush Bharti, Amauri Souza, Luigi Acerbi, and Samuel Kaski. Learning robust statistics for simulation- inference under model misspecification. arXiv preprint arXiv:2305.15871, 2023.
David Rios Insua and Fabrizio Ruggeri. Robust Bayesian Analysis. Springer Science Business Media, Berlin Heidelberg, 2012. ISBN 978-1-461-21306-2.
Pavel Izmailov, Sharad Vikram, Matthew D Hoffman, and Andrew Gordon Gordon Wilson. What are bayesian neural network posteriors really like? In International conference on machine learming, pp. 4629-4640, PMLR, 2021.
Xu Ji, Razvan Pascanu, R Devon Hjelm, Balaji Lakshminarayanan, and Andrea Vedaldi. Test sample accuracy scales with training sample density neural networks. Conference on Lifelong Learning Agents, pp. 629-646. PMLR, 2022.
Manoj Joshi, Ed Hawkins, Rowan Sutton, Jason Lowe, and David Frame. Projections of when temperature change will exceed C above pre-industrien levels. Nature Climate Change, (18):40-412, November 2011. ISSN 1758-678X, 1758-6798. doi: 10. 1..1030//ilimtte12200 URL httptttttt..... : /www. . natura.uue.... cww/crt aaw.etttiteetls.eeeeeeeeeeeeeeee /ncl imatel 261.
Noa Kallioinen, Topi Paananen, Paul-Christian Bürkner, and Aki Vehtari. Detecting and diagnosing prior and likeli- hood sensitivity with power-scaling. arXiv preprint 177 2021.
Diederik P. Kingma and Jimmy Lei Ba. Adam: A method for stochastic optimization. 3rd International Conference on Learning Representations, pp. 1-15, 2015.
Daniel Kristanto, Micha Burkhardt, Christiane Thiel, Stefan Debener, Carsten Giessing, and Andrea Hildebrandt.
The multiverse of data preprocessing and analysis in graph-base based fmri: A systematic literature review of analytical choices fed into a decision support tool for informed analysis. bioRxiv, pp. 2024-01, 2024.
Balaji Lakshminarayanan. Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty esti- mation using deep ensembles. Advances in neural information processing systems, 30, 2017.
June-Yr Yi Lee, Jochem Marotzke, Govindasamy Bala, Long Cao, Susanna Corti, John P Dunne, Francois Engelbrecht, Erich Fischer, John C Fyfe, Christopher Jones, et al. Future global climate: scenario-based projections and near- term information. In Climate chunge 2021: The physical science basis. Contribution of working group I to the sixth assessment report of the intergovermmentl panel on climate change, pp. 553-672. Cambridge University Press, 2021.
Veronika Lerche and Andreas Voss. Model complexity in diffusion modeling: Benefits of making the model more parsimonious. Frontiers in Psychology, 7(1324):1-14, 2016.
Friedrich Liese and Igor Vajda. On divergences and informations in statistics and information theory. IEEE Transac- tions Information Theory, 52( 4412, 2006.
16

---

Published in Transactions on Machine Learning Research (08/2024)
Jiashuo Liu, Zheyan Shen, Yue He, Xingxuan Zhang, Renzhe Xu, Han Yu, and Peng Cui. Towards out-of-distribution generalization: A survey. arXiv preprint arXiv:2108, 13624, 2021.
Yang Liu, Tim Althoff, and Jeffrey Heer. Paths explored. paths omitted, paths obscured: Decision points & selective reporting in end-to-end data analysis. In Proceedings of the 2020 CHI Conference on Human Factors Comput- ing Systems. ACM, 2020. doi: 10. 10.1145/33138311333777 3376533.
URL
David MacKay. Information theory, inference and learning algorithms. Cambridge University Press, 2003.
Jean-Michel Marin, Pierre Pudlo, Christian P Robert, and Robin Ryder. Approximate bayesian computational methods. Statistics and computing, 1266)1167 2012.
Chirag Modi, Shivam Pandey, Matthew Ho, ChangHoon Hahn, Bruno Blancard, and Benjamin Wandelt. Sensitivity analysis of simulation-base -based inference for galaxy clustering. arXiv preprint arXiv:2309. 15071, 2023.
Warren Morningstar, Cusuh Ham, Andrew Gallagher, Balaji Lakshminarayanan, Alex Alemi, and Joshua Dillon. Density of states estimation for out of distribution detection. In International Conference on Artificial Intelligence and Statisticx, pp. 3232-3240. PMLR, 2021.
Mahdi Pakdaman Nacini, Gregory Cooper, and Milos Hauskrecht. Obtaining well calibrated probabilities using bayesian binning. In Proceedings of the Twenty-Nint AAAI Conference on Artificial Intelligence, pp. 2901-29900
AAAI Press, 2015.
Eric Nalisnick, Akihiro Matsukawa, Yee Whye Teh, and Balaji Lakshminarayanan. Detecting out-of-distribution inputs to deep generative models using typicality. arXiv preprint arXiv:1906.029:0 2019.
Radford M Neal. Meme using hamiltonian dynamics. Steve Brooks, Andrew Gelman, Galin Jones, and Xiao-Li Meng (eds.), Handbook of markov chain monte carlo, chapter Chapman and Hall/CRC, 2011.
OSC. Estimating the reproducibility of psychological science. Science, 349(6251), 2015. doi: 10. 10.1126/sceence. aac4716. URL
wttpo////....//cciiiooooooooccoe aac4716 Published by the Open Science Collab-
oration (OSC).
Lorenzo Pacchiardi and Ritabrata Dutta. Generalized bayesian likelihood-free inference using scoring rules estimators. arXiv preprint arXiv:2104.03889 2021.
Pierre Pudlo, Jean-Michel Marin, Amaud Estoup. Jean-Marie Cornuet, Mathieu Gautier, and Christian Robert. Reliable abe model choice via random forests. Bioinformatics, 32(6)885,-866, 2016.
Stefan Radev. Deep Learning Architectures for Amortized Bayesian Inference in Cognitive Modeling. PhD thesis, Heidelberg University, 2021.
Stefan T Radev, Ulf K Mertens, Andreas Voss, Lynton Ardizzone, and Ullrich Köthe. Bayesflow: Learning complex stochastic models with invertible neural networks. IEEE transactions on neural networks and learning systems, 2020.
Stefan T Radev, Marco D' Alessandro, Ulf K Mertens, Andreas Voss, Ullrich Köthe, and Paul-Christian Bürkner. Amortized bayesian model comparison with evidential deep learning. IEEE Transactions on Neural Networks and Learning Systems, 2021a.
Stefan T Radev, Frederik Graw, Simiao Chen, Nico T Mutters, Vanessa M Eichel, Till Bärnighausen, and Ullrich Köthe. OutbreakFlow: Model-based Bayesian inference of disease outbreak dynamics with invertible neural net- works and its application to the COVID-19 pandemics in germany. PLoS computational biology, 2021b.
Stefan T. Radev, Marvin Schmitt, Valentin Pratz, Umberto Picchini, Ullrich Köthe, and Paul-Christian Bürkner. JANA: Jointly Amortized Neural Approximation of Complex Bayesian Models. In Robin J. Evans and Ilya Shpitser (eds.), Proceedings of the 39th Conference on Uncertainty in Artificial Intelligence, volume 216 of Proceedings of Machine Learning Research, pp. 1695-1706. PMLR, 2023a.
17

---

Published in Transactions on Machine Learning Research (08/2024)
Stefan T Radev, Marvin Schmitt, Lukas Schumacher, Lasse Elsemüller, Valentin Pratz, Yannik Schälte, Ullrich Köthe, and Paul-Christian Bürkner. Bayesflow: Amortized bayesian workflows with neural networks. arXiv preprint httv:230.110113. 2023b.
Roger Ratcliff. A theory of memory retrieval. Psychological review, 85(2):59, 1978.
Roger Ratcliff, Philip Smith, Scott D Brown, and Gail McKoon. Diffusion decision model: Current issues and history. Trends in cognitive sciences, 20(4):260-281, 2016.
Keywan Riahi, Detlef P. Van Vuuren, Elmar Kriegler, Jae Edmonds, Brian C. O'Neill, Shinichiro Fujimori, Nico Bauer, Katherine Calvin, Rob Dellink, Oliver Fricko, Wolfgang Lutz, Alexander Popp. Jesus Crespo Cuaresma, Samir Kc, Marian Leimbach, Leiwen Jiang, Tom Kram, Shilpa Rao, Johannes Emmerling, Kristie Ebi, Tomoko Hasegawa, Petr Havlik, Florian Humpenöder, Lara Aleluia Da Silva, Steve Smith, Elke Stehfest, Valentina Bosetti, Jiyong Bom, David Germatt, Toshihiko Masui, Joeri Rogelj, Jessica Strefler, Laurent Drouet, Volker Krey, Gun- nar Luderer, Mathijs Harmsen, Kiyoshi Takahashi, Lavinia Baumstark, Jonathan C. Doelman, Mikiko Kainuma, Zbigniew Klimont, Giacomo Marangoni, Hermann Lotze-Campen, Michael Obersteiner, Andrzej Tabeau, and Massimo Tavoni. The Shared Socioeconomic Pathways and their energy, land use, and greenhouse gas emis- sions implications: An overview. Global Environmental Change, 42:153-168, January 2017. ISSN 09593780. doi: 10. gloe:nii.gg. 01959378016300655
URL https://linkinghub el sevier. com/retrieve/ppi./
Malgorzata Roos, Thiago G Martins, Leonhard Held, and Havard Rue. Sensitivity analysis for bayesian hierarchical models. Bayesian Analysis, 10(2):321-349, 2015.
Marvin Schmitt, Paul-Christian Bürkner, and Köthe. Detecting model misspecification amortized bayesian inference with neural networks. Proceedings of the German Conference on Pattern Recognition (GCPR), 2023.
Comeelis Schröder and Jakob H Macke. innalttancuusddenffiiio of models and parameters of scientific simulators. arXiv preprint 2.0.::::00.111744 2023.
Adi Shamir, Odelia Melamed, and Oriel BenShmuel. The dimpled manifold model of adversarial examples in machine learning. arXiv preprint arXiv:2 10151, 2021.
Joseph P. Simmons, Leif D. Nelson, and Uri Simonsohn. False-positive psychology. ence, 22(11):1359-1366, 2011. 0 0956797611417622
Psychological Sci-
doi: 10.117709557996114476622. URL
Sara Steegen, Francis Tuerlinckx, Andrew Gelman, and Wolf Vanpaemel. Increasing transparency through multi- verse analysis. Perspectives on Psychological Science, 11(5):702-712, 2016. doi: 10.1177/745991666686777 URL httttpsss/ddi..or///....
6659637.
Sean Talts, Michael Betancourt, Daniel Simpson, Aki Vehtari, and Andrew Gelman. Validating bayesian inference algorithms with simulation-basse calibration. arXiv preprint arXiv:1804.06788 2018
Alvaro Tejero-Cantert Jan Boelts, Michael Deistler, Jan-Matthis Lueckmann, Conor Durkan, Pedro J Gonçalves, David S Greenberg, and Jakob H Macke. Sbi-a toolkit for simulation-based inference. arXiv preprint arXiv:2007.091 2020.
Aleksei Tiulpin and Matthew B. Blaschko. Greedy bayesian posterior approximation with deep ensembles. Trans- actions on Machine Learning Research, 2022. ISSN 2835-8856. httthttts//:/seeeeeee wwt.:eeettfrrrtt id=PIDuPJZVIN.
Aki Vehtari, Andrew Gelman, and Jonah Gabry. Practical bayesian model evaluation using leave-one-out cross- validation and waic. Stutistics and computing, 27:1413-1432, 2017.
Andreas Voss, Veronika Lerche, Ulf Mertens, and Jochen Voss. Sequential sampling models with variable boundaries
and normal noise: A comparison of six models. Psychonomic bulletin & review, 26(3):813-823 2019.
Eric-Jan Wagenmakers, Alexandra Sarafoglou, and Balazs Aczel. One statistical analysis must not rule them all. Nature, 605(7910):4233 425, 2022.
18

---

Published in Transactions on Machine Learning Research (08/2024)
Daniel Ward, Patrick Cannon, Mark Beaumont, Matteo Fasiolo, and Sebastian Schmon. Robust neural posterior estimation and statistical model criticism. Advances in Neural Information Processing Systems, 35:33845-338857 2022.
Florian Wenzel, Jasper Snoek, Dustin Tran, and Rodolphe Jenatton. Hyperparameter ensembles for robustness and uncertainty quantification. Advances in Neural Information Processing Systems, 33:6514 6527, 2020.
Jelte M. Wicherts, Coosje L. Veldkamp, Hilde E. M. Augusteijn, Marjan Bakker, Robbie C. M. van Aert, and Marcel A. L. M. van Assen. Degrees of freedom in planning, running, analyzing, and reporting psychological studies: A checklist to avoid p-hacking. Frontiers in Psychology, 7, 2016. doi: 10.3389ffssy.2226601182 URL
/etps://dci... htgp://o..orrrg/// 10. 3 Epsyg. 2016 6. 01832.
Eva Marie Wieschen, Andreas Voss, and Stefan Radev. Jumping to conclusion? a lévy flight model of decision making,
The Quantitative Methods for Psychology, 16(2):120- 132, 2020.
Andrew G Wilson and Pavel Izmailov. Bayesian deep learning and a probabilistic perspective of generalization. Advances in neural information processing systems, 33:4697-4708,, 2020.
Samuel Wiqvist, Pierre-Alexanrr Mattei, Umberto Picchini, and Jes Frellsen. Partially exchangeable networks and architectures for learning summary statistics in approximate bayesian computation. In International Conference Machine Learning, pp. 6798-6807. PMLR, 2019.
Samuel Wigvist, Jes Frellsen, and Umberto Picchini. Sequential neural posterior and likelihood approximation. arXiv preprint arXiv:2/02.06522 2021.
Mike Wu, Kristy Choi, Noah Goodman, and Stefano Ermon. Meta-amortized variational inference and learning. Proceedings of the AAAI Conference on Artificial Intelligence, 34(04)6404-6412, 2020. doi: 10. 10.160//aaii, v34i04 6111. URL https: doi wot. 609 aaai. v34104. 6111.
Jingkang Yang, Pengyun Wang, Dejian Zou, Zitang Zhou, Kunyuan Ding, Wenxuan Peng, Haoqi Wang, Guangyao Chen, Bo Li, Yiyou Sun, et al. Openood: Benchmarking generalized out-of-distribution detection. Advances in Neural Information Processing Systems, 35:32598-322 2022.
Yuling Yao, Bruno Régaldo-Sain Blancard, and Justin Domke. Simulation based stacking, 2023.
Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poczos, Russ R Salakhutdinov, and Alexander Smola. Deep sets. Advances in neural information processing systems, 30, 2017.
Mark D. Zelinka, Timothy A. Myers, Daniel T. McCoy, Stephen Po-Chedley, Peter M. Caldwell, Paulo Ceppi, Stephen A. Klein, and Karl Taylor. Causes Higher Climate Sensitivity CMIP6 Models. Geophys- ical Research Letters, 470)x220196. 2020. ISSN 1944-8007. doi: 00.10992019LL.88782... URL https://onl ne meeibbrrryy... wi ley. com/ doi /eci/aaa /abs/ 0 1009/20092290 9GL085782. 952001222111111...55...2
Fuzhen Zhuang, Zhiyuan Qi, Keyu Duan, Dongbo Xi, Yongchun Zhu, Hengshu Zhu, Hui Xiong, and Qing He. A comprehensive survey on transfer learning. Proceedings of the IEEE, 109(1):43-76, 2021. doi: 10.1109/jproc. 2020.3004555. URL https:tppp/..o //t/////ii/////oo wwrdddi.orgg..... 110.1110///00 1119///rooccco../ooooo
19

---

Published in Transactions on Machine Learning Research (08/2024)
Supplementary Material
A Frequently Asked Questions (FAQ)
Q: How can reproduce the results?
Code for reproducing all results from this paper is freely available httpp:///gghhib...mmaaeeeett AB..
Q: Can apply your sensitivity-awar approach to posterior predictive model comparison as well?
Yes! In this work, we focus on prior predictive model comparison, but our ideas directly apply to posterior predictive metrics (Gelman al., 2014), such as leave-one-out cross-validation (Vehtari al., 2017). We recommend the joint usage of a posterior and a likelihood network as proposed in Radev et al. (2023a) to achieve amortization in this application.
Q: Are there limits to the distributional shapes that can be explored in C₁. and Cp?
The only requirement for distributions in the likelihood and prior context is being able to simulate data from the resulting model. Besides that, SA-ABI gives modelers full flexibility in specifying any theoretically meaningful alternative formulations without concerns about analytically tractable likelihoods, unlike MCMC methods.
Q: How exactly did you encode the context variables Cr. and Cp in suitable format for the neural network in your experiments?
In Experiment and Experiment 3, each prior distribution over a parameter is continuously tempered by power- scaling. Consequently, Cp is encoded by vector holding the scaling factors for each prior component. In Experiment 2. both Cg and Cp consist of discrete choices and are therefore passed to the inference network in one-hot-encoded vectors.
B Methods: Additional Details
B.1 Hypothesis Testing for Quantitative Sensitivity
We can employ a sampling-base based (frequentist) hypothesis test to determine the probability of observed (D.II))) esti- mates (hitherto referred to as D) under the null hypothesis of zero difference between p(0 xx, C+) and p(0 x Cý). For this, mutation tests based on multiple draws from p(0 æ, C.). Based the approximate sampling distribution, then obtain a critical D value for fixed Type error probability and compare the observed one. The power of such test will generally be high when having access to many draws from p(0 x. Cx) and p(0 x, Cý), a requirement that is easily met in the context of ABI.
construct an approximate sampling distribution of fir under the null hypothesis via bootstrap per-
C Experiments: Implementation Details and Additional Results
C.1 Benchmarking Metrics
For the benchmarks conducted in Experiment 1 and Experiment 2, we measure three complementary performance metrics on unseen test data sets (D! دبر with known ground-truth parameters {o!²) J J==1- For each data set D!"), ) of posterior draws from the neural approximator 90 "D))). We summarize each metric into single measure across data and parameters for given neural approximator and setting (e.g..
we obtain a set {
7 0.5 in Experiment 1).
20

---

Published in Transactions on Machine Learning Research (08/2024)
We use the Mean Absolute Error (MAE) to measure the overall error between posterior draws 0 and ground-truth parameters 0
J
s 1
MAE = j
_ Σ
(00h) (o
o
(13)
Further, assess uncertainty calibration via the Expected Calibration Error (ECE). In Bayesian parameter estimation, all uncertainty regions (6 D) of the true posterior p(0 E (0, 1) (Bürkner et al., 2022), such that:
D) are by definition well-calibrated for any quantile
/ʃ 1 E ( P(D 0.) (00.)dd0,
-=
(14)
with [[-] denoting the indicator function. Simulation-baxe calibration (SBC; Talts et al., 2018) measures miscalibra- tion via deviations from this equality. We estimate the ECE via the median SBC error of 20 linearly spaced credible intervals with quantiles of q E [0.5%, 99.5%). Lastly, we measure Bayesian information gain via the median /ooostrroor//arrPrriii (Betancourt, 2018).
Posterior Contraction (PC) across data sets, defined as 1
C.2 Experiment 1: COVID-19 Outbreak Dynamics
Model Setup: We consider simple SIR model where individuals are either susceptible, S, infected, I, or recovered, R. Both infection and recovery are modeled with a constant transmission rate À and recovery rate μ4, respectively. The model is described by a system of ordinary differential equations (ODEs),
dS dt = A
S1 N
(15)
dl = X
ST
- I,
(16)
dt N
dR = II, dt
(17)
with N S + R denoting the total population size. In addition to the ODE parameters À and μa our model includes a reporting delay parameter and noise dispersion parameter th which jointly influence the (noisy) number of reported infected individuals via
flob)) NegBinomial( (T(mor)) 1b),
((eew)
(18)
with [new)) = X(S) The negative binomial distribution allows for modeling dispersion, i.e., variation of the variability independent of the mean, which is considered likely for early phases of the COVID-19 pandemic (Blumberg et al., 2014; Braumann al., 2021). For our implementation, we transform the parameterization in Equation 18 with mean /! ) and dispersion 1 to the numpy library's implementation of the negative binomial distribution with number of successes n and probability of success pt
n +
(19)
(new) +
(20)
1b( (1((er)) )
(new)
1-D
p σ
(21)
The fifth estimated model parameter the initial number of infected individuals lo-
We use the same prior specification as Radev et al. (2021b), which is displayed in Table 6 along with the respective power-scaling scheme. Figure 6 shows the behavior of the prior predictive distributions under different power-scaling values 7.
In Experiment 3, we use the RCF formulation Nacini et al. (2015) for probabilistic classification.
21

---

Published in Transactions on Machine Learning Research (08/2024)
y= 0.5
0.3
100
230
300
20
45
80
V 1.0
15
100
230
300
40
60
Y = 2.0
10
60
Figure 6: Experiment 1. Prior predictive distributions under different scaling parameters 7.
Table 6: Experiment 1. Power-scaled prior distributions for all parameters.
Parameter
Symbol
Power-scaled prior distribution
Transmission rate
LogNormal ( (0.4), /n)
Recovery rate of infected individuals
LogNormal (log (1/8), 0.2/ / 72)
Reporting delay (lag)
D
LogNormal (8), 0.2/ 73)
Initial number of infected individuals
To
Gamma(27 74 1, 20/74)
Noise dispersion
(5ponential55/7s)
Note. Our parameterization follows the numpy library's implementation of the respective distribution.
We use time-series data from the first weeks of the COVID-19 pandemic in Germany provided by the Center for Systems Science and Engineering (CSSE) at Johns Hopkins University, licensed under CC BY 4.0. 4
Neural Network and Training: Our neural network architecture follows a simplified version of the design imple- mented by Radev et al. (2021b): We use a recurrent network with gated recurrent units as summary network and a conditional invertible network as inference network.
All computations for this experiment were performed on a single-GPU machine with an NVIDIA RTX 3070 graphics card and AMD Ryzen 5 5600X processor. Simulating 16 384 training data sets took 8 seconds and subsequent offline training for 75 epochs took 6 minutes.
Additional Results: We further investigate SA-ABI and ABI for potential reductions in approximation performance due to amortized prior sensitivity in the medium simulation budget setting of 2 = 16 384. Figure 7 and Figure 8 show similar parameter recovery in the baseline 7 = setting, both limited by the small number T data points available. Figure and Figure 10 demonstrate that calibrated predictions are nevertheless mostly achievable, with equal patterns between SA-ABI (Figure 9) and standard ABI (Figure 10) for the baseline setting. Figure 11 additionally contains MMD hypothesis tests that clearly show sensitivity of the parameter posterior to the prior specification.
* //aps///rww..ee http://wwwwiiuusserrerrr
com http://SSSaaaaaaaaaaoooa SandDatadddaaaae SoaddDaatCCoVTD---- http:///o--////aee//////aae// /tt////er/csseeeeoe///// aa_ddaaa/sssssss a
d 19 me series/t ime_sss series_covv __con prrd__glbba cevv
22

---

Published in Transactions on Machine Learning Research (08/2024)
A 0,964
8 0.041
A 0.091
R3 0.327
R? 0.625
0,972
== 0.204
0.347
= 0.505
0.803
Estimated
Ground teuth
Ground tzuth
Ground truth
Ground teath
Ground teath
Figure 7: Experiment Simulation-based recovery of the context-aware neural approximator used in experiment for y 1.0 (simulation budget of 2 16 384).
H
A² 0.944
0. 0.036
0.110
R² 0:326
R² 0.620
0,372
0.189
0.358
0.584
0.802
-2
Ground truth
Ground truth
Ground truth
Ground truth
Ground truth
Figure 8: Experiment 1. Simulation-bass parameter recovery single-prior neural approximator trained with the same configuration and simulation budget (214 16384) as our experiment without Cp (i.e., on the baseline 7 1.0 setting).
Benchmark Details: All results of Experiment except the benchmark operate in standardized parameter space to align the different parameter scales. To eliminate the influence of standardization mismatches across the tested settings, the networks trained for the benchmark use the original (unstandardized) parameters. We further employ ensembles size = to check for potential approximator sensitivity affecting the stability the benchmarking results and observe stable results within the ensemble members (i.e., no approximator sensitivity). All networks are trained for 75 epochs.
23

---

Published in Transactions on Machine Learning Research (08/2024)
D
10
A ECDF
0.05
0.00
-0.05
0.0
0.5
1.0
0.0
1.0 0.0 Fractional rank statistic
1.0
0.5
1.0
0.0
0.5
1.0
10
Figure 9: Experiment 1. Simulation-basee calibration of the context-ane neural approximator used in our experi- ment for contexts y E {0.5, 1.0, 2.0} (simulation budget of 2 16 384).
D
1o
w
0.05
A ECDF
0.00
-0.05
0.0
0.5
1.0
0.0
0.5
1.0 0.0
0.5
1.0 0.0
0.5
1.0
0.0
0.5
1.0
Fractional rank statistic
Figure 10: Experiment 1. Simulation-basee calibration of a single-prior neural approximator trained with the same configuration and simulation budget (214 setting).
16 384) as our experiment but without Cp (i.e., on the baseline y 1.0
0.025 0.050 0.075 0.100 0.125 0.150 0.175
0.0
0.1
0.2
0.3
0.4
MMD
MMD
(a) 7 0.5 (red) vs. 7 1.0 (blue).
(b) 7 2.0 (red) vs 7 1.0 (bluc).
Figure 11: Experiment MMD hypothesis tests confirm sensitivity prior specification in the parameter space. The blue density represents samples under the null distribution of zero difference between 7 1.0 and the scaled posteriors (red; 7 0.5 or 7 2.0.), the yellow lines mark the area with 5 = 5% rejection probability.
24

---

Published in Transactions on Machine Learning Research (08/2024)
C.3 Experiment 2: Climate Trajectory Forecasting
Climate models (CM) are typically differential equations describing all relevant components of the Earth system and their couplings. Socioeconomic pathways (SSPs) provide comprehensive scenarios of future developments by describing qualitatively different trajectories of future emissions. Following standard practice, we append the associated radiative forcing in the year 2100 to the SSP identifier (e.g., SSP1-2.6 refers to the SSP1 scenario with radiative forcing of 2.6W/m²). Given CM, an SSP, and initial condition (IC), high-dimensional trajectories of many observables can be simulated from the underlying model. Here we restrict the analysis to the air temperature at the surface (TAS).
Data:
Simulations produce trajectories TAS SSP. SSP, IC (ž, t) where i is a spatial coordinate on the Earth's surface and is time between 1850 and 2100. From this, the area-weighted global mean surface air temperature (GSAT) can be computed, which is the key indicator global warming over pre-industrial levels.
All data used in this experiment is freely available to download: For the climate model simulation outputs, we use data from the Earth System Grid Federation. For the observational data set for 2022, we use data from Berkeley Earth, licensed under CC BY 4.0.6 We include all climate models that have archived at least 10 trajectories for the future emission scenarios SSPI, SSP2, and SSP3 in our analysis. We reshape all data to a 2.5 x 2.5 longitude-latitude grid and compute yearly differences to the baseline period (1951 to 1980). The warming threshold is defined relative to the pre-industrial period 1850 1900, from which we can directly obtain the -ttreo--hreshold parameter for each tuple of year, model, and trajectory ensemble.
Model Setup: Our model setup focuses on the time 0 until the 1.5%C warming threshold is reached, but can easily be adapted to arbitrary temperature thresholds. We realize a prior context Cp for 0 by two discrete prior choices: First, a weakly informative prior, 0 ~ (1-40, 41), that encompasses the full range of values present in the training data of simulated climate warming trajectories. Second, more informative Gaussian prior, A (10, 10), truncated to include only positive values. This prior is based the IPCC sixth assessment report stating that the central estimate of crossing the C. threshold lies in the early 2030s (Lee et al., 2021).
Table 7: Overview of the climate models included in each SSP emission scenario.
Climate Models
SSP1-2.6 SSP2-4.5
SSP3-7.0
ACCESS-ESMIS
CanESM5
CESM2
CNRM-ESM2-1
GISS-E2-I-G
IPSL-CM6A-LR
MIROC-ES21.
MIROC6
UKESM1-O-LL
The likelihood is obtained from the simulated trajectories of the climate models. For a given time-to- tot--to-treshold 0 and climate model (encoded in the likelihood context Cz), trajectories of the respective climate model are selected in ensembles of 10. We first identify the year of threshold exceedance of the mean global surface temperature across trajectory ensemble. Afterwards, random ensemble and trajectory are chosen. Finally, the likelihood algo- rithm returns the spatial temperature pattern that 0 years prior to the year of threshold exceedance the simulated trajectory.
/nttps://esgt.. http:///eg...lll....... www...llooo///
onttps://ber www.e/eerbbeeeeeebbbeeeeee...a.. org.or////o data/
"Averaging over trajectory ensembles smoothes the chaotic internal variability obtain stable estimate. In contrast, the IPCC defines year of threshold exceedance as the middle of 20-year averaging window. Diffenhaugh Barnes (2023) show that resulting forecasts insensitive the chosen definition.
25

---

Published in Transactions on Machine Learning Research (08/2024)
Neural Network and Training: We z-standardize data and parameters before passing them to the neural approx- imator. As summary network, we use a dense network that parallels the architecture used in Diffenbaugh & Barnes (2023): Inputs come in the form of 72x144 lemperature grids and are flattened. Two hidden layers of 25 units with ReLU activation are followed by output units of learned summary statistics. During training, we additionally em- play dropout regularization with dropout probability 0.4 on the initial layer of the summary network to mitigate overfitting. As inference network, we use conditional invertible neural network. Since normalizing flows require more than one dimension, we add dummy standard normal parameter.
Training the neural approximator took approximately 70 minutes consumer notebook with 6-core AMD Ryzen 5 5625U CPU and without dedicated graphics card, underscoring the wide applicability of our method.
For the individual ABI instances trained for the benchmark, each instance was trained on data of a specific scenario x climate model setting. We used the same network architecture for separate and joint training to enhance comparability, but note that the hyperparameters may not be optimal for the respective data size settings. Joint training was conducted 80 epochs, whereas chose smaller number of 15 epochs for the separate training mitigate overfitting.
Additional Results: To validate our approach, we compute the mean absolute error (MAE) of the point estimate (posterior mean) 9 to the true value 8* on a held-out validation set. Figure 12a shows good recovery across climate models, true time-to-threshold 0", and SSPs. To parallel the evaluation procedure of Diffenbaugh & Barnes (2023), who did not differentiate between the climate models, we use a flat prior via the prior context Op and marginal- ize predictions over all climate models contained in C₁.- Figure 12b shows that our sensitivity-aware approach does not sacrifice predictive accuracy and further enables the identification of biased estimates for MIROC6 in the SSPI- 2.6 scenario as the main limitation to better performance. Figure 13 provides an additional perspective via standard simulation-based recovery and calibration plots. Overall recovery is high, but calibration plots indicate notable under- confidence of the posteriors, implying that the networks systematically overestimate the variance of th-reshldd 0 predictions. We hypothesize that this is due underperforming summary network which nevertheless keep the same as in Diffenbaugh & Barnes (2023) for the sake of comparability.
26

---

Published in Transactions on Machine Learning Research (08/2024)
SSP1-2.6
SSP2-4.5
SSP3-7.0
Climate Model
ACCESS-ESMIL
CESM2
1.5cc
CNRM-ESM2-1
CanESM5
GISS-E2-1-G
IPSL-CM6A-LR
-40
MAE: 3.7
MAE: 2.6
MAE: 2.2
MIROC-ES2L
MIROC6
40
0
40
-40
40
-40
0
40
UKESM1 -O-LL
0
1.5+c
0
0
0
1.5
(a) Weakly informative prior. Recovery with Ct information about the respective climate model given is good (total MAE of 2.0) across climate models, true time-to-threshole 8*, and SSPs.
SSP1-2.6
SSP2-4.5
SSP3-7.0
Climate Model
ACCESS-ESM1-5
CESM2
CNRM-ESM2-
01.5
CanESM5
GISS-E2-1 E2-1-2-1-G
IPSL-CM6A-LA
MAE:
MAE: 3.9
MAE:
MIROC-ES2L
MIROC6
40
40
-40
40
-40
0
40
UKESM1-0-LA
01's C
0
1..5
0
(b) Weakly informative prior. Recovery without Cl information about the respective climate model given all predic- tions are not only obtained for appropriate climate model, but all climate models contained C2. afterwards averaged, This leads validation setup comparable to Diffenhaugh & Barnes (2023). The mean absolute error (MAE) 3.0 years for SSP3-7.0 indicates our approach does not sacrifice predictive accuracy in comparison Diffenbaugh & Barnes (2023), who report MAE between 2.7 and 3.8 years for the same task (eyeballed values from boxplots reported the appendix).
SSP1-2.6
SSP2-4.5
SSP3-7.0
Climate Model
40
ACCESS-ESME -SM--5
CESM2
01.5cC
CNRM BURM-ESM2-1
CanESM5
0
GISS-E2-1-G
IPSL-CM6A
40
MAE: 2.2
MAE: 2.0
MAE: 2.1
MIROC-ES2L.
MIROC6
-40
-4
-40
40
UKESM1 O--LL
01's'c
01.0a c
01.5aC
(c) Informative prior. Recovery with Cz information about the respective climate model given, here restricted positive 0 values due to the truncation of the prior. Recovery is good with a total MAE of 1.2.
Figure 12: Experiment 2. Recovery of time-to-threshold on held-out validation data.
27

---

Published in Transactions on Machine Learning Research (08/2024)
40
0.974 O, 1987
6.2
R 0,877
0.939
3.03
Estimatedd
D03F differanos
6:08
Estimateed
DC: aittermoe
2.81
0.02
-10
8355 Carffderce Bands
Rack ECDIF
-280
50% Confifence Bards
Rank ECDE
64
6.5
Fractional rank stanisesc
Practional rank statiatc
(a) Weakly informative prior.
(b) Informative prior.
Figure 13: Experiment 2. Standard simulation- based metrics of recovery and calibration on held-out validation data for both prior contexts Cp. All results are marginalized over the likelihood context space C2 (i.e., climate models and emission scenarios).
28

---

Published in Transactions on Machine Learning Research (08/2024)
C.4 Experiment 3: Comparing Hierarchical Models of Decision-Making
Model Sctup: The drift-diffusion model (DDM; Ratcliff et al., 2016; Ratcliff, 1978) models binary decision out- comes and their associated response times with the following stochastic ordinary differential equation:
dæ = dt + gydd Ç ~~
(22)
(23)
with dæ denoting the change in evidence accumulation, v denoting the rate of evidence accumulation, and & the noise of evidence accumulation. Additional parameters of the model include the non-decision time ty (c.g., encoding and motor response), the decision threshold a, and the bias towards decision option Zy-
The Lévy flight model (LFM; Voss al., 2019) relaxes the Gaussian noise assumption by using the more general c-stable distribution, leading to unknown analytical form of the likelihood:
dæ u dt + odg
(24)
ξ ~~ AlphaStable(a 0, = -/1 8 =0,
(25)
V2
with the additional stability parameter a which shall also be estimated.
Additionally, there is a debate about whether the model parameters v, Zr, and to should have fixed value over the course of an experiment (basic models) or be allowed to vary (full models) throughout the experiment (Lerche & Voss, 2016; Boehm et al., 2018). Therefore, we compare the following four models in this experiment:
Basic DDM (M1): The classic cour-parameter formulation with the model parameters to. a, and Zy-
Basic LFM (M2): Equals the basic DDM plus the additional stability parameter a controlling the tail behav- ior of the noise distribution.
Full DDM (M3): Equals the basic DDM plus inter-trial variability parameters Su, Sto and 8g,.-
Full LFM (M4): Equals the full DDM plus the additional stability parameter a.
As in Elsemüiller et al. (2023), we reanalyze data by Wieschen et al. (2020) (provided by the original authors) con- taining 40 participants with 900 decision trials each and assume uniform model prior (i.e., equal prior model proba- bilities). We use the same hierarchical priors as Elsemülller al. (2023), who provided a detailed table with all prior choices. Since they are central to our experiment, we reiterate the priors leading to om for each participant m here:
Na ((1.65,0.15/1)
a (0.3,
(26)
Q'm -Trrnnatee Ja, 1,2) for m 1,..,,,.. M.
Our experiment investigates the sensitivity to different Cp. CA. and Cp realizations. For Cr, we power-scale the hierarchical prior a with a wide range of y E [0.1, 10), which would have been infeasible with existing methods since they require values close to the mid-range value of no scaling, i.e., y 1 (Kallioinen al., 2021). To ensure equal amounts of widening and shrinking of the respective distributions during training, draw the scaling factors from independent uniform distributions in the log space Y exp((((log(O.)) log(10))). Figure 14a displays the prior predictive distribution of under different 7 configurations. We use 100 bootstrap samples on the trial level (i.c., within the observations of each participant) for the data context Cp and describe the details of the approximator context CA in the following section.
Neural Network and Training: All 20 members the employed deep ensemble constituting CA are set up and trained independently and identically. Each network uses hierarchical summary network consisting of two permuta- tion invariant deep set networks (Zaheer al., 2017) and standard feedforward network with a softmax output layer inference network.
29

---

Published in Transactions on Machine Learning Research (08/2024)
Simulations tall models) Empirical
10.0
Scaling of
10
0.1
1.0 1.3 150
175
232
100 135 LOD 175 242
101 125 L11 135 2.06
Response (seconds)
Scaling of P.
(b) Prior pushforward distributions of simulated response times per person contrasted with the empirical distributions per per- son. Negative response times indicate a decision for the lower decision boundary and positive times for the upper decision boundary.
(a) Prior predictive distributions of the a parameter under power-scaling. The yellow line represents the marginal den- sity for each configuration.
Figure 14: Experiment 3. Prior distributions. a) Prior predictive distributions of the parameter under maximum widening (y = 0.1), no scaling (y = 1), and maximum shrinkage (y = 10) of the hyperpriors Pox and d (b) Prior pushforward distributions of the response times simulated by the four compared models (marginalized over y) and the empirical response times.
As in Elsemüller et al. (2023), first pre-train each network smaller data sets of 40 simulated participants with 100 observations each, and afterwards fine-tune on the full data size of 40 simulated participants with 900 observations. We use 30 epochs for both phases and an Adam optimizer (Kingma & Ba, 2015) with cosine decay schedule (initial learning rates of 5 x 10-4 for pre-training and 5 x 10 for fine-tuning).
All computations for this experiment were performed on a single-GPU machine with an NVIDIA RTX 3070 graphics card and an AMD Ryzen 5 5600X processor. Simulating 40 000 pre-training and 8 500 fine-tuning data sets the Julia programming language (Bezanson al., 2017) took 23 minutes. Training the deep ensemble took 21 minutes for pre-traine pre-training and 45 minutes for fine-tuning per network.
Additional Results: Figure 14b displays the prior pushforward distribution of simulated response times and the empirical response times distributions. The informative priors from Elsemüller et al. (2023) assign high densities to the central regions of the empirical distribution. Nevertheless, the results of the typical set approach (Nalisnick et al., 2019; Morningstar et al., 2021) in Figure 15 flag the empirical data as -ut-of-distributiin of each deep ensemble member.
To ensure that the inclusion of Cp in the amortization scope does not lead substantially worsened approximation performance, we trained an additional deep ensemble without Cp. Table 8 displays validation performance and empirical approximations for the ensemble including Cp and Table for the ensemble without Cp. Including Cp does neither lead to substantial drop in performance nor qualitatively different model comparison results despite power-scaling over a wide range.
30

---

Published in Transactions on Machine Learning Research (08/2024)
Network 1
Network 2
Network 3
Network 4
Network 5
100
150
100
200
100
200
100
200
100
150
200
Network 6
Network
Network
Network
Network 10
100
150
100
150
100
150
100
100
150
Network 11
Network 12
Network 13
Network 14
Network 15
100
200
100
200
100
150
100
150
100
150
200
Network 16
Network 17
Network 18
Network 19
Network 20
100
150
200
150
200
100
150
200
100
200
100
200
Negative Log Density
Simulations' Entropy
Observed data
Simulations
Figure 15: Experiment 3. The typical set of learned summary statistics from the model simulations contrasted with the density of the learned summary statistics from the observed data (both distributions are marginalized over Cp but separated for each ensemble member of Ca). The empirical data is flagged as out-of out-of-distribution for each ensemble member.
31

---

Published in Transactions on Machine Learning Research (08/2024)
Table 8: Experiment 3. SA-ABI validation performance on 8 000 held-out simulations and predictions on the empir-
ical data of the deep ensemble trained on power- scaler Cp context from 0.1 to 10.
Validation Performance
Empirical Predictions
ECE
Brier Score
MAE
Accuracy
MM
M2
M3
M1
Network 1
0.01
0.01
0.02
0.99
0.00
0.00
1.00
0.00
Network 2
0.01
0.03
0.05
0.96
0.00
0.00
0.92
0.08
Network 3
0.01
0.01
0.01
0.99
0.00
0.00
0.17
0.83
Network 4
0.01
0.01
0.01
0.99
0.00
0.00
0.50
0.50
Network 5
0.01
0.01
0.01
0.99
0.00
0.00
0.96
0.04
Network 6
0.01
0.01
0.01
0.99
0.00
0.00
0.02
0.98
Network 7
0.01
0.01
0.01
0.99
0.00
0.00
0.75
0.25
Network 8
0.01
0.01
0.02
0.99
0.00
0.00
1.00
0.00
Network 9
0.01
0.01
0.01
0.99
0.52
0.03
0.25
0.21
Network 10
0.01
0.02
0.03
0.97
0.00
0.00
0.85
0.15
Network 11
0.01
0.01
0.01
0.99
0.00
0.00
0.37
0.63
Network 12
0.01
0.01
0.01
0.99
0.00
0.00
0.00
1.00
Network 13
0.01
0.01
0.01
0.99
0.00
0.00
0.99
0.01
Network 14
0.01
0.02
0.03
0.98
0.00
0.00
0.99
0.01
Network 15
0.01
0.01
0.01
0.99
0.00
0.96
0.00
0.04
Network 16
0.00
0.00
0.01
0.99
0.97
0.00
0.03
0.00
Network 17
0.01
0.02
0.02
0.98
0.00
0.00
0.62
0.38
Network 18
0.01
0.02
0.03
0.98
0.00
0.00
0.91
0.09
Network 19
0.01
0.01
0.01
0.99
0.00
0.04
0.49
0.46
Network 20
0.01
0.01
0.01
0.99
0.00
0.00
0.08
0.92
Average
0.01
0.01
0.02
0.99
0.07
0.05
0.54
0.33
Std. Deviation
0.00
0.01
0.01
0.01
0.24
0.21
0.40
0.36
32

---

Published in Transactions on Machine Learning Research (08/2024)
Table 9; Experiment 3. ABI validation performance on 000 held-out simulations and predictions on the empirical data the deep ensemble trained without Cp context.
Validation Performance
Empirical Predictions
ECE
Brier Score
MAE
Accuracy
MM
M2
M3
M1
Network 1
0.00
0.00
0.00
1.00
0.00
0.00
0.01
0.99
Network 2
0.00
0.01
0.01
0.99
0.00
0.00
0.00
1.00
Network 3
0.01
0.01
0.02
0.98
0.00
0.00
0.99
0.01
Network 4
0.01
0.02
0.03
0.98
0.00
0.00
0.60
0.40
Network 5
0.01
0.01
0.03
0.98
0.00
0.00
0.88
0.12
Network 6
0.00
0.00
0.00
1.00
1.00
0.00
0.00
0.00
Network 7
0.00
0.00
0.00
1.00
0.00
0.00
0.00
1.00
Network 8
0.01
0.01
0.02
0.98
0.00
0.00
0.00
1.00
Network 9
0.00
0.00
0.00
1.00
0.00
1.00
0.00
0.00
Network 10
0.00
0.00
0.00
1.00
0.72
0.16
0.11
0.00
Network 11
0.00
0.00
0.00
1.00
0.01
0.42
0.00
0.56
Network 12
0.01
0.02
0.04
0.97
0.00
0.00
0.91
0.09
Network 13
0.01
0.01
0.01
0.99
0.00
0.00
0.00
1.00
Network 14
0.00
0.00
0.00
1.00
0.00
0.00
1.00
0.00
Network 15
0.01
0.01
0.01
0.99
0.00
0.00
1.00
0.00
Network 16
0.00
0.00
0.00
1.00
0.00
0.00
0.98
0.02
Network 17
0.00
0.00
0.00
1.00
0.00
0.25
0.01
0.74
Network 18
0.00
0.00
0.00
1.00
0.00
0.00
0.86
0.14
Network 19
0.01
0.01
0.01
0.99
0.00
0.00
0.00
1.00
Network 20
0.01
0.02
0.04
0.97
0.00
0.00
0.88
0.11
Average
0.00
0.01
0.01
0.99
0.09
0.09
0.41
0.41
Std. Deviation
0.01
0.01
0.01
0.01
0.27
0.24
0.46
0.44
33


Reviews:
[{"review_id": "5G3R3C0b9m", "reviewer": "Reviewer_ME2Q", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\nThis is highly original work that has the potential to be really significant for evaluating and\nexploring models using amortised inference. The experiments are well-designed and effective at\ndemonstrating how sensitivity to the prior, data, and approximator looks like in several real-world\nexamples. The writing is very clear and straightforward to follow. This work should definitely be published.\n\nWeaknesses:\n\nSome of the results can be hard to interpret. For example for Figure 4, it's not clear what are these\nfour models and how do they differ. I couldn't find detailed answers even in the appendix. Also although\nscoring rules are mentioned, it doesn't seem any are used for the experiments. It would make the paper\nstronger if those can be used in some experiments. It would also be helpful to show that results are\nstill calibrated for all context variables.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces a method for using amortised Bayesian inference for sensitivity analysis. This is done by\ntraining deep ensemble networks which allow models and data to be readily changed and quickly obtaining a new\nposterior. The resulting posteriors seem no worse than what is obtained with normal amortized inference.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "More details on what were the models used for the experiments. It would also help to have an example which uses\nthe scoring rules just to show them in action.", "broader_impact_concerns": "No concerns about ethical implications of this work", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "aUw4dPqhp6", "reviewer": "Reviewer_6Cay", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\nI thoroughly enjoyed the paper and believe it will be of great interest to the TMLR audience, as well as the wider SBI/ABI community. It offers an easily applied and computationally efficient method to incorporate sensitivity analysis in ABI, which is especially important for neural-based methods. The approach is versatile enough to handle different instances of context variables, as well as allowing users to easily evaluate sensitivity to new context instances of likelihood and prior at inference time. Experiments 1 and 2 provide great demonstrations of the method's effectiveness in sensitivity analysis of prior and likelihood choices.\n\nWeaknesses: \n\nThe main limitation I identified is the lack of clarity with respect to the approximator sensitivity and its relationship to the issue of model misspecification and simulation gaps. Firstly, the authors later differentiate between closed-world and open-world settings. In the closed-world setting, which in this case refers to the setting where the data at inference time is sampled from the simulator hence we are certain that the model is well-specified, I think approximation sensitivity is clear and performance variability between ensemble members can give insights on the sensitivity of the neural approximator e.g. due to finite training or suboptimal convergence, as mentioned in the paper. \n\nHowever, in the open-world setting things are less clear. The paper refers to detecting approximator sensitivity as a way to get insights on the effect of simulation gaps due to a misspecified simulator relative to the data-generating process (this is mentioned in the abstract where approximator sensitivity refers to ‘unreliable approximation due to e.g. model misspecification' and later in Section 3.4 where approximator sensitivity is defined as `the variability of inferential results due to the approximation method employed'. ). It further hypothesizes that 'variability across the M ensemble members in the open world despite consistent performance in the closed world indicates a simulation gap'. There are two issues that need clarification: 1) In practice does this suggest that one should perform inference on simulated data and on observed data (potentially from a different model) to assess such variability? Is this also done in Experiment 3? and more importantly 2) It is generally not clear throughout the paper how approximator sensitivity is related to simulation gaps and how variability across performances of ensemble members indicate model misspecification. To explain myself further:\n\n- The above hypothesis is motivated by relevant works on the robustness of SBI methods in misspecified settings, such as Cannon et al. (2022), who highlight that in misspecified settings, the main issue affecting the performance of neural SBI is poor performance in OOD settings, such as in the presence of simulation gaps. Sensitivity to such gaps usually stems from the misspecified simulator and the poor generalization capabilities of the neural approximator, rather than the quality of the posterior approximation itself. In other words, the approximator might be accurately approximating the \"target\" posterior (i.e., the Bayesian posterior based on simulated data), but that posterior might not be the \"optimal\" posterior with respect to the data-generating process. This issue arises because the Bayesian posterior itself (which the approximation targets) is not robust to model misspecification. This raises the question of whether the approximation sensitivity refers to sensitivity to the quality of approximation of the Bayesian posterior as in the well-specified setting or to the sensitivity of the resulting inference due to model misspecification as in Cannon et al. 2022 and other citations in the paper? \n\n- The above point also leads to the question of how such sensitivity is detected. In the paper an ensemble of neural networks (of the same configuration) is used, and sensitivity is defined as the variation in inference results among the ensemble members. It is not entirely clear how variability in the performance of ensemble members would quantify sensitivity to model misspecification or data contamination. I can imagine a scenario where all ensemble members perform similarly but lead to misleading inference outcomes because they are all based on the same misspecified simulator.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces a method to efficiently integrate sensitivity analysis into Amortized Bayesian inference by using implicit context variables to represent different sources of perturbations. The resulting approach, called Sensitivity-Aware Amortized Bayesian Inference (SA-ABI), provides insights into the sensitivity of inference concerning the likelihood, prior, approximator, and data perturbations without the need to re-perform the entire analysis for each instance of the context variable. It is versatile, applicable to different types of Bayesian inference, including parameter estimation and model comparison, and is suitable for large-scale applications. The effectiveness of this methodology is demonstrated through real-world applications, showcasing its practical utility.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- [crucial] Clarification with respect to my points above throughout the paper on the type of approximator sensitivity addressed and its connection to simulation gap/ model misspecification. \n- [crucial] Figure 4 can benefit from a more detailed explanation which I think will help with the issues mentioned above. For example, the text reads ` Strikingly, Figure 4 reveals highly inconsistent predictions of the ensemble members '. Is this conclusion based on the high uncertainty of the posteriors? Does that directly link to high disparity between ensemble members or could it be due to other factors? A further explanation regarding the last two subplots would also be helpful to the reader, especially as $C_A$, $C_D$ and $C_P$ are all incorporated in different ways. For example, for the third subplot how do the illustrated posteriors take into account all the different data perturbations? A figure like 2b or 3a for at least a subset of different $C_D$ instances would be helpful here.\n- [non-crucial] In equation 8 how is the prior over $C_T$ normally chosen? Is it something simple like a uniform prior over different realisations of the context variable?\n- [non-crucial] Typo: above equation 6 in the two brackets I think section 3.3 should be 3.4 and section 3.4 should be 3.5", "broader_impact_concerns": "I don't see any broader impact concerns", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "fU4PsBaAg8", "reviewer": "Reviewer_dZmd", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n- The problem studied in this work (i.e. sensitivity analysis for SBI) is important and is of great interest within SBI community. The reviewer himself had also considered the same research question for a long time;\n- The method proposed is simple and elegant, with necessary implementation details well-explained;\n- The paper is mostly well-written and easy to follow. I personally enjoy reading the paper very much;\n- The proposed method is verified with several real-world examples. In addition, the author also include discussions on the practical implications found by their amortized sensitivity-analysis method (see e.g. the experiments on climate modeling), which is insightful.\n\n**Weaknesses**\n\nThe paper targets at an overlooked problem in SBI and proposes an elegant approach to fix it, with good empirical results. I don't really have too much to complain about. The only comments are:\n\n- Reproducibility: despite the important method developed and the interesting results, there seem to be no way to access the code of the proposed method nor the code of the simulator, which is a bit disappointing;\n\n- Explicit comparison to non-amortized sensitive analysis approach. The work may benefit from an explicit discussion on how their amortize method supports sensitive analysis that is not possible with non-amortized approach. This could be done by either highlighting in the methodology section or in the experiment.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a fully amortized approach for facilitating sensitivity analysis in simulation-based inference (SBI), an important but somewhat overlooked topic in SBI. The key idea is to treat the various sensitivity sources (e.g. different assumption of the underlying data generating process used in simulation, different choices of prior used in simulation, etc.) as context variables. These context variables serve as additional inputs to the neural network in amortized Bayesian inference, adding an outer level of amortization. Upon convergence, one can either quantitatively or qualitatively inspect the sensitivity w.r.t specific sources by varying the values of the corresponding input context variables, thereby enabling easy sensitivity analysis. Experiments on three real-world models in epidemics, climate science and cognitive science demonstrate the potential of the method.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Reproducibility: It is encouraged that upon acceptance, the author will provide the code of the method and the simulation models for reproducible research;\n- Presentation: It may also be helpful to group the topics in sec. 3 into two subgroups for further improving (the already very good) clarity. Furthermore, it *might* also be useful to provide a table summarizing different context variables considered in this work along with corresponding real-world examples.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: All reviewers mention that they enjoyed reading the paper. Some reviewers had some concerns about the clarity of some aspects of the methodology but this was cleared up in the authors' response.

I concur with the reviewers that this is a good paper that should be accepted. I do, however, have the following comments that I would like the authors to take into account in their final version:

- p2: "hundred thousand model configurations" Please clarify how distinct the different configurations are. For example, I imagine that in a hypothetical setup where we specify a prior on a mean, choosing a Gaussian and varying its mean results in scenarios that easier to amortise over than choosing different prior families (e.g. Bernoulli, spike-and-slab prior, and Poisson distribution) that results in more different prior predictive outputs. 

- Section 2.2: Regarding the Cranmer et al, 2020 citation for implicit likelihoods. Please clarify that this is just one possible reference, e.g. by adding a "e.g.". The earliest reference to my knowledge is https://www.jstor.org/stable/2345504 Another earlier review paper that discuss this are https://link.springer.com/article/10.1007/s11222-011-9288-2

- I think references [1] and [3] mentioned by Reviewer dZmd should be added to section 2.4. Reference https://openreview.net/forum?id=SRDuJssQud would also be a good addition to this section.

- [Important] Please add a discussion on the limits of the amortisation, in terms of the diversity of the different scenarios and scalability. I suppose as the number of the different scenarios and their diversity increases, the simulation budget needs to increase as well, with a rate depending on the similarity of the scenarios. Can you quantify how your results change as the diversity of different models considered increases?

- Please further add a discussion on the sensitivity of the setup to the choice of the neural network architecture that you use, and how you select them.
Decision: Accept (poster)