Consistency Models for Scalable and Fast Simulation-Based Inference
Authors: Marvin Schmitt
Valentin Pratz
Ullrich Koethe
Paul-Christian Bürkner
Stefan T. Radev
Venue: NeurIPS 2024 poster (2024)
Keywords: simulation-based inference
amortized inference
machine learning for science
probabilistic modeling
Bayesian inference
Primary area: probabilistic_methods
TL;DR: Consistency models are good and fast at solving Bayesian inverse problems.

Abstract: Simulation-based inference (SBI) is constantly in search of more expressive and efficient algorithms to accurately infer the parameters of complex simulation models.
In line with this goal, we present consistency models for posterior estimation (CMPE), a new conditional sampler for SBI that inherits the advantages of recent unconstrained architectures and overcomes their sampling inefficiency at inference time.
CMPE essentially distills a continuous probability flow and enables rapid few-shot inference with an unconstrained architecture that can be flexibly tailored to the structure of the estimation problem.
We provide hyperparameters and default architectures that support consistency training over a wide range of different dimensions, including low-dimensional ones which are important in SBI workflows but were previously difficult to tackle even with unconditional consistency models. 
Our empirical evaluation demonstrates that CMPE not only outperforms current state-of-the-art algorithms on hard low-dimensional benchmarks, but also achieves competitive performance with much faster sampling speed on two realistic estimation problems with high data and/or parameter dimensions.

Consistency Models for Scalable and Fast Simulation-Based Inference
Marvin Schmitt"
Valentin Pratz" Heidelberg University & ELIZA Germany
University of Stuttgart Germany
ma i www.rrviisccitttieme tt@ com
Ullrich Köthe
Bürkner TU Dortmund University
Stefan T. Radev Rensselaer Polytechnic Institute United States
Heidelberg University Germany
Germany
Abstract
Simulation-basee inference (SBI) is constantly in search of more expressive and efficient algorithms to accurately infer the parameters complex simulation mod- els. In line with this goal, we present consistency models for posterior estimation (CMPE), a new conditional sampler for SBI that inherits the advantages of recent unconstrained architectures and overcomes their sampling inefficiency at inference time. CMPE essentially distills a continuous probability flow and enables rapid few-shot inference with an unconstrained architecture that can be flexibly tailored to the structure of the estimation problem. We provide hyperparameters and de- fault architectures that support consistency training over wide range of different dimensions, including low-dimensional ones which are important in SBI work- flows but were previously difficult to tackle even with unconditional consistency models. Our empirical evaluation demonstrates that CMPE not only outperforms current state-of-thed -frtt--att------- algorithms on hard low-dimensional benchmarks, but also achieves competitive performance with much faster sampling speed on two realistic estimation problems with high data and/or parameter dimensions.
Introduction
Simulation-based inference (SBI) comprises family of computational methods for modeling the hidden properties of complex systems by means of simulation [1, 2]. In recent years, deep learning and generative models in particular have proven indispensable for scaling up SBI to challenging inverse problems [3-7]. Recently, multiple streams of neural SBI research have been capitalizing on the rapid progress in generative modeling of unstructured data by re-purposing existing generative architectures into general inverse problem solvers for applications in the sciences [8-11]. In line with the above trend, we contribute new contender to the growing suite of SBI methods for amortized Bayesian inference.
Within the class of deep generative models, score-based diffusion models [12-15] and flow matching algorithms [16, 17] have recently attracted significant attention due to their sublime performance as realistic generators. Diffusion models and flow matching strikingly flexible but require relatively expensive multi-step sampling phase to denoise samples [12]. To address this shortcoming, Song al. [18] proposed consistency models (CMs), which trained to perform few-step generation by design and have since emerged as standalone generative model family. Thus, CMs appear attractive
"equal contribution
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Reference
ACF
NSF
FMPE 10#
FMPE 30#
FMPE 1000#
CMPE 10#
CMPE 30
GMM
75mss
167ms
3058mm
40ms
1.36m
Two Moons
121ms
79mms
210ms
4095
62ms
102ms
Kinematics
210ms
78ms
228me
Figure 1: Experiments 1-3. 1000 posterior draws for unseen test instance per task, well as sampling time in milliseconds. All amortized neural approximators were trained with small budget of M 1024 simulations. The bottom shows the posterior predictive distribution in the kinematics task, and the pink cross-hair indicates the true end location x of the robot arm. Across all benchmarks. CMPE (Ours) yields the best trade-off between fast sampling speed and high accuracy. ACF: affine coupling flow, NSF: neural spline flow, FMPE: flow matching posterior estimation, CMPE: consistency model posterior estimation (Ours), K# denotes K sampling steps during inference.
as a backbone architecture for SBI workflows due to their unique combination of unconstrained architectures and fast sampling speed conceptual advantage over the current state-of-the-art in neural SBI.
However, paradoxically, CMs may struggle with low-dimensional estimation problems, which less important for image generation but crucial in typical SBI applications: These SBI applications often characterized relatively low-dimensional parameter spaces,
though the conditioning vectors (i.e., observables) might be high-dimensional [e.g., , 3, 19-21]. Additionally, the quality of conditional CMs as few-step Bayesian samplers has yet been explored empirically (e.g., in terms of probabilistic calibration and precision), even though this is crucial for their application in science and engineering. Lastly, while CMs for image generation are trained on enormous amounts of data, training data are typically scarce in SBI applications. In our empirical evaluations, we demonstrate that CMs are competitive with state-of- satee-of-hhee the-art SBI algorithms low-data regimes using our adjusted settings (see Appendix A for details).
In this paper, propose conditional CMs for SBI along with generalizable hyperparameter settings that achieve competitive performance with much faster sampling speed on both challenging low- dimensional benchmarks and high-dimensional applied problems. Our main contributions
We adapt consistency models to simulation-basee Bayesian inference and propose consis- tency model posterior estimation (CMPE);
2. We showcase the fundamental advantages of consistency models for amortized simulation- based inference: expressive unconstrained architectures and fast inference;
3. We demonstrate that CMPE outperforms NPE (normalizing flows) on three benchmark experiments (see Figure above), is competitive with FMPE (flow matching) on Bayesian image denoising, and surpasses both NPE and FMPE on challenging scientific simulator of tumor spheroid growth.
Preliminaries and related work
This section recaps simulation-based inference (SBI) via normalizing flows, flow matching, and score-based diffusion models. Readers familiar with these topics can safely fast-forward to Section 3.

---

2.1 Notation
In the following, the neural network training relies on a synthetic training set {(0(m) x0)) mm=1- which consists of parameter-data tuples. Each superscript marks one training example, namely, tuple of a latent parameter vector and an observable data set which was generated from the parameter vector. The total simulation budget for training the neural networks follows as M. We summarize the D-dimensional latent parameter vector of the simulator as A = (0I..... 0D). Further, each data set = [x=]== is matrix whose rows consist of N vector-valued observations. Accordingly, one parameter vector ((m²) yields one data set x (mm) from the simulator (see Section 2.2). In Bayesian inference, we aim infer the unknown parameters 0 of the mechanistic simulator, which are not to be confused with the trainable neural network weights Φ
}
2.2 Simulation-base inference (SBI)
Computer simulations play fundamental role across countless scientific disciplines, ranging from physics to biology, and from climate science to economics [2]. Simulators g(:,-) generate observables X as a function of unknown parameters 0 E O and latent program states & E [1]:
g) with 0 p(0): & p(E10)
The forward problem in Equation is typically well-understood through scientific theories instantiated as generative models. The inverse problem, however, is much harder, and forms the crux of Bayesian inference: reduce uncertainty about the unknowns A based observables x through the posterior distribution p(0 x) x p(0) p(x 0). The key property of neural SBI and early approximate Bayesian computation [ABC; 22, 23] methods is that they approach the inverse problem by sampling (i.e., simulating) from the forward model. Thus, the forward model acts as an implicit statistical model [24] that enables proper Bayesian inference in the limit of infinite simulations. In contrast, likelihood- methods (e.g., MCMC) depend on the explicit evaluation of the likelihood p(x 0).
We can classify neural SBI methods into sequential [e.g.. 10, 25-29] and amortized [e.g., 9-11, 30- 35] methods. Sequential inference algorithms iteratively refine the prior p(0) to generate simulations in the vicinity of the target observation. Thus, they are not amortized, each new observed data set requires a potentially costly re- re-training of the neural approximator tailored to the particular target observation. However, most sequential methods can be turned amortized by training the neural approximator to generalize over the entire prior predictive space of the model. This allows us to query the approximator with any new data set during inference. In fact, amortization across any component of the model, including multiple data sets [5] and contextual factors, such as the number of observations in data set [31], heterogeneous data sources [36], or even different probabilistic models [37, 38). Our current work is situated in the amortized setting.
be performed
2.3 Normalizing flows for neural posterior estimation
Neural posterior estimation (NPE) methods for SBI have traditionally relied on conditional discrete normalizing flows for learning a conditional neural density estimator P (0 x) from simulated pairs (0, x) of parameters and data [8, 39]. Normalizing flows can learn closed-form approximate densities via maximum likelihood training:
CNe = Ep(0,))) log P& (0 [x]]
(2)
Two prominent normalizing flow architectures in SBI are affine coupling flows [ACF; 40] and neural spline flows [NSF; 41J. A major disadvantage of these architectures their strict bijectivity requirement: it restricts the design space of the neural networks to invertible functions with cheap Jacobian calculations, hence the desire for more flexible architectures. the following, we will briefly recap recent works that have pioneered the use of multi-step unconstrained architectures for SBI, inspired by the success of such models in a variety of generative tasks [13, 15).
2.4 Flow matching for posterior estimation
Wildberger et al. [11] applied flow matching techniques [16, 17] in SBI, an approach abbreviated as flow matching posterior estimation (FMPE). FMPE based on optimal transport [42, 43], where the mapping between base and target distribution is parameterized by continuous process driven

---

by vector field on the sample space for each time step t E [O, 1]. Here, the distribution at t = 1 could be a unit Gaussian (O. and the distribution at t is the target posterior. The FMPE loss replaces the maximum likelihood term in the NPE objective (Fq. 2) with a conditional flow matching objective,
LFMPE: = Ep(x.0) [. [l: 0) (O, 4; x))l22 di
(3)
where Me denotes the conditional vector field parameterized by a unconstrained neural network with trainable parameters Φ, and ú; denotes marginal vector field, which can be as simple (0, ()) = 01 - 0 for all [17]. We obtain posterior draws 9g ~ p(0 x) by solving d0, (O, t; x:x] in reverse on E [o. 1). starting with noise samples 01 N I). We can use any ofe-the-shee shelf ODE
solver for transforming noise 0₁ into a draw 0, from the approximate posterior. In principle, the number of steps K' in the ODE solver can be adjusted by setting the step size di = 1/K. Decreasing the number of steps increases the sampling speed, but FMPE is not designed to optimize few-step sampling performance. We confirm this in our experiments with a rectified flow that encourages straight-path solutions [17].
2.5
Neural posterior score estimation
Another approach to iimulation-based inference with unconstrained networks lies in neural posterior score estimation INPSE; 10), which can either feature single-round inference (amortized) or multiple sequential inference rounds on particular data set (non-amortized). This stream of research uses conditional score-based diffusion models [12, 44] to learn the posterior distribution. With a slightly different focus, [9] factorize the posterior distribution and learn scores of the diffused posterior for subsets (down to a single observation) of larger data set. Subsequently, the information from the subsets is aggregated by combining the learned scores to approximate the posterior distribution of the entire data set. Crucially, both methods rely on the basic formulation of score-based diffusion models: They gradually diftuse the target distribution according to the following diffusion process,
d0, u(0,, 1; x)di- + (()dw,,
with drift coefficient μ1, diffusion coefficient a, time t [0, and Brownian motion {wa (0, At each time I, the current (diffused) distribution of 0 conditional on x is denoted as p. ( x). Crucially, the distribution at = 0 equals the target posterior distribution, (o x) = p(0 | x), and the distribution at t = T is set to be Gaussian noise (see below). Song et al. [12] prove that there exists an ordinary differential equation ("Probability Flow ODE") whose solution trajectories at time t are distributed according to (0 x))
1
dO,- = 1u(0 x;
(t) V po x dt,
(5)
2
where V logg Pr (0: xx) is the score function of (0) x). This differential equation is usually designed to yield a spherical Gaussian noise distribution pr x) se NN, after the diffusion process. Since we do not have access the target posterior p(0 x), score-based diffusion models train time-dependent score network & (,,, f, x) R V log Pa (0, x) via score matching and insert it into Eq. 5. Setting u(0,, f; x) = 0 and o, = /2t, the estimate of the Probability Flow ODE becomes d0; ts ( d) [45]. Finally, we can generate random draw from the noise distribution
Ir (0 T) and solve the Probability Flow ODE backwards for trajectory {0, ( 0)- The
end of the trajectory On represents a draw from the approximate posterior (0 x) R p(0 xx.
Numerical stability. The solver is usually stopped a fixed small positive number = e to prevent numerical instabilities [18], so we use O. to denote the draw from the approximate posterior. For simplicity, we will also refer to 0g as the trajectory's 's origin.
3 Consistency model posterior estimation
Diffusion models have one crucial drawback: At inference time, they require solving differential equation for each posterior draw which slows down their sampling speed. This is particularly troublesome SBI applications which may require thousands of samples for thousands of data sets [e.g., > M data sets in 7J. Consistency models (CMs; 18] address this problem with a new generative model family that supports both single-step and multi-step sampling. the following, we summarize key ideas of CMs [18, 46] with a focus on conditional sampling for the purpose of SBI.

---

3.1 Conditional consistency models
The consistency function f (0,, t; x) ト 0g maps points across the solution trajectory (0 (et)eeTTeel to the trajectory's s origin 0₂ given fixed conditioning variable (i.e., observation) x and the probability flow ODE in Eq. 5. To achieve this with established score-based diffusion model architectures, we can use unconstrained neural network (0, x) which is parameterized through skip connections of the form
(0, 1; x) Caxip(1) (t) + Coui (t) (0, x)
where Cskip() (1) and (1) are differentiable and fulfill the boundary conditions Cakip()) = and Cout (e) O. Consistency models are originally motivated distillation technique for diffusion models. However, Song et al. [18] show that training consistency models in isolation is possible, and we base our method on their direct approach.
Sampling. Once the consistency model has been trained, generating draws from the approximate posterior is straightforward by drawing samples from the noise distribution, Or ~ NN 7 which shall then be transformed into samples from the target distribution, like in a standard diffusion model. In contrast to diffusion models, however, we do not need to solve sequence of differential equations for this transformation. Instead, we can use the learned consistency function f& to obtain the one-step target sample 0~ (0r, T; x). What is more, inference with consistency models is not actually limited to one-step sampling. In fact, multi-step generation is possible with an iterative sampling procedure, which we will describe in the following. For sequence of time points E =1 11 < l2 < ...... <.. = T and initial noise 0x~ ~ NO, IT),, we calculate
O2 + (0 (411, x) + V 12
52
for k = K 11,1, 2..... where Z₁: ~ (N I) and K - 1 is the number of sampling steps [18, 46]. The resulting sample is usually better than one-step sample.
3.2 Consistency models for ioaullation-basee inference
Originally developed for image generation, consistency models can be applied to learn arbitrary dis- tributions. The unconstrained architecture enables the integration of specialized architectures for both the data x and the parameters 0. Due to the low number of passes required for sampling (in contrast to flow matching and diffusion models), more complex networks can be used while maintaining low inference time. In theory, consistency models combine the best of both worlds: unconstrained networks for optimal adaptation to parameter structure and data modalities, while enabling fast inference speed with few network passes. Currently, this comes at the cost of explicit invertibility, which limits the computation of posterior densities. More precisely, single-step consistency models do not allow density evaluations at an arbitrary parameter value A but only at a set S of approximate posterior draws _ ..... o @(S) However, this is sufficient for important downstream tasks like marginal likelihood estimation, importance sampling, or self-consistency losses. In contrast, multi- step consistency sampling defines a Markov chain which cannot be evaluated without an additional density estimator (see Section 3.5 for details). accordance with the taxonomy from Cranmer et al. [1], we call our method consistency model posterior estimation (CMPE). While we focus on posterior estimation, using consistency models for likelihood emulation is a natural extension of our work.
As consequence of its fundamentally different training objective, CMPE is not just faster version of FMPE. Instead, it shows qualitative differences FMPE beyond much faster sampling speed. In Experiment 4, we show that CMPE is less dependent on the neural network architecture than FMPE, making it a promising alternative when the optimal architecture for task is not known. Further, we consistently observe good performance in the low-data regime, rendering CMPE an attractive method when training data is scarce. In fact, data availability a common limiting factor for complex simulators in engineering [47] and science (e.g., molecular dynamics; 48).
3.3 Optimization objective
We formulate the consistency training objective for CMPE, which extends the unconditional training objective from Song and Dhariwal [46] with a conditioning variable x to cater to the SBI setting,
LCMPE Φ-) E (x()) d(u((), (41ix), u(s .4 x))
(8)
5

---

where A(t) is a weighting function, d(u, v) is distance metric, z ~ (0, is unit Gaussian noise, and the arguments for the distance metric amount the outputs of the consistency function obtained at two neighboring time indices 14 and 11+1,
u 1; = f& ( + 1,+12, f441: x), u(Φ x4;x)) = f (0 12 t4: x)-
The teacher's neural network weights Φ during each step via stopgra operator, Φb f- stopgrad(()). We follow Song and Dhariwal [46] using A(t;) = 1/(1++1 - 44) and d(u, v) VI
copy of the student's weights which are held constant
cD from the Pseudo-Huber metric family. Appendix A contains more details on discretization and noise schedules.
v -
3.4 Hyperparameter tuning
Consistency training introduces several additional hyperparameters, but there is limited theoretical guidance on how to select most of them. Within the scope of this paper, we rely on empirical search to identify the hyperparameters that are the most relevant for tuning efforts. As stated above, the maximum time determines the standard deviation the latent distribution. It should be larger than the magnitude of the target distribution [18]. Values that too low might lead to biased sampling, while values that are too high can make the training process more difficult. The minimum and maximum number of discretization steps So 81 during training also influence the final result in our experiments. For 8o. a sufficiently low value (e.g. 10) should be chosen. For s1, smaller values around 50 seem beneficial for stable training, especially when the number of epochs is low. When longer training is possible, increasing 81 might lead to improved accuracy. We summarize our recommended hyperparameter choices for SBI applications in Appendix C.
3.5 Density estimation
Using the change-of-variable formula, we can express the posterior density of a single-step sample as
[00r)
(=(0|x)) = pr = = (0, x)) det
(10)
a0
where (0, T; x) is the implicit inverse of the consistency function and 00r/00, the result- ing Jacobian. While we cannot explicitly evaluate f3 (0,, x). we can generate samples from Or ~ (0, and use autodiff for evaluating the Jacobian because the consistency surrogate fo is differentiable. Thus, we cannot directly evaluate the posterior density at arbitrary O. Yet, evaluating the density at set of S posterior draws [0 suffices for crucial downstream tasks, such as marginal likelihood estimation [49], neural importance sampling [21], or self-consistency losses [50].
For the purpose of multi-step sampling, the latent variables 0k, 0x.1..... 0, form a Markov chain with change-of variables (CoV) given by Köthe [51]:
p(O x,,x)
p(0 [x) = p(Ok) II
(11)
O(O
0x-1,
Due to Eq. the backward conditionals are given by p(0 1o. x) = N (f tx:x) (t2-
Unfortunately, the forward conditionals p(0,. 0x-1, x) are not available in closed form, but we could learn an amortized surrogate model q(O,. 0x-1) capable of single-shot density estimation based on a data set of execution paths {0..x} }. This method would work well for relatively low-dimensional 0, which are common throughout scientific applications, but may diminish the efficiency gains of CMPE in higher dimensions. We leave extensive evaluation of different surrogate density models
to future work.
3.6 Choosing the number of sampling steps
The design of consistency models enables one-step sampling (see Eq. 7). In practice, however, using two steps significantly increases the quality image generation tasks [46]. We observe that few-step sampling with approximately K = 5-15 steps provides the best trade-off between sample quality and compute for SBI, particularly in low-dimensional problems. This roughly comparable to the speed of neural one-step estimators like affine coupling flows or neural spline flows in Experiments

---

1-3 (see Figure 2a). The number of steps can be chosen at inference time, so practitioners can easily adjust this for given situation. We noticed that upproaching the maximum trained number of discretization steps can lead to overconfident posterior distributions. See Section C.5 for an empirical evaluation of the relation between sampling quality and number of sampling steps.
Empirical evaluation
Our experiments cover three fundamental aspects of SBI. First, we perform an extensive evaluation on three low-dimensional experiments with bimodal posterior distributions from benchmarking suites for inverse problems [52, 53J. The simulation-baset training phase is based on a fixed training set {(x(m) ((()) } Jm=1 of M tuples of data sets x(m)) and corresponding ground-truth parameters @(m)) Second, we focus on an image denoising example which serves as sufficiently high-dimensional case study in the context of SBI [32, 54]. Third, we apply our CMPE method to a computationally challenging scientific model of tumor spheroid growth and showcase its superior performance for complex simulator from cell biology [19]. We implement all experiments using the BayesFlow Python library for amortized Bayesian workflows [55].
Evaluation metries. We evaluate the experiments based on well-established metries to gauge the accuracy and calibration of the results. All metrics are computed on a test set of J unseen instances ((x())00))) ) In the following, S denotes the number of (approximate) posterior samples that draw for each instance J. First, the casy-to-understand root mean squared error (RMSE) metric quantifies both the bias and variance of the approximate posterior samples across the test set as
$ +E==v/ V E (o ol))² 2 Second, we estimate the squared maximum mean discrepancy (MMD; 56] between samples from the approximate vs. reference posterior, which is kernel-based distance between distributions. Third, as a widely applied metric in SBI, the C2ST score uses an MLP classifier to distinguish samples from the approximate and reference posteriors. The resulting test accuracy of the classifier is the C2ST score, which ranges from (samples are perfectly separable + bad posterior) 0.5 (samples are indistinguishable + perfect posterior). Finally, we assess calibration (SBC; [57]): All uncertainty intervals Ug (@ x) of the true posterior p(0 x) are well calibrated for every quantile q E (0,
uncertainty calibration through simulation-
/ʃ 1 E Ug(0 x)] 0.) p(0-)d0..xx
q
(12)
where [[-] is the indicator function. Discrepancies from Eq. 12 indicate deficient calibration of an approximate posterior. The expected calibration error (ECE) aggregates the median SBC error of central credible intervals of 20 linearly spaced quantiles g, averaged across the test set.
Contender methods. We compare affine coupling flows [ACF; 40], neural spline flows (NSF; 41], flow matching posterior estimation [FMPE; 11], and consistency model posterior estimation (CMPE: ours). We use identical unconstrained neural networks for FMPE and CMPE to ensure comparability. Appendix C lists the hyperparameters of all models in the experiments.
4.1 Experiment 1: Gaussian mixture model
We illustrate CMPE on 2-dimensional Gaussian mixture model with two symmetrical components [9, 50). The symmetrical components are equally weighted and have equal variance,
0,00,,1),
1 Wx( 0, !) + | -0,
(13)
where N/- μu, Σ) is a Gaussian distribution with location μl and covariance matrix E. The resulting posterior distribution is bimodal and point-symmetric around the origin (see Figure I top-left). Each simulated data set x consists of ten exchangeable observations {xi,... x10), and we train all architectures on M = 1024 training simulations. We use DeepSet [58] to learn summary statisties for each data set jointly with the inference task.
Results. As displayed in Figure 1, both ACF and NSF fail to fit the bimodal posterior with separated modes. FMPE successfully forms disconnected posterior modes, but shows visible overconfidence through overly narrow posteriors. The visual sampling performance of CMPE is superior to all other methods. this task, we observe that CMPE does not force us to choose between speed or performance relative to the other approximators. Instead, CMPE can outperform all other methods
7

---

1.0
0.9
123 mi
C2ST
0.8
C2ST
0.7
m5
0.7
4885 ms
0.6
0.10
0.15
0.20
1.15
2048
Sampling speed [seconds]
Simulation Budget
CMPE 10# (Ours)
ACF NSF
FMPE 30 FMPE
(Ours)
FMPE
CMPE 30# (Ours)
(a) Experiment (Gaussian Mixture Model)
(h) Experiment (Two Moons)
Figure 2: C2ST score of 4000 approximate posterior draws vs. reference posterior (lower is better) for J = 100 unseen test examples. (a) CMPE (Ours) outperforms all other methods through both faster and more accurate inference on the GMM benchmark (mean=tSD). (b) CMPE (Ours) with 10 sampling steps shows superior performance up to a training budget of 4096 instances on the Two Moons benchmark (SC.
simultaneously with respect to both speed and performance, as quantified by lower C2ST to the reference posterior across J 100 test instances (see Figure 2a). If we tolerate slower sampling, CMPE achieves peak performance at 1t0p-step FMPE by large margin, even though the latter is approximately 75 slower.
inference steps. Most notably, CMPE outperforms
4.2 Experiment 2: Two moons
This experiment studies the moons benchmark [27, 49, 52, 59]. The model is characterized by bimodal posterior with two separated crescent moons for the observed point X.bs = (0. ().0) which approximator needs recover. We repeat the experiment for different training budgets M E {512, 1024, 2048, 4096, 8192] to probe each method under varying data availability. While M 512 is very small budget for the two moons benchmark, M 8192 is considered sufficient.
Results. Trained on a simulation budget M = 1024 examples, CMPE consistently explores both erescent moons and successfully captures the local patterns of the posterior (see Figure 1, middle row). Both the affine coupling flow and the neural spline flow fail to fully separate the modes. Most notably, if aim to achieve fast sampling speed with FMPE by reducing the number of sampling steps during inference, FMPE shows visible overconfidence with 30 sampling steps and markedly deficient approximate posteriors with 10 sampling steps. In stark contrast, CMPE excels in the few-step regime. Figure 2b illustrates that all architectures benefit from a larger training budget. CMPE with 10 steps emerges the superior architecture in the low- and medium-data regime up to . 4096 training instances, FMPE with 1000 steps outperforms the other approximators by small margin for the largest training budget of M 8192 instances. Keep in mind, however, that 1000-step FMPE is approximately 30-70x slower than ACF, NSF, and CMPE in this task.
4.3 Experiment 3: Inverse kinematics
Proposed as a benchmark task for inverse problems by Kruse et al. [53], the inverse kinematics model aims reconstruct the configuration (0,,0,, 03, 04) = 0 E IR1 of multi-jointed 2D robot arm for given end position x E RR2 (see Figure 1, bottom left). The input to the forward process 0 - x are the initial height 0₃ of the arm's 's base, as well as the angles 02 03. 0g at its three joints. The inverse problem aims to determine the posterior distribution p(0 x). which represents all arm configurations that end at the observed 2D position x the arm's 's end effector.
Results. On the challenging small training budget of M 1024 training examples, CMPE with K 30 sampling steps visually outperforms all other methods with respect to posterior predictive performance while maintaining fast inference (see Figure 1). On the C2ST metric, CMPE outperforms normalizing flows (ACF, NSF), but in turn surpassed by 1000-step FMPE (see Figure 4b), which however 85x slower (FMPE: 6565ms, CMPE: 78ms). In contrast to the other empirical evaluations, the simulator this benchmark lacks aleatoric uncertainty [60] and the plausible parameter values tightly bounded through the geometry of the problem, which might explain this pattern.

---

4.4 Experiment 4: Bayesian denoising
This experiment demonstrates the feasibility of CMPE for a high-dimensional inverse problem, namely, Bayesian denoising on the Fashion MNIST data set [61]. The unknown parameter 0 78784 is the flattened original image, and the observa- tion x R8784 is a blurred and flattened version of the crisp image from simulated noisy camera [32, 49, 54]. We compare CMPE and FMPE a standard (60k training images) and a small data (2k training images) regime. As both methods allow for unconstrained architectures, we can assess the effect of the architectural choices on the results by evaluating both methods using suboptimal naïve architecture vs. the established U-Net [62].
TStart/op
Trousse
Pallover
Drass
Doul
Pursinstorr
Obsurvation
Mean
Std.Dww
Figure 3: Experiment 4. CMPE denoising
results on Fashion MNIST (U-Net backbone, K 2 sampling steps, 60 training im- ages). First row: Original image (target param- eters θ). Second row: Blurred image (obser- vations x). Third and fourth row: Mcans and standard deviations of the approximate posteri- ors. Note: For standard deviations, darker re- gions indicate larger variability in the outputs. Adapted from [49]. More in Appendix C.6.
Neural architectures. The naïve architecture con- sists of convolutional neural network [CNN; 63] to convert the observation into a vector of latent summary statistics. We concatenate input vector, summary statistics, and a time embedding and feed them into a multi-layer perceptron (MLP) with four hidden layers consisting of 2048 units each. This corresponds to a situation where the structure of the observation (i.e., image data) is known, but the structure of the parameters is unknown or does not inform a bespoke network architecture. In this example, however, we can leverage the prior knowledge that our parameters are images. Specifically, can incorporate inductive biases into our network architecture by choosing U-Net architecture which is optimized for image processing [i.e., an adapted version of 64]. Again, a CNN learns a summary vector of the noisy observation, which is then concatenated with a time embedding into the conditioning vector for the neural density estimator.
Results. We report the aggregated RMSE, MMD, and the time per sam- ple for both methods and architectures (Table 1). FMPE is not able to gen- erate good samples for the naïve ar- chitecture, whereas CMPE produces acceptable samples even in this subop- timal setup. This reduced susceptibil- ity to suboptimal architectures might become a valuable feature in high- dimensional problems where the struc-
Table 1: Experiment 4: RMSE and MMD between the ground-truth image vs. 100 draws from the approximators trained on 2 200 and 60 000 training images, aggregated over 100 test images. For MMD, we draw one denoised sample per test image and calculate the MMD between the denoised samples and the original images. Time per draw.
Models
RMSE J
(L SD) [x 10 60:000
Time 4
2000
60000
2000
|U-Net narve
FMPE
0.836
0.597
171.53
11.1
95.26 + 1.21
15.4ms 0.3ms
CMPE (Ours)
0,388
0.293
102
002.09:20
+ 3.21
57.90 + 1.59
ture of the parameters cannot be ex- ploited. The U-Net architecture en- ables good sample quality for both methods, highlighting the advantages of unconstrained architectures. The MMD values align well with visual assessment of the sample quality, therefore we deem it an informative metric to compare the results (see Section C.6 for visual inspection). The U-Net architecture paired with a large training set provides detailed and versatile samples of similar quality for CMPE and FMPE. CMPE enables 50-1000x faster inference than FMPE because it only requires two neural network passes for sampling, while achieving better (naïve) or competitive (U-Net) quality.
FMPE
0,278
0.217
17.38
+ 0.10
14.50 + 0.05
565 8ms
CMPE (Ouns)
0.311
0.12
1 0.05
4.5 Experiment 5: Tumor spheroid growth
We conclude our empirical evaluation with complex multi-scale model of 2D tumor spheroid growth [19, 65). The model has 7 unknown parameters which govern single-cell behavior (agent-based) as well as the extracellular matrix description (PDE-based). Crucially, running simulations from this model on consumer-grat grade hardware is rather expensive (Rs minute for a single simulation), so there is desire for methods that can provide reasonable estimates for a limited offline training budget.

---

Here, we compare the performance of the four contender methods in this paper on fixed training set of M 19 600 simulations with J 400 simulations as a test set to compute performance metrics. The contender methods are affine coupling flows (ACF), neural spline flows (NSF), and flow-matching posterior estimation (FMPE), as these are among the most popular choices for neural SBI to date. Our main contender is FMPE as free-form architecture, and we use the exact same inference network for both FMPE (L& - field network) and CM (fe consistency network), Across all methods, we use hybrid statistics of variable length into fixed length embedding vectors h(x). Appendix C.7 provides more details on the neural network architectures and training hyperparameters.
architecture to transform high-dimensional summary
Results. CMPE outperforms the alternative neu- ral methods via better accuracy and calibration, as indexed by lower RMSE and ECE on 400 unseen test instances (see Table 2). The speed of the sim- pler ACF is unmatched by the other methods. In direct comparison to its unconstrained contender FMPE, CMPE simultaneously exhibits (i) a slightly higher accuracy; (ii) drastically improved calibra- tion; and (iii) much faster sampling (see Figure 6 in the Appendix). For this simulator, FMPE did not achieve satisfactory calibration performance with up to K 100 inference steps. Table 2 reports the best FMPE results for K = 1000 steps.
Table 2: Experiment 5. RMSE, ECE, and sam- pling time are computed for 2000 posterior sam- ples, aggregated over 400 unseen test instances. Max ECE denotes the worst-case marginal cali- bration error across all 7 model parameters.
Model
RMSE L
Max ECE J
Time 4
ACF
0.589
0.021
1.07s
NSF
0.500
0.027
1.95s
EMPE 30
0.582
0.222
17.13s
FMPE 1080#
0.583
0.057
500.90s
CMPE 2# (Ours)
0.064
2.16s
CMPE 30 (Ours)
0.577
0,018
18.33s
5 Discussion
We presented consistency model posterior estimation (CMPE), a novel approach to perform accurate simulation-baat based Bayesian inference on large-scale models while achieving fast inference speed. CMPE enhances the capabilities of Matte-of-the-ar neural posterior estimation by combining few-step sampling with unconstrained neural architectures. To assess the effectiveness of CMPE, we applied it to a set of 3 low-dimensional benchmark tasks that allow intuitive visual inspection, as well as high- sigh-dimensional Bayesian denoising experiment and a scientific tumor growth model. Across our experiments, CMPE emerges as competitive method for simulation- based Bayesian inference, as evidenced by a holistic assessment of posterior accuracy, calibration, and inference speed.
Limitations. Our proposed CMPE method has two core limitations, which we specifically highlight again in this paragraph. First, consistency models do not directly yield tractable densities for arbitrary parameter values (see Section 3.5 for a remedy). Second, the relation between inference time (number of sampling steps K) and performance (posterior accuracy) is not monotonic, as clucidated in Experiment 1, (see Figure 2a). We provide more details and report the CMPE posterior quality (quantified by C2ST) as a function of the number of sampling steps for all three benchmark experiments in Section C.5. This phenomenon is not drawback per se, but rather a counter-intuitivy attribute of consistency models. In practice, it can easily be addressed by performing a brief sweep over the number of sampling steps (K 1,... Kmxx) during inference time and choosing an optimal K based on user-defined metrics (e.g., calibration or MMD on a validation set). Given the very fast inference speed of CMPE and the observed U-shaped relation between sampling steps and performance, this approach does not generally come with a relevant computational burden.
Perspectives. Future work might aim to further reduce the number of sampling steps towards one-step inference in SBI tasks to achieve even faster sampling speed in time-sensitive applications. This might be achieved via extensive automated hyperparameter optimization, tailored training schemes, or consistency model inference models [CTMs; 66] are a closely related generative model family, whose conditional formulation might prove useful for SBI as well. Overall, our results demonstrate the potential of CMPE as a novel simulation-bar -based inference tool. Its unique combination of highly expressive unconstrained neural networks with fast sampling speed renders CMPE new contender for SBI workflows in science and engineering, where both time and performance are essential.
compressed latent space. Furthermore, consistency trajectory
10

---

Acknowledgments
We thank Lasse Elsemüller for insightful feedback and Maximilian Dax for support with the imple- mentation of the gravitational wave experiment in the Dingo library. MS acknowledges funding from the Cyber Valley Research Fund (grant number. CyVy-RF-2021-10, and the Deutsche Forschungsge- meinschaft (DFG, German Research Foundation) under Germany's s Excellence Strategy EXC-2205 -39074001 (the Stuttgart Cluster of Excellence SimTTech). MS additionally thanks the Google Cloud Research Credits program and the European Laboratory for Learning and Intelligent Systems (ELLIS) PhD program for support. VP acknowledges funding by the German Academic Exchange Service (DAAD) and the German Federal Ministry of Education and Research (BMBF) through the funding programme "Konrad Zuse Schools of Excellence in Artificial Intelligence" (ELIZA) and support by the state of Baden-Wiine Württemberg through bwHPC. VP acknowledges support by the Bundesministerium fuer Wirtschaft und Klimaschutz (BMWK, German Federal Ministry for Economic Affairs and Climate Action) part of the German government's 7th energy research program "Innovations for the energy transition" " under the 03EТE0391 HiBRAIN project (Holistic method of combined data- and model-based Electrode design supported by artificial intelligence).
Code
The software code for the experiments is available in a public GitHub repository: https://gihubb.oo//ayysttttoo wtwaay/eseloooooo/oossseeeayeeee moodel ponuedelposterorr-eee esstimaiooo
Additionally, we provide backend-agnostic implementation of Consistency Model Posterior Estima- tion (CMPE) in the open source BayesFlow library for amortized Bayesian workflows [55]. Users can choose between PyTorch, TensorFlow, or JAX backend. BayesFlow available as open-source software on GitHub https://jiibbbb.ommmayyyeeeooo/p
and PyPI.
References
[1] Kyle Cranmer, Johann Brehmer, and Gilles Louppe. The frontier of simulation-based inference. Proceedings of the National Academy of Sciences, 2020.
[2] Alexander Lavin, Hector Zenil, Brooks Paige, et al. Simulation intelligence: Towards a new generation of scientific methods. arXiv preprint, 2021.
[3] Stefan T Radev, Frederik Graw, Simiao Chen, Nico T Mutters, Vanessa M Eichel, Till Bärnighausen, and Ulirich Köthe. Outbreakflow: Model-based bayesian inference of dis- ease outbreak dynamics with invertible neural networks and its application the covid-19 pandemics in germany. PLOS computational biology, 2021.
[4] Takashi Shiono. Estimation of agent- models using Bayesian deep learning approach of BlyesFlow. Journal of Economic Dynamics and Control, 10402, 2021.
[5] Pedro J Gonçalves, Jan-Matthi Lueckmann, Michael Deistler, et al. Training deep neural density estimators to identify mechanistic models of neural dynamics. Elife, 2020.
[6] Antoine Wehenkel, Jens Behrmann, Andrew C. Miller, Guillermo Sapiro, Ozan Sener, Marco Cuturi, and Jömn Henrik Jacobsen. Simulation- based inference for cardiovascular models, 2023.
[7] Mischa von Krause, Stefan Radev, and Andreas Voss. Mental speed is high until age 60 as revealed by analysis of over million participants. Nature Human Behaviour, 6(5):700-778, May 2022. doi: 10. (0338//41622-21-00000
[8] George Papamakarios, Eric Nalisnick. Danilo Jimenez Rezende, Shakir Mohamed, and Balaji Lakshminarayanan, Normalizing flows for probabilistic modeling and inference. J. Mach. Learn Res., 22(1), jan 2021. ISSN 1532-4435.
[9] Tomas Geffner, George Papamakarios, and Andriy Mnih. Compositional score modeling for simulation-based inference, 2022.
[10] Louis Sharrock, Jack Simons, Song Liu, and Mark Beaumont. Sequential neural score estima- tion: Likelihood-free inference with conditional score based diffusion models, 2022.
11

---

[11] Jonas Wildberger, Maximilian Dax, Simon Buchholz, Stephen Green, Jakob H Macke, and Bernhard Schölkopf. Flow matching for scalable iomulation-basee inference. In A. Oh. T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, volume 36, pages 16837-16864, 2023.
[12] Yang Song. Jascha Sool-Dickstein, Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2021.
[13] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10684-10660 10695, 2022.
[14] Jonathan Ho. Ajay Jain, and Picter Abbeel. Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems, volume 33, pages 6840-6851, 20200
[15] Georgios Batzolis, Jan Stanczuk, Bisola-Bibbane Schönlieb, and Christian Etmann. Conditional image generation with score-based diffusion models, 2021.
[16] Yaron Lipman, Ricky T. Q. Chen, Heli Ben-Hamu, Maximilian Nickel, and Matthew Le. Flow matching for generative modeling. In The 11th International Conference on Learning Representations, 2023. URL https: /wwp///eeeeeeeeeeeeeeeeeeee /
[17] Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow, 2022.
[18] Yang Song, Prafulla Dhariwal, Mark Chen, and llya Sutskever. Consistency models. In International Conference Machine Learning. 2023.
[19] Nick Jagiella, Dennis Rickert, Fabian J Theis, and Jan Hasenauer. Parallelization and high- performance computing enables automated statistical inference of multi-scale models. Cell systems, 4(2):19442000 2017.
[20] Amin -andllii-Kaaaaari.. Jamal Amani Rad, and Michael D. Nunez. A general integrative neurocognitive modeling framework jointly describe EEG and decision-makerr on single trials. Computational Brain and Behavior, 2023. doi: 10.1007/s4211330 11211113-222 122230222221--02-000700----
[21] Maximilian Dax, Stephen R. Green, Jonathan Gair. Michael Pürrer, Jonas Wildberger, Jakob H. Macke, Alessandra Buonanno, and Bernhard Schölkopf. Neural importance sampling for rapid and reliable gravitational
inference. Physical Review Letters, 130(17), April 2023. ISSN 1079-7114. doi: 10.1 10.1103/physrevlet. 130. 170.17. URL http: //tp///xxx// www.doi..rooooo orgg//.......// 1103/ PhysRevLett. 130. 171403.
[22] Donald B Rubin. Bayesianly justifiable and relevant frequency calculations for the applied statistician. The Annals of Statistics, pages 1151-1172, 1984.
[23] Simon Tavaré, David J Balding, Robert C Griffithss and Peter Donnelly. Inferring coalescence times from dna sequence data. Genetics, 145(2):505-518, 1997.
[24] Peter Diggle and Richard J Gratton. Monte carlo methods of inference for implicit statistical models. Journal of the Royal Statistical Society Series B: Statistical Methodology, 46(2): 193-212, 1984.
[25] George Papamakarios and lain Murray. Fast e-free inference of simulation models with bayesian conditional density estimation. Advances in neural information processing systems, 29, 2016.
[26] Jan-Matthis Lueckmann, Pedro J Gonçalves, Giacomo Bassetto, Kaan Öcal, Marcel Nonnen- macher, and Jakob H Macke. Flexible statistical inference for mechanistic models of neural dynamics. In st NeurIPS Conference Proceedings, 2017.
[27] David Greenberg, Marcel Nonnenmacher, and Jakob Macke. Automatic posterior transformation for likelihood-free inference. In International Conference on Machine Learning, 2019.
[28] Conor Durkan, Iain Murray, and George Papamakarios. On contrastive learning for likelihood- free inference. In International Conference Machine Learning. PMLR, 2020.
12

---

[29] Michael Deistler, Pedro J Goncalves, and Jakob H Macke. Truncated proposals for scalable and hassle-free simulation-basee inference. arXiv preprint, 2022.
[30] Lynton Ardizzone, Jakob Kruse, Sebastian Wirkert, Daniel Rahner, Eric W Pellegrini, Ralf S Klessen, Lena Maier-Hein, Carsten Rother, and Ullrich Köthe. Analyzing inverse problems with invertible neural networks. In Intl. Conf. on Learning Representations, 2019.
[31] Stefan T Radev, Ulf K Mertens, Andreas Voss, Lynton Ardizzone, and Ullrich Köthe. Bayesflow: Learning complex stochastic models with invertible neural networks. IEEE transactions neural networks and learning systems, 2020.
[32] Lorenzo Pacchiardi and Ritabrata Dutta. Score matched neural exponential families for likelihood-free inference. J. Mach. Learn. Res., 23:38-1, 2022.
[33] Grace Avecilla, Julie N Chuong, Fangfei Li, Gavin Sherlock, David Gresham, and Yoav Ram. Neural networks enable efficient and accurate simulation- inference evolutionary parameters from adaptation dynamics. PLoS Biology, 2065):33001633, 2022.
[34] Stefan Radev, Marco D Alessandro, Ulf K Mertens, Andreas Voss, Ullrich Köthe, and Paul- Christian Bürkner. Amortized bayesian model comparison with evidential deep learning. IEEE Transactions on Neural Networks and Learning Systems, 2021.
[35] Lasse Elsemüller, Martin Schnuerch, Paul-Christian Bürkner, and Stefan T. Radev. A deep learning method for comparing bayesian hierarchical models, 2023.
[36] Marvin Schmitt, Stefan T. Radev, and Paul-Christian Bürkner. Fuse lose it: Deep fusion for multimodal simulation-based inference, 2023. arXiv:2311.10677
[37] Lasse Elsemüller, Hans Olischläger, Marvin Schmitt, Paul-Christian Bürkner, Ullrich Köthe, and Stefan T. Radev. Sensitivity- aware amortized Bayesian inference, 2023. arXiv:2310.11 1122..
[38] Cornelius Schröder and Jakob Macke. Simultaneous identification of models and parameters of scientific simulators, 2023.
[39] Danilo Jimenez Rezende and Shakir Mohamed. Variational inference with normalizing flows. In Proceedings of the 32nd International Conference International Conference Machine Learning Volume 37, ICML 15, page 1530-1538. JMLR.org. 2015.
[40] Laurent Dinh, Jascha Sohl-Dicksted and Samy Bengio. Density estimation using real nvp. arXiv preprint 160iv:1605.08000 2016.
[41] Conor Durkan, Artur Bekasov, lain Murray, and George Papamakarios. Neural spline flows. Advances in neural information processing systems, 32, 2019.
[42] Cédric Villani. Optimal Transport, volume 338 of Grundlehren Der Mathemattsschn Wis- senschaften. Springer, Berlin, Heidelberg, 2009. ISBN 978-3-540-71049-33 978-3-540-770000--- doi: 10.1007/978---540-77000
[43] Gabriel Peyré and Marco Cuturi. Computational optimal transport, 2020. URL https: //arxiv.org abs////////////// 1803. 00567.
[44] Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. In Proceedings of the 33rd International Conference on Neural Information Processing Systems, 2019.
[45] Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-based generative models. Advances in Neural Information Processing Systems, 35: 26565-26574 2022.
[46] Yang Song and Prafulla Dhariwal. Improved techniques for training consistency models. In The Twelfth International Conference on Learning Representations, 2024. URL. https:
/ /openrevieww net worummid-nueyyyrrrrerrrro
13

---

[47] Monika E. Heringhaus, Yi Zhang, André Zimmermann, and Lars Mikelsons. Towards reliable parameter extraction in mems final module testing using bayesian inference. Sensors, 22(14): 5408, July 2022. ISSN 1424-8220. doi: 10.3390/$22144008 htt:ttp: :ttt:p://d/ ://:x//////// doi org/
10. (0.3330/s22144
[48] J.C.S Kadupitiya, Fanbo Sun, Geoffrey Fox, and Vikram Jadhao. Machine learning surrogates for molecular dynamics simulations of soft materials. Journal of Computational Science, 42: 101 107, 2020. doi: 10. (0.1016/j.oes.2020) 1000.01100 107.
[49] Stefan T. Radev, Marvin Schmitt, Valentin Pratz, Umberto Picchini, Ullrich Köthe, and Paul- Christian Bürkner. JANA: Jointly Amortized Neural Approximation of Complex Bayesian Models. In Robin J. Evans and Ilya Shpitser, editors, Proceedings of the 39th Conference on Uncertainty in Artificial Intelligence, volume 216 of Proceedings of Machine Learning Research, pages 1695-1706. PMLR, 2023.
[50] Marvin Schmitt, Daniel Habermann, Paul-Christian Bürkner, Ullrich Koethe, and Stefan T. Radev. Leveraging self-consistency for data-efficient amortized Bayesian inference. NeurIPS UniReps: the First Workshop on Unifying Representations in Neural Models, 2023.
[51] Ullrich Köthe. A review of change of variable formulas for generative modeling. arXiv preprint arXiv:2308.0240 02655,, 2023.
[52] Jan-Matthis Lueckmann, Jan Boelts, David S. Greenberg, Pedro J. Gonçalves, and Jakob H. Macke. Benchmarking simulation-bard based inference. arXiv preprint, 2021.
[53] Jakob Kruse, Lynton Ardizzone, Carsten Rother, and Ullrich Köthe. Benchmarking invertible architectures on inverse problems. 2021. Workshop on Invertible Neural Networks and Normalizing Flows (ICML 2019).
[54] Poornima Ramesh, Jat-his Lueckmann, Jan Boelts, Álvaro Greenberg, Pedro J Gonçalves, and Jakob H Macke. Gatsbi: Generative adversarial training for simulation-based inference. arXiv preprint arXiv:2203.0648/ 2022.
David S
[55] Stefan T. Radev, Marvin Schmitt, Lukas Schumacher, Lasse Elsemüller, Valentin Pratz, Yannik Schälte, Ullrich Köthe, and Paul-Christian Bürkner. Bayesflow: Amortized bayesian workflows with neural networks. Journal of Open Source Software, 8(89):5702, 2023. doi: 10.21 105/joss. 05702. URL https: /toii//o.///////o////ooo wwiddo...r.....oo .rgg 21 1 05/ joss 05702.
[56] A Gretton, K. Borgwardt, Malte Rasch, Bernhard Schölkopf, and AJ Smola. A Kernel Two- Sample Test. The Journal of Machine Learning Research, 13:723-773, 2012.
[57] Sean Talts, Michael Betancourt, Daniel Simpson, Aki Vehtari, and Andrew Gelman. Validating bayesian inference algorithms with simulation- calibration. arXiv preprint, 2018.
[58] Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poczos, Ruslan Salakhutdinov, and Alexander Smola. Deep sets, 2017.
[59] Samuel Wiqvist, Jes Frellsen, and Umberto Picchini. Sequential neural posterior and likelihood approximation. arXiv preprini, 2021.
[60] Eyke Hüllermeier and Willem Waegeman. Aleatoric and epistemic uncertainty in machine learning: an introduction to concepts and methods. Machine Learning, 110(3):457-066, 2021. ISSN 1573-0565.
[61] Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms, 2017.
[62] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. Nassir Navab, Joachim Hornegger, William M. Wells, and Alejandro F. Frangi, editors, Medical Image Computing and Computer-Asiisedd Intervention MICCA/ 2015, pages 234-241, Cham, 2015. Springer International Publishing. ISBN 978-3- 319-24574-4.
14

---

[63] Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. Nature, 521(7553):4366444
2015. ISSN 1476-4687. doi: (00038/natur115333 URL http: //x doi. ooi.org/10.
[64] Aakash Kumar Nain. Keras documentation: Denoising diffusion probabilistic model. https:
///eeaa.i//eeaaaaeeeee wttps//peeeee/a//aaaaaaaapa eexamp/ee/ggeeeaatii////pp/ppppp 2022. Accessed: 2023-11-27.
[65] PyABC. PYАBC documentation, multi-scale model:
Tumor
spheroid growth.
www../aateeeeo...oo/////atatttte multiscale agentt based.html, 2017.
Accessed: 2023-12-07.
[66] Dongjun Kim, Chich- Hsin Lai, Wei-Hsiang Liao, Naoki Murata, Yuhta Takida, Toshimitsu Uesaka, Yutong He, Yuki Mitsufuji, and Stefano Ermon. Consistency trajectory models: Learning probability flow ODE trajectory of diffusion. In The Twelfth International Confer- ence on Learning Representations, 2024. URL https:ttp///ccc... y8.DTDDD
hwtph::ttpppptteeee /ttp:///pppreeeeeee/eeeee.. net forum?ide
[67] Maximilian Dax, Stephen R. Green, Jonathan Gair, Jakob H. Macke, Alessandra Buonanno, and Bernhard Schölkopf. Real-time gravitational wave science with neural posterior estimation. Physical Review Letters, 127(24), 2021. ISSN 1079-7114, doi: 10.1103phyysrvleet. 127.241 103.
URL http: / dx. org. o.t........rr.......////eeeeeeee 117.22221......22.0....
15

---

A Consistency training details
Table 3 details all functions and parameters required for training. Except for the choice of parameters
sp, $1, E, Tmux and data we exactly follow the design choices proposed by Song and Dhariwal [46]. which improved our results compared to the choices in Song et al. [18]. Song and Dhariwal [46]
showed that that the Exponential Moving Average of the teacher's s parameter Φ should not be used in Consistency Training, therefore we omitted in this paper.
Table 3: Design choices: Table and values adapted our notation from Song and Dhariwal [46, Table 1].
Loss metric
d(x, V|xx y c²
Discretization scheme
N(k) min ,1, where K" =²= LK/ / + +1]
Noise schedule
t, where i ~ p(i) and p(i) erf [log(t;1]]- 1)eee erf
log(t:) Paee
V2PPa
v2Pas
Weighting function Skip connections
A(t,) = 1/(14+11 - t,)
Ckip (t) = /ma /(t(/((( ((t - c)² + 2an)) Coul (tun(t) = d e)/ Vodaa +
80 $1 50 except where indicated otherwise
0.00054 d, where dl is the dimensionality of
Parameters
P Peean = 1.11, Psu = 2.0
k 0...., K where K is the total training iterations
t E1/P + N(A)
1-1 (TIL
-2/0) P 7, 0.001, Tma 200
B
Evaluation metrics
Especially for multi-modal distributions, some of the standard metrics may behave in unexpected ways. This impedes interpretation of the results and can lead misleading conclusions. In this section, we want to highlight the peculiarities of some of the metrics, and highlight where caution is necessary when they are applied to the presented benchmarks.
B.1 C2ST
As an expressive classifier, C2ST is most sensitive to regions without distribution overlap. In our experiments, this results in pronounced punishment of overconfident posteriors. For limited training data and distributions without sharp edges (e.g., the Gaussian mixture model), some overconfidence usually given, as samples from the outer tails are simply not present in the training set. As the accuracy is used as measure, smaller number of very improbable samples (e.g., samples in the region between two modes) do not influence the results a lot.
In consequence, C2ST may deviate from the visual judgment in important ways. First, overconfident posteriors look more "precise", but are punished by C2ST. Second, samples between modes are very salient, but may not influence the results of C2ST lot.
For distributions with sharp edges and more less uniform density in given mode, c2st is very capable judging the overlap of two distributions, which then is good measure of quality. This applies the two-moons example, where budget leading to more accurate results (Figure 2b).
observe the expected behavior of larger training
B.2 MMD
By capturing and comparing the moments of two distributions, MMD provides a more holistic view than C2ST. It is also applicable to wider range of cases, as it generalizes to high dimensions. The exact behavior can be controlled by choosing the kernel function. our experiments, we observe MMD to be less sensitive for distributions with sharp edges (i.e., two moons), where C2ST seems to provide more reasonable results. We also observe a low sensitivity to samples between modes, leading to unexpected results compared to visual inspection.
16

---

B.3 Summary
Calculating meaningful metrics given only samples from two distributions is a challenging task. Ideally. the metrics would reward similar density and punish samples in the low-density regions of the reference distribution. Especially the latter seems to be hard to achieve. Even when metries work fine for small deviations from the ideal distribution, the behavior for different forms of larger deviations, as they occur in challenging problems or cases with low training budget, might not be easily grasped or interpreted. That makes it difficult to compare different methods producing qualitatively different deviations (e.g., overconfidence vs. underconfidence) from the reference distribution.
C Additional details and results
For neural network training, we used a Mac M1 CPU for Experiments 1-3, NVIDIA V100 GPU for Experiment 4, an NVIDIA RTX 4090 GPU for Experiment 5, and an NVIDIA H100 GPU for Experiment 6. The evaluation scripts were executed an NVIDIA V100 GPU for Experiments 3-4, on an NVIDIA RTX 4090 GPU for Experiment 5, and NVIDIA V100 GPU for Experiment 6. The computation time for Experiments 1-5 is approximately one day, with the evaluations (e.g., reference posteriors, C2ST computations, MMD computations taking up considerable amount of time. However, our proposed CMPE method is designed to be carried out on consumer-grade hardware for applied analyses on real data sets, which usually requires minutes hours of training and can be executed on CPUs depending on the application. In contrast, the large-scale analysis in Experiment 6 requires 2-3 days of training time on high- perforrrform computing infrastructure [see 11, for details on this setting].
Mac M1 CPU for Experiments 1-2, on
C.1 Experiment
Neural network details. All architectures use a summary network to learn fixed-length representa- tions of the i.i.d. data output. Both ACF and NSF use 4 coupling layers and train for 200 epochs with a batch size 32. CMPE relies on an MLP with 2 hidden layers of 256 units each, L2 regularization with weight 10-4 10% dropout and an initial learning rate of 10-4, The consistency model is instantiated with the hyperparameters Sy 10, $1 1280, Tiax 1. Training is based on 2000 epochs with batch size 64. FMPE and CMPE use an identical MLP, and the only difference is the initial learning rate of 10-5 for FMPE to alleviate instable training dynamics.
(x.,.... X10) We implement this via a DeepSet [58] with 6-dimensional
C.2 Experiment?
Neural network details. CMPE uses an MLP with 2 layers of 256 units each, L2 regularization with weight 10-3, 5% dropout, and initial learning rate of 5 10-4, The consistency model uses 80 = 10, 81 = 50, Timax = 10, and it is trained with batch size of 64 for 5000 epochs. Both ACF and NSF use 6 coupling layers of 128 units each, kernel regularization with weight 7 10-4 and train for 200 epochs with a batch size 32. FMPE uses the same settings and training configuration as CMPE.
C.3 Experiment 3
The three robot arm segments have lengths 0.5, 0.5, 1.0. The parameters = [6), 02 0g. 04) follow a Gaussian prior 0 ~ (O, I²1) with -² = L 4. 4). Neural network details. CMPE relies on an MLP with layers of 256 units each, L2 regularization with weight 10 5% dropout, and an initial learning rate of 5 x 10 - The consistency model uses = 10, $1 = 50, and trains for 2000 epochs with a batch size of 32. FMPE uses the same MLP and training configuration as CMPE. Both ACF and NSF use 6 coupling layers of 128 units each, kernel regularization with weight y 10-1 and train for 200 epochs with batch size 32. The reference posterior is obtained using approximate Bayesian Computation (ABC), with e = 0.002, which corresponds to 1/ 1/1000 of the arm length.
17

---

167 ms
139 ms
335 ms
0.9
0.70
78
0.65
73
0.8
ms
6545
512
2048
4096 Simulation Budget
8192
Simulation Budget
CMPE 10# (Ours)
ACF NSF
FMPE 30 FMPE
CMPE 10# (Ours)
ACF
FMPE FMPE
30#
CMPE 30# (Ours)
(Ours)
(a) Experiment (Gaussian Mixture Model)
(b) Experiment (Inverse Kinematics)
Figure 4: C2ST score of 4000 approximate posterior draws reference posterior (lower is better), we report SC over J = 100 unseen test examples. (a) For the GMM benchmark, we observe unexpected pattern for the dependency stays approximately constant across all methods, indicating that in this regime a higher training budget leads to inferior performance, for example due tendency overfit with more training data. It could also be sign that the C2ST is not a good quality metric for this benchmark, but the monotonically decreasing curve for higher-quality samples (i.e., more sampling steps) for FMPE in Figure 2a indicates that the behavior can probably be attributed to the training budget and not to the metric. See Appendix for more detailed discussion. (b) In this task, it is more challenging to achieve excellent C2ST scores because there is no aleatoric uncertainty in the data- generating process. CMPE outperforms ACF and NSF, FMPE performs best and can benefit most from the increased training budget.
the training budget. The C2ST score increases
18

---

C.4 Unstandardized wall-clock times
Table gives an overview of the observed wall-clock training times for the benchmark tasks in Experiments 1-3. These results have been measured a consumer-grade CPU (Apple Silicon MI processor). We emphasize that these do not constitute a principled timing benchmark under standardized conditions, and the training times between algorithms are not directly comparable. We train each algorithm for fixed number of epochs, rather than training until some pre-defined accuracy metric has been achieved. This is because no single reported metric holistically quantifies the accuracy in the investigated benchmark tasks, as explained in more detail Appendix B. Further, we observed degrading performance of both normalizing flow implementations for larger numbers of epochs despite dropout and regularization. Instead comprehensive benchmark timing test, the results shall show that CMPE does in fact require slightly longer training times compared to FMPE given the same neural network backbone. We attribute this to the student-teacher training scheme which necessitates more neural network evaluations (not gradient updates) during training. However, the much faster inference speed of CMPE constitutes significant advantage for real-time amortized Bayesian inference, which is why the drawback of longer neural network training is acceptable.
M 512
M
1024
M
2048
M
4096
M
8192
ACF
105
189
352
565
1056
NSF
118
203
359
646
1327
Gaussian Mixture
FMPE
715
1278
2008
3973
7508
CMPE
867
1471
2401
4390
10102
ACF
42
66
118
225
453
NSF
60
86
146
257
515
Two Moons
FMPE
175
289
498
890
1805
CMPE
195
323
571
1064
2173
ACF
43
70
124
234
439
NSF
63
99
169
297
552
Inverse Kinematics
FMPE
97
166
311
551
1095
CMPE
107
194
358
719
1397
Table 4: Wall-clock training times on the benchmark tasks. M denotes the available simulation budget for the neural network training stage.
19

---

1.0
C2ST + SE
0.9
0.8
0.7
0.6
0.5
10
15
20
25
30
35
40
45
50
# #Sampling steps
Two Moons
GMM
Inverse Kinematics
Figure 5: Experiment 1-3. C2ST score of 4000 approximate posterior draws vs. reference posterior (lower is better), we report mea over J 100 unseen test examples at different numbers of inference steps. The minimum C2ST value is achieved around 10-20 inference steps for every benchmark, after which the value increases again.
C.5 Empirical evaluation of optimal number of inference steps
For Experiments 1-3, evaluate the performance of CMPE (as indexed by C2ST) as function of the number sampling steps K during inference. Figure shows that there is no monotonic relationship where more compute leads to better results. Instead, we observe a U-shaped relationship with a performance maximum at = 10-20 sampling steps. This seemingly counter-intuitive phenomenon is consequence of the consistency training, which aims achieve few-step sampling and thus has no incentive to optimize the consistency function for a large number of sampling steps.
C.6 Experiment 4
The Fashion MNIST data set is released under MIT license [61]. We compare samples from 2 x 2 2 design: Two methods, CMPE and FMPE, are trained two network architectures, naive architecture and U-Net architecture, using 000 and 000 training images. Except for the batch size, which is 32 for 2 000 training images and 256 for 60000 training images, all hyperparameters are held constant between all runs. We use an Adam optimizer with a learning rate of 10 and cosine decay for 20 000 iterations, rounded up to the next full epoch. As both methods use the same network sizes, this results in an approximately equal training time between methods (approximately 1-2 hours Tesla V100 GPU), with significantly lower training time for the naïve architecture.
Neural network details. For CMPE, we use 80 = 10, 81 = 50, 22 0.25 and two-step sampling. For FMPE, we follow Wildberger al. [11] and 248 network passes. Note that this directly correlated with inference time: Choosing a lower value will make FMPE sampling faster, a larger value will slow down FMPE sampling. We show samples for each method (CMPE, FMPE), backbone architecture (MLP, U-Net), and simulation budget (2 000, 6 600) at the end of the appendix.
20

---

3.0
Lop Division Rato (8x)
2.0
1.0
-3.0
-2.5
-2.0
-1.5
-3.0
-1.5
-2.0
-is
-i.0
-3.0
-2.5
-2.0
-1.5
-10
-3.0
-2.
-2.0
-1.5 -1.0
Log Division Depth (6)
Log Division Depth (8))
Log Division Depth (8,)
Log Division Depth (8,)
(a) CMPE (Ours) 2 steps
(b) CMPE (Ours) 30 steps
(c) FMPE 30 steps
(d) FMPE 1000 steps
Figure 6: Experiment 5. Posterior draws for two parameters of the tumor growth model. Cross-hair and pink star indicate the ground-truth parameter that we aim recover for one fixed data x. The gray region depicts the prior distribution. CMPE (Ours) shows no visible bias even for two-step inference, but an improved sharpness when we increase the number of sampling steps 30, In contrast, FMPE with 30 sampling steps (comparable speed to CMPE) yields biased posterior. The posterior of FMPE with 1000 inference steps visibly improves but is orders of magnitude slower than CMPE with 30 steps.
C.7 Experiment 5
Neural network details. Both CMPE and FMPE use an MLP with 4 hidden layers of 512 units each and 20% dropout. CMPE and FMPE train for 1000 epochs with batch size of 64. The consistency model uses 80 = 10, 81 = 50, Tmax = 50. ACF and NSF employ a coupling architecture with learnable permutations, coupling layers of 128 units each, 20% dropout, and 300 epochs of training with batch size of 64.
The input data consists of multiple time series of different lengths. As proposed by Schmitt et al. [36] for SBI with heterogeneous data sources, we employ a late fusion scheme where the input modalities first processed by individual embedding networks, and then fused. First, the data tumor size growth curves enters an LSTM which produces a 16-dimensional representation of this modality. Second, the information about radial features is processed by temporal fusion transformer with attention heads, 64-dimensional keys, 128 units per fully-connected layer, and 10 1 1.2 regularization on both weights and biases. These latent representations then concatenated and further fed through a fully-connected two-layer perceptron of 256 units each, 10 L2 regularization on weights and biases, and 32 output dimensions.
Figure 7 shows calibration plots for all methods. We observe that our CMPE method with inference steps leads to the best calibration, with affine coupling flows and neural spline flows performing similarly with respect to the expected calibration error (ECE).
21

---

Affine Coupling Flow (ACF) RMSE 0.589, max ECE 0.021, inference time 1.07s
Fractional Rank Statstic
Neural Spline Flow (NSF)
RMSE 0.590, max ECE 0.027, inference time 1.95s
BPC
Conkbratte
Doricmmaten
Fractional Rank Stafistic
Flow Matching (FMPE), steps RMSE 0,582, max ECE 0. 222 inference time 17.13s
Fractional
Flow Matching (FMPE), 1000 steps
RMSE 0.583, max ECE 0.057, inference time 500.90s
Fractional Rank Statstic
Consistency Model (CMPE; Ours), 2 steps RMSE 0.616, max ECE 0.064, inference time 2.16s
Fractional Rank Statstic
Consistency Model (CMPE; Ours), 30 steps RMSE 0.577, max ECE 0.018, inference time 18.33s
Fractional Rank Statitic
Figure 7: Calibration plots for Experiment 5. The gray envelopes represent the 95% confidence bands for sufficient calibration. Inference times refers to the wall-clock time (in seconds) required to draw 2000 posterior samples. We emphasize the best performance for each metric in green, and the worst performance for each metric in darkred.
22

---

Method
CMPE FMPE
1000-50
de [Mрc]
4g [uus]
Figure 8: Pairplot of univariate and bivariate posterior draws obtained from CMPE (ours) and FMPE. FMPE clearly outperforms CMPE in this application, where the latter shows pathologically underexpressive posteriors that do not capture the nuances with acceptable detail (see text for details and hypotheses).
C.8 Experiment 6
In this additional experiment, we apply CMPE to gravitational wave inference of the binary black hole merger GW 150914. Our analysis parallels the study of Wildberger et al. [11] who apply FMPE to this challenging scientific task. To this end, we implement our CMPE algorithm in the Dingo library for analyzing gravitational wave data using neural posterior estimation [67]. Dingo implements sophisticated analysis pipeline that is carefully tailored to gravitational wave inference. As in previous experiments, we compare FMPE and CMPE based on identical neural network backbones with total of = 190 Million neural network weights, and we refer to Wildberger et al. [11] for detailed description of the experimental setup. The consistency model uses 50 10, $1 1280, Tmax 50, daaia = 1.
Similar Wildberger et al. [11], show the bivariate posterior marginals for subset of all 14 inference parameters in Figure 8. FMPE successfully captures the posterior geometry [which matches ground-truth posterior from slow reference method; 11]. In contrast, CMPE yields a pathologically underexpressive posterior which mostly resembles a multivariate Gaussian (with some minor mixture contributions for a). Yet, the maximum posteriori (MAP) estimates of CMPE and FMPE mostly align (up to CMPE covering both posterior modes for a), which indicates that CMPE does distill information from the model. We suspect that the pathologically poor performance of CMPE is related to (i) issues in our re- re-implementation of CMPE interfacing the Dingo library; or (ii) misaligned hyperparameters for this specific application. We will continue troubleshooting and invite the interested reader to watch the accompanying GitHub repository for updates.
23

---

Param. 0
Obs.
Sample
Sample
Sample
Sample
Sample
T-Shirt/Top
Trouser
Pullover
Dress
Coat
Sandal
Shirt
Sneaker
Bag
Ankle Boot
Figure 9; CMPE U-Net architecture 2 200 training images. CMPE denoising results from each class of Fashion MNIST obtained using a U-Net architecture and two-step sampling. A small training set of 2 000 images was used.
24

---

Param.
Obs.
Sample
Sample
Sample
Sample
Sample
T-Shirt/Top
Trouser
Pullover
Dress
Coat
Sandal
Shirt
Sneaker
Bag
Ankle Boot
Figure 10: CMPE U-Net architecture 60 000 training images. CMPE denoising results from each class of Fashion MNIST obtained using U-Net architecture and two-step sampling. A large training set of 60 000 images was used.
25

---

Param.
Obs.
Sample
Sample
Sample
Sample
Sample
T-Shirt/Top
Trouser
Pullover
Dress
Coat
Sandal
Shirt
Sneaker
Bag
Ankle Boot
Figure 11: FMPE- U-Net architecture 2 2000 training images. FMPE denoising results from each class of Fashion MNIST obtained using U-Net architecture. A small training set of 2 000 images was used.
26

---

Param.
Obs.
Sample
Sample
Sample
Sample
Sample
T-Shirt/Top
Trouser
Pullover
Dress
Coat
Sandal
Shirt
Sneaker
Bag
Ankle Boot
Figure 12: FMPE U-Net architecture 60 000 training images. FMPE denoising results from each class of Fashion MNIST obtained using U-Net architecture. A large training set of 60 000 images was used.
27

---

Param.
Obs.
Sample
Sample
Sample
Sample
Sample
T-Shirt/Top
Trouser
Pullover
Dress
Coat
Sandal
Shirt
Sneaker
Bag
Ankle Boot
Figure 13: CMPE Naïve architecture 2 200 training images. CMPE denoising results from each class of Fashion MNIST obtained using the naïve architecture described above and two-step sampling. A small training set of 000 images was used.
28

---

Param.
Obs.
Sample
Sample
Sample
Sample
Sample
T-Shirt/Top
Trouser
Pullover
Dress
Coat
Sandal
Shirt
Sneaker
Bag
Ankle Boot
Figure 14: CMPE Naïve architecture 60 000 training images. CMPE denoising results from each class of Fashion MNIST obtained using the naïve architecture described above and two-step sampling. A large training set of 60 000 images was used.
29

---

Param.
Obs.
Sample
Sample
Sample
Sample
Sample
T-Shirt/Top
Trouser
Pullover
Dress
Coat
Sandal
Shirt
Sneaker
Bag
Ankle Boot
Figure 15: FMPE Naïve architecture 000 training images. FMPE denoising results from each class of Fashion MNIST obtained using the naïve architecture described above. A small training set of 000 images was used.
30

---

Param.
Obs.
Sample
Sample
Sample
Sample
Sample
T-Shirt/Top
Trouser
Pullover
Dress
Coat
Sandal
Shirt
Sneaker
Bag
Ankle Boot
Figure 16: FMPE- Naïve architecture 000 training images. CMPE denoising results from each class of Fashion MNIST obtained using the naïve architecture described above. A large training set of 60 000 images was used.
31

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We claim that consistency models combine the expressiveness of unconstrained neural network architectures with fast sampling speed. We demonstrate this combination of desirable properties across all five experiments.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: We discuss two potential drawbacks of consistency models for simulation- based inference: (i) The relation between inference time (number of sampling steps) and performance (posterior accuracy) is not monotonie; and (ii) consistency models do not directly yield tractable densities for arbitrary parameter values. We mention these drawbacks throughout the main text and also included dedicated Limitations section. Further, we propose solutions to these issues in Section 3.6, Section and Section 3.5.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "LLimitations" section in their paper.
The paper should point any strong assumptions and how robust the results to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
32

---

3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proof' can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented
by formal proof's provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We disclose all details of the neural network architectures, neural network training, data (synthetic simulators or fixed data sets), and evaluation metrics.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (c.g., to registered users), but it should possible for other researchers have some path reproducing verifying the results.
33

---

reproducible, they 5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification:
We provide open access the code reproduce all results.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https:/ /atp:://npppe.oe cc/
public guides Codeeuuomis ssionn
ssmmmsosssspiyyyooooooooooooommc olicy) for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: / n ps cc/ ccc/pblls i c/ gu i des CodeSubdr i ss i onPol icy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We specify all training and test details in the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We quantify and report uncertainty in all experiments, which is fundamentally important for our application in probabilistic (Bayesian) inference.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
34

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: We report the compute resources and approximate execution time to reproduce the experiments in Appendix C.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: /tap:::://neuue /tw.rpssu..pp/.e cc
Answer: [Yes]
Justification: respect.
The research in this paper confirms with NeurIPS Code of Ethics in every
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative socictal impacts of the work performed?
Answer: [Yes]
Justification: We discuss the positive impact on scientific progress in the Introduction. As this paper is generally concerned with foundational research, the specific benefit or harm largely depends the underlying application.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
35

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no such risk as we do not release any large-scale models, generators, data sets.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We cite all original papers of used henchmarks, evaluation tasks, data sets, and metries.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
36

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms of use the package should be provided. For popular datasets, papers thoode com/ dat ssets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmodel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
37

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
38


Reviews:
[{"review_id": "Zv0HzNPJHP", "reviewer": "Reviewer_XVP6", "summary": "This paper adapts the recently introduced Consistency Models of\nSong et al to the task of Simulation-based Inference. Compared to\nthe previous approaches based on flow matching, this technique\nexhibits similar or better quality. All while being significantly\nfaster to sample from with fewer restrictions on the underlying\nneural architecture.", "questions": "Why for the Gaussian Mixture Model does the C2ST score go up as\nthe computational budget is increased? Is this just a property of\nthe dataset being so small?\n\nWould it be possible to have a plot that shows the evaluations as a\nfunction of wall-time? One of the major advantage of this method is\nit can get away with doing significantly less sampling, and it would\nbe nice to have a plot that showcases that", "limitations": "Yes they have been adequately discussed in paper.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "I am not familiar with any work that quite does this. While there do\nnot seem to be novel technical challenges required to adapt CMs for\nthe Simulation-based Inference. This is still the first.\n\nThis paper is very well written and organised. The contribution is described clearly and contextualised well\nin the related work.\n\nThe technical quality of this work is very good. All details are well-explained, and the experiments are\nreasonably chosen with bost smaller illustrative toy examples and larger more realistic ones.\n\nThe work is significant as the massive performance improvements and flexibility in architecture make\nthis something I expect many researchers to build upon.", "weaknesses": "I do believe this paper would benefit for slightly reworked plots since the main advantage is just how much more efficient of an inference method it is. I already appreciate that wall-times are included in many places, but something like Figure 2 would be even better if that was taken into account.\n\nIt appears to be a relatively straightforward application of CMs to SBI.\n\nI do think the paper would be stronger if the specific restrictions on the density estimation were mentioned in the main paper vs Appendix C\n\nThis paper would benefit from an additional larger example.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive review of our work. We appreciate that you find our paper well-written, clearly organized, of very good technical quality, and significant to the field. We have addressed your concerns and are optimistic that this will substantially increase the quality of our manuscript.\n\n---\n\n## W1: Plots \n\n> I do believe this paper would benefit for slightly reworked plots since the main advantage is just how much more efficient of an inference method it is. I already appreciate that wall-times are included in many places, but something like Figure 2 would be even better if that was taken into account.\n\nThank you, we agree that it is an excellent idea to present wall-clock times wherever possible to underscore the massive improvements in speed and performance through our CMPE method. \n\n**Question:** Could you please clarify where exactly you would like us to take wall times into account in Figure 2? Figure 2a contains the wall times (i.e., sampling speed) on the x-axis and Figure 2b contains the wall times (i.e., sampling speed) as labels for each inference method. \n\nIf “sampling speed” is too vague a term, we will happily replace it with a term like “inference wall time” or similar.\n\n---\n\n## W2: Straightforward application\n\n> It appears to be a relatively straightforward application of CMs to SBI.\n\nWe agree that the application of consistency models to SBI is conceptually straightforward, and remark that many of the recent advancements in neural SBI have been fueled by the general progress of generative neural networks (e.g., normalizing flows, flow matching, score-based diffusion, …). We demonstrate that consistency models lead to further progress, especially to a favorable trade-off between sampling time and quality. We further provide default hyperparameters as a starting point for a wide range of SBI applications, which will help practitioners use consistency models for SBI in real-world analyses. Upon publication, we will release an implementation of our CMPE in an open-source Python package which is already used by practitioners in application domains.\n\n---\n\n## W3: Density estimation remarks to the main text\n\n> I do think the paper would be stronger if the specific restrictions on the density estimation were mentioned in the main paper vs Appendix C\n\nWe agree and we will use the additional page in the camera-ready version to discuss the main points of the density estimation restrictions in the main text. \n\n---\n\n## W4: Additional larger example\n\n> This paper would benefit from an additional larger example.\n\nThanks for this recommendation. Do you have a specific example in mind that would underscore the benefits of CMPE? Based on the comment of reviewer GeMw, we are currently applying CMPE to gravitational wave inference, where the observations (i.e., conditioning variables) are very high-dimensional. Would this address your comment?\n\n---\n\n## Q1: Computational budget and performance\n\n> Why for the Gaussian Mixture Model does the C2ST score go up as the computational budget is increased? Is this just a property of the dataset being so small?\n\nWe are not completely sure, but your hypothesis aligns with how we tend to interpret it. In this benchmark, we observe that for an increasing computational budget, there is an increased tendency to produce overconfident posteriors. In Hermans et. al [1], Figure 2, we see that the amount of overconfidence can increase in the low-budget regime (e.g. for Spatial SIR/SNPE), but that this isn’t the usual pattern. As can be seen in that figure, usually, the posteriors become less overconfident with an increased computational budget, as one would assume. Therefore we assume that the C2ST score will decrease again for significantly larger budgets.\n\n[1] Hermans, J., Delaunoy, A., Rozet, F., Wehenkel, A., Begy, V., & Louppe, G. (2022). A Crisis In Simulation-Based Inference? Beware, Your Posterior Approximations Can Be Unfaithful. Transactions on Machine Learning Research. https://openreview.net/forum?id=LHAbHkt6Aq \n\n---\n\n## Q2: Evaluations as a function of wall-time\n\n> Would it be possible to have a plot that shows the evaluations as a function of wall-time? One of the major advantage of this method is it can get away with doing significantly less sampling, and it would be nice to have a plot that showcases that\n\nWe share your opinion that an evaluation as a function of wall time is important to assess the faster inference speed of CMPE while maintaining high inference quality. We provide such a plot in Figure 2a ($x$ axis: inference wall-time, $y$ axis: posterior quality) for the Gaussian Mixture Model experiment and, following your suggestion, will use the additional page in the camera-ready version to present more plots of this type for other experiments.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ROySlwdIWr", "reviewer": "Reviewer_GeMw", "summary": "Adopting the idea from consistency models in the generative process, the authors propose its application for posterior estimation, introducing a new type of model for simulation-based inference (SBI). The proposed method, consistency models for posterior estimation (CMPE), enjoying the benefits of the consistency models, can support unconstrained architecture and is more efficient in the sampling process. Empirically, the authors conducted experiments on low-dimensional datasets, including Two Moons, GMMs, and Kinematics, and high-dimensional datasets, including Fashion MNIST.", "questions": "Would you provide a comparison of the training time? And the performance regarding the hyper-parameters of the CMPE.", "limitations": "N/A", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "Overall, the paper is well-written and easy to follow.\n\nThe authors try to address the computational efficiency problem of the current SBI methods, which is essential.\n\nThe authors provided code and implementation details of the experiments, implying high reproducibility of their results.", "weaknesses": "W1: The current experiments might be based on relatively simple datasets. While these are useful for initial validation, they may not adequately demonstrate the model's ability to handle real-world complexities and high-dimensional data. The authors could consider more complex datasets with complex distributions, like multimodal distributions. GW150914 in https://arxiv.org/pdf/2305.17161 might be a good case to try.\n\nW2: The training time for consistency models might be problematic for scaling to high-dimensional distributions, and its stability has been known as a problem for complex distribution.\n\nW3: My main concern is that the motivation for applying a consistency model for SBI might be weak, as consistency models are known to have stability issues for learning complex distributions. The current experiments only verify the computational advantage in the sampling process. However, the increase in the training time and the instability of training might pose more difficulties for real-world applications.\n\nW4: The performance of the consistency models relies heavily on tuning a series of hyper-parameters, which might be a problem for generalization.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thorough assessment of our work, and for the actionable issues you pointed out. We appreciate that you found our paper well-written, easy to follow, relevant for the SBI field, and reproducible through the code we shared.\n\n---\n## W1: Data set selection\n> W1: The current experiments might be based on relatively simple datasets. [...] The authors could consider more complex datasets with complex distributions, like multimodal distributions. GW150914 might be a good case to try.\n\nWe agree with you about the need for adequate empirical analyses to evaluate SBI methods. In fact, all three benchmarks in Experiment 1 have multimodal posterior distributions (the inverse kinematics is multimodal in parameter space and we just plot the posterior predictive in 2D data space which is not multimodal). In the updated manuscript, we emphasize this more clearly.\n\nAs you acknowledge in your summary, our Fashion MNIST task is quite high-dimensional (784D parameter and data spaces) for typical SBI applications in science. Other tasks, like GW150914, have high-dimensional observations, but low-dimensional parameters. Based on your recommendation, we will apply our CMPE method to GW150914. Due to the long simulation and pre-processing time to generate the training set, we are unlikely to have results during the discussion period. If you have any aspects that we should account for, please let us know and we'll consider them.\n\n---\n## W2/W3: Training time and stability\n> W2: The training time for consistency models might be problematic for scaling to high-dimensional distributions, and its stability has been known as a problem for complex distribution.\n\nThank you for the good points, which we will pay more attention to in the revised version of the paper. The training time for consistency models seems comparable to that of flow matching. Moreover, while constrained architectures (i.e., normalizing flows) need fewer iterations until convergence, the free-form architecture of consistency models (and flow matching) makes for a much faster neural network pass *within every single iteration*. See our response below regarding the good stability of CMPE in SBI.\n\n> W3: My main concern is that the motivation for applying a consistency model for SBI might be weak, as consistency models are known to have stability issues for learning complex distributions. [...]\n\nIn our experiments, we did not observe issues regarding instability or infeasible training times for CMPE (if anything, CMPE appeared more stable than normalizing flows without additional tricks, such as heavy-tailed latent distributions). However, we acknowledge that other studies have found problems for high-dimensional image generation tasks, where training times are a common bottleneck due to the sheer scale of the problem.\n\nIn our setting of amortized inference, we are typically not very concerned about slightly longer training times, which are usually in the magnitude of minutes to two-digit hours. Instead, we want to achieve maximally accurate, well-calibrated, and fast *inference*. As we demonstrate in our experiments, CMPE achieves this by uniting fast inference speeds (like normalizing flows) with high accuracies (like flow matching). In Experiment 1, we repeated the neural network training for different simulation budgets and did not observe training runs that went “off-rails” and would indicate stability concerns.\n\n---\n\n## W4: Hyperparameters\n\n> W4: The performance of the consistency models relies heavily on tuning a series of hyper-parameters, which might be a problem for generalization.\n\nWe agree that consistency models introduce additional hyperparameters compared to flow matching and normalizing flows, with the number of hyperparameters being similar to that of modern score-based diffusion samplers. In the paper, we propose a set of hyperparameters that performed well throughout our experiments, and which we suggest as defaults for SBI tasks. We will use the additional page of an eventual camera-ready version to add a section that discusses hyperparameters and explicitly states our recommendations for SBI.\n\n---\n## Q1: Training time\n\n> Would you provide a comparison of the training time?\n\nWe observe that the training times of CMPE are of the same magnitude as FMPE. In a rough estimate, all else being equal CMPE tends to be 10-25% slower during training. However, using the same number of epochs is an arbitrary choice, it would be better to compare training times required to reach a certain “quality”. As this is hard to define and would probably be very noisy, we did not perform such an experiment to avoid invalid conclusions.\n\nWe did not have the time to repeat the entire experimental suite ad-hoc during the rebuttal period but could recreate approximate training times for the inverse kinematics benchmark based on timestamps of our checkpoint directories. We will repeat the benchmark experiments in due time and add the properly measured times to the paper. M denotes the simulation budget. All times in seconds.\n\n| Model| M=512 | M=1024|M=2048|M=4096|M=8192|\n|-|-|-|-|-|-|\n| AC|—| 67| 112| 217|424|\n| NSF| 61| 96|155| 289|495|\n| CMPE (Ours)|128|219| 428| 789| 1483|\n| FMPE|105|193|360| 629| 1236|\n\n---\n## Q2: Hyperparameters\n> And the performance regarding the hyper-parameters of the CMPE.\n\nAs there are several interacting hyper-parameters, a quantitative analysis of the performance concerning the different hyper-parameters would be highly expensive and probably hard to interpret. We searched this space in an earlier project stage and found a set of parameters that consistently yield good performance in SBI. We gathered some insights on which hyper-parameters are important and are most likely to influence the results. In our experiments, we identified $s_0$, $s_1$, and $T_{\\text{max}}$ as the most relevant for sharpness and calibration. We will add a section on hyper-parameters and our recommendations for SBI in a camera-ready version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Fm8LCh4vVU", "reviewer": "Reviewer_bX9J", "summary": "The paper proposes to use a conditional consistency model for amortized likelihood-free inference. Empirical evaluation shows that this approach compares favorably in terms of inference time as well as performance against competing methods.", "questions": "1. Could you provide the coverage curves from which ECE is calculated?\n2. Fig 2 b), Fig 4 a) and b) - it looks like the standard deviation around the points is 0. Is it so small that one cannot see it in the plot or is it missing from the figures?\n3. Why is ECE reported only for Experiment 5?", "limitations": "Yes, the limitations of the proposed method are described.", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 2, "strengths": "- The paper is the first to propose the use of consistency models, which is a fairly new family of models, in the likelihood-free inference task.\n- The quality of the experiments conducted and the results obtained is high.\n- The paper is well written. The methods used are well described with references to sources.\n- The presented results may encourage practitioners to use the described method.\n- The authors identify the lack of density evaluation from the consistency model as a weakness of the proposed method, and discuss potential solutions to overcome it.", "weaknesses": "- The main weakness of this work is the lack of novelty. It comes down to applying an existing model that shows very good performance in computer vision to likelihood-free inference, which has not been published yet. However, someone has to be first, and this work is well suited to this purpose due to its high quality.\n- Little attention is paid to all the \"design choices\" summarized in Table 3. For comparison, reference [41], on which the authors base their work, presents an analysis of the impact of the hyper-parameters used.\n- The authors do not evaluate the proposed solutions to the lack of density evaluation limitation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thorough review and the excellent questions you raise. We appreciate that you attest excellent scores regarding presentation as well as soundness, and we are delighted by your optimism that our paper may encourage practitioners to use CMPE in their work.\n\n---\n\n## W1: Lack of novelty\n> The main weakness of this work is the lack of novelty. It comes down to applying an existing model that shows very good performance in computer vision to likelihood-free inference, which has not been published yet. However, someone has to be first, and this work is well suited to this purpose due to its high quality.\n\nWe appreciate your observation regarding the novelty of our approach. While it is true that our study transfers an existing model to the domain of likelihood-free inference, we believe that continuous exploration and benchmarking of algorithms across different fields are crucial. This is especially true since likelihood-free inference offers a unique test bed for an objective assessment of the statistical performance of conditional generative models and thus informs the mainstream applications about appropriate design choices.\n\n---\n\n## W2: Design choices\n> Little attention is paid to all the \"design choices\" summarized in Table 3. For comparison, reference [41], on which the authors base their work, presents an analysis of the impact of the hyper-parameters used.\n\nThank you for this remark. We will use the additional page of the camera-ready version to add a section that discusses hyper-parameters and explicitly states our recommendations for SBI. In our experiments, we identified $s_0$, $s_1$, and $T_{\\mathrm{max}}$ to be the most relevant to tune in order to achieve good sharpness and calibration. We will highlight this and more prominently refer to reference [41] for a discussion of the remaining hyper-parameters, to keep the paper concise.\n\n---\n\n## W3: Density evaluation limitation\n> The authors do not evaluate the proposed solutions to the lack of density evaluation limitation.\n\nYou are correct, and we are now more explicit about the density evaluation limitation. While this is not a problem for posterior sampling, it will require more research until consistency models can be used, for example, as surrogate likelihoods in tandem with MCMC samplers. \n\n---\n\n## Q1: ECE coverage plots\n> Could you provide the coverage curves from which ECE is calculated?\n\nThank you for this excellent suggestion! We computed coverage curves for all methods and compiled them in the PDF that we attached to the “Author Rebuttal” at the top of the OpenReview page (could not attach to this reviewer-specific rebuttal). Further, we will add the coverage curves in the updated paper version.\n\n---\n\n## Q2: Uncertainty bands in figures\n> Fig 2 b), Fig 4 a) and b) - it looks like the standard deviation around the points is 0. Is it so small that one cannot see it in the plot or is it missing from the figures?\n\nThe uncertainty bars are vanishingly small, which is a consequence of the stable evaluation (low variation across the test data set). We will revise the figure to increase the visibility of the uncertainty bars.\n\n---\n\n## Q3: ECE reporting\n> Why is ECE reported only for Experiment 5?\n\nIn experiments 1–3, we have access to ground-truth posterior samples and thus report more powerful metrics that quantify the distance between the approximate and ground-truth posteriors (i.e., C2ST and MMD). \n\nIn Experiments 4 and 5, we only have ground-truth parameters (no full posteriors) and thus resort to metrics that do not compare against full posteriors. Posterior inference on images is notoriously ill-calibrated for some pixels, hence we do not report ECE in Experiment 4. Here, visual inspection is a powerful tool to compare different methods, which is why we display images. \n\nExperiment 5 is a scientific application where calibration is achievable and of paramount importance. We compute the probabilistic calibration via simulation-based calibration (SBC; [1]). We currently report ECE in addition to RMSE which quantifies bias and variance and will also add the corresponding coverage curves [2, 3] to the paper (see your question above).\n\n[1] Talts et al. (2018. Validating Bayesian Inference Algorithms with Simulation-Based Calibration. https://arxiv.org/abs/1804.06788\n\n[2] Säilynoja et al (2022). Graphical test for discrete uniformity and its applications in goodness-of-fit evaluation and multiple sample comparison. https://doi.org/10.1007/s11222-022-10090-6\n\n[3] Radev et al. (2023). JANA: Jointly amortized neural approximation of complex Bayesian models. https://proceedings.mlr.press/v216/radev23a.html", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KaO7lpUTR1", "reviewer": "Reviewer_iNhZ", "summary": "The paper adopts consistency models for simulator-based inference tasks, highlighting its expressive free-form architectures and fast inference as main advantages. The new method, called consistency model posterior estimation, was shown to outperform Neural Posterior Estimation (NPE) and be competitive with Fully Neural Posterior Estimation (FMPE) methods in terms of amortized inference quality.", "questions": "Abstract\n\n(line 1): Consider revising the first sentence \"Simulation-based inference (SBI) is constantly in search of more expressive...\" to be more concrete.\n\n(line 5): Clarify what is meant by 'free-form architectures'.\n\n(line 5-6): Some SBI methods have long inference times, and 'overcoming sampling inefficiency at inference time' may be bad for them --perhaps some clarification is needed.\n\n(line 8-9): 'Providing an architecture' slightly contrasts with the advantage of having a free-form architecture, so some justification is need.\n\n(line 10): Could you specify the dimensions this line refers to and explain why low-dimensional simulator parameters were difficult to handle with unconditional consistency models and why not simply use other SBI methods instead?\n\nIntroduction\n\n(line 20): Could you rephrase to avoid broad generalizations about SBI research, as there are many recent works focused, for instance, on choosing summary statistics with little relation to generative modeling?\n\n(line 31): Could you clarify the term 'free-form architecture'?\n\n(line 36): Please define 'target spaces' in the context of SBI applications.\n\n(lines 41-43): I don't understand the significance of this sentence. Do authors imply that they explore empirically the quality of conditional CMs and it is one of the contributions? It follows the sentence about the empirical evaluations in the paper, so I am a little bit confused.\n\nSection 2\n\n(line 59): I am not familiar with the term 'simulation-based training'. Perhaps the point of the term is to emphasize the synthetic nature of the data generated by the simulator or some other reason?\n\n(line 66, 70): I am not sure why simulation models are presented as 'programs'. This may be confusing as it is not common for the SBI literature in general. Later authors use the term 'program states', but it has very little to do with the setting they operate in. Therefore I would propose to either avoid such terminology or explicate the setting they operate in (if it differs from the 'classical' SBI setting).\n\nTypos: (line 71): 'g( \\cdot, \\cdot)'. (line 74): 'reasoning' (line 79): 'sequential and amortized'. (line 95): 'it' (line 124): 'they'\n\n(line 101): Please provide a reference for 'optimal transport'.\n\nSection 3\n\n(line 151): x is presented as “a fixed conditioning variable x”, but it is already used for observations in the SBI context. Could you clarify in the text if they are the same?\n\n(line 177): Could you at least briefly detail the ‘hardships for computing the posterior density’ in the main text?\n\n(Eq 8): Could you provide justification or references for this objective?\n\nExperiments\n\n(line 234): Could you provide a reference, where the hyper-parameters for the comparison methods are listed?", "limitations": "The authors have adequately addressed the limitations of their work in a separate section.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper demonstrates a thorough experimental analysis of the method, with diverse case studies and a comprehensive list of comparison methods. This suggests that the proposed method is reasonable and applicable for simulator-based inference. The method appears to be novel and addresses a well-established task, improving the speed of inference and simplifying hyper-parameter selection. The submission is mostly technically sound, with the main claims supported. The authors are transparent about the limitations of their work and have published the code, which is commendable.", "weaknesses": "The main weaknesses lie in the presentation and the overall results of the paper. The paper is quite difficult to follow, and the overall contribution seems relatively small. The adoption of the consistency model for simulator-based inference (SBI) appears straightforward, and some methodological decisions, such as those in Section 3.3, lack justification. The significance of the contribution may be understated due to the presentation issues in Sections 1-3, which need significant improvement to strengthen the proposal for this new method. Detailed suggestions for improvement are provided below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your assessment of our work, as well as the detailed list of questions and comments. We responded to your general points below and incorporated all items of your list of questions/edits in our manuscript. We hope that this addresses your concerns regarding the paper’s presentation and justification. \n\nIf any remaining reservations prevent you from recommending our paper for acceptance, please let us know and we are happy to further engage in a conversation during the author-reviewer discussion period.\n\n---\n\n## Clarifying question to the reviewer \n\nIn your summary, you write that our CMPE method is “competitive with Fully Neural Posterior Estimation (FMPE)”. \nAre you referring to Flow Matching Posterior Estimation (FMPE), and “Fully Neural Posterior Estimation” is a typo?\nWe observe that our CMPE method is consistently 50–100x faster than FMPE, performs comparably with FMPE in 2 of 5 experiments (w.r.t. accuracy, calibration, or distance to a ground-truth posterior), and even outperforms FMPE in the other 3 experiments. Hence, the conclusion “competitive with FMPE” is an understatement in light of the empirical results we present.\n\n---\n\n## W1: Presentation\n> The main weaknesses lie in the presentation and the overall results of the paper. The paper is quite difficult to follow [...]. The significance of the contribution may be understated due to the presentation issues in Sections 1-3, which need significant improvement to strengthen the proposal for this new method. Detailed suggestions for improvement are provided below.\n\nThank you for voicing concerns about the presentation of our paper. The other reviewers remark that the paper may have a large significance for the SBI community. Therefore, we are particularly interested in making the paper accessible to a broad audience and will gladly make the necessary edits to ensure this.\n\nWithin the scope of this review process, we are surprised to see the stark contrast to the assessment of the other three reviewers, who explicitly remark that the paper is well-written (scores: good–good–excellent) and technically sound (scores: excellent–fair–excellent).  We are keen to improve the accessibility of the paper. To this end, we have addressed all of your detailed comments below and updated the manuscript accordingly.\n\n---\n\n## W2: Justification of decisions\n> The adoption of the consistency model for simulator-based inference (SBI) appears straightforward, and some decisions, such as those in Section 3.3, lack justification.\n\nThank you for this observation. Given that consistency models are relatively new, there is no theory (yet) that allows detailed reasoning about their hyper-parameters. Therefore, the decisions were made based on empirical observations. Most of the design decisions were made identical to those presented in [41], with some changes to fit the SBI settings. We will use the additional page of an eventual camera-ready version to add a section that discusses hyper-parameters and explicitly states our recommendations for SBI.\n\n---\n\n## Answers to the detailed list as \"Official Comment\"\n\nDue to space constraints, we have to post our answers to your detailed list of comments and edits as an **Official Comment** object in OpenReview. Please kindly let us know if that is not visible to you.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all reviewers for their assessment of our work. All reviewers agree that conditional consistency models are a promising technique for amortized simulation-based inference, that the topic is relevant to the NeurIPS community, and that our paper is technically sound.

Below, we give a detailed rebuttal for each reviewer, and we have incorporated all comments into a revised version of the manuscript. Thank you for your time and service in reviewing our work, which substantially strengthened the paper and hopefully renders it a valuable contribution to NeurIPS 2024.

**Note.** We observe a strong discrepancy in the assessment of our presentation and clarity of writing: Reviewers XVP6, GeMw, and bX9J conclude that our paper is well-written and easy to follow. The reviewers express this with explicit comments and high scores for presentation (good–good–excellent). In contrast, reviewer iNhZ lists the presentation as the paper’s major weakness. Based on this discrepancy, we would appreciate it if the AC could provide their opinion in this regard as well. We would like to point out that reviewer iNhZ has kindly provided a detailed list of minor edits and clarifications; we have addressed all of these items and hope that this alleviates the remaining reservations.

Meta-review: The paper proposes to use a conditional consistency model for amortized likelihood-free inference.  Their experimental evaluation shows that the inference speed of this approach is much better that existing methods.  While some reviewers initially has some concerns about the experimental evaluation and the clarity of presentation, their concerns seem to have been adequately addressed in the rebuttal phase.
Decision: Accept (poster)