Consistency Models for Scalable and Fast Simulation-Based Inference

Authors: marvin  schmitt
valentin  pratz
paul-christian b ürkner
stefan t radev
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 17:38:50

Abstract: Simulation-based inference (SBI) is constantly in search of more expressive and efficient algorithms to accurately infer the parameters of complex simulation models.
In line with this goal, we present consistency models for posterior estimation (CMPE), a new conditional sampler for SBI that inherits the advantages of recent unconstrained architectures and overcomes their sampling inefficiency at inference time.
CMPE essentially distills a continuous probability flow and enables rapid few-shot inference with an unconstrained architecture that can be flexibly tailored to the structure of the estimation problem.
We provide hyperparameters and default architectures that support consistency training over a wide range of different dimensions, including low-dimensional ones which are important in SBI workflows but were previously difficult to tackle even with unconditional consistency models. 
Our empirical evaluation demonstrates that CMPE not only outperforms current state-of-the-art algorithms on hard low-dimensional benchmarks, but also achieves competitive performance with much faster sampling speed on two realistic estimation problems with high data and/or parameter dimensions.

Main paper: 
 introduction: 
 Simulation-based inference (SBI) comprises a family of computational methods for modeling the hidden properties of complex systems by means of simulation [1, 2] . In recent years, deep learningand generative models in particular -have proven indispensable for scaling up SBI to challenging inverse problems [3] [4] [5] [6] [7] . Recently, multiple streams of neural SBI research have been capitalizing on the rapid progress in generative modeling of unstructured data by re-purposing existing generative architectures into general inverse problem solvers for applications in the sciences [8] [9] [10] [11] . In line with the above trend, we contribute a new contender to the growing suite of SBI methods for amortized Bayesian inference.Within the class of deep generative models, score-based diffusion models [12] [13] [14] [15] and flow matching algorithms [16, 17] have recently attracted significant attention due to their sublime performance as realistic generators. Diffusion models and flow matching are strikingly flexible but require a relatively expensive multi-step sampling phase to denoise samples [12] . To address this shortcoming, Song et al. [18] proposed consistency models (CMs), which are trained to perform few-step generation by design and have since emerged as a standalone generative model family. Thus, CMs appear attractive as a backbone architecture for SBI workflows due to their unique combination of unconstrained architectures and fast sampling speed -a conceptual advantage over the current state-of-the-art in neural SBI.However, paradoxically, CMs may struggle with low-dimensional estimation problems, which are less important for image generation but crucial in typical SBI applications: These SBI applications are often characterized by relatively low-dimensional parameter spaces, even though the conditioning vectors (i.e., observables) might be high-dimensional [e.g., 3, [19] [20] [21] . Additionally, the quality of conditional CMs as few-step Bayesian samplers has not yet been explored empirically (e.g., in terms of probabilistic calibration and precision), even though this is crucial for their application in science and engineering. Lastly, while CMs for image generation are trained on enormous amounts of data, training data are typically scarce in SBI applications. In our empirical evaluations, we demonstrate that CMs are competitive with state-of-the-art SBI algorithms in low-data regimes using our adjusted settings (see Appendix A for details).In this paper, we propose conditional CMs for SBI along with generalizable hyperparameter settings that achieve competitive performance with much faster sampling speed on both challenging lowdimensional benchmarks and high-dimensional applied problems. Our main contributions are: 
notation: 
 In the following, the neural network training relies on a synthetic training set {(θ (m) , x (i) )} M m=1 , which consists of parameter-data tuples. Each superscript m marks one training example, namely, a tuple of a latent parameter vector and an observable data set which was generated from the parameter vector. The total simulation budget for training the neural networks follows as M . We summarize the D-dimensional latent parameter vector of the simulator as θ ≡ (θ 1 , . . . , θ D ). Further, each data set x ≡ {x n } N n=1 is a matrix whose rows consist of N vector-valued observations. Accordingly, one parameter vector θ (m) yields one data set x (m) from the simulator (see Section 2.2). In Bayesian inference, we aim to infer the unknown parameters θ of the mechanistic simulator, which are not to be confused with the trainable neural network weights ϕ.Computer simulations play a fundamental role across countless scientific disciplines, ranging from physics to biology, and from climate science to economics [2] . Simulators g(•, •) generate observables x ∈ X as a function of unknown parameters θ ∈ Θ and latent program states ξ ∈ Ξ [1] :EQUATIONThe forward problem in Equation 1 is typically well-understood through scientific theories instantiated as generative models. The inverse problem, however, is much harder, and forms the crux of Bayesian inference: reduce uncertainty about the unknowns θ based on observables x through the posterior distribution p(θ | x) ∝ p(θ) p(x | θ). The key property of neural SBI and early approximate Bayesian computation [ABC; 22, 23] methods is that they approach the inverse problem by sampling (i.e., simulating) from the forward model. Thus, the forward model acts as an implicit statistical model [24] that enables proper Bayesian inference in the limit of infinite simulations. In contrast, likelihood-based methods (e.g., MCMC) depend on the explicit evaluation of the likelihood p(x | θ).We can classify neural SBI methods into sequential [e.g., 10, [25] [26] [27] [28] [29] and amortized [e.g., 5, 9-11, [30] [31] [32] [33] [34] [35] methods. Sequential inference algorithms iteratively refine the prior p(θ) to generate simulations in the vicinity of the target observation. Thus, they are not amortized, as each new observed data set requires a potentially costly re-training of the neural approximator tailored to the particular target observation. However, most sequential methods can be turned amortized by training the neural approximator to generalize over the entire prior predictive space of the model. This allows us to query the approximator with any new data set during inference. In fact, amortization can be performed across any component of the model, including multiple data sets [5] and contextual factors, such as the number of observations in a data set [31] , heterogeneous data sources [36] , or even different probabilistic models [37, 38] . Our current work is situated in the amortized setting.Neural posterior estimation (NPE) methods for SBI have traditionally relied on conditional discrete normalizing flows for learning a conditional neural density estimator p ϕ (θ | x) from simulated pairs (θ, x) of parameters and data [8, 39] . Normalizing flows can learn closed-form approximate densities via maximum likelihood training:EQUATIONTwo prominent normalizing flow architectures in SBI are affine coupling flows [ACF; 40] and neural spline flows [NSF ; 41] . A major disadvantage of these architectures is their strict bijectivity requirement: it restricts the design space of the neural networks to invertible functions with cheap Jacobian calculations, hence the desire for more flexible architectures. In the following, we will briefly recap recent works that have pioneered the use of multi-step, unconstrained architectures for SBI, inspired by the success of such models in a variety of generative tasks [13, 15] .Wildberger et al. [11] applied flow matching techniques [16, 17] in SBI, an approach abbreviated as flow matching posterior estimation (FMPE). FMPE is based on optimal transport [42, 43] , where the mapping between base and target distribution is parameterized by a continuous process driven by a vector field µ on the sample space for each time step t ∈ [0, 1]. Here, the distribution at t = 1 could be a unit Gaussian N (0, I) and the distribution at t = 0 is the target posterior. The FMPE loss replaces the maximum likelihood term in the NPE objective (Eq. 2) with a conditional flow matching objective,EQUATIONwhere µ ϕ denotes the conditional vector field parameterized by a unconstrained neural network with trainable parameters ϕ, and u t denotes a marginal vector field, which can be as simple as u t (θ t | θ) = θ 1θ for all t [17] . We obtain posterior draws θ 0 ∼ p(θ | x) by solving dθ t = -µ(θ t , t; x) in reverse on t ∈ [0, 1], starting with noise samples θ 1 ∼ N (0, I). We can use any off-the-shelf ODE solver for transforming noise θ 1 into a draw θ 0 from the approximate posterior. In principle, the number of steps K in the ODE solver can be adjusted by setting the step size dt = 1/K. Decreasing the number of steps increases the sampling speed, but FMPE is not designed to optimize few-step sampling performance. We confirm this in our experiments with a rectified flow that encourages straight-path solutions [17] .Another approach to simulation-based inference with unconstrained networks lies in neural posterior score estimation [NPSE; 10], which can either feature single-round inference (amortized) or multiple sequential inference rounds on a particular data set (non-amortized). This stream of research uses conditional score-based diffusion models [12, 44] to learn the posterior distribution. With a slightly different focus, [9] factorize the posterior distribution and learn scores of the diffused posterior for subsets (down to a single observation) of a larger data set. Subsequently, the information from the subsets is aggregated by combining the learned scores to approximate the posterior distribution of the entire data set. Crucially, both methods rely on the basic formulation of score-based diffusion models: They gradually diffuse the target distribution according to the following diffusion process,EQUATIONwith drift coefficient µ, diffusion coefficient σ, time t ∈ [0, T ], and Brownian motion {w t } t∈[0,T ] . At each time t, the current (diffused) distribution of θ conditional on x is denoted as p t (θ | x).Crucially, the distribution at t = 0 equals the target posterior distribution, p 0 (θ | x) ≡ p(θ | x), and the distribution at t = T is set to be Gaussian noise (see below). Song et al. [12] prove that there exists an ordinary differential equation ("Probability Flow ODE") whose solution trajectories at time t are distributed according to p t (θ | x),EQUATIONwhere ∇ log p t (θ t | x) is the score function of p t (θ | x). This differential equation is usually designed to yield a spherical Gaussian noise distribution p T (θ | x) ≈ N (0, T 2 I) after the diffusion process. Since we do not have access to the target posterior p(θ | x), score-based diffusion models train a time-dependent score network s ϕ (θ t , t, x) ≈ ∇ log p t (θ t | x) via score matching and insert it into Eq. 5. Setting µ(θ t , t; x) = 0 and σ t = √ 2t, the estimate of the Probability Flow ODE becomes dθ t = -ts ϕ (θ t , x, t)dt [45] . Finally, we can generate a random draw from the noise distribution θ T ∼ N (0, T 2 I) and solve the Probability Flow ODE backwards for a trajectory {θ t } t∈[T,0] . The end of the trajectory θ 0 represents a draw from the approximate posterior p 0 (θ 0 | x) ≈ p(θ | x).Numerical stability. The solver is usually stopped at a fixed small positive number t = ε to prevent numerical instabilities [18] , so we use θ ε to denote the draw from the approximate posterior. For simplicity, we will also refer to θ ε as the trajectory's origin. 
consistency model posterior estimation: 
 Diffusion models have one crucial drawback: At inference time, they require solving a differential equation for each posterior draw which slows down their sampling speed. This is particularly troublesome in SBI applications which may require thousands of samples for thousands of data sets [e.g., >1M data sets in 7]. Consistency models [CMs; 18] address this problem with a new generative model family that supports both single-step and multi-step sampling. In the following, we summarize key ideas of CMs [18, 46] with a focus on conditional sampling for the purpose of SBI.The consistency function f : (θ t , t; x) → θ ε maps points across the solution trajectory {θ t } t∈[T,ε] to the trajectory's origin θ ε given a fixed conditioning variable (i.e., observation) x and the probability flow ODE in Eq. 5. To achieve this with established score-based diffusion model architectures, we can use a unconstrained neural network F ϕ (θ, t; x) which is parameterized through skip connections of the formEQUATIONwhere c skip (t) and c out (t) are differentiable and fulfill the boundary conditions c skip (ε) = 1 and c out (ε) = 0. Consistency models are originally motivated as a distillation technique for diffusion models. However, Song et al. [18] show that training consistency models in isolation is possible, and we base our method on their direct approach.Sampling. Once the consistency model has been trained, generating draws from the approximate posterior is straightforward by drawing samples from the noise distribution, θ T ∼ N (0, T 2 I), which shall then be transformed into samples from the target distribution, like in a standard diffusion model. In contrast to diffusion models, however, we do not need to solve a sequence of differential equations for this transformation. Instead, we can use the learned consistency function f ϕ to obtain the one-step target sample θ ε = f ϕ (θ T , T ; x). What is more, inference with consistency models is not actually limited to one-step sampling. In fact, multi-step generation is possible with an iterative sampling procedure, which we will describe in the following. For a sequence of time pointsEQUATIONfor k = K -1, K -2, . . . , 1, where z k ∼ N (0, I) and K -1 is the number of sampling steps [18, 46] . The resulting sample is usually better than a one-step sample.Originally developed for image generation, consistency models can be applied to learn arbitrary distributions. The unconstrained architecture enables the integration of specialized architectures for both the data x and the parameters θ. Due to the low number of passes required for sampling (in contrast to flow matching and diffusion models), more complex networks can be used while maintaining low inference time. In theory, consistency models combine the best of both worlds: unconstrained networks for optimal adaptation to parameter structure and data modalities, while enabling fast inference speed with few network passes. Currently, this comes at the cost of explicit invertibility, which limits the computation of posterior densities. More precisely, single-step consistency models do not allow density evaluations at an arbitrary parameter value θ but only at a set S of approximate posterior draws {θ (1) ε , . . . , θ (S) ε }. However, this is sufficient for important downstream tasks like marginal likelihood estimation, importance sampling, or self-consistency losses. In contrast, multistep consistency sampling defines a Markov chain which cannot be evaluated without an additional density estimator (see Section 3.5 for details). In accordance with the taxonomy from Cranmer et al. [1] , we call our method consistency model posterior estimation (CMPE). While we focus on posterior estimation, using consistency models for likelihood emulation is a natural extension of our work.As a consequence of its fundamentally different training objective, CMPE is not just a faster version of FMPE. Instead, it shows qualitative differences to FMPE beyond a much faster sampling speed. In Experiment 4, we show that CMPE is less dependent on the neural network architecture than FMPE, making it a promising alternative when the optimal architecture for a task is not known. Further, we consistently observe good performance in the low-data regime, rendering CMPE an attractive method when training data is scarce. In fact, data availability is a common limiting factor for complex simulators in engineering [47] and science [e.g., molecular dynamics; 48].We formulate the consistency training objective for CMPE, which extends the unconditional training objective from Song and Dhariwal [46] with a conditioning variable x to cater to the SBI setting,EQUATIONwhere λ(t) is a weighting function, d(u, v) is a distance metric, z ∼ N (0, I) is unit Gaussian noise, and the arguments for the distance metric amount to the outputs of the consistency function obtained at two neighboring time indices t i and t i+1 ,EQUATIONThe teacher's neural network weights ϕ -are a copy of the student's weights which are held constant during each step via a stopgrad operator, ϕ -← stopgrad(ϕ). We follow Song and Dhariwal [46] using λ(t i ) = 1/(t i+1t i ) and d(u, v) = ∥u -v∥ 2 2 + c 2c from the Pseudo-Huber metric family. Appendix A contains more details on discretization and noise schedules.Consistency training introduces several additional hyperparameters, but there is limited theoretical guidance on how to select most of them. Within the scope of this paper, we rely on empirical search to identify the hyperparameters that are the most relevant for tuning efforts. As stated above, the maximum time T determines the standard deviation of the latent distribution. It should be larger than the magnitude of the target distribution [18] . Values that are too low might lead to biased sampling, while values that are too high can make the training process more difficult. The minimum and maximum number of discretization steps s 0 , s 1 during training also influence the final result in our experiments. For s 0 , a sufficiently low value (e.g. 10) should be chosen. For s 1 , smaller values around 50 seem beneficial for stable training, especially when the number of epochs is low. When longer training is possible, increasing s 1 might lead to improved accuracy. We summarize our recommended hyperparameter choices for SBI applications in Appendix C.Using the change-of-variable formula, we can express the posterior density of a single-step sample asEQUATIONwhere f -1 ϕ (θ, T ; x) is the implicit inverse of the consistency function and ∂θ T /∂θ ε is the resulting Jacobian. While we cannot explicitly evaluate f -1 ϕ (θ ε , T ; x), we can generate samples from θ T ∼ N (0, T 2 I) and use autodiff for evaluating the Jacobian because the consistency surrogate f ϕ is differentiable. Thus, we cannot directly evaluate the posterior density at arbitrary θ. Yet, evaluating the density at a set of S posterior draws {θ ε } S s=1 suffices for crucial downstream tasks, such as marginal likelihood estimation [49] , neural importance sampling [21] , or self-consistency losses [50] .For the purpose of multi-step sampling, the latent variables θ K , θ K-1 , . . . , θ 1 form a Markov chain with a change-of-variables (CoV) given by Köthe [51] :EQUATIONDue to Eq. 7, the backward conditionals are given by p(θ k-1 | θ k , x) = N (f ϕ (θ k , t k ; x) , (t 2 k -ε 2 )I). Unfortunately, the forward conditionals p(θ k | θ k-1 , x)are not available in closed form, but we could learn an amortized surrogate model q(θ k | θ k-1 ) capable of single-shot density estimation based on a data set of execution paths {θ 1:K }. This method would work well for relatively low-dimensional θ, which are common throughout scientific applications, but may diminish the efficiency gains of CMPE in higher dimensions. We leave an extensive evaluation of different surrogate density models to future work.The design of consistency models enables one-step sampling (see Eq. 7). In practice, however, using two steps significantly increases the quality in image generation tasks [46] . We observe that few-step sampling with approximately K = 5-15 steps provides the best trade-off between sample quality and compute for SBI, particularly in low-dimensional problems. This is roughly comparable to the speed of neural one-step estimators like affine coupling flows or neural spline flows in Experiments 1-3 (see Figure 2a ). The number of steps can be chosen at inference time, so practitioners can easily adjust this for a given situation. We noticed that approaching the maximum trained number of discretization steps can lead to overconfident posterior distributions. See Section C.5 for an empirical evaluation of the relation between sampling quality and number of sampling steps. 
empirical evaluation: 
 Our experiments cover three fundamental aspects of SBI. First, we perform an extensive evaluation on three low-dimensional experiments with bimodal posterior distributions from benchmarking suites for inverse problems [52, 53] . The simulation-based training phase is based on a fixed training set {(x (m) , θ (m) * )} M m=1 of M tuples of data sets x (m) and corresponding ground-truth parameters θ (m) * . Second, we focus on an image denoising example which serves as a sufficiently high-dimensional case study in the context of SBI [32, 54] . Third, we apply our CMPE method to a computationally challenging scientific model of tumor spheroid growth and showcase its superior performance for a complex simulator from cell biology [19] . We implement all experiments using the BayesFlow Python library for amortized Bayesian workflows [55] . We provide reproducible code for all experiments in the Supplementary Material or an anonymized online repository (Link).Evaluation metrics. We evaluate the experiments based on well-established metrics to gauge the accuracy and calibration of the results. All metrics are computed on a test set of J unseen instances {(x (j) , θ (j) * )} J j=1 . In the following, S denotes the number of (approximate) posterior samples that we draw for each instance J. First, the easy-to-understand root mean squared error (RMSE) metric quantifies both the bias and variance of the approximate posterior samples across the test set as1 J J j=1 1 S S s=1 θ (j) s -θ (j) * 2 .Second, we estimate the squared maximum mean discrepancy [MMD; 56] between samples from the approximate vs. reference posterior, which is a kernel-based distance between distributions. Third, as a widely applied metric in SBI, the C2ST score uses an MLP classifier to distinguish samples from the approximate and reference posteriors. The resulting test accuracy of the classifier is the C2ST score, which ranges from 1 (samples are perfectly separable → bad posterior) to 0.5 (samples are indistinguishable → perfect posterior). Finally, we assess uncertainty calibration through simulation-based calibration (SBC; [57] ): All uncertainty intervals U q (θ | x) of the true posterior p(θ | x) are well calibrated for every quantile q ∈ (0, 1),EQUATIONwhere IWe illustrate CMPE on a 2-dimensional Gaussian mixture model with two symmetrical components [9, 50] . The symmetrical components are equally weighted and have equal variance,EQUATIONwhere N (• | µ, Σ) is a Gaussian distribution with location µ and covariance matrix Σ. The resulting posterior distribution is bimodal and point-symmetric around the origin (see Figure 1 top-left). Each simulated data set x consists of ten exchangeable observations {x 1 , . . . , x 10 }, and we train all architectures on M = 1024 training simulations. We use a DeepSet [58] to learn 6 summary statistics for each data set jointly with the inference task.Results. As displayed in Figure 1 performance relative to the other approximators. Instead, CMPE can outperform all other methods simultaneously with respect to both speed and performance, as quantified by lower C2ST to the reference posterior across J = 100 test instances (see Figure 2a ). If we tolerate slower sampling, CMPE achieves peak performance at K = 10 inference steps. Most notably, CMPE outperforms 1000-step FMPE by a large margin, even though the latter is approximately 75× slower.This experiment studies the two moons benchmark [27, 49, 52, 59 ]. The model is characterized by a bimodal posterior with two separated crescent moons for the observed point x obs = (0, 0) ⊤ which an approximator needs to recover. We repeat the experiment for different training budgets M ∈ {512, 1024, 2048, 4096, 8192} to probe each method under varying data availability. While M = 512 is a very small budget for the two moons benchmark, M = 8192 is considered sufficient.Results. Trained on a simulation budget of M = 1024 examples, CMPE consistently explores both crescent moons and successfully captures the local patterns of the posterior (see Figure 1 , middle row). Both the affine coupling flow and the neural spline flow fail to fully separate the modes. Most notably, if we aim to achieve fast sampling speed with FMPE by reducing the number of sampling steps during inference, FMPE shows visible overconfidence with 30 sampling steps and markedly deficient approximate posteriors with 10 sampling steps. In stark contrast, CMPE excels in the few-step regime. Figure 2b illustrates that all architectures benefit from a larger training budget. CMPE with 10 steps emerges as the superior architecture in the low-and medium-data regime up to M = 4096 training instances. FMPE with 1000 steps outperforms the other approximators by a small margin for the largest training budget of M = 8192 instances. Keep in mind, however, that 1000-step FMPE is approximately 30-70× slower than ACF, NSF, and CMPE in this task.Proposed as a benchmark task for inverse problems by Kruse et al. [53] , the inverse kinematics model aims to reconstruct the configuration (θ 1 , θ 2 , θ 3 , θ 4 ) = θ ∈ R 4 of a multi-jointed 2D robot arm for a given end position x ∈ R 2 (see Figure 1 , bottom left). The input to the forward process g : θ → x are the initial height θ 1 of the arm's base, as well as the angles θ 2 , θ 3 , θ 4 at its three joints. The inverse problem aims to determine the posterior distribution p(θ | x), which represents all arm configurations θ that end at the observed 2D position x of the arm's end effector.Results. On the challenging small training budget of M = 1024 training examples, CMPE with K = 30 sampling steps visually outperforms all other methods with respect to posterior predictive performance while maintaining fast inference (see Figure 1 ). On the C2ST metric, CMPE outperforms normalizing flows (ACF, NSF), but is in turn surpassed by 1000-step FMPE (see Figure 4b ), which is however 85× slower (FMPE: 6565ms, CMPE: 78ms). In contrast to the other empirical evaluations, the simulator in this benchmark lacks aleatoric uncertainty [60] and the plausible parameter values are tightly bounded through the geometry of the problem, which might explain this pattern. Neural architectures. The naïve architecture consists of a convolutional neural network [CNN; 63] to convert the observation into a vector of latent summary statistics. We concatenate input vector, summary statistics, and a time embedding and feed them into a multi-layer perceptron (MLP) with four hidden layers consisting of 2048 units each. This corresponds to a situation where the structure of the observation (i.e., image data) is known, but the structure of the parameters is unknown or does not inform a bespoke network architecture. In this example, however, we can leverage the prior knowledge that our parameters are images. Specifically, we can incorporate inductive biases into our network architecture by choosing a U-Net architecture which is optimized for image processing [i.e., an adapted version of 64]. Again, a CNN learns a summary vector of the noisy observation, which is then concatenated with a time embedding into the conditioning vector for the neural density estimator. We conclude our empirical evaluation with a complex multi-scale model of 2D tumor spheroid growth [19, 65] . The model has 7 unknown parameters which govern single-cell behavior (agent-based) as well as the extracellular matrix description (PDE-based). Crucially, running simulations from this model on consumer-grade hardware is rather expensive (≈ 1 minute for a single simulation), so there is a desire for methods that can provide reasonable estimates for a limited offline training budget.Here, we compare the performance of the four contender methods in this paper on a fixed training set of M = 19 600 simulations with J = 400 simulations as a test set to compute performance metrics. The contender methods are affine coupling flows (ACF), neural spline flows (NSF), and flow-matching posterior estimation (FMPE), as these are among the most popular choices for neural SBI to date. Our main contender is FMPE as a free-form architecture, and we use the exact same inference network for both FMPE (µ ϕ -field network) and CM (f ϕ -consistency network). Across all methods, we use a hybrid LSTM-Transformer architecture to transform high-dimensional summary statistics of variable length into fixed length embedding vectors h(x). Appendix C.7 provides more details on the neural network architectures and training hyperparameters. 6 in the Appendix). For this simulator, FMPE did not achieve satisfactory calibration performance with up to K = 100 inference steps, so Table 2 reports the best FMPE results for K = 1000 steps. 
discussion: 
 We presented consistency model posterior estimation (CMPE), a novel approach to perform accurate simulation-based Bayesian inference on large-scale models while achieving fast inference speed. CMPE enhances the capabilities of state-of-the-art neural posterior estimation by combining few-step sampling with unconstrained neural architectures. To assess the effectiveness of CMPE, we applied it to a set of 3 low-dimensional benchmark tasks that allow intuitive visual inspection, as well as a high-dimensional Bayesian denoising experiment and a scientific tumor growth model. Across our experiments, CMPE emerges as a competitive method for simulation-based Bayesian inference, as evidenced by a holistic assessment of posterior accuracy, calibration, and inference speed.Limitations. Our proposed CMPE method has two core limitations, which we specifically highlight again in this paragraph. First, consistency models do not directly yield tractable densities for arbitrary parameter values (see Section 3.5 for a remedy). Second, the relation between inference time (number of sampling steps K) and performance (posterior accuracy) is not monotonic, as elucidated in Experiment 1, (see Figure 2a ). We provide more details and report the CMPE posterior quality (quantified by C2ST) as a function of the number of sampling steps for all three benchmark experiments in Section C.5. This phenomenon is not a drawback per se, but rather a counter-intuitive attribute of consistency models. In practice, it can easily be addressed by performing a brief sweep over the number of sampling steps (K = 1, . . . , K max ) during inference time and choosing an optimal K based on user-defined metrics (e.g., calibration or MMD on a validation set). Given the very fast inference speed of CMPE and the observed U-shaped relation between sampling steps and performance, this approach does not generally come with a relevant computational burden.Perspectives. Future work might aim to further reduce the number of sampling steps towards one-step inference in SBI tasks to achieve even faster sampling speed in time-sensitive applications. Table 3 details all functions and parameters required for training. Except for the choice of parameters s 0 , s 1 , ε, T max and σ data we exactly follow the design choices proposed by Song and Dhariwal [46] , which improved our results compared to the choices in Song et al. [18] . Song and Dhariwal [46] showed that that the Exponential Moving Average of the teacher's parameter ϕ -should not be used in Consistency Training, therefore we omitted it in this paper. 1 ].d(x, y) = ∥x -y∥ 2 2 + c 2 -c Discretization scheme N (k) = min(s 0 2 ⌊k/K ′ ⌋ , s 1 ) + 1 where K ′ = ⌊K/(log 2 ⌊s 1 /s 0 ⌋ + 1)⌋ Noise schedule t i , where i ∼ p(i) and p(i) ∼ erf log(ti+1)-Pmean √ 2Pstd -erf log(ti)-Pmean √ 2Pstd Weighting function λ(t i ) = 1/(t i+1 -t i ) Skip connections c skip (t) = σ 2 data /((t -ε) 2 + σ 2 data ), c out (t) = σ data (t -ε)/ σ 2 data + t 2Parameters s 0 = 10, s 1 = 50 except where indicated otherwise c = 0.00054 √ d, where d is the dimensionality of θ P mean = -1.1, P std = 2.0 k ∈ 0, . . . , K -1, where K is the total training iterationst i = ε 1/ρ + i-1 N (k)-1 T 1/ρ max -ε 1/ρ ρ , ρ = 7, ε = 0.001, T max = 200Especially for multi-modal distributions, some of the standard metrics may behave in unexpected ways. This impedes an interpretation of the results and can lead to misleading conclusions. In this section, we want to highlight the peculiarities of some of the metrics, and highlight where caution is necessary when they are applied to the presented benchmarks.As an expressive classifier, C2ST is most sensitive to regions without distribution overlap. In our experiments, this results in a pronounced punishment of overconfident posteriors. For limited training data and distributions without sharp edges (e.g., the Gaussian mixture model), some overconfidence is usually given, as samples from the outer tails are simply not present in the training set. As the accuracy is used as a measure, a smaller number of very improbable samples (e.g., samples in the region between two modes) do not influence the results a lot.In consequence, C2ST may deviate from the visual judgment in important ways. First, overconfident posteriors look more "precise", but are punished by C2ST. Second, samples between modes are very salient, but may not influence the results of C2ST a lot.For distributions with sharp edges and more or less uniform density in a given mode, c2st is very capable in judging the overlap of two distributions, which then is a good measure of quality. This applies to the two-moons example, where we can observe the expected behavior of a larger training budget leading to more accurate results (Figure 2b ).By capturing and comparing the moments of two distributions, MMD provides a more holistic view than C2ST. It is also applicable to a wider range of cases, as it generalizes to high dimensions. The exact behavior can be controlled by choosing the kernel function. In our experiments, we observe MMD to be less sensitive for distributions with sharp edges (i.e., two moons), where C2ST seems to provide more reasonable results. We also observe a low sensitivity to samples between modes, leading to unexpected results compared to visual inspection.Calculating meaningful metrics given only samples from two distributions is a challenging task. Ideally, the metrics would reward similar density and punish samples in the low-density regions of the reference distribution. Especially the latter seems to be hard to achieve. Even when metrics work fine for small deviations from the ideal distribution, the behavior for different forms of larger deviations, as they occur in challenging problems or cases with low training budget, might not be easily grasped or interpreted. That makes it difficult to compare different methods producing qualitatively different deviations (e.g., overconfidence vs. underconfidence) from the reference distribution.For Neural network details. CMPE uses an MLP with 2 layers of 256 units each, L2 regularization with weight 10 -5 , 5% dropout, and an initial learning rate of 5 • 10 -4 . The consistency model uses s 0 = 10, s 1 = 50, T max = 10, and it is trained with a batch size of 64 for 5000 epochs. Both ACF and NSF use 6 coupling layers of 128 units each, kernel regularization with weight γ = 10 -4 , and train for 200 epochs with a batch size 32. FMPE uses the same settings and training configuration as CMPE.The three robot arm segments have lengths 0.5, 0.5, 1.0. The parameters θ = (θ 1 , θ 2 , θ 3 , θ 4 ) follow a Gaussian prior θ ∼ N (0,EQUATION). Neural network details. CMPE relies on an MLP with 2 layers of 256 units each, L2 regularization with weight 10 -5 , 5% dropout, and an initial learning rate of 5 × 10 -4 . The consistency model uses s 0 = 10, s 1 = 50, and trains for 2000 epochs with a batch size of 32. FMPE uses the same MLP and training configuration as CMPE. Both ACF and NSF use 6 coupling layers of 128 units each, kernel regularization with weight γ = 10 -4 , and train for 200 epochs with a batch size 32. The reference posterior is obtained using approximate Bayesian Computation (ABC), with ε = 0.002, which corresponds to 1/1000 of the arm length. For the GMM benchmark, we observe an unexpected pattern for the dependency on the training budget. The C2ST score increases or stays approximately constant across all methods, indicating that in this regime a higher training budget leads to inferior performance, for example due to a tendency to overfit with more training data. It could also be a sign that the C2ST is not a good quality metric for this benchmark, but the monotonically decreasing curve for higher-quality samples (i.e., more sampling steps) for FMPE in Figure 2a indicates that the behavior can probably be attributed to the training budget and not to the metric. See Appendix B for a more detailed discussion. (b) In this task, it is more challenging to achieve excellent C2ST scores because there is no aleatoric uncertainty in the data-generating process. CMPE outperforms ACF and NSF. FMPE performs best and can benefit most from the increased training budget.Table 4 gives an overview of the observed wall-clock training times for the benchmark tasks in Experiments 1-3. These results have been measured on a consumer-grade CPU (Apple Silicon M1 processor). We emphasize that these do not constitute a principled timing benchmark under standardized conditions, and the training times between algorithms are not directly comparable. We train each algorithm for a fixed number of epochs, rather than training until some pre-defined accuracy metric has been achieved. This is because no single reported metric holistically quantifies the accuracy in the investigated benchmark tasks, as explained in more detail in Appendix B. Further, we observed degrading performance of both normalizing flow implementations for larger numbers of epochs despite dropout and regularization. Instead of a comprehensive benchmark timing test, the results shall show that CMPE does in fact require slightly longer training times compared to FMPE given the same neural network backbone. We attribute this to the student-teacher training scheme which necessitates more neural network evaluations (not gradient updates) during training. However, the much faster inference speed of CMPE constitutes a significant advantage for real-time amortized Bayesian inference, which is why the drawback of longer neural network training is acceptable. M = 512 M = 1024 M = 2048 M = 4096 M =For Experiments 1-3, we evaluate the performance of CMPE (as indexed by C2ST) as a function of the number of sampling steps K during inference. Figure 5 shows that there is no monotonic relationship where more compute leads to better results. Instead, we observe a U-shaped relationship with a performance maximum at K = 10-20 sampling steps. This seemingly counter-intuitive phenomenon is a consequence of the consistency training, which aims to achieve few-step sampling and thus has no incentive to optimize the consistency function for a large number K of sampling steps.The Fashion MNIST data set is released under MIT license [61] . We compare samples from a 2 × 2 × 2 design: Two methods, CMPE and FMPE, are trained on two network architectures, a naïve architecture and a U-Net architecture, using 2 000 and 60 000 training images. Except for the batch size, which is 32 for 2 000 training images and 256 for 60 000 training images, all hyperparameters are held constant between all runs. We use an AdamW optimizer with a learning rate of 5 • 10 -4 and cosine decay for 20 000 iterations, rounded up to the next full epoch. As both methods use the same network sizes, this results in an approximately equal training time between methods (approximately 1-2 hours on a Tesla V100 GPU), with a significantly lower training time for the naïve architecture.Neural network details. For CMPE, we use s 0 = 10, s 1 = 50, σ 2 data = 0.25 and two-step sampling. For FMPE, we follow Wildberger et al. [11] and use 248 network passes. Note that this is directly correlated with inference time: Choosing a lower value will make FMPE sampling faster, a larger value will slow down FMPE sampling. We show samples for each method (CMPE, FMPE), backbone architecture (MLP, U-Net), and simulation budget (2 000, 6 000) at the end of the appendix. The gray region depicts the prior distribution. CMPE (Ours) shows no visible bias even for two-step inference, but an improved sharpness when we increase the number of sampling steps to 30. In contrast, FMPE with 30 sampling steps (comparable speed to CMPE) yields a biased posterior. The posterior of FMPE with 1000 inference steps visibly improves but is orders of magnitude slower than CMPE with 30 steps.Neural network details. Both CMPE and FMPE use an MLP with 4 hidden layers of 512 units each and 20% dropout. CMPE and FMPE train for 1000 epochs with a batch size of 64. The consistency model uses s 0 = 10, s 1 = 50, T max = 50. ACF and NSF employ a coupling architecture with learnable permutations, 6 coupling layers of 128 units each, 20% dropout, and 300 epochs of training with a batch size of 64.The input data consists of multiple time series of different lengths. As proposed by Schmitt et al. [36] for SBI with heterogeneous data sources, we employ a late fusion scheme where the input modalities are first processed by individual embedding networks, and then fused. First, the data on tumor size growth curves enters an LSTM which produces a 16-dimensional representation of this modality. Second, the information about radial features is processed by a temporal fusion transformer with 4 attention heads, 64-dimensional keys, 128 units per fully-connected layer, and 10 -4 L2 regularization on both weights and biases. These latent representations are then concatenated and further fed through a fully-connected two-layer perceptron of 256 units each, 10 -4 L2 regularization on weights and biases, and 32 output dimensions.Figure 7 shows calibration plots for all methods. We observe that our CMPE method with 30 inference steps leads to the best calibration, with affine coupling flows and neural spline flows performing similarly with respect to the expected calibration error (ECE). In this additional experiment, we apply CMPE to gravitational wave inference of the binary black hole merger GW150914. Our analysis parallels the study of Wildberger et al. [11] who apply FMPE to this challenging scientific task. To this end, we implement our CMPE algorithm in the Dingo library for analyzing gravitational wave data using neural posterior estimation [67] . Dingo implements a sophisticated analysis pipeline that is carefully tailored to gravitational wave inference. As in previous experiments, we compare FMPE and CMPE based on identical neural network backbones with a total of ≈ 190 Million neural network weights, and we refer to Wildberger et al. [11] for a detailed description of the experimental setup. The consistency model uses s 0 = 10, s 1 = 1280, T max = 50, σ data = 1.Similar to Wildberger et al. [11] , we show the bivariate posterior marginals for a subset of all 14 inference parameters in Figure 8 . FMPE successfully captures the posterior geometry [which matches a ground-truth posterior from a slow reference method; see 11]. In contrast, CMPE yields a pathologically underexpressive posterior which mostly resembles a multivariate Gaussian (with some minor mixture contributions for α). Yet, the maximum a posteriori (MAP) estimates of CMPE and FMPE mostly align (up to CMPE covering both posterior modes for α), which indicates that CMPE does distill information from the model. We suspect that the pathologically poor performance of CMPE is related to (i) issues in our re-implementation of CMPE interfacing the Dingo library; or (ii) misaligned hyperparameters for this specific application. We will continue troubleshooting and invite the interested reader to watch the accompanying GitHub repository for updates.T-Shirt/Top Param. • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. Guidelines:Sample Sample Sample Sample Sample Trouser Pullover Dress Coat Sandal Shirt Sneaker Bag Ankle Boot• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: We discuss the positive impact on scientific progress in the Introduction. As this paper is generally concerned with foundational research, the specific benefit or harm largely depends on the underlying application.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: The paper poses no such risk as we do not release any large-scale models, generators, or data sets.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes]Justification: We cite all original papers of used benchmarks, evaluation tasks, data sets, and metrics.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [NA]Justification: The paper does not release new assets.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA]Justification: The paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA]Justification: The paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.• Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix:  


Decision: The paper proposes to use a conditional consistency model for amortized likelihood-free inference.  Their experimental evaluation shows that the inference speed of this approach is much better that existing methods.  While some reviewers initially has some concerns about the experimental evaluation and the clarity of presentation, their concerns seem to have been adequately addressed in the rebuttal phase.

Reviews: review: paper_summary: paper_summary: This paper adapts the recently introduced Consistency Models of
Song et al to the task of Simulation-based Inference. Compared to
the previous approaches based on flow matching, this technique
exhibits similar or better quality. All while being significantly
faster to sample from with fewer restrictions on the underlying
neural architecture.
strength_weakness: strength_weakness: I do believe this paper would benefit for slightly reworked plots since the main advantage is just how much more efficient of an inference method it is. I already appreciate that wall-times are included in many places, but something like Figure 2 would be even better if that was taken into account.

It appears to be a relatively straightforward application of CMs to SBI.

I do think the paper would be stronger if the specific restrictions on the density estimation were mentioned in the main paper vs Appendix C

This paper would benefit from an additional larger example.
questions: questions: Why for the Gaussian Mixture Model does the C2ST score go up as
the computational budget is increased? Is this just a property of
the dataset being so small?

Would it be possible to have a plot that shows the evaluations as a
function of wall-time? One of the major advantage of this method is
it can get away with doing significantly less sampling, and it would
be nice to have a plot that showcases that
limitations: limitations: Yes they have been adequately discussed in paper.
paper_summary: paper_summary: Adopting the idea from consistency models in the generative process, the authors propose its application for posterior estimation, introducing a new type of model for simulation-based inference (SBI). The proposed method, consistency models for posterior estimation (CMPE), enjoying the benefits of the consistency models, can support unconstrained architecture and is more efficient in the sampling process. Empirically, the authors conducted experiments on low-dimensional datasets, including Two Moons, GMMs, and Kinematics, and high-dimensional datasets, including Fashion MNIST.
strength_weakness: strength_weakness: W1: The current experiments might be based on relatively simple datasets. While these are useful for initial validation, they may not adequately demonstrate the model's ability to handle real-world complexities and high-dimensional data. The authors could consider more complex datasets with complex distributions, like multimodal distributions. GW150914 in https://arxiv.org/pdf/2305.17161 might be a good case to try.

W2: The training time for consistency models might be problematic for scaling to high-dimensional distributions, and its stability has been known as a problem for complex distribution.

W3: My main concern is that the motivation for applying a consistency model for SBI might be weak, as consistency models are known to have stability issues for learning complex distributions. The current experiments only verify the computational advantage in the sampling process. However, the increase in the training time and the instability of training might pose more difficulties for real-world applications.

W4: The performance of the consistency models relies heavily on tuning a series of hyper-parameters, which might be a problem for generalization.
questions: questions: Would you provide a comparison of the training time? And the performance regarding the hyper-parameters of the CMPE.
limitations: limitations: N/A
paper_summary: paper_summary: The paper proposes to use a conditional consistency model for amortized likelihood-free inference. Empirical evaluation shows that this approach compares favorably in terms of inference time as well as performance against competing methods.
strength_weakness: strength_weakness: - The main weakness of this work is the lack of novelty. It comes down to applying an existing model that shows very good performance in computer vision to likelihood-free inference, which has not been published yet. However, someone has to be first, and this work is well suited to this purpose due to its high quality.
- Little attention is paid to all the "design choices" summarized in Table 3. For comparison, reference [41], on which the authors base their work, presents an analysis of the impact of the hyper-parameters used.
- The authors do not evaluate the proposed solutions to the lack of density evaluation limitation.
questions: questions: 1. Could you provide the coverage curves from which ECE is calculated?
2. Fig 2 b), Fig 4 a) and b) - it looks like the standard deviation around the points is 0. Is it so small that one cannot see it in the plot or is it missing from the figures?
3. Why is ECE reported only for Experiment 5?
limitations: limitations: Yes, the limitations of the proposed method are described.
paper_summary: paper_summary: The paper adopts consistency models for simulator-based inference tasks, highlighting its expressive free-form architectures and fast inference as main advantages. The new method, called consistency model posterior estimation, was shown to outperform Neural Posterior Estimation (NPE) and be competitive with Fully Neural Posterior Estimation (FMPE) methods in terms of amortized inference quality.
strength_weakness: strength_weakness: The main weaknesses lie in the presentation and the overall results of the paper. The paper is quite difficult to follow, and the overall contribution seems relatively small. The adoption of the consistency model for simulator-based inference (SBI) appears straightforward, and some methodological decisions, such as those in Section 3.3, lack justification. The significance of the contribution may be understated due to the presentation issues in Sections 1-3, which need significant improvement to strengthen the proposal for this new method. Detailed suggestions for improvement are provided below.
questions: questions: Abstract

(line 1): Consider revising the first sentence "Simulation-based inference (SBI) is constantly in search of more expressive..." to be more concrete.

(line 5): Clarify what is meant by 'free-form architectures'.

(line 5-6): Some SBI methods have long inference times, and 'overcoming sampling inefficiency at inference time' may be bad for them --perhaps some clarification is needed.

(line 8-9): 'Providing an architecture' slightly contrasts with the advantage of having a free-form architecture, so some justification is need.

(line 10): Could you specify the dimensions this line refers to and explain why low-dimensional simulator parameters were difficult to handle with unconditional consistency models and why not simply use other SBI methods instead?

Introduction

(line 20): Could you rephrase to avoid broad generalizations about SBI research, as there are many recent works focused, for instance, on choosing summary statistics with little relation to generative modeling?

(line 31): Could you clarify the term 'free-form architecture'?

(line 36): Please define 'target spaces' in the context of SBI applications.

(lines 41-43): I don't understand the significance of this sentence. Do authors imply that they explore empirically the quality of conditional CMs and it is one of the contributions? It follows the sentence about the empirical evaluations in the paper, so I am a little bit confused.

Section 2

(line 59): I am not familiar with the term 'simulation-based training'. Perhaps the point of the term is to emphasize the synthetic nature of the data generated by the simulator or some other reason?

(line 66, 70): I am not sure why simulation models are presented as 'programs'. This may be confusing as it is not common for the SBI literature in general. Later authors use the term 'program states', but it has very little to do with the setting they operate in. Therefore I would propose to either avoid such terminology or explicate the setting they operate in (if it differs from the 'classical' SBI setting).

Typos: (line 71): 'g( \cdot, \cdot)'. (line 74): 'reasoning' (line 79): 'sequential and amortized'. (line 95): 'it' (line 124): 'they'

(line 101): Please provide a reference for 'optimal transport'.

Section 3

(line 151): x is presented as “a fixed conditioning variable x”, but it is already used for observations in the SBI context. Could you clarify in the text if they are the same?

(line 177): Could you at least briefly detail the ‘hardships for computing the posterior density’ in the main text?

(Eq 8): Could you provide justification or references for this objective?

Experiments

(line 234): Could you provide a reference, where the hyper-parameters for the comparison methods are listed?
limitations: limitations: The authors have adequately addressed the limitations of their work in a separate section.

Comments: title: Final feedback
RE: Rebuttal
Response to rebuttal
Response to the authors
Final Rebuttal Response
Rebuttal Response
Acknowledgement of the rebuttal
Response to detailed list (Part 2)
Response to detailed list (Part 1)
comment: Dear authors, thank you for your response. It's good to know that my assessment was correct. And I appreciate your merit in not cherry-picking the results. I will raise my score to 5.
I am satisfied with the author response. I think adding the gravitational wave example would be sufficient as a larger example.

It might be worth exploring the relationship between computational budget and performance, but I don't think it's essential for this paper. Since the example is fairly synthetic it can't hurt to see if it behaves better if you just generate more points for the GMM.
Thank you for your clarifications and for explicitly encouraging a constructive discussion. Regarding Experiment 5, we understand your concern and appreciate the opportunity to address it.

We agree with your assessment of this experiment: Depending on the modeling goals, the choice of neural network architecture is not as unanimous as in the other experiments. If modelers desire maximum performance w.r.t. bias/variance (RMSE) and calibration (ECE), CMPE is the best choice according to the assessed metrics. If the modeler wants to optimize for training and inference time at the cost of a slightly worse RMSE and calibration, affine coupling flows (ACF) are the method of choice. In this experiment, we observe the typical trade-off between speed (ACF) and performance (CMPE; ours). As often in computational methods, we empirically see diminishing returns; the performance advantage of CMPE in this task comes with a substantially longer training and inference time compared to ACF. This might be related to properties of the task, such as the unimodal posterior geometry: ACF is known to be efficient in unimodal posteriors while struggling to fully separate multiple posterior modes (see the ‘bridges’ between modes in Figure 1 for ACF). In the other experiments, CMPE empirically shows a clearer superiority in the performance-vs-time trade-off.

> [...] this result as a big portion of your experiments [...]

We acknowledge that this experiment does not paint as clear a picture in favor of CMPE as the other experiments. However, it is important to us to refrain from selective reporting and we chose to include this experiment to show the nuanced modeling choices that researchers face when selecting an inference method in SBI. We will use the additional page in an eventual camera-ready version to discuss this aspect in more detail based on Experiment 5.

We hope that this additional context addresses your concern and again want to express our appreciation for the active discussion you sparked.
We appreciate the constructive discussion and are grateful that you adjusted your score to honor the efforts we've made to address your concerns. We are optimistic that these changes have substantially strengthened the paper. Thank you for your time and service in reviewing our work.
Dear authors, I notice that we are closing to the end of the discussion period. I am hoping to let you know that I am still waiting for your response for adjusting my decision.
To clarify my above concern, the results you demonstrate in Experiment 5 are what I worry about. Compared to ACF, the best result you report slightly decreases the RMSE from 0.589 to 0.577, while your inference time increases from 1.07s to 18.33s, and your training time increases from 424 seconds to 1236 seconds. Since you emphasize that the goal of the consistency model is to improve inference efficiency while maintaining high accuracy, I find this experiment does not support your argument.

While I am trying to evaluate your results in a holistic way, this result as a big portion of your experiments makes me concerned. I would appreciate if you can help me further address this concern.
I thank the authors for their clarifications. After revisiting the experiments in light of their comments, I recognize that FMPE indeed performs poorly on tasks requiring mode separation. This may be due to suboptimal hyperparameter settings, though as I understand the CMPE hyperparameters were not adjusted either. It only seems fair to conclude that CMPE indeed performs better for multimodal posteriors.

While I still believe comparing 1000 steps of FMPE against 30 steps of CMPE in terms of speed is unfair, I now better understand the authors' point that their method achieves comparable performance in shorter time and may even outperform FMPE in accuracy with more steps. A suggestion for the revised version would be to include all evaluation metrics in one table for easier comparison, though I understand this may be unreasonable to expect in the short term of the discussion period. Overall, the method offers a clear empirical performance gain with about a 30% increase in training time, which seems justified.

I appreciate the significant effort the authors have made to improve the presentation and address my critical points. In fairness, I have adjusted my score accordingly:

(Old score → New score)
Presentation: 2 → 3
Rating: 3 → 5
Confidence: 3 → 4
Thank you for replying to our rebuttal, we appreciate your time and the helpful clarifications you provide.

We are grateful for your efforts to point out unclarities and thank you for the recommendations you provided. We are keen to improve the paper’s clarity and therefore gladly incorporated all proposed edits in our manuscript. We did not mean to discount your critique of our presentation but merely wanted to point out the heterogeneity in the reviewers’ assessment of that aspect. We apologize if we failed to bring this message across in our initial rebuttal.

In the following, we would like to respond to your helpful clarifications and answers (we shortened some block quotes due to the character limit in OpenReview).

---
> Regarding the speed comparison, you use FMPE with 1000 sampling steps while your method uses 30 sampling steps when claiming that your method is faster. If we consider speed alone, this comparison might seem unfair.

Because CMPE and FMPE use identical neural networks to ensure a fair comparison, the number of inference steps to achieve good results is the natural hyperparameter to influence the speed-accuracy trade-off of the methods. In Experiments 1-3, the performance of few-step FMPE is unacceptably bad (see for example Figures 1 and 2).
This is one of the main points from the paper: CMPE can reach a performance with few steps that FMPE can only achieve with many steps. Therefore, with CMPE we can get away with fewer inference steps, resulting in increased sampling speed. So what is important is (number of sampling steps needed)x(time per step). As the difference in necessary sampling steps dominates, the difference in the time per sampling step is not as important here.

---
> Table 2 shows that FMPE with 30 sampling steps is marginally faster than your method in Experiment 5. Additionally, in the same Table, FMPE with 30 sampling steps outperforms FMPE with 1000 sampling steps, suggesting potential overfitting in the latter case 

While the RMSE of FMPE with 30 steps is slightly lower (better) than the RMSE of FMPE with 1000 steps, the calibration is extremely bad, as quantified by max ECE in Table 2 and also in the calibration curves that we provide in the PDF of the general rebuttal (at the top in OpenReview). Therefore, we respectfully disagree with the statement that FMPE with 30 sampling steps outperforms FMPE with 1000 sampling steps when we account for uncertainty quantification.

---
> Regarding the justification of decisions [...] this needs to be clearly stated when you introduce those decisions. Others using your method will need guidance or some intuition on how to adjust different components of your method if it fails on some tasks

We wholeheartedly agree with this and are happy to dedicate more space to the justifications in an eventual camera-ready version which allows one more page.

---
> Regarding lines 5-6, [...] Some SBI methods, such as most ABC methods, may have very slow inference times, potentially taking hours depending on the simulator. Therefore, the statement "overcoming sampling inefficiency at [simulator-based] inference time" might not be advantageous in the general SBI sense -- you clearly meant something different, so I proposed to clarify that.

We indeed misunderstood your initial comment and apologize for the inconvenience. Thank you for the clarification, and we are glad to realize that we are on the same page here.

---

> Regarding response to my question "why not simply use other SBI methods instead?" [...]

Thank you for this clarification, we misunderstood the initial comment and apologize for that. We will expand the section on SBI methods with those fast-and-high-accuracy methods for lower-dimensional data. We agree that the requirements of a modeler differ depending on the tasks and NPE/FMPE/CMPE are not the right hammer for every nail.

---

> Regarding "For context: presenting simulation models as programs with latent program states dates back to the seminal paper 'The frontier of simulation-based inference'", it is worth noting that simulator-based inference, also known as "likelihood-free inference", dates back at least to the 1990s and originates from statistics. Including references to these earlier seminal papers could provide a more comprehensive historical context.

Thank you for this clarification and suggestion. We will gladly include references to these earlier seminal papers to paint a comprehensive historical context of the LFI/SBI field.
Thank you for your response, we appreciate that you take the time to engage in the discussion period.

We do not quite understand your verdict. Our timing experiments show that CMPE training takes about as long as FMPE training, but our method has up to 75$\times$ faster inference in Experiments 1-3 and up to 1000$\times$ faster inference in Experiment 4. In Experiments 1-3, our inference takes only slightly longer than NSF and ACF, but has much better accuracy. Thus, our method occupies an interesting location on the Pareto frontier of the different modeling objectives and cannot be dismissed as insignificant.

Please keep in mind that the training times we reported in the rebuttal are based on approximate timings with a fixed number of epochs (not on some early stopping criteria). This means that the small difference between CMPE and FMPE training times is well within the margin of error of the makeshift timing setup. 

We invite you to reconsider your verdict in the holistic context of all empirical evaluations. Thank you for your time and service in reviewing our work.
I thank the authors for addressing my questions and sharing the full ECE coverage plots. They made me realize that in the absence of the possibility of density evaluation, the reported ECE is based on per-dimension coverage.

I keep my rating.
I would like to thank the authors for their professional rebuttal. I feel it is necessary to clarify certain critical points raised regarding my critique of the presentation and perceived performance of your method:

1. In my review, I made a concerted effort to point out specific areas of unclarity in the presentation of your method, particularly in critical sections of the text. I explicitly mentioned that these issues could lead to the contribution being understated, which influenced my decision to lower my confidence score.  Therefore, I believe it is fair to mention that the presentation has affected my assessment of your work and to list it as a primary weakness. Presentation is inherently subjective, and discrepancies between reviewers on this point are not uncommon. I encourage you to refer to the detailed comments in my Questions section for precise feedback on the presentation.

2. I acknowledge the mistake in the abbreviation. I correctly referred to FMPE but inaccurately expanded it to "Fully Neural Posterior Estimation" instead of "Flow Matching Posterior Estimation". This was a typo, I stand corrected. I trust that the authors did not imply my misunderstanding of the method by questioning whether I meant a different approach.

3. I think that your method is competitive with FMPE and I don't think it's an understatement. Regarding the speed comparison, you use FMPE with 1000 sampling steps while your method uses 30 sampling steps when claiming that your method is faster. If we consider speed alone, this comparison might seem unfair. Table 2 shows that FMPE with 30 sampling steps is marginally faster than your method in Experiment 5. Additionally, in the same Table, FMPE with 30 sampling steps outperforms FMPE with 1000 sampling steps, suggesting potential overfitting in the latter case. As for accuracy, based on your empirical evaluations, CMPE does not consistently outperform FMPE; it varies depending on the scenario. As mentioned in my review, I believe your method is competitive and reasonable. As you state in line 365, "CMPE emerges as a competitive method for SBI", and I agree with this assessment without intending to understate your work.

I hope this clarifies my position. Below are specific responses to the rebuttal:

* Regarding the justification of decisions, your response notes that "there is no theory (yet) that allows detailed reasoning about their hyper-parameters" and that "the decisions were made based on empirical observations". However, it still requires some justification in the text. As a reader, I am unable to understand this unless you explicitly point it out. It is perfectly acceptable to adjust design decisions empirically, but this needs to be clearly stated when you introduce those decisions. Others using your method will need guidance or some intuition on how to adjust different components of your method if it fails on some tasks.

* Regarding lines 5-6, I am not suggesting that SBI methods with long inference times are desirable. The initial sentence reads, "a new conditional sampler for SBI that ... overcomes their sampling inefficiency at inference time". I interpreted "inference time" as "[simulator-based] inference time". Some SBI methods, such as most ABC methods, may have very slow inference times, potentially taking hours depending on the simulator. Therefore, the statement "overcoming sampling inefficiency at [simulator-based] inference time" might not be advantageous in the general SBI sense -- you clearly meant something different, so I proposed to clarify that.

* Regarding response to my question "why not simply use other SBI methods instead?" (line 10), there are simpler SBI methods (e.g., BOLFI, ABC, KDEs) that perform well in lower-dimensional cases. Typically, higher-dimensional cases are problematic for SBI, but there are numerous fast and high-accuracy methods for lower-dimensional cases, which may outperform NPE and FMPE in those scenarios.

* Regarding "For context: presenting simulation models as programs with latent program states dates back to the seminal paper 'The frontier of simulation-based inference'", it is worth noting that simulator-based inference, also known as "likelihood-free inference", dates back at least to the 1990s and originates from statistics. Including references to these earlier seminal papers could provide a more comprehensive historical context.
I thank the authors for the rebuttal and additional experiments. The newly added experiments partly confirm my concern that the training efficiency of CMPE is adverse compared to the other methods. At the same time, the relative gain in the inference time is not significant. I am inclined to keep my current rating.
> (lines 41-43): I don't understand the significance of this sentence. Do authors imply that they explore empirically the quality of conditional CMs and it is one of the contributions? It follows the sentence about the empirical evaluations in the paper, so I am a little bit confused.

Yes, it is one of the contributions that we are the first to explicitly study the uncertainty quantification of consistency models (e.g., via simulation-based calibration), which is typically not a concern in the literature on generative image modeling. We agree that the ordering of the sentences is suboptimal here. We will swap the sentences to:

Additionally, the quality of conditional CMs as Bayesian samplers has not yet been explored empirically (e.g., in terms of probabilistic calibration and precision), even though this is crucial for their application in science and engineering. Lastly, while CMs for image generation are trained on enormous amounts of data, training data are typically scarce in SBI applications. In our empirical evaluations, we demonstrate that CMs are competitive with state-of-the-art SBI algorithms in low-data regimes using our adjusted settings (see Appendix A for details).

---

> (line 59): I am not familiar with the term 'simulation-based training'. Perhaps the point of the term is to emphasize the synthetic nature of the data generated by the simulator or some other reason?

Correct, the training scheme is based on synthetic simulations by the simulator. We change the sentence to:

*In the following, the neural network training relies on a synthetic training set [...]*

---

> (line 66, 70): I am not sure why simulation models are presented as 'programs'. [...]

**For context:** Presenting simulation models as programs with latent program states dates back to the seminal paper “The frontier of simulation-based inference” [1] and it is equivalent to the classical SBI setting. We agree with your concern and will remove the ‘program’ notion of simulation models in the manuscript to avoid confusion.

[1] https://www.pnas.org/doi/10.1073/pnas.1912789117 

---

> Typos: (line 71): 'g( \cdot, \cdot)'. (line 74): 'reasoning' (line 79): 'sequential and amortized'. (line 95): 'it' (line 124): 'they'

Fixed, thank you.

---

> (line 101): Please provide a reference for 'optimal transport'.

Thanks, we added [1] and [2] as a reference for optimal transport.

[1] Villani, C. (2009). Optimal Transport. https://link.springer.com/book/10.1007/978-3-540-71050-9

[2] Peyré, G., & Cuturi, M. (2019). Computational Optimal Transport. Foundations and Trends in Machine Learning, 11(5–6), 355–607.https://arxiv.org/abs/1803.00567 

---
> (line 151): x is presented as “a fixed conditioning variable x”, but it is already used for observations in the SBI context. Could you clarify in the text if they are the same?

In SBI, the observations $x$ are in fact the (fixed) conditioning variables for the neural density estimator. Edit to the text: 

*[...] given a fixed conditioning variable (i.e., observation) $\mathbf{x}$.*

---
> (line 177): Could you at least briefly detail the ‘hardships for computing the posterior density’ in the main text?

Thanks, edited to: 
*Currently, this comes at the cost of explicit invertibility, which limits the computation of posterior densities. More precisely, single-step consistency models do not allow density evaluations at an arbitrary parameter value $\boldsymbol{\theta}$ but only at a set of $S$ approximate posterior draws* $\\{\boldsymbol{\theta}\_{\varepsilon}^\{(1)\}, \ldots, \boldsymbol{\theta}_{\varepsilon}^\{(S)\}\\}$. 
*However, this is sufficient for important downstream tasks like marginal likelihood estimation, importance sampling, or self-consistency losses. In contrast, multi-step consistency sampling defines a Markov chain which cannot be evaluated without an additional density estimator (see Appendix C for details).*

---
> (Eq 8): Could you provide justification or references for this objective?

This optimization objective is given in Eq. 5 of Song and Dariwal [1]. The paper was recently published at ICLR, the new reference is given below. We adapted the equation to feature a conditioning variable, as required in SBI, and made minor changes to match our notation. We have included the reference to [1] in the updated version of the manuscript:

*We formulate the consistency training objective for CMPE, which extends the unconditional training objective from Song & Dhariwal (2024) with a conditioning variable $\mathbf{x}$ to cater to the SBI setting,
`<Equation 8>`,
where [...]*

[1] Song, Y., & Dhariwal, P. (2024). Improved Techniques for Training Consistency Models. The Twelfth International Conference on Learning Representations. https://openreview.net/forum?id=WNzy9bRDvG

---
> (line 234): Could you provide a reference, where the hyper-parameters for the comparison methods are listed?

Thanks, we added: 

*Appendix D lists hyperparameters of all models in the experiments.*
> (line 1): Consider revising the first sentence "Simulation-based inference (SBI) is constantly in search of more expressive..." to be more concrete.

Thank you, we edited the sentence to:

*Research in simulation-based inference (SBI) aims to develop algorithms that can accurately estimate the unknown parameters of complex simulation models while minimizing computational time and the amount of required training data.*

---

> (line 5): Clarify what is meant by 'free-form architectures'.

**TL;DR:** We will change the term free-form to unconstrained.

**Detailed answer:** The term “free-form architecture” refers to an architecture of a generative neural network that is not subject to specific design constraints. We use this term following Draxler et al. (2024; [1]). Dax et al. (2023, page 1; [2]) call this “unconstrained architecture”, which is synonymous. For example, most normalizing flow architectures (affine coupling flow, neural spline flow) require a specialized layout that allows cheap computation of the Jacobian determinant, which restricts the design space of the neural networks. Hence, normalizing flows typically don’t allow free-form (unconstrained) architectures.  In contrast, flow matching and consistency models do not require a special neural network architecture, which means that they allow “free-form architectures” that can be more expressive and tailored to the specific data at hand [1]. We acknowledge that the term “free-form” is currently less widespread and will instead use “unconstrained” in the manuscript.

[1] Draxler et al. (2024). Free-form Flows: Make Any Architecture a Normalizing Flow. https://arxiv.org/abs/2310.16624

[2] Dax et al. (2023). Flow Matching for Scalable Simulation-Based Inference. https://arxiv.org/abs/2305.17161 

---

> (line 5-6): Some SBI methods have long inference times, and 'overcoming sampling inefficiency at inference time' may be bad for them --perhaps some clarification is needed.

Could you please clarify what you mean by this remark? We argue that SBI methods with long inference times are generally not desirable if we can achieve equal performance with shorter inference times. For example, we empirically show that flow matching (FMPE) is accurate but slow, and consistency models are as accurate (or more accurate depending on the task) but up to 100x faster.

---

> (line 8-9): 'Providing an architecture' slightly contrasts with the advantage of having a free-form architecture, so some justification is need.

While consistency models allow for unconstrained free-form architectures, we would like to provide readers and analysts with sensible default architectures as a starting point. We will clarify this sentence in the abstract. 

---

> (line 10): Could you specify the dimensions this line refers to and explain why low-dimensional simulator parameters were difficult to handle with unconditional consistency models [...]?

This line refers to the dimensionality of the parameter space in SBI. In the public review/rebuttal by Song et al. (2023, [1]), the authors report that consistency models are particularly efficient in high-dimensional applications and are expected to struggle in low-dimensional tasks. However, they do not give a theoretical reason for this. In our work, we mitigate this shortcoming through a revised set of default hyperparameters which led to excellent performance in a range of SBI tasks, and this is one contribution of our paper.

[1] https://openreview.net/forum?id=WNzy9bRDvG&noteId=EQQd87ImQk

> [continued] …  and why not simply use other SBI methods instead?

Our comparisons feature multiple state-of-the-art SBI methods (NPE with affine coupling flows, NPE with neural spline flows, FMPE), and we show that consistency models outperform these other SBI methods in almost all cases by achieving fast and accurate inference.

---

> (line 20): Could you rephrase to avoid broad generalizations about SBI research, as there are many recent works focused, for instance, on choosing summary statistics with little relation to generative modeling?

Thank you, we agree with you and propose the following clarification:

*Recently, multiple streams of neural SBI research have been capitalizing on the rapid progress in generative modeling of unstructured data by re-purposing existing generative architectures into general inverse problem solvers for applications in the sciences [7–10].*

---

> (line 31): Could you clarify the term 'free-form architecture'?

We changed the term to “unconstrained”. See your comment and our response above for more context.

---

> (line 36): Please define 'target spaces' in the context of SBI applications.

Thanks, we will replace ‘target spaces’ with ‘parameter spaces’ to match the standard SBI terminology. If you prefer another expression, please let us know and we will be happy to adjust it.