Stacking Variational Bayesian Monte Carlo
Authors: Francesco Silvestrin
Chengkun LI
Luigi Acerbi
Venue: Accepted by TMLR ()

Abstract: Approximate Bayesian inference for models with computationally expensive, black-box likelihoods poses a significant challenge, especially when the posterior distribution is complex. Many inference methods struggle to explore the parameter space efficiently under a limited budget of likelihood evaluations. Variational Bayesian Monte Carlo (VBMC) is a sample-efficient method that addresses this by building a local surrogate model of the log-posterior. However, its conservative exploration strategy, while promoting stability, can cause it to miss important regions of the posterior, such as distinct modes or long tails.
In this work, we introduce Stacking Variational Bayesian Monte Carlo (S-VBMC), a method that overcomes this limitation by constructing a robust, global posterior approximation from multiple independent VBMC runs. Our approach merges these local approximations through a principled and inexpensive post-processing step that leverages VBMC's mixture posterior representation and per-component evidence estimates. Crucially, S-VBMC requires no additional likelihood evaluations and is naturally parallelisable, fitting seamlessly into existing inference workflows. We demonstrate its effectiveness on two synthetic problems designed to challenge VBMC's exploration and two real-world applications from computational neuroscience, showing substantial improvements in posterior approximation quality across all cases. Our code is available as a Python package at https://github.com/acerbilab/svbmc.

Published in Transactions on Machine Learning Research (11/2025)
Stacking Variational Bayesian Monte Carlo
Francesco Silvestrin
Deportment of Computer Science University of Helsinki
Chengkun Li
Chengkunnli@hclliing
Deportment of Computer Science University of Melsinki
Luigi Acerbi
haigi.acerbi@@hessikk..
Department of Computer Science University of Helsinki
Reviewed OpenReview: https: oppprevvepperevieuu. net/ forum? id= www.:a/awaaaaaadeeccoooeoooaoaoa
Abstract
Approximate Bayesian inference for models with computationally expensive, black-box like- lihoods poses significant challenge, especially when the posterior distribution is complex. Many inference methods struggle to explore the parameter space efficiently under a limited budget of likelihood evaluations. Variational Bayesian Monte Carlo (VBMC) is a sample- efficient method that addresses this by building a local surrogate model of the log- prsposterror. However, its conscrvative exploration stratogy, while promoting stability, can cause to miss important regions of the posterior, such as distinct modes or long tails. In this work, we introduce Stacking Variational Bayesian Monte Carlo (S-VBMC), a method that overcomes this limitation by constructing robust, global posterior approximation from multiple inde- pendent VBMC runs. Our approach merges these local approximations through a principled and inexpensive post-processing step that leverages VBMC's mixture posterior representa- tion and per-component evidence estimates. Crucially, S-VBMC requires no additional like- lihood evaluations and is naturally parallelisable, fitting seamlessly into existing inference workflows. We demonstrate its effectiveness on two synthetic problems designed to chal- lenge VBMC's exploration and two real-world applications from computational neuroscience, showing substantial improvements in posterior approximation quality across all cases. Our code is available as a Python package at https:://pp thub. com/accebilab/svbmne
1 Introduction
Bayesian inference provides a powerful framework for parameter estimation and uncertainty quantification. but it is usually intractable, requiring approximate inference techniques (Brooks et al. 2017). Many scientific and engineering problems involve black-box models (Sacks et al. 1989 Kennedy O'Hagan 2001), where likelihood evaluation time-consuming and gradients cannot be casily obtained, making Iraditional approximate inference approaches computationally prohibitive.
2011 Blei et al.
A promising approach to tackle expensive likelihoods is to construct statistical surrogate model that approximates the target distribution, similar in spirit to surrogate approaches to global optimisation using Gaussian processes, generally known as Bayesian oplimisation (Rasmussenn & Williams 2006 Garneu (023) However, unlike Bayesian optimisation, where the goal is to find single point estimate (the global optimum
here the aim is reconstruct the shape of the entire postcrior distribution. In this sotting, attempting
build single global surrogate model may lead numerical instabilities and poor approximations when the target distribution is complex multi-modal, without ad hoc solutions (Wang & Li 2018 Järvenpää et al.

---

Published in Transactions on Machine Learning Research (11/2025)
Li et al. 2025). Local or constrained surrogate models, while more limited in scope, tend to be more stable and reliable in practice (Fl Gammal et al. 2023 Järvenpää & Corander 2024).
2021
Variational Bayesian Monte Carlo (VBMC; Acerbi 2018 exemplifies this local approach, using active sam pling to train a Gaussian process surrogate for the unnormalised log- posterior on which it performs variational inference. VBMC adopts a conservative exploration strategy that yields stable, local approximations (Acerbi 2019). Compared other perogatte-bsee approaches, the method offers a versatile set of features: it returns the approximate posterior the model evidence (ELBO) via Bayesian quadrature (Ghahramani & Rasmussen 2002), useful for model selection; and it can handle noisy log-likelihood evaluations 4Acerbi 2020), which arisc in simulation-based models through estimation techniques such as inverse binomiaa sampling (van Opheusden et al. 2020 and synthetic likelihood (Wood] 2010: Price et al. 2018). However, VBMC's limited likelihood evaluation budget combined with its local exploration strategy can leave vulnerable to potentially missing regions of the target posterior particularly for distributions with distinct modes or long tails.
tractable distribution (a mixture of Gaussians); it provides lower bound for
In this work, propose practical, yet effective approach to constructing global surrogate models while overcoming the limitations of standard VBMC by combining multiple local approximations. We introduce Stacking Variational Bayesian Monte Carlo (S-VBMC), method for merging independent VBMC inference runs into a coherent global posterior approximation. S-VBMC inherits its parent algorithm's operating regime and intended for low- to moderattedimmussoonal problems (D S 10). Our approach leverages VBMC's unique properties its mixture posterior representation and per-component Bayesian quadrature estimates of the ELBO to combine and reweigh each component through a simple post-processing step, Figure 1 shows example of two separate posteriors and the combined ("stacked") result obtained with S-VBMC.
Separate VBMC posteriors
Stacked posterior
0.01
0.01
0.00
0.00
-0.01
00.01
1e-5
2e-5
3e-5
1e-5
2e-5
3e-5
02
02
Two separate VBMC posteriors (left, shown as blue and orange contours) and the resulting stacked posterior via S-VBMC (right, red contour) for a neuronal model with real data (see Section 4.5): showing the marginal distribution of two out of the five model parameters.
Figure 1:
Crucially, method requires additional evaluations of either the original model the surrogate. This approach is easily parallelisable and naturally fits existing VBMC pipelines that already employ multiple independent runs (Huggins et al. 2023 While our method could theoretically extend to other variational approaches based on mixture posteriors, VBMC is uniquely suitable for it as re-estimation of the ELBO would otherwise become impractical with expensive likelihoods (see Section 3
Related work. Our work addresses the challenge of building global posterior approximations by com- bining local solutions from the VBMC framework (Acerbi 2018 2019 2020). While the idca of combining posterior distributions has been explored before, previous approaches dilfer substantially in their goals and methodology.
"Stacking" was first introducod in the context of supervisod learning as a method for model averaging (Wolpert] 1992). Given a set of predictive models, the idea was to use a weighted average of their outputs, with the weights optimised to minimise the leave-one-out squared error. This approach has then been
2

---

Published in Transactions on Machine Learning Research (11/2025)
which the authors named Bayesian slacking. This still relies on leave-one-out strategy to optimise predictive performance, which requires access to the likelihood per data point, while S-VBMC optimises the ELBO on the full dataset, allowing treatment of the log-joint as a black box.
adapted to average Bayesian predictive distributions Yao et al. 2018:
2022
Another relevant approach is variational boosting (Guo al. 2016 Miller al. 2017 Campbell Li 2019 This method builds on black-box variational inference (Ranganath 2014) and entails iteratively running series of variational optimisations (number of components) of mixture posterior distribution. This allows practitioners to obtain arbitrarily complex (and accurate) posteriors, trading compute time for inference accuracy. However, the process is inherently sequential, whilst S-VBMC can be implemented as simple post-processing step, allowing individual (VBMC) inference runs to happen in parallel, offering significant computational advantages, further substantiated by VBMC's surrogate-based approach and sample efficiency, which make it particularly suitable for problems with expensive likelihoods.
the whole dataset, with each iteration increasing the complexity
Parallel computations have, the other hand, been leveraged in number of tdivide-and-ooqqeee barrussingly parallel approximate inference techniques, starting from embarrassingly parallel Markov Chain Monte Carlo (MCMC) NNiiwanger et al. 2014]. All these methods are based on dividing the data into subsets which are processed separately to obtain a set of "sub-pasteriors" which are then merged in various
ways to recreate the full (approximate) posterior (Wang & Dunson 2013 Wang et al. 2015 Nemeth & 2022
Sherlock 2018 Srivastava et al.) 2018: Scott et al.
De Souza et al.] 2022; Chan et al. 2023). These methods are mostly motivated by the need to process very large datasets (Scott et al. 2022) or by privacy
concerns requiring federated learning (Liang et al. 2025). In contrast, S-VBMC is motivated by need to capture complex posteriors that elude single inference runs. Therefore, individual runs all use the complete dataset. This allows our method not to rely on the quality and representativeness of the sub-datasets, and to remain robust to individual run failures.
Outline. We first introduce variational inference and VBMC (Section then present our algorithm for stacking VBMC posteriors (Section 3 We demonstrate the effectiveness of our approach through experi- ments on two synthetic problems and two real-world applications that are challenging for VBMC (Section 4 We then discuss observed bias buildup in the ELBO estimation and propose practical hcuristic to coun- teract it (Section 5/. We finally discuss our results (Section G and conclude with closing remarks (Section 7
Appendix A contains supplementary materials.
2
Background
2.1
Bayesian Inference
Given a dataset D, and a model parametrised by the vector 0 E RRD describing how 'D was generated, Bayesian inference represents a principled framework to infer the probability distributions over 0. This is achieved through Bayes' rule:
r0 (1) p(0)p(D) p(D)
(1)
where p(0) represents the prior over model paramoters, p(D 0) the likelihood and p(D) = J p()())(( 0)d0 the normalising constant, also called marginal likelihood model evidence. This latter quantily is particularly useful for Bayesian model selection (MacKay 2003), but the integral is often intractable, requiring some approximation technique (Brooks et al. 2011 Blei et al. 2017). In the following section, we discuss one of such techniques.
2.2 Variational Inference
Considering the setup outlined in Eq. [ variational inference Blei et al. 2017) is a technique that approx- imates the true posterior P(0 with parametric distribution 44 (0), where denotes the optimisable
3

---

Published in Transactions on Machine Learning Research (11/2025)
variational parameters. This is achieved by maximising the evidence lower bound (ELBO):
ELBO(Ø) Eu [ + (g)(0)),
(2)
where the first term is the expocted log-joint distribution (the joint being likelihood times prior) and the second term is the entropy of the variational posterior. Maximising Eq. 2 is equivalent to minimising the Kullback-Leibler divergence between (0) and the true posterior, as
Is (0)
1k (0) p(0) D)] Egs log
p(0 (D)
(3)
E4 log (0, D)] + Egg p(D) + Egg [log qø (0)),
and, since the model evidence p(D) is already a constant,
logp(D) DKL. (k).(uu(0) || v(0 D) BBg log (0, D)] + H [1+(
(4)
=EBBO()).
Crucially, since Dxr.(q l p] 2 0, the ELBO provides a lower bound on the log model evidence log p(D) (hence the name), with equality when the approximation matches the true posterior (i.e., when DKL || pl = 0), thus constituting uscful metric for model sclection.
2.3 Variational Bayesian Monte Carlo (VBMC)
2020) is a sample-efficient technique to obtain a variational approximation of a target density with only small number of likelihood evaluations, often of the order of a few hundred. VBMC uses a Gaussian process (GP) surrogate of the log-joint, Bayesian quadrature to calculate the expected log-joint, and active sampling to decide which parameters to evaluate next. As in-depth knowledge of the inner workings of VBMC is not necessary to undersland our core contribution, here we only describe its relevant aspects. An interested reader should refer to the original papers (Acerbi 2018 2020) or Appendix A.1 where we providc a more detailed description of the algorithm.
VBMC (Acerbi 2018
As mentioned above, VBMC uses a GP as a surrogate of the larget, which often an unnormalised log- posterior (i.e., the log-joint).. Crucially, VBMC performs variational inference on the surrogate
f(0) log (((()))(()
(5)
instead of the true, expensive model. The efficacy of VBMC hinges the quality of such approximation. The variational posterior in VBMC is defined as a flexible mixture of K components,
K
(0)
wkgk (0),
(6)
where Ok is the k-th component (a multivariate normal) and wy its mixture weight, with Ek=1 ui 1 and
Wk 2 0. Plugging in the mixture posterior, the ELBO (Eq. 2 becomes:
ELBO
Σ Ewplg log (o))(D))))(O) (0)
Σ R=1
wolk + 7. (0)
(7)
k=1
where we defined the k-th component of the expected log-joint as:
J Equ. [log (D(0))(()) Eyy. [f(0)] f..
(8)
The efficacy of VBMC stems from the fact that Eq. has closcd-form Gaussian expression via. Bayesian Ghahramani & Rasmussen, 2002), which yields posterior mean fę and covariance matrix Jay (Acerbi 2018). The entropy of mixture of Gaussians does not have analytical solution, but
quadrature O'Hagan 1991
4

---

Published in Transactions on Machine Learning Research (11/2025)
gradients can be estimated via Monte Carlo. Thus, using the posterior mean of Eq. as plug-in estimator for the expected log-joint of each component, Eq. 7 can be efficiently optimised via stochastic gradient ascent [Kingma & Ba 2014).
VBMC differs from other approaches that directly try to apply Bayesian quadrature to solve Bayes' rule (e.g.. Ghahramani Rasmussen, 2002 Osborne et al. 2012 Gunter et al., 2014 Adachi al. 2022 in that instead it leverages Bayesian quadrature estimate the ELBO. This a simpler problem, since instead of attempting to solve the global integral of Eq. [ namely the integral of prior times likelihood, where the prior is often diffuse, it mainly deals with the local integrals in Eq. namely the expected log-joint, i..., the integral of the approximate posterior times log-joint, where the approximate posterior is under our control and often more localised, and the entropy, which can often be estimated or approximated for known distributions.
3 Stacking VBMC
In this work, we introduce Stacking VBMC (S-VBMC), novel approach to merge different variational posteriors obtained from different runs on the same model and dataset. Crucially, these runs can happen in parallel, with no information exchange required. The core idea is that single global surrogate of the log-joint f(0) might be inaccurate in some regions of the posterior. Therefore, it would be beneficial to leverage the combination of M local surrogates {fm (0) instead, each yiclding good approximation of the target in different parameter region.
3.1 Optimisation objective
Given M independent VBMC runs, one obtains M variational posteriors
IΦ... (0)
Σ Wm.kqk.d (0),
(9)
each with Km Gaussian components, well as M different im vectors, per Eq. with im (Im,1...., Im.k..). Our approach consists of "stacking" the Gaussian components of all posteriors Is... (0). leaving all individual components' parameters (means and covariances) unchanged, and reoptimising all the weights. The full set of new mixture weights is denoted by the vector w {wmm,) m { The complete set of parameters for the final stacked posterior, 9g denoted by Φ. This set consists of the frozen parameters (all means and covariances) from the original components {9.. and the newly optimised weights w.
Thus, given the stacked posterior
M Km
13
(0)
W 1h (0);
(10)
1=1
we optimise the global evidence lower bound with respect the weights w,
M
Km
ELBO gLBBkee (w) =
Σ
Σ
W Wm
Im.k + 7 99 [g)
(11)
m=
It should be noted that cach VBMC run applics its own parameter transformation gm.(.) during infer- ence (Acerbil 2020), so variational posteriors are returned in transformed coordinates Ie.,. (gm(0)). For clarity, in this section, we present all quantities a common parameter space of and VBMC and V-MCC posteriors expressed accordingly as go... (0), and (0), respectively. In practice, when applying our stacking approach, we account for these per-run transformations via chango-of variaalle (Jacobian) corrections; see Appendix A.2 for details.

---

Published in Transactions on Machine Learning Research (11/2025)
3.2 Algorithm
As the entropy term in Eq. 11 does not have closed-form solution, it needs to be estimated via Monte Carlo sampling. To do this, we take samples and estimate the entropy
from each component of the stacked posterior, X qk om
s=1
M
Km
8 (x(a) (xmxk)
H [0)(0)] 2
Wm,h
log Us
(12)
5
m
k=1
sml
This works because
H [0)
-Bng log (0)
=-f 9 (0) log 93 (0)
M Km
Wm,kleee om (0) 1o (0))0
(13)
m=
M Km
Wm. / om (0) log 96 (0)d
m=
k=1
Wm,k Be [log
where we approximate the expectation via Monte Carlo,
Eggr..e [og az (0)
1 S
Σ
log oz (x(n)))
(14)
Importantly, with this formulation, the entropy is differentiable with respect to the weights w.
Since weights must be non-negative and sum one (i.e., they live on the probability simplex), for the purpose of optimisation follow standard practices in computational statistics (e.g., Carpenter al. 2017): we reparameterise the weights with unconstrained logits a and compute the objective with softiax weights,
exp(am.k)
Wm,kk
(15)
Em= Σtei exp(am((.))
We initialise the logits as
km,k log(wm.t) + ELBO(
max (log(Wm wx) ELBO( )), Sm M ISk'KK,.
(16)
where the last term ensures that max(a) = 0, preventing underflow in the exponentials. This is equivalent to initialising the weights as
Wm.k oc www.eexpEELBO(()).)))
(17)
effectively weighting each individual posterior by its approximate evidence (i.e., the exponential of the corresponding ELBO). This assigns higher initial weight to the components coming from the best runs, providing a good starting point for the optimisation process.
We can then use the Adam optimiser (Kingma & Ba 2014 to optimise the stacked ELBO. An overview of
the procedure is outlined in Algorithm 0
Notably, the optimisation can be performed as pure post-processing step, requiring neither evaluations of the original likelihood p(D(0) nor of the surrogate models fn. only that the estimates fn are stored, as in current implementations (Huggins et. al. 2023

---

Published in Transactions on Machine Learning Research (11/2025)
Algorithm S-VBMC
Input: outputs of M VBMC runs (Φm, ELBO
1. Stack the posteriors to obtain mixture of Σm= Km Gaussians
Reparametrise the weights with unconstrained logits
3. Initialise the logits as in Eq. 16
repeat
(a) Compute the normalised weights w as in Eq. 15
(b) Estimate ELBOstacked (W) as Eq. with the entropy approximated as in Eq. 12
(c) Compute dELBOstcceddl da
(d) Update with step of Adam Kingma & Ba 2014
until ELBO ()) converges
return Φ, staked
Our stacking method hinges the key feature of VBMC of providing accurate estimates mn While in principle Eq. 11 could apply any collection of variational posterior mixtures, without efficient way of caleulating each Ia (Eq. {), optimisation of the stacked ELBO would require many likelihood evaluations, which would be prohibitive for problems with expensive, black-box likelihoods.
In the following, we demonstrate the efficacy of this approach.
4 Experiments
In this section, we describe our experimental procedure (Section 4.1 baselines (Section 4.2 evaluation metrics (Section 4.3), and results for synthetic (Section 4.4) and real-world problems (Section 4.5 Finally, we briefly discuss computational costs in Soction
4.6 Additional results are reported the appendix, including additional experiments (Appendix A.3) more extensive tables of results (Appendix A.4). and example visualisations of posterior approximations (Appendix A.5
4.1 Procedure
We first tested our method on synthetic problems, designed be particularly challenging for VBMC, and then two real-world datasets and models (see below for full descriptions). We considered both noiseless problems (exact estimation) and noisy problems where Gaussian noise with 3 is applied to each likelhood measurement, emmlating what practitioners might find when estimating the likelihood via simulation (van Opheusden et al.] 2020). For each benchmark, we considered 100 VBMC runs obtained with the pyvbmc Python package (Hnggins et al. 2023 that satisfiod the following conditions:
1. the algorithm had converged (as assessed by the pyvbme software);
max (JR,A) 5, so all the estimates f had an associated variance lower than ISKCK
The reason for the latter is that S-VBMC's efficacy is strongly dependent accurate VBMC estimates of the real I from Bayesian quadrature (see Sections 2.3 and 3.2), and we sought to filter out "poorly converged" runs where this might not be the case. The results of this filtering procedure can be found in Appendix A.4.I. We performed all VBMC runs using the default settings (which always resulted in posteriors with components, K, 50 Y m E (I.....)) and random uniform initialisation within plausible parameter bounds (Acerbi 2018). To investigate the effect of combining a different number of posteriors, adopted bootstrapping approach: from these 100 runs, randomly sampled and stacked

---

Published in Transactions on Machine Learning Research (11/2025)
with S-VBMC a varying number of runs (between 2 and 40) twenty times each, and computed the median with corresponding 95% confidence interval (computed from 10000 bootstrap resamples) for all metrics, described below. For all benchmark problems, the entropy is approximated as in Eq. 12 with S = 20 during optimisation, and a final estimation (reported all tables and figures) is performed with $ = 100 after convergence. The ELBO is optimised using Adam (Kingma & Ba 2014) with learning rate set. to 0.1.
All the experiments presented in this work were run on an AMD EPYC 7452 Processor with 16GB of RAM.
4.2
Baseline methods
Black-box variational inference. We used black-box variational inference (BBVI; Ranganath et al. 2014] as baseline for all our benchmark problems.
Our implementation follows Li et al. (005 For gradient-free black-box models, we cannot use the repa- rameterisation trick (Kingma & Welling 2013) to estimate ELBO gradients. Instead, we employ the score function estimator (REINFORCE; Ranganath et al. 2014) with control variates to reduce gradient variance.
The variational posterior is parameterised as mixture of Gaussians (MoG) with either K = 50 K 500 components, matching the form used in VBMC. We initialise the component means near the origin by adding Gaussian noise (a = 0.1) and set all component variances to 0.01. We optimise the ELBO using Adam (Kingma & Ba 2014) with stochastic gradients, performing grid search over Monte Carlo sample sizes 10, }00} and learning rates (0.01, 0.001}. We select the best hyperparameters based on the estimated ELBO.
For a fair comparison with S-VBMC, we set the target evaluation budget for a BBVI run to 2000(D+2) and 3000( + 2) evaluations for noiseless and noisy problems, respectively, matching the maximum evaluations used by 40 VBMC runs in total.
Naive Stacking. As a further bascline, we implemented a "naive" version of our stacking approach, consisting of simple averaging of the individual VBMC posteriors. For this, we rewrite the stacked posterior
M
95 (0)
=
Wm Iarm (0)
(18)
m=1
and simply set
W1 = We2
Win = 1/M.
(19)
We call this approach "Naive Stacking" (NS).
As for S-VBMC, we ran NS by randomly sampling varying number of runs (between 2 and 40) twenty times each, and then computed the median with corresponding 95% confidence intervals for all metrics.
4.3 Metrics
Following Acerbi 2020
Li et al. 2025
we evaluate our method using three metrics:
The absolute difference botween true and estimated log marginal likelihood (ALML), where values considered negligible for model selection (Burnham & Anderson 2003).
The mean marginal total variation distance (MMTV), which measures the average (lack of) overlap between true and approximate posterior marginals across dimensions:
D Σ
MMTV q) 2D
(pa(z)) qa(xx)) dxa.
(20)
where Ps and qa denote the marginal distributions along the d-th dimension.
8

---

Published in Transactions on Machine Learning Research (11/2025)
3. The "Gaussianised" symmetrised KL divergence (GsKL), which evaluates differences in means and covariances between the approximate and true posterior:
1
GsKT(p.
2D [DL (ND)))W)) PRL (WLl(Woll
(21)
where N Mpp] denotes a Gaussian with the same mean and covariance as p.
We consider MMTV 0.2 and GsKT. < g as target thresholds for reasonable posterior approximation Li et al. 2025). Ground-truth estimates of the log marginal likelihood and posterior distributions are obtained through numerical integration, extensive MCMC sampling, or analytical methods as appropriate for each problem.
4.4 Synthetic problems
GMM target. Our synthetic GMM target consists of mixture of 20 bivariate Gaussian components arranged in four distinet clusters. The cluster centroids were positioned at -8), (-7,7), (6,-6) and 55. Around each centroid, we placed five Gaussian components with means drawn from (W I), where Me is the respective cluster centroid and I is the 2x2 identity matrix. Each component was assigned unit marginal variances and a correlation cocfficient of +0.5 (randomly selected with equal probability). This configuration produces an irregular mixture structure that requires substantial number of components to approximate accurately. All components were assigned equal mixing weights. The resulting distribution is illustrated in Figure 2 (top pancls).
VBMC
S-VBMC (5 posteriors)
S-VBMC (20 posteriors)
10
10
10
0
-10
-10
-10
-10
10
-10
0
0
VBMC
S-VBMC (5 posteriors)
S-VBMC (20 posteriors)
10
10
10
S
0
S
0
-10
-10
-10
-10
10
-10
0
10
-10
0
10
0
0
0
Figure 2: Examples of overlap between the ground truth and the posterior when combining different numbers of VBMC runs on the GMM (top panels) and ring (bottom panels) synthetic benchmarks. The red points indicate samples from the posterior approximation, with the target density depicted with colour gradicnts in the background.
In our experiments, we used both a noiseless version of this target (i.e., exact target evaluation) and a noisy one, where we applied i..d. Gaussian noise (a = 3) to each log- likeliohdooo evaluation.
Ring target. Our second synthotic target is a ring-shaped distribution defined by the probability density function
Pring (0, ) x exp ( (r R)2
(22)
2

---

Published in Transactions on Machine Learning Research (11/2025)
where r (81 c1 (02 c2)² represents the radial distance from centre (c1, R is the ring radius,
and a controls the width of the annulus. We set R = 8, = 0.1, and centred the ring at (c1,22 (1, -2). The small value of a produces narrow annular distribution that challenges VBMC's exploration capabilities. The resulting distribution is shown Figure (bottom pancls).
As with the GMM target, we used both a noiseless version of this benchmark and noisy one (a = 3) in our experiments.
(a) GMM (D 2, noiseless)
3.0
0.44
13.00
L.M
MMTV
C.KL
0.1
0.2
2.0
0,001
0.001
0.0
1
8
16
24
32
40
16
24
32
40
14
16
24
32
40
1.4
16
24
32
40
N. of runs
(b) GMM (D
2,
10
D.4
ELUO
3.0
LML
MMTV
0.3
OsKL
0.1
0.1
0.01
2.0
0.0
1
16
24
32
40
14
16
24
32
40
1.4
16
32
40
1
18
24
322
N. of runs
(c) Ring (D 2, noiseless)
0.8
0.0
BL80
L.M.
MMTV
CoKL
-2.5
0.1
0.2
0.01
-7.5
0.001
24
32
40
16
nuns
(d) Ring (D = 2.
3)
0.8
EL.0
LML
MMTV
0.4
GoKE
0.2
0.01
-2
nons
Figure 3: Synthetic problems. Metrics plotted as function of the number of VBMC runs stacked (median and 95% confidence interval). Metrics are plotted for S-VBMC (bluc), VBMC (red), and NS (grey). The best BBVI results are shown in green. The black horizontal line in the ELBO panels represents the ground- truth LML, while the dashed lines on ALML, MMTV, and GsKL denote desirable thresholds for each metric (good performance below the threshold; see Section 4.3)
Results. Results in Figure 3 and Table A.3 show that merging more posteriors Icads to stcady im- provement in the GsKL and MMTV metrics, which measure the quality of the posterior approximation. Remarkably, S- S-VBMC proves to be robust to noisy targets, with minor differences between noiseless and noisy scttings. In all our synthetic benchmarks (both noiseless and noisy settings), we observe that the values of the GsKL and MMTV metries, which directly compare to the ground-truth target densities, reach good values and start to plateau, least. to improve at much slower pace by the time 10 VBMC
10

---

Published in Transactions on Machine Learning Research (11/2025)
posteriors are stacked. This suggests that value of M = 10 (or even M ~ 6 for noiseless problems, where metrics tend to converge faster) would be sufficient to obtain accurate stacked posteriors.
S-VBMC outperforms the BBVI baseline on the ring-shaped synthetic target. The BBVI baseline performs well and is only marginally worse compared to S-VBMC only on the GMM problem, where it effectively managed to capture the four clusters (see Figure 2 for a visualisation). While NS exhibits a similar im- provement pattern with increasing values of M, S-VBMC consistently outperforms it, with differences being larger in noiseless settings.
As expected by design, individual VBMC runs tended to explore the two synthetic target distributions only partially, leading to poor performance. Still, the random initialisations allowed different runs to discover different portions of the posterior, allowing the merging process to cover the whole larget (see Figure 2
Finally, we observe that, in noisy settings, while the ELBO keeps increasing, the ALML, error (difference between ELBO and true log marginal likelihood) initially decreases but then increases again as further components are added, a point which will discuss later.
4.5 Real-world problems.
Neuronal model. Our first real-world problem involved fitting five biophysical parameters of a detailed compartmental model of a hippocampal CA1 pyramidal neuron. The model was constructed based on experi- mental data comprising a shree-dimensional morphological reconstruction and electrophysiologiaal recordings of neuronal responses to current injections. The deterministic neuronal responses were simulated using the NEURON simulation environment (Hines Carnevale 1997 Hines et al. 2009). applying current step inputs (hat matched the experimental protocol. The model's parameters characterise key biophysical properties: intracellular axial resistivity (01), leak current reversal potential (02), somatic leak conductance (0s), den- dritic conductance gradient (04, per μm), and a dendritic surface scaling factor (0z). Based on independent measurements of membrane potential fluctuations, observation noise was modelled as a stationary Gaussian process with zero mean and a covariance function estimated from the data. The covariance structure was captured by the product of a cosine and an exponentially decaying function. For a similar approach applicd to cerebellar Golgi cells, see (Szoboszlay el al. (2016).
This model allowed exact log- log-likelihood evaluations, and no noise was added.
Multisensory causal inference model. Perceptual causal inference involves determining whether mul- tiple sensory stimuli originate from common source, problem of particular interest in computational cog- nitive neuroscience (Körding et al. 2007). Our second real-world problem involved fitting a vustba causal inference model to empirical data from representative participant (S1 from Acerbi et al. 2018). In each trial of the modelled experiment, participants seated in moving chair reported whether they per- ceived their movement direction (svet) as congruent with an experieentalyy-maiipuuee field (Svis). The model assumes participants receive noisy sensory measurements, with vestibular informa- tion Zvest ~ (Svees, )i and visual information Zvis ~ [N ((s (c), where o and oi represent sensory noise variances. The visual coherence level c was experimentally manipulated across three levels (Now: Cmed, 9high). The model assumes participants judge the stimuli as having common cause when the absolute difference between sensory measurements falls below threshold к, with lapse rate À accounting for random responses. The model parameters 0 comprise the visual noise parameters ovia avis (Cme(C) ovis(Chigh) vestibular noise &vest, lapse rate X, and decision threshold ε ((Aerbi et al.) 2018).
looming visual
We fitted this model assuming log-likelihood measurement noise (a = =3, which we applied to each log- likelihood evaluation).
The results in Figure g and Table A.4 confirm our earlier findings of improvements across the posterior metrics. We also find that S-VBMC is robust to noisy targets for real data, with performance that improves with the increasing number of stacked runs in the multisensory model problem.
Results.
Similar to what we observed for the synthetic problems, we find that stacking =110 VBMC posteriors seems be sufficient to vastly improve posterior quality (as indexed by the GsKL, and MMTV metries), compared to
11

---

Published in Transactions on Machine Learning Research (11/2025)
single VBMC run (M 1); with relatively small additional improvements for M 10. The only exception to this is the GsKT, metric in the neuronal model, where the lower confidence intervals start to go below the desirable threshold (see Section 4.3) only if at least 18 posteriors are merged, and the full confidence interval never fully stabilises below such a threshold. However, this was our most challenging benchmark problem, as evidenced by the very poor perlormance obtained by both single-run VBMC and BBVI. Thus, while S-VBMC here does not achieve a posterior approximation fully on par with gold-standard MCMC (used to obtain the ground-truth posterior in this problem), the stacking process still vastly improves posterior quality at much lower computational cost.
Finally, similarly to the synthetic problems, we see that S-VBMC performs consistently better than standard VBMC and BBVI across all metrics in both scenarios. Despite similar improvement patterns with increasing values of M, as observed above, S-VBMC consistently outperforms NS, with starker differences in the noisy setting (f.e., the multisensory model).
Neuronal model (D 5, noiseless)
-7450
100
0.6
100
-7460
EL8O
7470
MMTV
0.4
GAKL
10
7480
0.2
-7490
0.1
7600
0.0
352
16
(b) Multisensory (D 6,
3)
0.20
1414
0.15
EL30
440
LMI.
MMTV
GoLL
0.05
450
0.00
1.4.
16
24
32
40
14
24
32
40
16
24
32
144 8
16
24
32
40
N. of mins
BBVI
NS
VBMC
S-VBMC
Figure 4: Real-world problems. Metrics plotted as a function of the number of VBMC runs stacked (median and 95% confidence interval). Metrics plotted for S-VBMC (bluc), VBMC (red), and NS (grey). The best BBVI results shown in green. The black horizontal line in the ELBO panels represents the ground- truth LML., while the dashed lines on ALML, MMTV, and GsKL. denote desirable thresholds for each metric (good performance is below the threshold; see Section 4.3). The BBVI error bar in the plot displaying the ELBO in the neuronal model (top left) is truncated for clarity.
4.6 Computational overhead
Here present details about the additional computational cost (quantified as compute time) introduced by S- S-VBMC lop of VBMC. Additional runtime analyses can be found in Appendix A.3.2
Figure Fiillustaats how S- VBMC introduces relatively small computational overhead, even when comparing the where the M VBMC runs happen vastly improves the resulting posterior, shown above and in Appendices A.4.2 and A.5 adds small amount of post-processing time to VBMC for all our benchmark problems (~ 5-15% overhcad).
cost of S-VBMC with the average cost of one VBMC run, under the idcalised condition parallel In particular, running our algorithm with M Rs 10 which
Put Logether, our results confirm that S-VBMC yields high returns in terms of inference performance at very marginal cost in terms of compute time.
In practice, completing M VBMC runs will be more expensive due additional parallelisation costs, making S- V-MC's relative overhead smaller than what report here.
12

---

Published in Transactions on Machine Learning Research (11/2025)
GMM (D = 2, noiseless)
Ring (D 2, noiseless)
Neuronal model (D = 5, noiseless)
Compute time (8)
150
Compute time (s)
Compute time (s)
200
750
100
500
100
50
250
4
16
24
32
40
1
4
16
24
32
40
16
24
32
40
N.
N. of
N. of runs
GMM (D = 2, 0==== 3)
Ring (D 2, o 3)
Multisensory (D = 6, 0
3)
Compute time (s)
Compute time (s)
Compute time (s)
600
2000
400
200
1000
200
0
4
16
24
32
40
1
4
16
24
32
40
1
8
16
24
32
40
N. of
N. of
N. of runs
VBMC
S-VBMC (post-process)
Figure 5: Compute time of single VBMC run (red) and post-processing time only (i.e., computational overhead) of S-VBMC (blue) plotted as a function of the number of VBMC runs stacked (median and 95% confidence interval, computed from 10000 bootstrap resamples). Each subplot represents a different benchmark problem. The values plotted here correspond to the actual computation times of the experiments described in Sections and 4.5
5 ELBO estimation bias
As briefly mentioned Section 4.4 in our results we observe that, in noisy settings, while the ELBO keeps increasing as more VBMC runs are stacked, the ALML error also increases, after an initial docrease (sco the second column of Figures and 4h. This apparently odd result can be explained by the presence of positive bias build-up in the estimated ELBO. This bias is visible in Figure 3 (b) and (d), first column, and Figure 4 (b), first column, in that the estimated ELBO from S-VBMC on these problems slightly "overshoots" the ground truth LML (the S-VBMC estimates, blue dots, end above the black horizontal line). As discussed in Section 2.2 the ELBO is always lower than the LML, with equality when the approximate posterior perfectly corresponds to the true postcrior. However, in these results, the estimated ELBO grows larger than the ground-truth LML. As this cannot be true, there must be a positive bias in the ELBO estimate. As one can see the aforementioned figures (first column), this bias builds up more and more VBMC runs stacked. What characterises these problems is the fact that they likelihood (as opposed to exact likelihood evaluations used in the other problems, where the estimate does not overshoot).
stochastic estimator for the
While this bias surprisingly does not affect other posterior quality metrics, which keep improving (or plateau) with increasing M, it might constitute an issue when using ELBOstacced for model comparison. In this section, we provide a simplified model for how the bias might statistically arise terms of the "winner's curse" and then provide an effective heuristic to counteract the bias.
5.1 Origin of bias
Here we analyse the ELBO overestimation observed in our results through a simplified example that illus- trates one potential mechanism for this bias. In short, we suggest this occurs because all In,k are noisy istims of the true expected log-joint contributions, causing S-VBMC to overweigh the most overestimated
13

---

Published in Transactions on Machine Learning Research (11/2025)
mixture components an effect that increases with the number of components M. While other factors may contribute, this analysis provides insight into why the bias tends to increase with the number of merged VBMC runs.
For the sake of argument, consider M VBMC runs that return idetical posteriors q (0) = ... = -4 (0): each with a single component. The stacked posterior takes the form:
M
(z(0)
Σ
Im Im (0).
(23)
m==
For each singlencomponet posterior, the expected log-joint approximated as
Im Eggm [log p (D)())p(()) se Ego... [fm (0)
(24)
where J' (0) is the surrogate log-joint from the m-th VBMC run. Since all posteriors share identical param- eters, their entropies are equal:
H l0b, (0)] H (162 (0)) H. 19o (0)].
(25)
The stacked posterior is thus mixture of identical components with different associated values In- The optimal mixture weights w depend solely the noisy estimates of Im:
Im Egg... [mm( Eng.. (D)))(
(26)
where 6n (O Jm) represents estimation noise with variance Jm Since all posteriors are identical and derived from the same data and model, differences in expected log-joint estimates arise purely from noise deriving from the Gaussian process surrogates fu.
Given that entropy remains constant under merging, in this scenario optimising ELBOstaccod reduces to selecting the posterior with the highest expected log-joint estimate. If we denote Imax max Im, the optimal ELBO becomes
ELBO stacked = imax + H 9 [gp(0)]
(27)
Since the true expected log-joint is identical across posteriors, the optimisation selects the most overestimated value. The magnitude of this overestimation increases with both M and the observation noise for fm: introducing positive bias in ELBO stacked that grows with the number of stacked runs and is more substantial for surrogates obtained from noisy log- likeliilloood observations.
While this simplified scenario does not capture the complexity of practical applications - where posteriors have multiple, non-overlapping components it illustrates a fundamental issue: if we model each f,,k as the sum of the true m,, and noise, the merging process will favour overestimated components, biasing the final ELBOstacked estimate upward.
This hypothesis is substantiated by our results, as we only observe noticeable bias in problems with noisy targets, where levels of noise in the VBMC estimation of m are non-negligible (note that VBMC outputs an estimate of such noise, see Section 2
5.2 Debiasing heuristic
We propose here heuristic approach to counteract the bias in the ELBO estimate.
First, as bascline we can estimate the per-component ground-truth expected log-joint Im.k, and the full expected log-joint summed over components, using Monte Carlo sampling on the true log-joint instead of the VBMC estimates (see Eqs. and []]. This estimate is unbiased and does not increase with M, as opposed the one estimated by S- VBMC, as shown for all noisy benchmarks in Figure 0 However, this requires numerous additional evaluations of the model's log- likelihooood, which is assumed to be expensive, it is not in general viable solution.
14

---

Published in Transactions on Machine Learning Research (11/2025)
One potentially effective heuristic to counteract the bias at least prevent it from increasing with M would be to cap its value post-hoc to some reasonable quantity. This heuristic has the advantage of being straightforward and inexpensive to implement, and, crucially, of not involving any tweaks to the ELBO optimisation process, thus not requiring any additional hyperparameters.
Expected log-joint (S-VRMC)
GMM,
Ring,
Multisensory,
-0.8
451.0
0.0
Expected log-joint
-1.0
Expected log--ooin
Expected log-joint
451.5
-02-
-12
-044
-4520
-14
-0.6
-1.6
452.5
-0.8
8
16
2.4
16
24
32
24
32
40
N. of runs
N. of
N. rf
Euuckll
Essc
Figure 6: Expected log-joint as estimated by S-VBMC (Estacked: blue) compared to that estimated via Monte Carlo sampling with numerous additional evaluations of the true log-joint (FMc, black, which we use as the ground truth) for our experiments with noisy targets. Dots represent the median value (of the 20 S-VBMC runs) and error bars 95% confidence intervals (computed from 10000 bootstrap resamples).
If we define the expected log-joint estimmted by S-VBMC
Estacked
Lm I
(28)
the 'capped' ELBO will be
ELBO (capped) Estacked + H (9 (g)(6))
(29)
stacked
where
Estacked min (Estacked, Ecap)
(30)
Note that, as mentioned in Section 3.1 each VBMC performs its parameter transformation Im. (-) during inference (Acerbi 2020). which prevents meaningful comparisons of the expected log-joint and its components across runs. Thhoouhhuut this section, we refer to and use the corrected VBMC estimates lm,k as expressed in the common parameter space of 0. More dotails on this correction can be found in Appendix(A22 A.2 In what follows, we discuss two candidates for Ecap.
Capping with median expected log-joint (run-wise). As a possible candidate for Ecaps we considered the median of the VBMC estimates of the expected log-joint from individual runs. So, if
F
Wm,k Pink
(31)
is the VBMC estimate of the expected log-joint from the m-th individual run, then
Emedlan median Emm)
(32)
15m$M
15

---

Published in Transactions on Machine Learning Research (11/2025)
Capping with median expected log-joint (component-wise). As the value of Emedian might be unstable and heavily dependent on individual VBMC runs (especially for low values of M), we also consider the median of the expected log-joints with respect to all individual components,
meddian = median (fm,)). ISmS IERSKm
(33)
As each VBMC run typically has Km - 50, even with = 2 we would have a stacked posterior with 100 components, which should ensure more stability in the estimate.
Debiasing results. We applied both these corrections to all our experiments with noisy targets (described in Sections 4.4 and 4.50, with results shown in Figure 7 We observe that both solutions are effective in preventing the ELBO bias buildup for three benchmark problems. The bias itself still present for the ring and multisensory targets, but it remains roughly constant with increasing values of M, and in both cases is very limited. We also observe that using median Feep yields the most stable solution, with debiased ELBO values fluctuating less, and confidence intervals being considerably less wide. While this is true for all three scenarios, it is particularly evident in the multisensory benchmark, where the ELBO capped with Emedian tends to fluctuate wildly, sometimes leading to ELBO overestimation, and sometimes to ELBO underestimation. In contrast, capping with mediian cnsures a reliably contained bias (< 0.5). These results suggest that the latter method more reliable across benchmarks and should therefore be preferred by practitioners.
ELBO debiasing (S- (S-V)
GMM, o=
Ring, ==
Multisensory, 0=
-443
35
25
-44..0
ELBO
ELBO
ELBO
3.0
20
444.5
2.
1.5
445.0
24
8
16
24
32
40
24
8
16
24
32
40
2.4
16
24
32
40
N. of runs
N. of runs
N. of runs
T
No capping
Eap Emsdier
Eap = Imoton
f
ELBOMC
Figure 7: Effects of debiasing heuristics noisy targets, with dots representing the median value and error bars 95% confidence intervals (computed from 10000 bootstrap resamples) across 20 S-VBMC runs. Each panel displays the uncorrected ELBO as output by S-MBMC (blue), the capped ELBO using Emedian (orange), the capped ELBO using Imediaa (purple), and the ground-truth F.BO obtained via Monte Carlo ELBOMC (black), for distinct benchmark problems. In each panel, the black dotted line is the ground-truth log marginal likelihood.
6 Discussion
The core problem we tackled in this work is that certain target posteriors might have properties (e.g.. multimodality, long tails, long and narrow shapes) that pose significant challenges to global surrogate-based approaches to Bayesian inference like VBMC. VBMC particular was developed to tackle problems with expensive likelihood functions, and thus relies on limited target evaluation budget and an active sampling strategy (Acerbi, 2019), which, while effective in many cases (Acerbil 2018 2020), makes it vulnerable leaving high-density probability regions unexplored (e.g., missing a mode). Therefore, attempting to build
16

---

Published in Transactions on Machine Learning Research (11/2025)
single, global surrogate approximation of complex targets might be cunter-productiivy Our proposed solution relies on the idea that, as an alternative, we can rely on collection of local surrogates to build a better, global posterior.
In this work, we introduced S-VBMC, a simple approach for merging independent VBMC runs in a princi- pled way to yield a global posterior approximation. We tested its effectiveness on synthetic problems that we expected VBMC to have trouble with, such as targets with multiple modes and long, narrow shapes, as well as on two real-world problems from computational neuroscience. We also probed S-VBMC's robustness to noise, by adding Gaussian noise to log- likelihood evaluations for some of our benchmark targets. This approach mimics realistic scenarios where the log- loglliklihood itself is not available, but can be estimated via simulation (Wood] 2010; van Opheusden et al. 2020 Järvenpää et al., 2021: Acerbi 2020). As expected, our results show that individual VBMC runs fail to yield good posterior approximations in our challenging prob- lems. Conversely, S-VBMC is remarkably effective across benchmarks, with all metrics steadily improving as more VBMC runs are merged, with minor differences between noisy and noiscless scttings. Importantly, we built S-VBMC to be robust to variability in the quality of individual VBMC posteriors. We initialise the weights w using each run's ELBO (Eq. [6), which immediately downweights low-ELBO runs, and the subsequent optimisation further reduces their influence. To verify this robustness, in all benchmarks we deliberately retained low-ELBO (but converged) runs and only excluded those flagged as non-converged by pyvbmc (Huggins et al. 2023 and poorly converged ones, with large uncertainty associated with the esti- mates Le (see Section 4.1. J. Despite this, the stacked posterior still improved as more runs were combined, suggesting that Low-ELBO VBMC posteriors are naturally assigned negligible weight.
S-VBMC inherits the limitations of VBMC and other surrogate-based inference methods, such applicability to relatively low-dimensional target posteriors (up to s 10 dimensions; see AAerbi 2018 2020 Järvenpää et al. 2021 While this fundamental constraint of the Gaussian process surrogtte remains, it is possible that, in moderately high dimensions (D 2 10 - 10), diverse initialisations might allow different runs to capture complementary regions of the posterior, yiclding incremental gains even when cach surrogate is only locally accurate. However, we do not expect S-VBMC to overcome the core scaling issues of the surrogate itself, and a careful empirical study of sigher-dimensional cases is left for future work. With that being said, S-VBMC effectively addresses some of VBMC's limitations, such as dealing with more complex postcrior shapes and multimodality.
Our results show that S-VBMC represents a practical and effective approach for performing approximate Bayesian inference when the target log-likelihood is expensive or the posterior distribution exhibits specific features. In scenarios where the likelihood is fast to evaluate and noiseless, and the posterior largely unimodal, traditional inference methods such as Markov Chain Monte Carlo (MCMC) are likely to outperform S-VBMC and should remain the default choice. While methods like MCMC represent a gold standard in most cases, it is worth noting that standard MCMC algorithms can struggle to efficiently explore complex posteriors with multiple, well-separated modes, for which there is no off- the-tf,-shelf,, easy solution. In contrast, S- S-VBMC's strategy of combining multiple independent local approximations, when paired with a diverse set of starting points, makes it better suited to explore challenging global structures. This suggests S-VBMC may be viable even in scenarios where the likelihood is not prohibitively expensive, but the posterior geometry is difficult for sequential samplers to navigate due to isolated modes.
Moreover, S- S-VBMC integrates seamlessly into the established best practices for its parent algorithm. For robustness and convergence diagnostics, performing several independent VBMC runs from different starting points is already recommended (Huggins et al. 2023). S-VBMC leverages this existing diagnostic workflow, providing a principled method to not only assess the inference landscape but to combine these runs into an improved, global posterior approximation at a small added computational cost. We intentionally designed S-VBMC as a simple post-processing step that requires no modification to the core VBMC algorithm. This simplicity is a key strength, as it preserves the embarrassingly parallel nature of rumning multiple indepen- dent inferences. While one could devise more sophisticated methods involving, for example, communication between runs to promote diversity and exploration, such approaches would sacrifice the case of implemen- tation and seamless integration that make S-VBMC practical tool for practitioners. As our results show, this approach is most impactful for problems with features like multiple modes ( (GMM target), long and nar-
17

---

Published in Transactions on Machine Learning Research (11/2025)
row shapes (ring target), heavy tails (neuronal model, see Appendix A.5 where individual runs explore different facets of the posterior.
To investigate the practical convenience of our approach, we measured and reported the computational overhead of S- G-VBCC. Our results, combined with our previous analyses, suggest that our approach yields considerable improvements in terms of posterior quality with a small added cost terms of compute time, assuming VBMC runs can be executed in parallel. In light of this, we recommend using M 10 as default choice, as it offers a strong accuracy-coss trade-off: most of the gains materialise by M = 10, which in our experiments added modest = 5 15% overhead cost, assuming the VBMC runs are executed in parallel. As mentioned carlier, our method is, in principle, applicable for stacking mixture posteriors produced by any inference scheme, not strictly limited to VBMC. However, without a closed-form solution for the expected joints of the individual components each run ([Im)==1 estimates of which are available in VBMC), Monte Carlo estimates are required, greatly inflating the number of necessary likelihood evaluations, and thus the computational cost. This wouldn't be the case for inexpensive likelihoods, but, as discussod above, VBMC would not necessarily be the primary recommended approach.
Finally, we discussed the main noticeable downside of S-VBMC, namely an observed bias in the ELBO estimation building up with increasing numbers of merged VBMC runs in problems with noisy targets, and proposed a simple heuristic (applicable in an inexpensive post-procsssing step) that mitigates this. Even though this bias buildup didn't seem to affect posterior quality in our experiments, it would constitute a problem one were to use the (inflated) ELBO estimates for model comparison. Our debiasing approach reduces this bias a smaller magnitude and prevents it from building up, so we encourage practitioners to adopt it (or some other debiasing technique) when using S-VBMC estimates of the ELBO for model comparison in problems with noisy likelihoods.
We should note that, although we discussed a plausible candidate source for the ELBO bias build-up in Section 5.1 this work does not contain a thorough investigation of this phenomenon and its causes. Some of the bias could come from the internals of the VBMC algorithm itself, which could explain the presence of a residual bias (see Section 5.2). Furthermore, our debiasing method simply consists of a post-processing heuristic, which is attractive for its simplicity and speed, but preserves the bias build-up mechanism during optimisation and inference. Precise identification of bias sources could allow their neutralisation at inference time, possibly leading the algorithm to shift its focus from overestimating the expected log-joint to opt rpliiiiiin an unbiased stacked ELBO. Nonetheless, an interesting finding is that in our benchmark problems the bias phenomenon did not affect posterior approximation quality as measured by our metrics, which kept improving steadily and approaching ground-truth posteriors while stacking more VBMC runs.
7 Conclusion
In this work, we introduced S-VBMC, a simple, novel approach for stacking variational posteriors generated by separate, possibly parallel, VBMC runs, and tested on a set of challenging targets. We further suggested an effective and inexpensive method to address one main drawback of our approach (f.e., the ELBO bias build-up). Our results, both in terms of performance and compute time, show its practical convenience for VBMC users, especially when tackling particularly challenging inference problems.
Acknowledgments
This work was supported by Research Council of Finland (grants 358980 and 356498). The authors wish to thank the Finnish Computing Competence Infrastructure (FCCI) for supporting this project with com- putational and data storage resources. The authors also acknowledge the research environment provided by ELLIS Institute Finland.
The model referred to as the "neuronal model" in the main text and appendices was developed by Dániel Terbe, Balázs Szabó and Szaboles Káli (HUN-REN Institute of Experimental Medicine, Budapest, Hungary), using data collectod by Miklós Szoboszlay in Zoltán Nusscr's laboratory (HUN-REN Institute of Experimental Medicine, Budapest, Hungary). The authors thank these researchers for making their data and model available for the work described in this paper.
18

---

Published in Transactions on Machine Learning Research (11/2025)
References
Luigi Acerbi. Variational Bayesian Monte Carlo. Advances in Neural Information Processing Systems, 31: 8222-8232, 2018.
Luigi Acerbi. An exploration of acquisition and mean functions in Variational Bayesian Monte Carlo. In Symposium Advances in Approrimate Boyesion Inference, pp. 1-10. PMLR, 2019.
Luigi Acerbi. Variational Bayesian Monte Carlo with noisy likelihoods. Advances in Newral Information Processing Systems, 03:8211-8222 2020.
Luigi Acerbi, Kalpana Dokka, Dora E Angelaki, and Wei Ji Ma. Bayesian comparison of explicit and implicit causal inference strategies in multisensory heading perception. PLoS Computational Biology, 170:1100, 2018.
Masaki Adachi, Satoshi Hayakawa, Martin Jørgensen, Harald Oberhauser, and Michael A Osborne. Fast Bayesian inference with batch Bayesian quadrature via kernel recombination. Advances in Neural Infor- mation Processing Systems, 85:16583-16544 2022.
David M Blei, Alp Kucukelbir, and Jon D McAuliffe. Variational inference: A review for statisticians. Journal of the Americon Statistical Association, 112(518))859-877 2017.
Steve Brooks, Andrew Gelman, Galin Jones, and Xiao-Li Meng. Handbook of Markov Chain Monte Carlo. CRC press, 2011.
Kenneth P Burnham and David R. Anderson. Model selection and multimodel inference: a practical information- tieerrettt approach. Springer Science & Business Media, 2003.
Trevor Campbell and Xinglong Li. Universal boosting variational inference. Advances in Neural Information Processing Systems, 32:3484-3495, 2019.
Bob Carpenter, Andrew Gelman, Matthew D Hoffman, Daniel Lee, Ben Goodrich, Michael Betancourt, Marcus Brubaker, Jiqiang Guo, Peter Li, and Allen Riddell. Stan: A probabilistic programming language. Journal of Statistical Software, 76:1-32, 2017.
Ryan SY Chan, Murray Pollock, Adam M Johansen, and Gareth O Roberts. Divide-and-conquer fusion. Journal of Machine Learning Research, 24(193):1-85 2023.
Daniel A De Souza, Diego Mesquita, Samuel Kaski, and Luigi Acerbi. Parallel MCMC without embarrassing failures. International Conference on Artificial Intelligence and Statistics, pp. 1786 180661804,11 2022.
Jonas El Gammal, Nils Schöneberg, Jesús Torrado, and Christian Fidler. Fast and robust. Bayesian inference using Gaussian processes with GPry. Journal of Cosmology and Astroparticle Physics, 2023(10):021, 2023.
Daniel Foreman-Mackey. Corner.py: Scatterplot matrices in Python. Journal of Open Source Software, 1 (2):24, 2016.
Roman Garnett. Bayesian optimization. Cambridge University Press, 2023.
Zoubin Ghahramani and Carl Rasmussen. Bayesian Monte Carlo. Advances in Neural Information Processing Systems, 15:505 512, 2002.
Tom Gunter, Michael A Osborne, Roman Garnett, Philipp Hennig, and Stephen J Roberts. Sampling for inference in probabilistic models with fast Bayesian quadrature. Advances in Neural Information Processing Systems, 27:2789-222 2779977777 2014.
Fangjian Guo, Xiangyu Wang, Kai Fan, Tamara Broderick, and David B Dunson. Boosting variational inference. arXiv preprinl arXix:611 1021161..05500 1.05559, 2016.
Michael Hines, Andrew Davison, and Eilif Muller. NEURON and Python. Frontiers Neuroinformatics, 3:391, 2009.
19

---

Published in Transactions on Machine Learning Research (11/2025)
Michael L Hines and Nicholas T Carnevale. The NEURON simulation environment. Neural Computation, 9 (6):1179-133 1209, 1997.
Bobby Huggins, Chengkun Li, Marlon Tobaben, Mikko Aarnos, and Luigi Acerbi. PyVBMC: Efficient Bayesian inference in Python. Journal of Open Source Software, 8(86):5428, 2023.
Marko Järvenpää and Jukka Corander. Approximate Bayesian inference from noisy likelihoods with Gaussian process emulated MCMC. Journal of Machine Learning Research, 25(366):1-555 2024.
Marko Järvenpää, Michael U Gutmann, Aki Vehtari, Pekka Marttinen, et al. Parallel Gaussian process surrogate Bayesian inference with noisy likelihood evaluations. Bayesian Analysis, 16(1):147-178, 2021.
Marc C Kennedy and Anthony O'Hagan. Bayesian calibration of computer models. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 63(3):425-464, 2001.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. Proceedings of the 3rd
International Conference on Learning Representations, 2014.
Diederik P Kingma and Max Welling, Auto-encoding variational Bayes. Proceedings of the 2nd International Conference on Learning Representations, 2013.
Konrad P Körding, Ulrik Beierholm, Wei Ji Ma, Steven Quartz, Joshua B Tenenbaum, and Ladan Shams. Causal inference in multisensory perception. PLoS One, 2(9))e943, 2007.
Chengkun Li, Grégoire Clarté, Martin Jørgensen, and Luigi Acerbi. Fast post-process Bayesian inference with variational sparse Bayesian quadrature. Statistics and Computing, 35(6):167, 2025.
Jiajun Liang, Qian Zhang, Wei Deng, Qifan Song, and Guang Lin. Bayesian federated learning with Hamil- tonian Monte Carlo: Algorithm and theory. Journal of Computational and Graphical Statistics, 34(2): 509-518, 2025.
David JC MacKay. Information theory, inference and learning algorithms. Cambridge University Press, 2003.
Andrew C Miller, Nicholas Foti, and Ryan P Adams. Variational boosting: Iteratively refining posterior approximations. In Proceedings of the 94th International Conference on Machine Learning, volume 70, pp. 2420 2429. PMLR, 2017.
Willie Neiswanger, Chong Wang, and Eric Xing. Asymptotically exact, embarrassingly parallel MCMC. Uncertainty in Artificial Intelligence - Proceedings of the 30th Conference, UAI 2014, 2014.
Christopher Nemeth and Chris Sherlock. Merging MCMC subposteriors through Caussian-proces approxi- mations. Bayesian Analysis, 13(2):507-55) 530, 2018.
Anthony O'Hagan. Bayes Hermite quadrature. Journal of Statistical Plonning and Inference, 29(3):245 260, 1991.
Michael Osborne, Roman Garnett, Zoubin Ghahramani, David Duvenaud, Stephen Roberts, and Carl Rasmussen. Active learning of model evidence using Bayesian quadrature. Advances in Neural Information Processing Systems, 25:46-55 54, 2012.
Leah F Price, Christopher Drovandi, Authony Lee, and David Nott. Bayesian synthetic likelihood. Journal of Computational and Graphical Statistics, 27(1):1-11, 2018.
Rajesh Ranganath, Sean Gerrish, and David Blei. Black box variational inference. In Arlificial Intelligence and Statistics, pp. 814-822. PMLR, 2014.
C. Rasmussen and C. K. I. Williams. Ganssian Processes for Machine Learning. MIT Press, 2006.
Jerome Sacks, Susannah Schiller, and William Welch. Designs for computer experiments. Technometrics, 31 31(1):11-47 1989.
20

---

Published in Transactions on Machine Learning Research (11/2025)
Steven L Scott, Alexander W Blocker, Fernando V Bonassi, Hugh A Chipman, Edward I George, and Robert E. McCulloch. Bayes and big data: The consensus Monte Carlo algorithm. In Big Data and Information Theory, pp. 8 18. Routledge, 2022.
Sanvesh Srivastava, Cheng Li, and David В Dunson. Scalable Bayes via barycenter in Wasserstein space. Journal Machine Learning Rescarch, 19(8):1-35, 2018.
Miklos Szoboszlay, Andrea Lórincz, Frederic Lanore, Koen Vervaeke, R Angus Silver, and Zoltan Nusser. Functional properties of dendritic gap junctions in cerebellar Golgi cells. Newron, 90(5):1048-1056, 2016.
Bas van Opheusden, Luigi Acerbi, and Wei Ji Ma. Unbiased and efficient log-likelihood estimation with inverse binomial sampling. PLoS Compulational Biology, =6(12):e008483,, 2020.
Hongqiao Wang and Jinglai Li. Adaptive Gaussian process approximation for Bayesian inference with expensive likelihood functions. Neural Compulation, 30(11):3072-300) 2018.
Xiangyu Wang and David Dunson. Parallelizing MCMC via Weierstrass sampler. ar preprinl orXiv:1312.4605, 2013.
Xiangyu Wang, Fangjian Guo, Katherine Heller, and David Dunson. Parallelizing MCMC with random partition trees. Advances in Neural Information Processing Systems, 28:451 459, 2015.
David H Wolpert. Stacked generalization. Neural Networks, 5(2):241 259, 1992.
Simon N Wood. Statistical inference for noisy nonlinear ecological dynamic systems. Nature, 466(7310): 1102 1104, 2010.
Yuling Yao, Aki Vehtari, Daniel Simpson, and Andrew Gelman. Using stacking to average Bayesian predictive distributions (with discussion). Boyesion Analysis, 13(3):917-1007, 2018.
Yuling Yao, Aki Vehtari, and Andrew Gelman. Stacking for non-mixing Bayesian computations: The curse and blessing of multimodal posteriors. Journal of Machine Learning Research, 23(79):1-45 2022.
21

---

Published in Transactions on Machine Learning Research (11/2025)
A Appendix
This appendix provides additional details and analyses to complement the main text, included in the following sections:
An overview of Variational Bayesian Monte Carlo,
A.1
A description of how S-VBMC handles VBMC's parameter transformations,
A.2
Additional experiments,
A.3
Full experimental results, A.4
Example posterior visualisations,
A.5
A.1 An overview of Variational Bayesian Monte Carlo
In this appendix we briefly describe Variational Bayesian Monte Carlo (VBMC). This is a simple overview of the various components the algorithm, and a full in-depth description of these beyond the scope of this appendix. For further details, see Acerbi (2018 2020
As mentioned in Section 2.3 VBMC addresses the problem of expensive likelihoods with black-box propertics by using a surrogate for the log-joint (see Eq. 5). Like many other surrogate- approaches (Garnetl, 2023), VBMC uses Gaussian process (GP) to approximate its expensive target (Rasmussen & Williams, 2006 GPs are stochastic processes such that any finite collection f( follows a multivariato normal distribution. A GP is fully specified by a mean function
m(x) = [[](
(A.1)
a covariance function (also called a kernel)
<(x,x') = cov cf [x(),, f(x)),
(A.2)
and an observation noise model or likelihood. In VBMC, as in most surrogate modelling approaches, the likelihood is assumed to be Gaussian, which affords closed-form GP posterior computations. Specifically, given & training set y, with X being the observed input locations, y the corresponding observed
function values, and S a diagonal covariance matrix representing observation noise, the posterior mean and covariance functions for a test. input location x are, respectively,
(x(()) = x(x, X (A X, X) + -y (y-))(yyymmm) -m()) + m(x)
(A.3)
and
(x, x)) = x x) x(x,, ()n + -))
(xx
(A.4)
For further details on GPs and their use in machine learning, see Rasmussen & Williams (2006)
Crucially, GP surrogate does not yield a usable posterior approximation, the integral of Eq. (Bayes' rule) remains intractable. Even if the integral can be solved, it does not yield usable approximation of the posterior, such as the ability to draw samples from it. To address this point, VBMC makes use of Bayesian
quadrature, a method for obtaining Bayesian estimates of intractable integrals (0 OO'Hagan 1991
Ghahramani
& Rasmussen 2002 Given an integral
J J- / f(x)T(x)dx
(A.5)
where f is the target function and π is a known probability distribution, if a GP prior is specified for f. the integral J Gaussian random variable with posterior mean
Ey[J] = / (x(x)
(A.6)
22

---

Published in Transactions on Machine Learning Research (11/2025)
and variance
V / I xx) (x)
(A.7)
If f has a Gaussian kernel and x(x is a mixture of Gaussians which is the case for the VBMC approximate posterior both integrals have closed-form solutions.
As mentioned in Section VBMC leverages Bayesian quadrature to estimate the ELBO by building a surrogate model of the log-joint f(0) x (og)())p(D()). In particular, the GP model f uses an exponentiated quadratic (more commonly known, if incorrectly, as squared exponential) kernel and a negative quadratic The latter ensures integrability of the function and is equivalent to an inductive bias towards Gaussian posteriors, but note that does not limit the modelled target be a Gaussian, as the GP can model arbitrary deviations from the mean function.
mean function (Acerbi 2018
2019
Therefore, putting everything logether, the posterior mean of the surrogate ELBO can be calculated as
By [LBOO Ey(Es[f()) + H [0)
(A.8)
where E, E& [[)]] is the posterior mean of the GP surrogate of the log-joint (É.e., the expected value of the expected log-joint), Here the expected log-joint takes the form
[+[ - (())
(A.9)
which, since 2 (0) is a mixture of Gaussians, affords closed-fomm solutions for its posterior mean and variance, as well as its gradients.
To build good surrogate approximation of the log-joint, a number of likelihood evaluations are needed to train the GP. Assuming the likelihood is expensive to compute, it is desirable to keep this number relatively contained. VBMC tackles this through an iterative process. In each iteration, VBMC selects the next points to evaluate by optimising an acquisition function that trades off exploration of new posterior regions and exploitation of high-density regions Acerbi 2019 2020). It then uses the resulting log-joint values to refine the GP surrogate, which is turn used refine the variational posterior via Bayesian quadrature. The VBMC algorithm begins with only two mixture components in a warm-up stage used to construct an initial surrogate model (see (see Acerbi 2018). After the initial warm-up iterations are concluded, VBMC starts adding new components to the mixture to refine the posterior approximation. This process of acquiring new points, using them to improve the surrogate model, and then refining the variational approximation (possibly increasing the number of components), is repeated until a convergence criterion is met or until the likelihood evaluation budget is exceeded.
23

---

Published in Transactions on Machine Learning Research (11/2025)
A.2 Change-of-vee -aange-of-variables variables corrections in S-VBMC
Problem setting. Due to the variational whitening feature introduced in Acerbi (202)), each VBMC run operates in its transformed parameter space, whereas in Acerbi (2018) all VBMC runs shared the same transformed parameter space (a fixed transform for bounded variables). An interested reader should refer to Acerbi (2020) for more information about variational whitening, and an in-depth discussion of this is beyond the scope of this appendix.
In the context of the m-th VBMC run, we call 9m (-) the function determining the parameter transformation, Im (0) the transformed parameters, and 4bm the parameters of the variational posterior expressed in the transformed space, Orhm (0)).
Crucially, VBMC returns approximate posteriors in the transformed space, and, since each run is executed independently, and has its own transformation (m,,,, the densities obtained with the different approximate posterior parameters qem.. (0)) are not directly comparable. To calculate the stacked ELBO, we need to operate the common (original) parameter space, in which the parameters are expressed.
Concretely, this means applying appropriate corrections the densities used evaluate the stacked ELBO, In the following, provide brief introduction to such corrections and discuss how they be applied to compute the entropy and the expected log-joint (i.e., the two terms of the stacked ELBO, see Eq. 11 in the common parameter space. Importantly, these corrections do not depend on the mixture weights of the stacked posterior, preserving the differentiability of our objective function with respect to w.
The Jacobian correction. Parameter transformations can be handled via the Jacobian correction. Given random variable æ, its probability Pa (x) and transformation T(x), we have
Pa Px(x) = Py (T(x)) det θT(x) p
(A.10)
dæ
where the correction is applied with the absolute value of the determinant of the Jacobian of T(x). Conve- niently, the pyvbmc software (Huggins et al., 2023) provides a function to evaluate the (log) absolute value of the determinant of the Jacobian of the inverse transformation (evaluated at (m)),), which, in this toy example, would mean
(T)))
Pa pa(x)
(A.11)
det (T(x) (T(
To tidy our notation, we will call our corrections term for the m-th transformation
Iyml (9mm (0))
(g) det
(A.12)
0gm ())
Bringing this back our case, the variational posterior of the m-th VBMC run can be written as and the stacked posterior as
Km Wm,k mm. (gm (0))_m((()0 (0)),
9 (0) =
(A.13)
M Km (0) = Σ Wm.k om. (gm (0)) (gm (0)).
(A.14)
In line with the notation used in the main text, om and & are the parameters of the m-th VBMC posterior and of the stacked posterior, respectively, expressed the common parameter space. With the exception of Wm and w (which are not affected by the transformation), these parameters are unknown, but, we will show in the following paragraphs, they not needed to estimate the stacked ELBO..
24

---

Published in Transactions on Machine Learning Research (11/2025)
The corrected entropy. One term of the stacked ELBO is the entropy
Hlgz] = Eg [log 9% (0)),
(A.15)
for which closed-form solution is available. We estimate it via Monte Carlo in Eq. 12 of the main tcxt, but crucially evaluate all component densities in the original space using the correction shown in Eqs. A.13 and A.14 Concretely, for each component (e,bb.. we draw S samples in the m-th transformed
space, mm,kk mm and map them to the original space, (*) = Im (2. Then, for every sample
(s)
X m, and for every component qk' Am!s we compute the per-component log- degdensity in the original space via
log q mm (x (xm,)) = log qus .mm (9m (x (xx - log (9m' (x (xx)
(A.16)
and then aggregale
Kmn Wm Je o (x
log q3 m,,k
(5)
log
(A.17)
via log-sun-exp. Then these values can be plugged into Eq. 12 of the main text to estimate the entropy. Importantly, the transformations do not depend on the mixture weights, so the entropy remains differentiable with respect to w.
The corrected expected log-joint. Let Lmm be the VBMC estimate (computed in the run's transformed coordinates) of the component-wise expected log-joint,
Lm Egn. [log Pme (D, Im
(D, (m (0))),
(A.18)
where Pm (D, 9m (..m())) (0)) is the log-joint reparametrisod with the m-th transform. To express this expectation in the common parameter space we apply the correction determined by (0)
m = Lmm,k E. [log Jm (gm (0)) s Eg (Dg)(D,))):
(A.19)
In practice we estimate the (per-component) Jacobian term Eq. A.19 using the same samples (e)
mmn te s-1 employed for the entropy and setting
1 s
Egg.
me.. wmm [log Jm (g (0)) =0
Σ log Jm (1
(A.20)
S am
After applying these corrections, and those the entropy described above, we can calculate the corrected stacked ELBO as in Eq. of of the main text.
Importantly, the im described here (f.e., the corrected ones) are the values we use in Section 5.2 for debiasing the stacked ELBO in noisy settings.
25

---

Published in Transactions on Machine Learning Research (11/2025)
A.3 Additional experiments
A.3.1 S-VBMC variant
To probe the benefits of optimising the ELBO with respect to the weights of the individual components, we performed additional experiments comparing the version of S-VBMC presented in the main text (*all- weights") with S-VBMC variant where we only reweigh the weights of cach individual VBMC posterior ("posterior-only"). Specifically, considered a stacked posterior written as:
93 (1(())
Σ
Wm 9or. (0).
(A.21)
For this "posterior-only" variant, we optimised the global ELBO with respect to the weights Wm assigned to each posterior. This is similar to the naive stacking approach seen in the main paper (Eq. 18]. with the difference that the posterior weights are now optimised. We ran this method for all the benchmark problems
described in Sections 4.4 and 4.5 using the same bootstrapping procodure described in Section 4.1
The results of this comparison are reported in Figures A.1 and A.2 and in further detail in Appendix A.4.2 We observe that optimising with respect to @ ("posterior-only") performs well, with both MMTV and GsKL motries stcadily improving with increased numbers of stacked postcriors. In fact, for most problems, "posterior-only" S-VBMC performs comparably to the "all-weights" variant presented in the main paper, which optimises all components weights w. Still, the "all-weights" variant performs slightly better in the GsKL metric and in some challenging scenarios (e.g., the multisensory model), so it remains our base recommendation, paired with the debiasing approach described in Section 5
A.3.2 Additional runtime analyses
Here we report the total runtime cost (in seconds) of BBVI for all our benchmark problems compared to that of running VBMC 40 times and stacking the resulting posteriors with S-VBMC. We consider this particular number of runs because the BBVT larget evaluation budget was set to match that of S-VBMC with M = 40. For both, we report the median and 95% confidence interval, computed from 10000 bootstrap resamples. For S-VBMC, each resample consisted of 40 VBMC runs and one S-VBMC run, then the S-VBMC runtime was added to that of the VBMC run with the highest runtime. This follows from the assumption that VBMC is run 40 times in parallel, and S-VBMC can be launched the moment. the last VBMC run has converged.
It is important to note that, is common in the surrogate-based literature (Acerbil 2018 Wang & Li 2018 Acerbi 2020: Järvenpää et al. 2021 El Gammal et al.) 2023 Järvenpää & Corander, 2024). in this work, we demonstrated the efficacy of our method on several problems where function evaluations are not computationally expensive, as a full benchmark with multiple expensive models is highly impractical. Therefore, wall-clock lime needs to be interpreted carefully as a metric when comparing methods with different likelihood evaluation costs.
We can directly compare VBMC and S-VBMC, as we did in Section 4.6 bocause by construction they use the same backbone method and have the same evaluation costs (S-VBMC adds small post-processing cost, which, crucially, does not depend on the cost of likelihood evaluation). Conversely, comparisons to VBM-VBMC methods become highly problem-dependent. The typical solution would consist of matching the number of function evaluations (as we did, see Section 4.2) for which non-surrogate-bar based baselines would be at a significant disadvantage, as demonstrated in previous work (Acerbi 2018 2020).
These considerations are crucial to interpret these results, displayed in Table A.1 As expected, BBVI is much faster than S- S-VBMC on problems with fast likelihood evaluation, but soon as the likelihood becomes more expensive (= 0.7 seconds per evaluation for the neuronal model) the cost of non-VBMC methods increases dramatically, illustrating the kind of scenarios VBMC was developed to solve in the first place.
Finally, it is worth noting that, as shown in Figures g and and Tables A.3 and A.4 BBVI performs substantially worse than S-CBMC across our examples, particularly in our rcal-world problems. Therefore, even where there may be runtime advantages (with the caveats discussed above), these come at a considerable cost in terms of posterior quality.
26

---

Published in Transactions on Machine Learning Research (11/2025)
(a) GMM (7) 2, noiseless)
3.0
0.2
11.00
2.8
4.M.
0.1
MMTV
0.1
2.6
0.01
0.1
0.01
2.4
0.001
0.001
0.0
0.000
16
24
32
40
16
24
32
40
16
24
322
40
224
32
40
N. of muns
(b) GMM (D 2,
3)
0.4
3.5
B1.30
A LML
MMTV
0.2
0.1
0.1
0.01
2.5
0.1
(c) Ring (D 2. noiseless)
0.6
2.00
FE.00
LML
MNTV
0.4
0.1
0.2
0.01
1.50
0.0
(d) Ring (D) 2,
E1.80
2.5
LML
MMTV
GVKL
0.2
2.0
0.1
0.01
1.5
0.0
16
24
32
24
32
40
16
24
32
4
24
32
N. muns
"pesterioe-onl"" S-VBMC
"all-weighs"" S-VBMC
Figure A.1: Performance comparison between the two versions of S-VBMC ("all-weights" and "posterior- only") on synthetic problems. Metrics plotted function of the number of VBMC ruMS stacked (median and 95% confidence interval, computed from 10000 bootstrap resamples) for S-VBMC when the ELBO is optimised with respect to "all-weights" (blue) and "posterior-only" weights (yellow). The black horizontal line in the ELBO panels represents the ground-truth LML, while the dashed lines on ALML, MMTV, and GsKL denote desirable thresholds for each metric (good performance is below the threshold; see Section
27

---

Published in Transactions on Machine Learning Research (11/2025)
(a) Neuronal model (D
5. noiseless)
- 7450
0.4
10
0.3
7452
ELBO
MMTV
GoKL
0.2
A
0.1
0.1
-7456
0.0
16
24
32
40
8
24
322
40
16
24
82
40
4
224
192
-00
N. runs
(b) Multisensory
3)
0.20
11.00
-444.0
LML
MMTV
0.10
Gakl
-444.5
0.05
runs
"posterox-oulf" S-YBMC
"all-weights" S-YBMC
Figure A.2: Performance comparison ween the two versions S- S-VBMC ("all-weights" and "posterior- only") on real-world problems. Metrics are plotted as function of the number of VBMC runs stacked (median and 95% confidence interval, computed from 10000 bootstrap resamples) for S-VBMC when the ELBO is optimised with respect to "all-weights" (blue) and "posterior-only" weights (yellow). The black horizontal line in the ELBO panels represents the ground-truth LML,, while the dashed lines on ALML, MMTV, and GsKL denote desirable thresholds for each metric (good performance is below the threshold; see Scction 4.3
Table A.1: BBVI runtime (in seconds) compared to that of 40 (parallel) VBMC runs and their subsequent stacking with S-VBMC. Values show median with 95% confidence interval in brackets. Bold entries indicate the best median performance (i.e., lowest compute time).
Algorithm
Benchmark
BBVI runtime
VBMC + S-VBMC runtime (s)
GMM (noiseless)
9.9 [9.1,10.3]
458.3 [128.3,510.8]
GMM (a = 3)
12.8 [12.2,13.7]
857.8 (759.0.954.8]
Ring (noiseless)
12.2 [11.7,12.6]
559.3 [516.9.744.2]
Ring (a =3)
14.0 [13.4,15.0]
1269.9 1126.0,1557.4
Neuronal model (noiseless)
8497.0 24112,,8617.8
1561.8 [1445.1,1900.
Multisensory model (a
24.5 [21.2.27.3]
3149.5 [2616.3,3525.1
28

---

Published in Transactions on Machine Learning Research (11/2025)
A.4 Full experimental results
A.4.1 Filtering procedure
Here we briefly present the results of our filtering procedure, described Section 4.1 As shown in Table A.2 VBMC had considerable trouble when run on the neuronal model, with over half the runs failing converge (as assessed by the pyvbmc software, Huggins al. 2023 suggosting complex, non-trivial postcrior structure which is reflected in the poor performance of other inference methods (see Figure and Table Convergence issues were also found with the noisy Ring target, although to lesser extent. Once non- converged runs were discarded, our second filtering criterion (i.e., excluding poorly converged runs with excessive uncertainty associated with the Is ostimates) led to considerably fewer exclusions overall, with only the Ring target. being somewhat affected (8% and 13% of runs discarded in noiseless and noisy settings, respectively).
All our VBMC runs were indexed, and, for experiments, we used the 100 filtered runs with the lowest indices.
Table A.2: Result of our filtering procedures. This table shows the total number of VBMC runs we performed ("Total"), those that did not converge ("Non-converged") and converged poorly ("Poorly converged") out of the total, and the ones that passed both filtering criteria ("Remaining").
VBMC runs
Benchmark
Total
Non-converged
Poorly converged
Remaining
GMM (noiseless)
120
3
115
GMM (a = 3)
150
145
Ring (noiscless)
120
108
Ring (a 3)
149
34
15
100
Neuronal model (noiseless)
300
159
140
Multisensory (a 3)
150
149
A.4.2 Posterior metrics
We present comprehensive comparison of S-VBMC against VBMC, NS and BBVI in Tables A.3 and A.4 complementing the visualisations in Figures 3 4 A.
and A.2 We consider both the version of S-VBMC described in the main text (where the ELBO is optimised with respect to the component weights w, "all- weights"), and the one described in Appendix A.3.1 (where the ELBO is optimised with respect to the posterior weights W, "posterior-only").
For both synthetic problems (Table A.3 and real-world problems (Table A.4 S-VBMC generally demon- strates consistently improved posterior approximation metrics compared to the baselines. However, we observe an increase in ALML error with larger numbers of stacked runs in problems with noisy targets. This increase likely stems from the accumulation of ELBO estimation bias, phenomenon analysed in detail in Section 5
29

---

Published in Transactions on Machine Learning Research (11/2025)
Table A.3: Comparison of S-VBMC, VBMC, and BBVI performance on synthetic benchmark problems. Values show median with 95% confidence intervals (computed from 10000 bootstrap resamples) in brackets. Bold entries indicate best median performance; multiple entries are bolded when confidence intervals overlap with the best median. For compactness, we label the S-VBMC version described in the main text "w..tt. w", (indicating that the ELBO is optimised with respect to "all-weights" w), and the version described in Appendix A.3.1 "w.r.t. "", (indicating that the ELBO is optimised with respect to W, or "posterior-only").
Benchmarks
GMM
Ring
Algorithm
ALML
MMTV
GsKT.
ALML
MMTV
GsKT.
Noiseless
BBVL, MaG (AC - 550)
0.059 [0035.0000]
0.0083
0.51 [0.0000))]
0.72 Ja
BBVI, MoG (K = 500)
0.053 0.020.0.11
0.052 00.445..07]
0,0087 (0.02550.003)
8.3 (6.9.22
0.47 (0.46.0499)
0.67 [0.56.0.8]]
VBMC
1.4 (1.7..44
0.54 [o.soc.
13 [100.4]]
1.2 [3.2.3.af]
0.53 (0.51,0.31)
9.4 [7.2.14
NS (10 runs)
0.091 (0.07.011]]
0.15 10.12.0.177 17]
0.051
00.004.0.070
0.16 (0.00.0.24 24]
0.19 (0.18,0:22)
00.02.0.051)
NS (20 runs)
0.047 0.007,0.009)
0.11
00.00.00-11]
0.027
(0.018.0.035)
0.11 p.077,0 1.
0.18 00166,0
0.028 (o.018.0.010)
S-VBMC (w.nee. 6; runs)
0.0042 [1.005700000r]]
0.032
0.0017
0.08
ju
0.36 [uts..u]]
0.0065
S-VBMC (wire.
runs)
0.0059 (000
0.031
(0.001.0..om
0.0011
(000006t.00000)
0.04
0.15
(0.14.015))
0.002
(..003...000a))
S-VIMMC (w.rt. w. 10 runs)
0.0089
0.036
0.0015
0.034
0.14
(1.44.1144)
0.0013
(1.00001 (
S-VBMC (W.F.t. W. 20 rume)
0.00446
(0.2028.0.007) 0002))
0.031
(o.028, o. 0.006
0.0013
jo 0.0009
0.022
0.019,20020)
0.14
(0.18.0.44)
0.0011
(0.0009..0.0014)
Noisy (o 3)
BBVL, MoG (K 50)
0.23 11.01.444
0.13 0.002.0.14
0.03 1.01.0.131
4.3 (3.3.4-7)
0.51 [0.47,0.54)
1.1 10,65
BBVI, McG (K 500)
0.27 [o.ossu.45]]
0.1 (0
0.019 (uu0)))
4.7 [4.5.5]
0.93 (0.01,084)
48 [as, cof
VBMC
0.98 [./8.1133
0.44 [0-43.0.44]
[8.5.13]
1.3 [t.1.1.5]
(0.57.0.65]
38 [24.0]]
NS (10 runs)
0.11 (0.005.013]]
0.17 10
0.066 [p.cas.0.331]
0.082 (1.006
0.23 (0.21,0216))
0.056 jounss,o.000
NS (20 runs)
0.056 (00068.0.0e0)
0.1 0.00.0.12]
0.017 (0.011
0.19 (0.10.0.23)
0.18 (0.28.0:2]
0.023 (0.00.0.000
V-VBMC (w.r.s
runs)
jo [0.16.004
0.13 14
0.012 (0.0000,0031]
0.23 0.11..028)
0.23 (0.31,0:24]
0.02 (0,01,0.000]
S-VBMC (w.r... @. 20 rums)
0.32 [0.28.0.4]
0.089
(o.078.0.30 0..00
0.0082 (0.004.0.015)
0.37 (0.54.0.44)
0.18 (0.17.0.19)
0.005-1 j0.004.20011]
S-VBMC (wirs. w. 10 rune)
(0.27.0.4))
0.11 00.003.0 13]
0.016 [0.000s.0004]]
0.39 (0.34,0 45)
0.2 jo 100091)
0:013
jo opeauc 025]
S-VBMC (w.rs. 20 runs)
0.53 [033.0..61]
0.09 (0.0s_j.00c]
0.0049
0.68 (0.33,0:71
0.17 (0.37.0.0))
0.0045
Table A.4: Comparison of S-VBMC, VBMC, and BBVI performance on neuronal and multisensory causal inference models. Bold entries indicate best median performance; multiple entries are bolded when confidence intervals overlap with the best median. See the caption of Table A.3 for further details.
Benchmarks
Multisensory model (a = 3)
Neuronal model
Algorithm
ALML
MMTV
GsKL.
ALML
MMTV
GsKL.
BBVI, MoG (K 50)
1.7 [1.3.4%]]
0.11 10.097.0.1]
0.17 [0.16.0.2)
44 [
0.6 10.06,0.60 64]
20 [17,2]]
BBVI, MoG (K 500)
1.8 [16.2.5]
0.31 30.28.0.30
0.53 [0.48.0.55]
170 [140.260]
0.67 [0.04.07]
21 [18.20]
VBMC
0.32 (0.23.0.77
0.18 10.17.0.19
0.21 [0.17.0.23]
3 [3.2.]]
0.32 0.33
140 (97.190)
NS (10 runs)
0.46 [0.41.0.52]
0.12 [0.11.0.1]]
0.072 [0.026.0.078]
1.8 [1.8.19]
0.17 [.11..011
1.2 [0.27.1.4]
NS (20 runs)
0.55 (0.5.0.90)
0.1 [0.006.0.11]
0.06 (o.082 0.068
[1.7.1.9]
0.17 [0.15.0.18]
[0.35,1.6
S- V-MC (w.r.. W. 10 runs)
0.73 [0.63.0.86)
0.11 [0.097.0.12]
0.062 (0.052.0.074]
[1.7.1.8]
0.14 [0.12.0.18]
0.67 [0.36.1.1]
S-VBMC (w..tt. W, 20 runs)
0.88 [0.96.0.95]
0.1 10.000.0.11]
0.047
[0.044.0.00
1.5 [L.5...]]
0.11 [0.087.0.13]
0.3 (0.337.0.57]
S-VBMC (w...t. W, 10 runs)
0.93 [0.e9.1]
0.091 [0.0660.094]
0.042
[0.038.0.05]
1.7 [1.5.1.7]
0.14 [0.11.0.10]
0.47 [0.17.0.79]
S- S-VBMC (w.r.t. W, 20 runs)
1.2 [L.21.3]]
0.079 p.076.0.051]
0.039 [0.03.0.044]
1.5 [L.5.1.G]
0.12 [0.092.0.13]
0.48 (0.059.0.54)
30

---

Published in Transactions on Machine Learning Research (11/2025)
A.5 Example posterior visualisations
We use corner plots (Foreman-Mackey 2016 to visualise exemplar posterior approxinations from different algorithms, including S- S-BMC, VBMC and BBVI. These plots depict one-dimensional marginal distributions and all pairwise two-dimensional marginals of the posterior samples. Example rosults (chosen at random among the runs reported Section | and Appendix A.4) are shown in Figures A.3 A.4 A.5 and A.6
VBMC consistently improves the posterior approximations over standard VBMC and generally outperforms BBVI, showing a closer alignment with the target posterior.
10.
02
82
0
02
01
0
(a) VBMC
(b) S-VBMC (20 runs)
10
10
O2
02
100
02
θ1
0
(c) VBMC (noisy)
(d) S-VBMC ruts, noisy)
31

---

Published in Transactions on Machine Learning Research (11/2025)
10
82
82
-0
10
0
02
0
0
(e) BBVI, MoG (K 50)
(l) BBVI, MoG (K 500)
10
10
Oz
02
01
02
01
02
(g) BBVI, MoG (K = 50), noisy
(h) BBVI, MoG (K = 500), noisy
Figure A.3: GMM (D = 2) example posterior visualisation. Orange contours and points represent postcrior
samples obtained from different algorithms, while the black contours and points represent ground truth samples.
32

---

Published in Transactions on Machine Learning Research (11/2025)
02
82
0
22
01
Oz
(a) VBMC
(b) S-VBMC (20 runs)
O2
02
6
θ1
02
θ
(c) VBMC (noisy)
(d) S-VBMC (20 runs, noisy)
33

---

Published in Transactions on Machine Learning Research (11/2025)
02
82
0
02
01
00
(e) BBVT, MoG (K = 50)
(f) BBVT, MoG (K = 500)
82
02
02
01
22
(g) BBVT, MoG 50), noisy
(h) BBVI, MoG (K 500), noisy
Figure A.4: Ring (D = 2) example posterior visualisation. See the caption of Figure A.3 for further details.
34

---

Published in Transactions on Machine Learning Research (11/2025)
83
25
8 COk
-44
845
e,
82
8y
(a) VBMC
83
324
Le-5
87
6
B
(b) S-VBMC (20 runs)
35

---

Published in Transactions on Machine Learning Research (11/2025)
25
63
24
84
0.04
9,
82
By
8
(c) BBVI, MoG (K = 50)
0
85
12
0.
02
03
6
85
(d) BBVI, MoG (K = 500)
Figure A.5: Neuronal model (D 5) example posterior visualisation. further details.
Sec the caption of Figure A.3 for
36

---

Published in Transactions on Machine Learning Research (11/2025)
87
+
5
87
8,
86
(a) VBMC
8
87
81
81
8.
86
(b) S-VBMC (20 runs)
37

---

Published in Transactions on Machine Learning Research (11/2025)
85
(c) BBVI, MoG (K = 50)
7
81
0.
0
8
(d) BBVI, MoG (K 500)
Figure A.6: Multisensory model (D a 3) example posterior visualisation. See the caption of Figure A.3 for further details.
38


Reviews:
[{"review_id": "v2wTqvxlkb", "reviewer": "Reviewer_ajye", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper addresses approximate Bayesian inference in settings where the likelihood is expensive to evaluate, making standard posterior inference methods infeasible. It builds on the variational Bayesian Monte Carlo (VBMC) framework, which uses a Gaussian process surrogate to model the log joint and estimates the ELBO with Bayesian quadrature. The main contribution is S-VBMC, an approach that combines multiple independent VBMC runs using a stacking procedure, yielding a mixture of mixtures posterior approximation. This design aims to improve robustness in capturing multimodality and long-tailed distributions, while remaining compatible with black-box likelihoods.\n\nStrengths\n\n* The idea of combining multiple VBMC runs through stacking is conceptually simple and easy to implement, requiring only an optimization of mixture weights over existing VBMC outputs.\n* The method is flexible and can capture multimodal and long-tailed posterior structures better than a single VBMC run.\n* The approach is naturally parallelizable: the bulk of the computation lies in independent VBMC runs, with the stacking step adding minimal overhead.\n* Experiments demonstrate competitive performance compared to BBVI, VBMC, and naive stacking baselines.\n\nWeaknesses:\n\n* The procedure to optimize the weights on the simplex is not well detailed.\n* The impact of the number of VBMC runs (posteriors) and the number of mixture components within each posterior is not well explored, leaving unclear intuition on how these choices affect performance.\n* Since VBMC runs are independant, there is a risk of redundancy if multiple runs converge to similar local approximations.\n* The empirical evaluation is mostly limited to low-dimensional settings; it remains unclear how well the method scales to higher-dimensional problems.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The claims are supported by accurate and convincing evidence, with experiments conducted in settings relevant to approximate Bayesian inference under expensive likelihoods. The results show clear improvements over baselines, and the paper also acknowledges and analyzes limitations, such as bias from over-estimation.", "audience_yn": "Yes", "audience_explanation": "The findings address a core challenge in approximate Bayesian inference with expensive likelihoods, a problem of broad interest to the TMLR audience. The proposed method is relevant to both methodological researchers and practitioners, as it improves posterior approximation while remaining practical and broadly applicable.", "requested_changes": "Several points could strengthen the paper:\n* While the naive stacking and \"posterior-only\" variants are useful ablations, it would also be valuable to compare against a baseline where VBMC is run directly with $\\sum_{m=1}^M K_m$ components, to better understand trade-offs in cost and performance.\n* Since the weight vector lies on the simplex, its optimization cannot rely on standard SGD. A more detailed explanation of the optimization procedure would help clarify this step.\n* The impact of both the number of posteriors and the number of components per posterior is not deeply discussed, yet this information would be important for practitioners. Related to this, it would be insightful to analyze how stacking handles poor-quality posteriors and whether the weight optimization effectively downweights them.\n* Finally, additional experiments in higher dimensions would be highly informative. As the authors acknowledge, dimensionality is a known limitation of VBMC. It would be interesting to see whether stacking can extend its applicability in this regard.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "a328KvvZre", "reviewer": "Reviewer_vbeX", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper considers a variational inference problem, where the likelihood evaluation is assumed to be expensive. \nAuthors considers the variational Bayesian Monte Carlo (VBMC) algorithm, wherein an posterior approximation by a Gaussian mixture is constructed by modelling the log joint using a Gaussian process, and estimating the ELBO with the Bayesian quadrature. \n\nThe main contribution of the paper is investigating an algorithm, called S(tacked)-VBMC, that combines a mixture of Gaussian mixtures from independent runs of VBMCs, in order to produce a better approximation to the posterior.\n\nStrengths: \n* The algorithm basically builds on top of any estimator of the log joint associated with a given variational family. The use of VBMC is supported by an existing software, and relevant to the blackbox setting. \n* The algorithm is simple to implement since it just requires an optimisation of the mixing weights that combine the learned Gaussian mixtures from the independent runs of VBMCs. \n\nWeaknesses: \n* As there is no communication between VBMC runs, it is unclear if one can obtain an improved estimate, since VBMC runs might end up exploring the same region, and produce similar variational approximations. Commenting on how to set up the runs would be useful.", "claims_and_evidence_yn": "Yes", "claims_explanation": "Overall, the paper shows the merit of performing the post processing of VBMC runs with through experiments. \n\nThe proposed algorithm is compared with BBVI by Ranganath et al, 2014, a standard benchmark, and a naive stacking where all Gaussian mixtures are given equal weights. \nIt is shown that while it is simple to implement, the proposed method outperforms these baselines in performance metrics such as ELBO. \n\nThe authors also acknowledge an issue with the proposed method: it tends to overestimate the ELBO (and the marginal likelihood) as the number of VMBC runs increases, when the evaluation of the log joint involves errors. \nThe authors propose a heuristic approach (capping the log joint estimate using an statistic of integral estimates) to counter this, and shows that it somewhat remedies this issue.", "audience_yn": "Yes", "audience_explanation": "The expensive likelihood setting is one of the challenging areas for applications (e.g., models with expensive simulators), and thus the work is aligned with TMLR's audience scope.", "requested_changes": "* The notation for the integral of the log joint and its estimator should be distinct, as it is confusing. For example, the hat notation is used in Eq. 24. Likewise, the RHS of Eq.8 should be given a symbol like $\\hat{I}_k$, since this is the $\\mathbf{I}_m$ vector that appears in Section 3 (the integral o the LHS is unavaiable). Other parts such as Eq. 26 should also be fixed, \n* Adam does not take into account the constraint on the weights (i.e., they form a probability vector). I would like a comment on this--how do the authors ensure this constraint? An easy fix would be to reparametrise the weights (see, e.g., [this paper](https://www.jmlr.org/papers/v26/23-1166.html)). \n* Having a sentence like \"we consider the following two candidates for $E_{\\mathrm{cap}}$ under Eq. 28 would help the transition. \n* It is unclear the computational cost of  a single run of VBMC is matched with S-VBMC -- can one achieve the same performance given the same computational cost with sufficiently flexible Gaussian mixtures? If this is challenging, it would highlight the point of using the proposed method.", "broader_impact_concerns": "N/A", "additional_comments": "The proposed algorithm does not take advantage of the variance of the integral estimate (I think this is the point using a Bayesian quadrature). Taking into account the uncertainty could perhaps be useful for countering the overestimation issue; for example, we could consider not giving too much of weights to those with high uncertainty.", "extra_scores": {}, "extra_text": {}}, {"review_id": "r9pJNG91Pm", "reviewer": "Reviewer_yRB2", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In many applied probabilistic models, the likelihood is difficult or expensive to evaluate, which makes the Bayesian posterior hard to obtain using the off-the-shelf methods. Variational Bayesian Monte Carlo (VBMC) provides an elegant solution: the unnormalized posterior is approximated by a surrogate Gaussian process (GP) in a similar way to Bayesian optimization, and then variational inference with mixture distributions is performed to approximate the surrogate model. This approximation is efficient with analytical computations with the GP. With VBMC, it is also possible to control the number of model evaluations to construct the GP model according to the budget.\n\nHowever, the approximation in VBMC is not perfect. In practice it may not cover the tails of long-tailed distributions, and it can miss areas in distributions with multiple modes. Bayesian stacking was invented to address these issues. This work proposes S-VBMC, which borrows ideas from Bayesian stacking to construct the approximate posterior with a mixture of imperfect posteriors with multiple VBMC runs. The pipeline has two phases: (1) M parallel VBMC instances are performed; (2) the mixing weights of the M posteriors are obtained using variational inference. \n\nIn the experiments, it is demonstrated that S-VBMC outperforms BBVI, VBMC and naive stacking on both synthesis and real data measured by multiple metrics. In addition, if phase (1) is performed in perfect parallel (which is achievable), the computing overhead from phase (2) is small compared to phase (1). The experiments also reveal a bias in the pipeline: as M is increased, the error to the true posterior does not goes to zero.\n\nThis work then explores reasons to explain the bias and ways to mitigate it. The intuition is that VBMC produces approximate models randomly, and stacking tends to select mixture components that overestimate ELBO. Then, capping techniques are introduced to correct the bias, which are found useful in the experiments.", "claims_and_evidence_yn": "Yes", "claims_explanation": "(1) With synthesis data, this work shows the problems of VBMC and how S-VBMC addresses the problems. With real data, there are also enough empirical evidences to compare the proposed methods with existing approaches. The work also experiments with the computation overhead, which is valuable to know about.\n\n(2) To demonstrate the problems of overestimation, the work uses a simplified example but the example is representative of common scenarios. I think the example is a clear theoretical evidence and motivates the solutions well.", "audience_yn": "Yes", "audience_explanation": "There are different assumptions in Bayesian computation. The proposed method is suitable for models with expensive likelihood evaluations. Applied researchers may find the method interesting when combined with their works. \n\nAnother interesting aspect is the parallel computation of Bayesian pipeline. Though not explicitly stated in the manuscript, this method has the potential of using large scale computing power. It is possible to develop better softwares that utilize modern hardwares with the method.\n\nThe negative result - the bias of stacking of random objectives, is particularly educational. Now people have been using approximate algorithms for intractable problems. The issues this work met may happen to other ideas. I think there may be interesting statistical problems in this space.\n\nNevertheless, the scope of the works is narrow. The work claims \"could theoretically extend to other variational approaches based on mixture posteriors\", but even if the claim is correct, a method that helps variational method with mixture posteriors is not very generalizable. Unless there are more discussions or experiments in this direction, I think this is the major limitation.", "requested_changes": "Throughout the work there is a variable $K_m$ that controls the number of mixture components per VBMC run. But the selection of this variable, and how it affects the results, are not discussed. Is there a reason to vary $K_m$ for different $m$? How are they selected in the experiments? I'd like to understand the questions better. Specifally in (12), it also assigns higher initial weights to posteriors with smaller $K_m$, which encourages worse settings. Probably it is a design choice, but there should be some discussion about why $K_m$ is in the denomenator.\n\nI'd like to see the hardware specification of the experiments. It would be particular interesting if there are GPU experiments that demonstrate the scalability of the methods.", "broader_impact_concerns": "I don't perceive potential broader impact concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "xCoBsdSLHS", "reviewer": "Reviewer_7953", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper is interested in the setting of approximate Bayesian inference under black-box likelihoods, where it is difficult or expensive to obtain gradients of the log posterior. The paper consider the idea of variational Bayesian Monte Carlo, which trains a surrogate for the log target via variational Bayes --  more specifically, they use the idea of \"stacking\" to construct a global surrogate model. Stacking merges independent VBMC inference runs into a global posterior approximation and uses a mixture approximation family. The posteriors are then reweighted to obtain the final posterior estimate; this is done by optimizing a global ELBO objective w.r.t. the weights. \n\nStrengths:\n* The posterior approximation is flexible, amenable to multimodality and long tails. The approach seems very sensible as a way to solve problems with the VBMC method. \n* The posterior results in comparison to the baselines seem to be more accurate.\n* The manuscript is fairly clear and pleasant to read. \n\nWeaknesses:\n* I don’t see a cost vs fidelity comparison of the S-VBMC method vs the other baselines — I think these might not be directly comparable, but it would still be useful to get a sense of the relative cost, in addition to Figure 5.\n* The experiments focus on lower-dimensional settings, e.g., D=1,2,5,6. While I think there are many interesting problems in this regime, I think other methods can also work well in this regime. It would be interesting to see some higher dimensional examples. \n* The fidelity of the stacked posterior depends on how well the posterior can be approximated locally in many different areas; it is perhaps unclear how many local procedures need to be run in order to get a good result.", "claims_and_evidence_yn": "Yes", "claims_explanation": "I think the paper satisfies the TMLR criteria in terms correctness of claims. I think the method takes an interesting approach (VBMC) and gives a solution to how to solve a problem with its posterior approximations by using stacking.", "audience_yn": "Yes", "audience_explanation": "This paper is of interest to many communities:\n* approximate Bayesian inference\n* simulation-based inference \n* variational inference\n\nThus, I think it is very relevant to a large subset of the TMLR community.", "requested_changes": "Minor changes:\n* I think it could be worth considering moving Figure 1 earlier -- I don't think it requires much background to understand, and it gets the point about the paper across very fast. \n* One clarifying point: it seems like the original VBMC \n* I would like to see more of a discussion of total cost vs other baselines besides a single VBMC run  \n* I'm not sure I fully understand the problem Section 5 is solving. Can this bias be expanded on in more depth, e.g., referencing specific figures and discussing the issues in more depth?\n* In practice, how should one choose the number of components and the number of posteriors?\n* It seems like the method is meant to focus on lower-dimensional posteriors, due to the dependence of VBMC on Gaussian process surrogates and Bayesian quadrature; perhaps this can be clarified and/or made more explicit earlier in the paper, e.g., in the introduction, especially for readers who might be unfamiliar with VBMC.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)