Revisiting Unbiased Implicit Variational Inference
Authors: Tobias Pielok
Bernd Bischl
David Rügamer
Venue: ICML 2025 poster (2025)
Keywords: semi-implicit variational inference; path gradient; importance sampling; conditional normalizing flows
Primary area: probabilistic_methods->variational_inference
TL;DR: We propose a competitive new SIVI method by identifying key shortcomings of the Unbiased Implicit Variational Inference method.

Abstract: Recent years have witnessed growing interest in semi-implicit variational inference (SIVI) methods due to their ability to rapidly generate samples from complex distributions. However, since the likelihood of these samples is non-trivial to estimate in high dimensions, current research focuses on finding effective SIVI training routines. 
Although unbiased implicit variational inference (UIVI) has largely been dismissed as imprecise and computationally prohibitive because of its inner MCMC loop, we revisit this method 
and show that UIVI's MCMC loop can be effectively replaced via importance sampling and the optimal proposal distribution can be learned stably by minimizing an expected forward Kullback–Leibler divergence without bias. Our refined approach demonstrates superior performance or parity with state-of-the-art methods on established SIVI benchmarks.

Revisiting Unbiased Implicit Variational Inference
Tobias Pielok" Bernd Bischl
David Rügamer 1.
Abstract
p(e)
gizlsa)
Recent years have witnessed growing interest in semi-implicit variational inference (SIVI) meth- ods due their ability to rapidly generate sam- ples from complex distributions. However, since the likelihood of these samples is non-trivial to estimate high dimensions, current research focuses on finding effective SIVI training rou- tines. Although unbiased implicit variational in- ference (UTVI) has largely been dismissed imprecise and computationally prohibitive be- cause of its inner MCMC loop, we revisit this method and show that UIVI's MCMC loop can effectively replaced via importance sampling and the optimal proposal distribution can be learned stably by minimizing an expected forward BratbackkLeiblee divergence without bias. Our refined approach demonstrates superior perfor- mance parity with state-of-the-art methods on established SIVI benchmarks.
1.75
1.50
2
122
EA
EB
1.00
gizlss)
0.75
0.50
0.25
0.00
-4
0.2
0.4
0.6
0.8
1.0
q(z)
4-
ZA
6
(idiaa) r(z|z>)
2
2
1
ZB
2
-2
1
-4
1. Introduction
0.00
Figure 1. We sample from a semi-implicit distribution (==) by sampling from the latent distribution p(e) and subsequently from the conditional distribution g(z(e). The simple distributions p(E) and (==(e) can induce complicated distribution g(z) and conse- quently potentially even more complicated reverse conditional distribution q(<|z), AISIVI learns coss-cooerinng representation ((E|z) of (())2) estimate Va log ((2) in high dimensions.
Bayesian inference, such as sampling-buss based or variational inference, is an important foundation for constructing uncer- tainty quantification measures for machine learning models. In variational inference (VI), samples are generated from a target distribution function Ps with the associated random variable z, which can only be evaluated but not directly sampled from and possibly unnormalized. This could be, e.g., Bayesian posterior distribution or the canonical dis- tribution w.r.t. a physical system. For this, family Q₂ over distributions with tractable sampling procedure is chosen, and a divergence measure D where D quantifies the dis- similarity between two distributions. The target distribution Ps can then be approximated by finding o E Q₂ which is closest to Pz w.r.t. D, i.e., 4x E arg ming EQ, D(qz,P2)
1.1. Implicit Variational Inference
In contrast to VI, where we assume that 9N E Q2 is an explicit distribution, i.c., we can evaluate gz: for implicit VI (IVI) we have no direct access to Ø and can only produce samples from (z, i.e., d= is an implicit distribution. Rep- resentative examples explicit and implicit distributions are normalizing flows (NFs) and neural samplers, which transform random variable via arbitrary neural network (NN), respectively. While NFs can be trained stably, they are known to smooth out sharp target distributions. In con- trast, neural samplers can model highly complex and sharp distributions but are notoriously hard to train. This naturally suggests combining them.
'Department of Statistics, LMU Munich, Munich, Germany "Munich Center for Machine Learning (MCML), Munich, Ger- many. Correspondence to: Tobias Pielok <tobias, cibbias,piclokeee @stat.uni muenchen.de>.
Proceedings of the nnnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Revisiting Unbiased Implicit Variational Inference
Semi-implicit variational inference (SIVI; Yin & Zhou, 2018) offers a compromise between VI and IVI. Since we sample from semi-implicit distribution dx by sampling the parameters y of an explicit distribution' Izly from an im- plicit distribution dy. its representative capabilities come close to those of an implicit distribution, but q of a semi- implicit distribution can estimated in a principle manner.
exist random variable v taking values in H C Rdd where
dæ E N and differentiable function g Y H 2 s.t.
6, ~ p,,m = -[S((())))))) n) ~ q=
(4)
mcha(e.))
where Pe,n is the joint distribution of the independent ran- dom variables e and 7 which does not depend on Φ. From this, it directly follows that
More formally, assuming that the target z Ps is a con- tinuous random variable taking values in Z Rd where dz E N, we approximate its probability density function via an uncountable mixture of densities s.t.
Exng, [ap())] = Ee Benn (as(ha(e, m))]
(5)
where as(x) Z + R is differentiable function with parameters Φb Hence, under our assumptions, the reparametrization trick can be applied to dz-
g (2) = Eyroy [ (2)()
For SIVI, the random variable taking values in Y S RR where dy E N is drawn via a neural sampler, i.e.,
2.2. Path gradient estimator and DKL minimization
We choose to minimize the reverse Kullback-Leibler diver- gence DKL, i.c.
Pe = y (c)
where e is a latent random variable taking values in E 3 Rd where dg and fø E is a NN with parame- ters & Rdd where de N. Consequently, since every ef defines yx the distribution family 2₂ is also parametrized by the NN parameters Φ. With Eq. 1 and Eq. 2, we also directly get that
g= 9=(2)) Ps
DKL(g=llPz) = Bzruq
log
(6)
On the one hand, one of the main advantages of DKL is that if we can evaluate Is we can compute unbiased estimates of the gradients w.r.t. the parameters Φ of 9z, which is especially useful when stochastic gradient descent methods are employed to minimize the objective. On the other hand, the reverse DKL is known to underestimate the variance the variational distribution ys is not sufficiently expressive (Andrade, 2024). However, for SIVI, this is rarely relevant, as the variational distribution dz is highly expressive due to its implicit nature.
(=(2) Repp [())((
(3)
where (z)e) = (=1u
1.2. Our Contributions
In this work, we focus on the efficient estimation of the score gradient V log 9z. which enables us to train SIVI models even in high dimensions. For this, we propose using importance sampling (IS) with an adaptively informed proposal distribution Tel= modeled by conditional normal- izing flow (CNF). We show that T<|= del= debiases our score gradient estimate and propose a stable training routine of the CNF via an expected forward Kullback-Leibh divergence. Our contribution advances both mathematical insights of SIVI and contributes two new algorithms.
Since (= is amenable to the reparametrization trick, we can follow Roeder et al. (2017) to formulate low-variance gradient estimator of DKL: the so-called path gradient esti- mator
V Ee Vz (log (1)g)=(>) logp=(>))
x=ho(E,n) (7)
vohhle.
While this result also appeared in the context of SIVI as an intermediate result in Titsias & Ruiz (2019), its far-reaching implications were not discussed since this expression was not of interest for the authors" final derivation (see Sec- tion 2.3). Not only does the path gradient estimator in Eq. 7 reduce the variance of the gradient estimation, but it also vastly reduces the computational demand in contrast the reparametrization trick, for which we would need to esti- mate the gradient
2. Background and Missed Opportunities
2.1. Reparametrizable Semi-implicit Distributions
In this work, we assume2 that the reparamctrization trick (Kingma & Welling, 2014) is applicable to 4x1y> y. i.e., there
"usually& common, unimodal distribution such as, e.g., nor- mal distribution
"e could even lessen assumption by only assuming that implicit reparametrization gradients he computed (Figurnov al., 2018), but this is not the focus this paper.
V a log qu (ho(e,
(8)
V log [Ezppe (holeh ()))
2

---

Revisiting Unbiased Implicit Variational Inference
This simple observation leads to surprisingly well- performing approach, which we will introduce in Section 3.
where Jr.- is the Jacobian of the inverse function of Tg- A conditional NF (CNF) is a differentiable map To : U x Z + U, (E, + To z) such that for every z E Z holds that Te (=(:,) is NF.
2.3. Unbiased Implicit Variational Inference
The problematic term of the path gradient estimator in Eq. is the score gradient Vz log 9- (z), for which no analytical expression exists. Titsias & Ruiz (2019) proved for UIVI
A plethora of different NFs have been proposed over the last years. In this work, we use affine coupling layers as in- troduced in RealNVP (Dinh et al., 2017) because sampling and evaluating their likelihood is equally computationally efficient, and they can be scaled up to high dimensions (An- drade, 2024). Specifically, we use a conditional variant of affine coupling layers similar one introduced in Lu & Huang (2020). While this is natural choice, other combi- nations could provide additional performance gains as also discussed in Section 6.
that
BENgE
V log 92 (=]e)] Vx (x).
(9)
i.e., if we can produce samples from the intractable con- ditional distribution Ge|z: we can compute an unbiased es- timate of the score gradient Vz log (z). Titsias & Ruiz (2019) propose to sample 2, E ~ 9z.e and use MCMC with target distribution -e|=- The MCMC chains are initialized at E because it already stems from the stationary distribution lel=- However, we can not use e directly since this would violate the independence assumption, which needed for an unbiased estimate in Eq. 9. Therefore, MCMC has long as the sample produced by the i-th chain ef is indepen- dent of e. Titsias& & Ruiz (2019) argue that only a few steps of MCMC are needed since the chains are already initialized at the stationary distribution. However, as it can be seen in Figure 1, Tel= is likely multimodal with regions of vanishing probability potentially occurring between the modes due to the implicit and possibly very complicated nature of 9.- In such cases, very long chains would be needed to effec- tively break the dependence between and <,, rendering the already computationally intensive method as prohibitive. Furthermore, note that the number of chains cannot reduce the bias introduced by the prevailing dependence between and e/- In light of these observations, we propose novel method in Section 3 to fix the encountered shortcomings.
3. Method
Starting from Eq. 7 we can rewrite the problematic score term, i.e.,
V logg Vx log [Eepp [g)))(e)]]
(11)
Thus, straightforward Monte Carlo (MC) estimator of the score gradient is
k < aateleles)
SMCx(2)) = Vz log
(12)
h
Lid. where (2,61) 61) 92,c and 6 ~ Pe, = 2.... k. This is a
consistent estimator of the score gradient Vz log Oz (z) and for large its bias
EccD, [VSSMC,(2)] - V log (9=(2)) ~
V Verp, [2]]c 2(k 1) 11-1)))-2 92 (z)3
(13)
V
(see Appendix A.1 for the proof). Note that including 6 introduces additional bias but strongly reduces the variance since 6 ~ Gel= A closely related estimator was derived in Molchanov al. (2019), but their estimator purely based the reparametrization trick and does not benefit from the advantages discussed in Section 2.2 and Section 3.2.
2.4. Conditional Normalizing Flows
Normalizing flows (NF; see, e.g., Papamakarios al., 2021) leverage the change of variable method to model complex distributions by repeatedly transforming random variable stemming from simple error distribution. More specifi- cally, for a random variable u taking values in U S Rdo where dy E N and differentiable and invertible transfor- mation To U - U with parameters A E ④ E R it holds that
Although we would expect that for high dimensions, the contribution of qz)e (2(c,) resulting from uninformed e; to nearly negligible to our estimator, 8MC,, performs sur- prisingly well.
Based on the previous observation, devise new impor- tance sampling (IS) version of Eq. 11, given as follows:
u ~ u 6 T=(uu) + 6~ Ge:
(10)
ge(c) = Pu( PT (c)) det (e)
Vg logaz
Pe (e(()gale((1e))
(14)
'We know 9-12 up normalizing constant, i.e., Zz,e, which suffices for MCMC
V log
E e i
==z
3

---

Revisiting Unbiased Implicit Variational Inference
The idea of enhancing SIVI with importance sampling was also proposed by Sobolev & Vetrov (2019), but their ap- proach is more expensive than ours due to the joint optimiza- tion of the proposal distribution and the SIVI model, ren- dering more expressive conditional models, such as CNFs, infeasible in practice.
This follows from the fact that
Ex~qa [DKL(Ae|zllTe
(20)
4e/= (E|z)
Ex Bergg log
(21)
Tel (E(x(E|2))
ez.e c)
=Ez,erga. log
(22)
(c|= (x)
Importance Sampling Estimator Based Eq. 14, can estimate Vx log (z) using the following score gradi- ent estimator
Dx, TT/ (=)
(23)
From this, assuming that Telx sufficiently flexible, di- rectly follows that at the global optimum Te|= of the expected forward DKT. it holds that
Pe(Es) (((((:e)
(1), = Vx log
(15) Z==
k
(e)
Uz.e
qz.c = Tdl= = [0]=
(24)
Being of particular importance for understanding our find- ing, we also include the proof of Proposition 3.1 in the following.
where E; Te 22 1....,
k. We show in Appendix A.2 that this estimator is consistent when supp(1e)=) C supp(Te(=). To also make this estimator efficient, we need to generate samples Telz and evaluate their likelihood ef- ficiently. A suitable option this case is to model Tel= with a sequence of conditional affine coupling layers (see Section 2.4).
3.1. Proof of Proposition 3.1
First note that
Pe(c,)
Pa (=(c))2(2)
(z(2)
Since we optimize Tel= and yz alternately, are interested in the optimal Tzlz for a fixed 9z This leads us to the fol- lowing proposition:
(25)
4el= (E:z() Pe(E:)asjc(((es) {z|e(z|E;)
With this, we get that
Proposition 3.1. Choosing Te|= 9c|= debiases our pro- posed score gradient estimate $15,k, i.e.,
k Pe(e: Pe((s))sse((2(s)
1
Beragel= V~ log
(26)
1:
(c/=(C](z)
1
k
Pe(e:)gs(c(((ee) 2ejE
h Li=1
pelei) V Vz0z/e (z|e))
Beiwgete Ve log
(27)
T:
())
(16)
Di=1 (ei (2) a= (2(e:)
Pe(e:)
i=1
Vz 4= (2).
Dimi Besrues (elx(e;13)
k
pele:) V # (2(c,
(28)
1/k: Ek= (=(z)
We prove Proposition 3.1 in Section 3.1. Hence, propose to learn Telz by minimizing the expected forward Kullback- Leibler divergence Ez~g= (xc]((9,]l for which we can estimate its gradient w.r.t. to the parameters 0 of the NE without bias since
Σ E Eerngglz
k
(=(()) (x(ee) (z(e(=(e,)
V log a21o (2(e;)
k. (2(2)
(29)
E Be Melja
(30)
k
Velle~gg [DKL(ec==lTTess)
(17)
(cl2)
Eq
V qz()..
(31)
= Ezngg Eengni Vo log
(18)
(E|2)
3.2. Training Under Memory Constraints
--R Vo lologra (E)2)
(19)
One of the main advantages of our proposed score gradi- ent estimators 8MC. and 818. is that increasing k; i.e., the number of samples e;, does not increase the computational cost backpropagation w.r.t. the parameters of our SIVI model & because we follow the path gradient. This insight motivates the following procedure, which allows us train
which holds because az.e does not depend on The follow- ing proposition assures the validity of our procedure:
Proposition 3.2. Minimizing Ezng. (9LL(ee=l is equivalent to minimizing (x,( (z).

---

Revisiting Unbiased Implicit Variational Inference
our SIVI models with constant memory requirements inde- pendent of k.
Algorithm 1 BSIVI
Input: target density Pz, batch size m, number of latent samples k with k > m, SIVI model his
First, note that both our score gradient estimators can be written s.t.
,.... m, j= 1......, k
repeat
Cj Pe. Mi Pn
8(2) = (2)
with
(32)
2; h n:)
8; = Vz, 2: logsumexp
(flog (xle z=l...k)
((2, 2) = =log
w(a w(e [(ee)
(33)
q: - st to..gradient(s)) 21
loss 1/m Σ (4:
log (2:))
opt (loss, Φ)
where choosing w<6) = 1 or (<e
[<][c]]]] results in SMC.A and &18,s, respectively. Since evaluating ( computationally non -ntenntee because of the inner neural sampler, we could, in principle, process very large e batches. However, since our memory is constrained, we need way to aggregate score gradient estimators computed on different e batches.
pe(e.)
until has converged
3.3.1. BSIVI
As new baseline method, we propose base SIVI (BSIVI), which minimizes the reverse Kullback-eibler divergence (x=(I=lIp=) by following the path gradient of Eq. 7. For the score gradient V log 9s we plug-in *MC,((z), This method exploits the fact that we can rapidly sample from SIVI model, and 8MC, can be computed with constant memory independent of i discussed in Section 3.2. The a algorithm is summarized in Algorithm 1. We use BSIVI to ablate the use of importance sampling, which our main method is built upon.
Efficient Aggregation on Batch Level Assume we have computed the score gradient estimates $1, $2 with associated log probability density estimates f,, of the e, batches of sizes j b and b, respectively, with j, b E N. Then, we show in Appendix A.3 that if we aggregate these estimates s.t.
f3(2, 2) = logaddexp (f1(2, 2) + log,, (34)
log(í + 1),
3.3.2. AISIVI
Furthermore, we propose adaptively informed SIVI (AISIVI), which alternates between minimizing the ex- pected forward KL divergence Ezwga [KL [<(=) and the reverse KL divergence DKI. (1=)))>>)))))))))))(()))) by following the path gradient of Eq. 7. For the score gradient Vz log 4x. we plug-in $18 & (z). which uses Tels as the proposal distri- bution. This alternating training is possible since $1s, (x) is a consistent estimator of the score gradient for any Telx with supp(ge;=) C supp(Te(=). Since the forward DKL mass covering, can expect that the support assumption is al always fulfilled. This means, in contrast to UIVI, we do not need exact* samples from le|z and the bias and variance of our estimate decreases" with increasing k. Also, sampling from the CNF Tel= is comparatively cheap, and the samples are guaranteed to be independent.
and
s3(z) = 0151 (2) + (2) with
j j+1, ) (35)
j
a1 = exp (z, - l2(22 2) + log
a2 = exp (l)(z, (z(2,
log(i + 1))
then 83 and la are the corresponding estimates of the com- bined E batches.
Also, note that we keep most of our operations in the log space to make the procedure numerically stable. For ex- ample, we use the Logaddexp(41) lows to numerically stable compute (og(exp((i) +exp(())))xx()))) and the logsumexp trick to compute & and &g themselves. Applying this algorithm iteratively allows us to process an arbitrarily large number of samples E; while keeping the memory requirement constant. As direct consequence, we note that our score gradient estimation is completely parallelizable.
operation, which al-
4. Related Literature
Yin & Zhou (2018) propose to use semi-implicit distribu- tions for VI and train their models by sandwiching the ELBO. Titsias & Ruiz (2019) introduce another objective based on ELBO and derive an associated unbiased gradient
3.3. Algorithms
Hooweerr we can greatly reduce the hias the better we match
Following the previous findings, we propose two new algo- rithms for SIVI.
Le z with Telz
Thii is not the case for UTVT regarding the number of chains
5

---

Revisiting Unbiased Implicit Variational Inference
Algorithm 2 AISIVI
Truth
AISIVI
BSIVI
Input: target density Pz, batch size m, number of latent samples k, SIVI model he, CNF Tel=
Banana
1.,....,7?. - J = ,.....
repeat
-2.5
0.0
2.5
-2.5
0.0
2.5
-2.5
0.0
2.5
G Pe: 77 ~ Pn
Zi = h (ci; 97:) Ei=1 log (G)
X-shape
2.5
lossnow = -1
0.0
= opt (lOSSRows 0)
-2.5
-2.5 0.0
2.5
-2.5 0.0 2.5
-2.5 0.0
2.5
Gi Telx (1zi)
log w = Pe (ep))e)))))))) Teg (zi)
Multimodal
2.5
log Wi, = stop graddient(loo) wi,)) Wi.))
0.0
log (z) E4,) = log W + log (2
-2.5
8, =z, logsumexppppp (1 Te (2i (1 | j=1....)))
-2.50.0 2.5
-2.50.0 2.5
-2.50.0 2.5
Q: st stoopggaadeen (5:)
loss = 1 1/m - (%: log Po (2:))
Φ opt ploss, Φ)
Figure 2. Histograms based 100000 samples produced the true distribution, AISIVI, and BSIVI
until @ has converged
estimator, which, however, depends on expensive MCMC simulations. Sobolev & Vetrov (2019) also improved upon Yin Zhou (2018) by introducing an importance sam- pling distribution; however, using expressive models such CNFs remains infeasible for their approach. In recent years, new approaches based on different objectives have been proposed that seem to outperform methods based on the ELBO. Yu & Zhang (2023) propose minimizing the Fisher divergence, but their minimax formulation proves difficult to train compared to the standard minimization problems mentioned above.
function space (Sun al., 2019; Ma al., 2019; Pielok al., 2023). These approaches frequently incorporate implicit inference mechanisms within their frameworks.
Several approaches have improved variational inference by incorporating importance sampling. IWAE (Burda et al., 2016) introduces tighter bound through multiple eepporaance-weihteee samples, while NVI (Zimmermann et al., 2021) extends this idea using nested objectives learn better proposal distributions. Our work builds this line by integrating importance sampling into the SIVI framework to improve expressivity and stability.
Building upon Yu & Zhang (2023), Cheng al. (2024) use the kernel Stein discrepancy as the training objective, which turns the minimax problem into standard minimization problem. We will refer to their method as KSIVI. Lim & Johansen (2024) proposed Particle Semi-Implicit Variational Inference (PVI), which is a particle approximation of Faclidean-Wasserstenn Wasserstein gradient flow. Both Cheng et al. (2024) and Lim & Johansen (2024) showed strong empirical evidence supporting their methods.
5. Experiments
In the following, we analyze the performance of our pro- posed methods AISIVI and BSIVI under different data narios. We start by comparing our two methods well- known toy examples that serve as a first sanity check (Sec- tion 5.1). We then compare our methods with the state-of- the-art methods KSIVI and PVI on 22-dimensional prob- lem in the context (Section 5.2, which serves as another common benchmark example for SIVI. Finally, we move to a 100-dimensional problem related to conditioned diffusion process (Sec- tion 5.3). We implemented AISIVI and BSIVI in PyTorch (Paszke et al., 2019). All experiments are performed on a Linux-based server A5000 server with 2 GPUs, 24GB VRAM, and Intel Xeon Gold 5315Y processor with 3.20 GHz.
While beyond the scope of this work, we note that SIVI has been successfully extended to multilayer architectures, yielding improved performance as demonstrated by Yu al. (2023).
Bayesian logistic regression model
Beyond SIVI, another line of research explores variational inference with fully implicit distributions (Mescheder et al., 2017; Shi et al., 2018; Feng et al., 2017). These methods often encounter training challenges, such as instability in- troduced by adversarial learning or rensity-ratio estimation.
Another related direction performs inference directly in
6

---

Revisiting Unbiased Implicit Variational Inference
Table DKL (p. q) of different toy examples (rows) using the two proposed methods (columns).
Table 2. KSIVI serves as the gold standard, with AISIVI reaching it in 10K iterations. The other SIVI variants are compared based on their estimated log marginal likelihood, given a comparable computational budget to AISIVI. The log marginal likelihood is estimated using 1000 high-quality SGLD samples, while each variant's s estimate is computed using 60,000 samples,
NAME
AISIVI (DRt.
BSIVI (Dki.)
BANANA
0.0853
0.3022
MULTIMODAL
0.0044
0.0017
X-SHAPE
0.0072
0.0034
МETHOD
1 LoG ML
TRAINING TIME [S]
ITERATIONS
KSIVI
74521
0.6K
100K
5.1. Toy examples
AISIVI
74062
1.4K
10K
IWHVI
67667
1.5K
10K
First, we train BSIVI and AISIVI on the three common two-dimensional test densities Banana, X-Shape, and Multi- modal proposed by Cheng al. (2024). Their respective definitions can be found in Table 3 in the Appendix B. For both methods, we use the same NN architecture and train them for 4000 iterations. For the NF of AISIVI, we use conditional affine coupling layers.
BSIVI
60556
1.5K
10K
PVI
53121
1.4K
10K
UIVI
40207
1.5K
10K
full batch for the score gradient computation of the target density.
Results It can be seen in Figure 2 that AISIVI and BSIVI can capture the three densities nearly equally well. Only for the Banana benchmark, AISIVI outperforms BSIVI notably (Table 1).
Results The marginal and pairwise density estimates in Figure 3 highlight that all methods perform nearly equally well since no systematic over- or underestimation of the variance can be observed. We also compare with the ground truth all pairwise correlation coefficients of given by
5.2. Bayesian Logistic Regression
cov (3 B()))
Next, we perform Bayesian logistic regression the WAVEFORM dataset as proposed by Yin & Zhou (2018). For the target variables yı E {0, (0,11,i= -- I,..., N with N = 400 and the feature vectors H; E IR2,, the log- ligeiiili is given by
Pr.
, + J. (36)
/ovv (10), Cav (BU)B(b)
where B()) is a vector containing the i-th coordinate of all B samples.
The scatter plot in Figure 4 provides visual summary of the correlation coefficients and the relation between those of different IVI methods and the ones of SGLD as considered ground truth. The results illustrate that PVI and KSIVI exhibit a slightly reduced spread compared to our proposed methods, indicating a marginally better fit. However, overall, the performance of all methods remains comparable.
log pgg(......N @ 21,,..,,,33))
Σ v11,, - log (1 + exp (1,zT))8))
exp
where G E 22 is the variable want to infer. We set the prior distribution of , to a normal distribution, i.e., p(B) -T) with a 0.01. In line with Cheng et al. (2024), we estimate the ground truth by simulating par- allel stochastic gradient Langevin dynamics (SGLD Welling & Teh, 2011) for 400,000 iterations, 1000 samples, and step size of 0.0001. We use the same NN architecture for all methods and use the best hyperparameters for PVI and KSIVI proposed by the respective authors for this bench- mark. We train AISIVI and BSIVI for 10,000 iterations and use e, batch sizes of 9182 and 91, 820 respectively. The large batch size of BSIVI is possible and computationally feasible because of the considerations discussed in Sec- tion 3.2. All methods use batch size m 128 the latent dimension is set to 10, i.e., E 10 For the NF of AISIVI, we use 16 conditional affine coupling layers. We use the
'T
5.3. Conditioned Diffusson Process
We adopt the Bayesian inference setting proposed in Cheng et al. (2024), which is based on the Langevin stochastic differential equation (SDE):
dæ, 10z, (1 - x2) + dw 0 Ost< 1,
(37)
where xg = 0 and we is a one-dimensional standard Brown- ian motion. This SDE models the motion of particle in an energy potential with Brownian fluctuations (Detommaso et al., 2018).
Following (Cheng et al., 2024), we discretize the SDE using the
scheme with step size At = 0.01, yielding a 100-dimensional latent variable
httpps: httpss////r/eveeeeeeee ics. wtt..uci.eddm////// machi ne lea wewaennnndaaaaaaeeeeee.. htt.eee//waeoormmm
x = (IAtt T2At: ",,
7

---

Revisiting Unbiased Implicit Variational Inference
KSPVI
ASM
BSM
Figure 3. Comparision of marginal and pairwise density estimates of ((a), Pray where the SGLD estimates are marked in black
KSIVI
PVI
gradient Langevin dynamics (SGLD) simulation with 1000 independent particles, a step size of 0.0001, and 100,000 iterations to generate 1000 ground truth samples.
0.50
0.25
Estimate
For this benchmark, also include IWHI ablate the ef- feet of their joint training approach compared to sequen- tial training. For their method, we use a conditional Gaus- sian model, where the conditional parameters are predicted by neural network, as their joint training setup makes more complex conditional models as continuous normal- izing flows (CNFs)-infeasee) against UIVI to compare our importance sampling-base based en- hancement with their original MCMC-based approach. For all methods, we use the same NN architecture. For KSIVI, we use the hyperparameters proposed by the authors for this benchmark. For PVI, we use 100 particles. To ensure fair comparison, we fixed the outer batch size (number of sampled z) for all SIVI methods and adjusted the inner batch size (number of sampled e) until we achieved approx- imately the same iterations per second as AISIVI. The e; batch sizes for AISIVI, BSIVI, and IWHI are 256, 40960, and 7000, respectively. The latent dimension is 100 for all SIVI variants. For the NF of AISIVI, we use 32 conditional affine coupling layers.
0.00
0.25
-050
AISIVI
BSIVI
Additionally, we evaluate
0.50
0.25
Estimate
0.00
0.25
-0.50
005
0.0
0.5
SGLD
SGLD
Figure 4. Scatter plot of every pairwise correlation coefficient P.. between the estimates and SGLD.
which gives rise to the prior distribution Pprio((a)) The observations are perturbed 20 time points, given by
y (J5At, ... 3100sa))
Results The results of the experiment is depicted in Fig- ure 5. We observe that KSIVI and AISIVI are closest to SGLD while UIVI, PVI, and BSIVI tend to underestimate the variability of the process. In Table 2, we report the estimated log marginal likelihoods of the SIVI variants along with their associated training times. Notably, only method, AISIVI, approaches the performance of the state-of- oftthe--rttttttt KSIVI. While IWHI also performs well, it does not match AISIVI, highlighting the benefits of more expressive proposal model. For UIVI, we were limited to an inner batch size 2 due to computational constraints, which led to noticeably weaker performance. Nevertheless, this
where
Vok ~ (f) 1 1 20
(38)
with 0.1, defining the likelihood function p(y|=).
Given y, our goal is to infer the posterior
p(zly) Porior (z)))((p)))) )
(39)
To approximate the posterior, we reapply the approach in (Cheng al., 2024) by running long-run parallel stochastic
8

---

Revisiting Unbiased Implicit Variational Inference
SGLD
AISM
5.00
tree path
sumple patn
contedirnce interval
observation
-4.50
-475
-1.50
0.0
6.2
3.6
5.6
3.0
3.0
0.
0.
0.4
0.8
0.8
1.0
0.0
0.2
0.4
0.6
0.0
1.0
50
62
0.4
0.6
0.0
3.6
LIVI
0.00
-0.25
-1.50
-1.25
Figure 5. Approximations of KSIVI, PVI, IWHVI, UIVI, AISTVI, and BSIVI discretized conditioned diffusion process are shown. The red dots represent the observations, the magenta line the ground truth estimated via parallel SGLD, and the blue line the estimated posterior mean. The shaded region shows the 95 marginal posterior confidence interval at each discretization step
comparison shows that AISIVI successfully adapts UIVI's core ideas a way that makes them more computationally efficient and competitive.
methods, suitable combination could further notably en- hance performance.
Additionally, the proposed method does not inherently of- fer exploration capabilities, which may limit its ability model multi-modal distributions. However, note that always combine temperature annealing strategy (Rezende & Mohamed, 2015) with our approach, but a more princi- pled procedure would be desirable. While this limitation is common in related work, addressing it in future research could enhance the applicability of AISIVI.
6. Conclusion
In this paper, proposed a novel SIVI framework, AISIVI, which revitalizes the ELBO as the training objective. This is possible because the bias and variance the ELBO gradi- ents can be severely reduced by using importance sampling and the optimal proposal distribution can be stably learned with a CNF. We provided the respective efficient Monte Carlo gradient estimators. The numerical experiments sup- port the efficiency and effectiveness claim of AISIVI.
Impact Statement
This paper presents work whose goal is to advance the field of machine learning. There are many potential societal consequences of our work, none of which we feel must be specifically highlighted here.
In particular, our experiments on the high- nigh-dimensional diffu- sion example suggest that it can beneficial not to rely on a kernel method, which known to be scalable to very large dimensions. Our method thus represents an easy-to-use and scalable alternative current state-of-the-art SIVI methods with on-par performance.
References
Andrade, D. Stabilizing training of affine coupling lay- ers for high-dimensional variational inference. Machine Learning: Science and Technology, 5, 12 2024. doi: 10. 1088//63222213//aattt
Limitations and Outlook
This work marks an initial attempt to integrate the strengths of semi-implicit distributions and normalizing flows. How- ever, given the numerous normalizing flow frameworks, certain alternative combinations may lead to improved per- formance. Future research could explore these possibilities to identify more effective configurations. While our method shows on par performance with current state-of-the- SIVI
Burda, Y., Grosse, R. B., and Salakhutdinov, R. Impor- tance weighted autoencoders. In Bengio, Y. and LeCun, Y. (eds.), 4th International Conference on Learning Rep- resentations. ICLR 2016, San Juan, Puerto Rico. May 2-4, 2016, Conference Track Proceedings, 2016. URL

---

met/forune Revisiting Unbiased Implicit Variational Inference
http://arxiv. haaprg/abb//a 00519. 9.
gence, EAAI 2020, New York, NY, USA, February 7- 12, 2020, pp. 5005-5012. AAAI Press, 2020. doi: 10.1699/AAALV344449990 URL https:/ /ttps://doi... orq/10. 1609 20116669/aaaa aaa.aaaaaa.............
Cheng. Z., Yu, L., Xie. T.. Zhang, S., and Zhang, C. Kernel semi-implicit variational inference. In Forty- first International Conference on Machine Learning, 2024. URL htttps://opeereeeeeee id=w5oUs0thO1.
Ma, C., Li, Y., and Hemnaddz--obatt, J. M. Variational implicit processes. In Chaudhuri. K. and Salakhutdinov, R. (eds.), Proceedings of the 36th International Confer- ence on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 4222-4233, PMLR, 09- .mlr.
Detommaso, G., Cui, T., Spantini, A., Marzouk, Y., and Scheichl, R. A stein variational newton method. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, NIPS' 18, pp- 9187-9197, Red Hook, NY, USA, 2018. Curran Asso- ciates Inc.
15 Jun 2019. URL https:: press/ v / ma 1 9b html.
Mescheder, L., Nowozin, S., and Geiger, A. Adversarial variational bayes: unifying variational autoencoders and generative adversarial networks. In Proceedings of the 34th International Conference on Machine Learning Volume 70, 17, pp. 2391-24001 JoL.. 2017.
Dinh, L.,, Sohl-Dickstein, J., and Bengio, S. Density estimation using real NVP. In International Confer- ence on Learning Representations, 2017. URL https: //openreview... weteeeewwwweeeeeeeeeeee....eeeee forum? d=HkpbbH91
Molchanov, D., Kharitonov, V., Sobolev, A., and Vetrov, D. Doubly semi-implicit variational inference. In Chaud- huri, K. and Sugiyama, M. (eds.), Proceedings of the Twenty-Second International Conference Artificial In- telligence and Statistics, volume 89 of Proceedings of Machine Learning Research, pp. 2593-2602. PMLR, 16- 18 Apr 2019. URL https: http::://prroceee ppw.eedinceeegseeeeee mmlr. press/ vvv9//89/// h///mmlllccc www.cooovllllccmmmmmmmmmmmm 9a. html.
Feng, Y., Wang, D., and Liu, Q. Learning to draw samples with amortized stein variational gradient descent. In Eli- dan, G., Kersting, K., and Ihler, A. (eds.), Proceedings of the Thirty-Third Conference Uncertainty in Artificial Intelligence, UAI 2017, Sydney, Australia, August I 11-1,, 2017. AUAI Press, 2017. URL http: /auai.org// uai2017 hha20177pxxcceddingg http://ewddddnnaeppppppspeeeeeee /ptppxs/pppppeeppeeee pd..
Figurnov, M., Mohamed, S., and Mnih, A. Implicit reparameterizatioo gradients. In Bengio, S., Wallach, H., Larochelle, H., Grauman, K., Cesa-Bianchi, N., and Garnett, R. (eds.), Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018. URL https://proceedingg.. neurips. oc ht/ppppe__filesspaperr//////ee/p 92280666431100777711100022222229 pdf.
Owen, A. B. Monte Carlo theory, methods and examples. https: / /at//erttwweeeeeeeeeeeeeee/eeeee su. doma iasiin//meeee /mc// 2013.
Papamakarios, G., Nalisnick, E., Rezende, D. J., Mohamed, S., and Lakshminarayanan, B. Normalizing flows for probabilistic modeling and inference. J. Mach. Learn Res., 22(1). January 2021. ISSN 1532-4435.
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z.,, Gimelshein, N., Antiga, L., Desmaison, A., Kopf, A., Yang, E., DeVito, Z., Raison, M., Tejani, A., Chilamkurthy, S., Steiner, B., Fang, L., Bai, J., and Chintala, S. Pytorch: An imperative style, figh-peeformance deep learning library. In Advances in Neural Information Processing Systems 32, pp. 8024- 8035. Curran Associates, Inc., 2019.
Kingma, D. P. and Welling, M. Auto-encoding variational bayes. In Bengio, Y. and LeCun, Y. (eds.), 2nd Interna- tional Conference on Learning Representations, ICLR 2014, Bansf, AB, Canada, April 14-16, 2014, Conference Track Proceedings, 2014. URL http: /tap::///axxvv.../ arw.v.......///./....../... abs/1312 6114.
Pielok, T., Bischl, B., and Rügamer, D. Approximate bayesian inference with stein functional variational gra- dient descent. In The Eleventh International Confer- ence on Learning Representations, 2023. https: / //openreveww.ro net/ tereet/forumte wtt.//oormmddda// aMq.eYM4.
Lim, J. N. and Johansen, A. M. Particle semi-implicit variational inference. In The eigrty- Annual Conference on Neural Information Processing Systems, 2024. URL htttps://openreeiiee wwwp:nnttururrrr id=p3gMGKHMKM
Rezende, D. and Mohamed, S. Variational inference with normalizing flows. In Bach, F. and Blei, D. (eds.), Pro- ceedings of the 32nd International Conference on Ma- chine Learning, volume 37 of Proceedings of Machine Learning Research, pp. 1530,1538, Lille, France, 07- 09 Jul 2015. PMLR. URL ht //t::///rreeee //proceedingg.. mir mir.press/vSi wtt.eeeeee////eeeeeeeeee 5, html.
Lu, Y. and Huang, B. Structured output learning with con- ditional generative flows. In The Foorth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intel- ligence Conference, IAAI 2020, The Tenth AAAI Sym- posium on Educational Advances in Artificial Intelli-
rewendel
10

---

Revisiting Unbiased Implicit Variational Inference
Roeder, G., Wu, Y., and Duvenaud, D. K. Sticking the landing: Simple, lower-variance gradient estimators for variational inference. In Guyon, I., Luxburg, U. V., Bengio, S., Wallach, H., Fergus. R., Vishwanathan, S., and Garnett, R. (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL cc co/paper_fiiee / 1100@fffsafaal594aaff33343333333 Paper. pdf.
ence Learning Representations, 2023. URL https: / net. cett//oorunee
Yu, L., Xie, T., Zhu, Y., Yang, T., Zhang, X., and Zhang, C. Hierarchical semi-implicit variational inference with ap- plication diffusion model acceleration. In Proceedings of the 37th International Conference Neural Informa- tion Processing Systems, NIPS '23, Red Hook, NY, USA, 2023. Curran Associates Inc.
Zimmermann, H., Wu, H., Esmacili, B., and van de Meent, J. Nested variational inference. In Ranzato, M., Beygelzimer, A., Dauphin, Y. N., Liang, P., and Vaughan, J. W. (eds.), Advances in Neural Informa- tion Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021. NeurlPS 2021, December 6-14, 2021, virtual, pp. 20423- 20435, 2021. URL https://proceeding neurips. wttpppppe//////1//////pp /hash// Mba920008Baaee111100000000000 Abstract. html.
Shi, J., Sun, S., and Zhu, J. Kermel implicit variational infer- ence. In International Conference on Learning Represen- tations, 2018. URL http:::///ppeeeeiiee n
Sobolev, A. and Vetrov, D. P. Importance weighted hierarchical variational inference. In Wallach, H., Larochelle, H., Beygelzimer, A., d'Alché-Buc, F., Fox, E., and Garnett, R. (eds.). Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL https://proceedingg neurips.. oc cc/ htcpppppe_filllessppeer////////1
773766622201165388775546666666 Paper.. pdf.
Sun, S., Zhang, G., Shi, J., and Grosse, R. FUNCTIONAL VARIATIONAL BAYESIAN NEURAL NETWORKS.
In International Conference on Learning Representations, 2019. URL https: http:///wppeeeeeeeetetteeeee forum? id=rkxacs0gy7.
Titsias, M. K. and Ruiz, F. Unbiased implicit vari- ational inference. In Chaudhuri, K. and Sugiyama, M. (eds.), Proceedings of the Twenty-Second Interna- tional Conference on Artificial Intelligence and Statis- tics, volume 89 of Proceedings of Machine Learn- ing Research, pp. 167-176. PMLR, 16-18 Apr 2019. URL https http::///ooceedeeeeeeeeee mlr, .aarrprec press/v89/ titsiasl9a html.
Welling, M. and Teh, Y. W. Bayesian learning via stochastic gradient langevin dynamics. In International Confer- ence on Machine Learning. 2011. URL https: app. semanti wwcenntisschllo wwccccsrc......occcoocccc.. aepoolar.rrrro ortooooooroo/oooooooo/oo 21189993..
Yin, M. and Zhou, M. Semi-implicit variational in- ference. In Dy, J. and Krause, (eds.), Proceed- ings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learn ing Research, pp. 5660-5669. PMLR, 10-15 Jul 2018. URL https: : httpps://prrooeeeo http///oceednggsss//ee mlr press/v80/ yin1 8...t...m..... html.
Yu, L. and Zhang, C. Semi- cemi-implicit variational inference via score matching. In The Eleventh International Confer-

---

Revisiting Unbiased Implicit Variational Inference
A. Proofs
For the proofs, we assume that the objects of interest are sufficiently regular s.t. we can change the order of integration, summation, and differentiation.
A.I. 8MC. consistent estimator and its bias approximation
We approximate the bias of (x),((2) V=log (t Ei (z)c((le;)) by using the delta method. First we note for
large & that sMC,e(2) Re V= log
(z|e(2|(s)
With the second-order Taylor approximation around (=(z)
(=)
Beiope
k Eizz Uxle (2/e()
we get that
( E (xc g=(z))²
k
= log (1=(2)) I Ei=o (z/c(2/))) (x(x)
log
k - _ -=
(40)
(x(2)
2 (z)2
From this, it follows that
)
Venp, (1/e((
E Berxpe
log
Σ (
~ log (1x(2))
(41)
k
2(k 1) 9: (2)
Consequently, get for large that
E Reppp
V VMM)(((2)] - Vx log (0x(2)) Berope
[ [2 - Vx log (gx(2))
(42)
Venp, 19 (z|c)]
-v
(43)
2(k 1) 2x (2)
which, in general, is non-zero.
To prove the consistency of SMC,A, we observe since log is continuous function that
lim SMC, k
V
lim 1
Σ 9
n.s. Vx log (EE-~e [(=)e(=|e)]) V Vogog(ax1)) (1x(2))
(44)
i.e.
P lim SMC,
Vz V=log((=(2))) 1.
(45)
k
A.2. $18,6 is consistent estimator
To prove the consistency of $18,k, observe since log is continuous function that
pe(e)dsle((le:) (e))((
[ec (c(())=)((2(ea) Te)(e,(2)
lim $18, V log
lim 8
a.s V log
Ee
(46)
==zz
For valid proposal distribution, it must hold that Tel= must be non-zero where Pr. Uzle = is greater than zero (Owen, 2013). Consequently, the support of Tel= must also contain the support of qe|= = lese In this case
lim $1S. 60000
log (9=(2)). i.e., IP
lim $IS $0000
V log (1=(=))) =
(47)
12

---

Revisiting Unbiased Implicit Variational Inference
A.3. 83 gives the correct score gradient estimator regarding all E, samples Assume got e batch of size (j 1)- and have computed the following estimators
s1(2) V= 2)
with
(48)
===
j.b w((, (<<e|
(1 2) log
(49)
j.b
52(2) Vzl2 2)
with
(50)
===
(1+1)).6
(z, log
δ Σ (((e
(51)
i=j-b1
These estimates can be aggregated such that
fz E) logaddexp (61 x) + log (2,,z - log(i + 1),
(52)
( 1).b
log
Σ
w(E, ((6]2)0z//o ((e:EE)0sc((2les)
(53)
(j+1).b
s2(2) = a1)1(z) + (2)2(2z) with
(54)
0n exp (z( 2) + log
(55)
j+ 1 02 exp (f2(2, 2) fz(2. B) - log(j + 1))
(56)
(57)
For the score gradient estimate, it follows that
1
$3 = exp (l2(2, x))
j+1 exp exp (f1(2,
exp (k)((,zzz)) yzj+ +11 exp exp (62(2,
Vz j+
(58)
2-z
V exp (ls(2, 2)) H=z
(59)
exp (fz(2, E))
Velo(2,2) Z) E=z
(60)
B. Implementation Details
Table 3 summarizes the details for the toy example discussed in Section 5.1.
Table 3. Densities of the toy examples
NAME
DENSITY
PARAMETERS
BANANA
(mI, 22 + 1)T N(O.
5=
1
0.9
0.9
MULTIMODAL
2 ~ (z5 1I + O.5NN(21 12
(-2,0)
122
2
1.8
2
-1.8
X-SHAPE
2~ 0.5 O,E.) + 0.5N[22 0,E2)
Σ1
Σn
-.8
13


Reviews:
[{"review_id": "KDiHiMkoAk", "reviewer": "Reviewer_Ssns", "summary": "This paper revisits Unbiased Implicit Variational Inference (UIVI), which has been largely dismissed due to its computational cost and imprecision from the inner MCMC loop. The authors propose replacing MCMC with importance sampling. By minimizing the expected forward Kullback–Leibler divergence, they ensure an unbiased estimation of the score gradient, making SIVI more efficient in high-dimensional settings. Authors provide detailed derivations of their proposed methods with appropriate proofs and algorithms. Experimental results show that their approach outperforms or matches state-of-the-art SIVI methods, advancing both theoretical understanding and practical implementation of variational inference.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "1. **Experiment 1 Comparison**: We agree that no comparison is needed for Experiment 1, as it primarily serves as a sanity check rather than a competitive benchmark.        \n\n2. **Correlation in Experiment 2**: We computed all possible correlations separately for the true $\\beta$ and the estimated $\\beta$, considering each coordinate individually. A negative correlation, $\\rho_{ij}$, means that $\\beta_i$ and $\\beta_j$ (where $\\beta$ represents here for example the true $\\beta$ in this case) are negatively correlated. After obtaining these correlations for both the true and estimated $\\beta$, we matched them so that the best possible outcome would form a diagonal line, ensuring alignment between the learned and true $\\beta$. In Eq. 36, this can also be seen since the correlations are always computed with respect to a fixed parameter vector. We will adapt the manuscript to make this process clearer for the reader. Our results demonstrate the state-of-the-art performance among SIVI variants. We will clarify this point again in the paper and thank the reviewer for the comment.\n\n\n3. **Experiment 3 Evaluation**:  \n> “From the results and the figures, it seems like all methods are doing equally well.” \n\nIn Experiment 3, we included a [table](https://figshare.com/s/f45697639e8908df84dd?file=53348411) demonstrating that only our proposed method (AISIVI) performs on par with the state-of-the-art KSIVI method. Additionally, we have improved the [figure](https://figshare.com/s/f45697639e8908df84dd?file=53348420) to better highlight that AISIVI is the only method capturing variance as effectively as KSIVI. \n\n4. **Code Availability**: We have inquired with the area chair about the permissibility of providing the code currently hosted on (Anonymous) Github (but the rebuttal rules only allow figures and tables to be linked). \n\n5. **Notation and Equation Clarifications**: The use of subset notation (⊆) is indeed correct, given our definition of $\\mathcal{R}$. However, we recognize that this notation might lead to misunderstandings and will adapt it to a more standard form. We have also incorporated the suggested clarification for Eq. (2).\n\n6. **KL Divergence Normalization**: As far as we know, there is no standard way to normalize KL divergence. However, to provide better intuition, we have added a [figure](https://figshare.com/s/f45697639e8908df84dd?file=53348417) depicting different training states alongside their KL values, helping to contextualize the reported numbers.\n\n\nWe hope these clarifications address your concerns and demonstrate our commitment to improving the manuscript. Thank you again for your valuable feedback. Should the reviewer find the response satisfactory, we would appreciate reconsidering the initial score. Otherwise, we remain fully committed to addressing any remaining concerns during the second author response phase.", "claims_and_evidence": "Good.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "Glanced at the derivations but not into their details.", "experimental_designs_or_analyses": "* For experiment 1, there is no comparison with other methods. Maybe it is unnecessary since all methods might doing well on this.\n* For experiment 2 and 3, although there are comparisons with other methods, I didn't find a clear benefit of the proposed methods. From the results and the figures, it seems like all methods are doing equally well. Then what are the real benefits of the proposed methods?", "relation_to_broader_scientific_literature": "/", "essential_references_not_discussed": "/", "other_strengths_and_weaknesses": "* The method derivation in this paper is good and solid.\n* The visualization of the experiment part is clear.", "other_comments_or_suggestions": "* A lot of $\\in$ should be $\\subseteq$ in this paper.\n* Eq. (2) $f_\\phi(\\boldsymbol \\epsilon) \\sim q_{\\boldsymbol y}$ is a confusing equation. Should be $y = f_\\phi(\\boldsymbol \\epsilon)$.", "questions_for_authors": "* For experiment 2, authors plot the measured correlation of $\\beta$ in Fig. 3. What does a negative correlation mean? If the authors want to show that after matching with the true parameter of SGLD, the authors should plot the learned $\\beta$ and the true $\\beta$ after matching and showing that they form a diagonal line. But what authors plot is a pairwise correlation of the learned $\\beta$ and the true $\\beta$. If plotting correlation, I would hope to see more close to 1 correlations, after matching the learned with the true. Could authors explain more about this figure and what it implies. Thanks!\n* In table 1, a close to 0 KL divergence means a good approximated distribution. However, the overall magnitude of the KL divergence may vary, depending on the nature of the true distribution. Is there a way to normalize the KL divergence? Otherwise, people don't know whether a number like 0.8 is close to 0 or not good enough, which will not be helpful compared with a direct visual comparison.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sMPrTzEmdd", "reviewer": "Reviewer_dhXk", "summary": "Estimating the gradient of the KL divergence between SIVI models and (unnormalized) densities is the core difficulty for training SIVI models. Many efforts have been made to partially solve this problem using, e.g., MCMC, kernel methods, Monte Carlo sampling, etc.\nThis paper presents a new method for training semi-implicit variational inference (SIVI) models based on importance sampling (IS).\nThe most contribution of this paper is employing a learnable reverse model $\\tau(\\epsilon|z)$ to provide a variance-reduced estimate of the gradient.\nThe experiments show that the proposed method has a comparable inference accuracy as baseline methods.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thorough review and insightful feedback. We greatly appreciate your suggestions, and we have addressed the following points:\n\n1. **High-dimensional Applicability**: We argue that the definition of \"high-dimensional\" depends on the specific goal. Finding an adequate approximation is achievable in higher dimensions, but our target for Experiment 3 was the true distribution. As shown in the newly added [table](https://figshare.com/s/f45697639e8908df84dd?file=53348411) and improved [figure](https://figshare.com/s/f45697639e8908df84dd?file=53348420), only our method and KSIVI from all SIVI variants come close to matching this distribution. Thus, for this scenario, 100 dimensions can be considered sufficiently high, as no other SIVI variant can achieve such good results. We will clarify in the final manuscript what we mean by 'high-dimensional' in our setting to avoid any confusion.\n\n\n2. **Computational Cost**: We have now included UIVI in Experiment 3, and as you suggested, we’ve focused on the log marginal likelihood. It is clear that UIVI’s performance does not come close to AISIVI, even for a similar simulation budget. Note that we can only afford an inner batch size of 2 because of the expensive inner MCMC loop, which further emphasizes our point about drastically improved computational costs. Similar findings are reported in Appendix D of SEMI-IMPLICIT VARIATIONAL INFERENCE VIA SCORE MATCHING by Longlin Yu et al., which further supports our argument. Thank you for strengthening our paper with this suggestion.\n\n\n3. **Reparameterization Trick and IWHVI**: We have corrected our previous claim, and indeed, the reparameterization trick can be applied to IWHVI. We appreciate your clarification on this point.\n\n\n4. **Optimization Strategy**: The alternate optimization scheme is a key aspect of our method and one of the main contributions. By using the path gradient and alternating optimization, we avoid backpropagating through the proposal distribution. This allows us to process arbitrarily large inner batches, whereas a simultaneous optimization approach is limited by memory constraints. This design enables us to use a more computationally expensive proposal distribution (conditional normalizing flow), and as shown in our updated experiments, AISIVI outperforms IWHVI in terms of performance within a similar computational budget. This now serves as an ablation study comparing simultaneous and alternating training approaches. We have also included IWHVI as a strong baseline, as suggested.\n\n\n5. **Typo**: We have corrected the typo in Eq. 15, as you pointed out.\n\n\nThank you again for your valuable feedback and for helping to improve the clarity and strength of our paper. We hope these changes address your concerns effectively. Should the reviewer find the response satisfactory, we would appreciate reconsidering the initial score. Otherwise, we remain fully committed to addressing any remaining concerns during the second author response phase.", "claims_and_evidence": "There are two important claims that are not verified by the experiments.\n- **On the high-dimensional application**. Line 57-59 explicitly states that 'In this work, ... enable us to train SIVI models even in high dimensions'. Line 151 states that \"they can be scaled up to high dimensions\". However, the largest dimension of the numerical examples is 100, which is not so large and is well studied by baseline methods (KSIVI). The authors should provide more high-dimensional evidence (e.g., AISIVI performs well and baseline methods fail) to support their claim.\n- **On the computational cost**. The authors emphasized the computational drawback of UIVI in Section 2.3 and \"propose a novel method to fix the shortcomings\" (Line 124). However, no detailed computational cost comparison, or the efficiency/accuracy trade-off is reported, making their claim unconvincing.", "methods_and_evaluation_criteria": "Although using important sampling to estimate the gradient is a somewhat straightforward, the authors make theoretical analysis and careful designation which can be considered as novel contributions. However, there are some problematic point in the methodology:\n- Regarding the comparison against IWHVI (Soblev \\& Vetrov, 2019), the author claims IWHVI is not applicable to the reparameterization trick. However, reparameterization trick is possible for IWHVI, as both the SIVI model and the reverse model are reparametrizable. I would like to see more explanation on this point.\n- The authors use an alternate optimization strategy for the SIVI model and importance distribution. A more natural choice is to optimize these two models simultaneously, and this strategy has been applied in VAE and IWHVI. It would be better to explain why an alternate optimization is considered here.", "theoretical_claims": "I've checked the correctness of the theoretical claims.", "experimental_designs_or_analyses": "### **About the experimental designation**\n- **Missing high-dimensional application**. Please see 'claims and evidence.'\n- **Missing computational cost comparison**. Please see 'claims and evidence.'\n\n### **About the baselines**\nAs least the following two variants of SIVI should be considered:\n- UIVI. This paper \"revisits\" UIVI (see the title), by using important sampling to \"fix the encountered shortcomings of UIVI\" (Line 124). Therefore, UIVI should be considered as a very important baseline for the method.\n- IWHVI. This paper has the same idea of using importance sampling to estimate the ELBO in optimizing SIVI.\n\n### **About the metrics**\nFrom the figures, AISIVI achieves very indistinguishable results compared to the baseline methods. In such a case, quantitative comparison (e.g., KL divergence, Wasserstein distance, ELBO, marginal likelihood) would be necessary to compare different methods.", "relation_to_broader_scientific_literature": "Using importance sampling to improve the gradient estimation of SIVI is a meaningful effort, as this is a long-standing and difficult problem of SIVI.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "I have no other comments.", "other_comments_or_suggestions": "One typo: There should be $\\epsilon_i$ in the numerator of Eq. 15.", "questions_for_authors": "I have no other questions.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nGMZ9jzV3T", "reviewer": "Reviewer_yGYH", "summary": "This paper proposes a new method to reduce the bias of semi-implicit VI (SIVI). The key idea is  to estimate the problematic term $\\nabla_z \\log q(z) = \\nabla_z \\log E_{\\epsilon} [q(z|\\epsilon) ]$ using importance sampling, where the proposal distribution is a normalizing flow. The normalizing flow is learned to match the posterior distribution $q(\\epsilon|z)$. If learned perfectly, it completely debiases the estimator. The authors propose a neat way for how the IS can be summed efficiently for large batches. Several experiments on different inference settings have been performed where the papers that the proposed method can compete with SOTA inference methods.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful feedback and valuable suggestions. We have made several improvements to address your concerns:\n\n1. **Improved Figures and Tables**: We have revised Figures 2, 3, and 4 for better clarity and ensured they are color-blind-friendly. Specifically, we have revised the [figure of Experiment 3](https://figshare.com/s/f45697639e8908df84dd?file=53348420) to better illustrate the performance differences. Additionally, we have added a [table](https://figshare.com/s/f45697639e8908df84dd?file=53348411) for clearer comparisons. The updated results now show that in Experiment 3, AISIVI and KSIVI outperform all other methods in terms of log marginal likelihood.\n\n\n2. **Additional Baselines**: We have expanded our comparisons to include UIVI and IWHVI. Furthermore, for the multimodal example, we added a [normalizing flow baseline](https://figshare.com/s/f45697639e8908df84dd?file=53348414). The results demonstrate that normalizing flows either smooth out low-density regions or introduce additional fragments (as observed with the neural spline flow). \n\n3. **Convergence Speed and Stability**: KSIVI remains the fastest in terms of convergence speed, but our method is the only one that matches its performance. Interestingly, AISIVI does not require very large inner batches if the proposal distribution (e.g., a normalizing flow) is sufficiently flexible and learns faster per iteration than KSIVI. Our framework efficiently supports computationally expensive proposal distributions, such as CNFs, since we do not need to backpropagate through the proposal distribution when computing path gradients. This property is one of our main novel contributions. However, without a proposal distribution, large batch sizes are indeed necessary, as demonstrated by BSIVI.\n\n\n4. **Computational Cost Comparison**: Following the suggestion to investigate computational costs, we have now performed comparisons based on the Conditioned Diffusion Process task for all methods. To facilitate a fair comparison, we now use the total iterations of KSIVI (outer loop iterations × inner loop iterations) instead of the outer iterations as done in Sec 5.3. While this shows that AISIVI learns much faster per iteration than KSIVI (as seen in the table), it also highlights the extraordinarily fast implementation of KSIVI when comparing run times. Overall, the comparison clearly shows that only the gold standard KSIVI and our AISIVI are able to reach log marginal likelihood values >70,000, while all other methods lag behind significantly. This confirms the visual comparison of Fig. 4 in the paper and demonstrates that such performance levels can be achieved without relying on a kernel-based approach. \n\n\n5. **Citations**: We will incorporate all the suggested references and are very grateful for your recommendations, which further enhance our paper’s positioning within the broader literature.\n\n\nThank you again for your constructive feedback, which has helped us refine and strengthen our work. We hope these improvements address your concerns.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "The evaluation Criteria makes sense", "theoretical_claims": "I only checked 3.1 which to the best of my understanding, is correct.", "experimental_designs_or_analyses": "The experimental design makes sense", "relation_to_broader_scientific_literature": "I must admit I am not very familiar with the literature in this regard. I have listed a few papers that seem relevant which are not cited here.", "essential_references_not_discussed": "[1] Ma, Chao, Yingzhen Li, and José Miguel Hernández-Lobato. \"Variational implicit processes.\" International Conference on Machine Learning. PMLR, 2019.\n\n[2] Yu, Longlin, et al. \"Hierarchical semi-implicit variational inference with application to diffusion model acceleration.\" Advances in Neural Information Processing Systems 36 (2023): 49603-49627.\n\n[3] Shi, Jiaxin, Shengyang Sun, and Jun Zhu. \"Kernel implicit variational inference.\" arXiv preprint arXiv:1705.10119 (2017).\n\n[4] Zimmermann, Heiko, et al. \"Nested variational inference.\" Advances in Neural Information Processing Systems 34 (2021): 20423-20435.", "other_strengths_and_weaknesses": "***Strengths*\n\n- The paper reads very well. The motivation of the paper makes sense and it offers a nice introduction to SIVI and the problems with it.\n\n- While the proposed approach might be simple, it makes a lota of sense and is a perfectly valid contribution for ICML. \n\n- The experiments are very systematic and well designed, on par with many other VI papers. \n\n**Weaknesses**\n\n- I must say the results seem abit underwhelming. One issue is that ofcourse most of the results figures (2,3,4) are difficult to read and judge, in the sense that is diffcult to compare the performance of a method to the ground truth as well as other methods. There are no tables with the exception of the toy experiment. It looks to me that everything performs roughly the same, while KSVI and PVI perform a bit better. I understand that beating SOTA is not a requirement but it should be explained abit given that these methods are used as baselines that what is the (potential) advantage here? Also how well the spline flow itself would have performed by itself (simply minimizing $KL[q|p]$ where $q$ is the NF.)\n\n- Related to the previous issues, some discussion of speed of convergence, its stability, and hyperparameter sensitivity would have been a good improvement. Having an expectation inside the gradient term (Eq. 14) seems very sensitive. I would expect $K$ to have to be large? How sensitive was the choice of $K$?", "other_comments_or_suggestions": "- In Figure 4, the magenta line and the blue line are not very color blind friendly and are hard to separate I would advise to use different coloring scheme . Also please use legends and label the x and y axis in your figures.", "questions_for_authors": "See weaknesses and Strengths", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BZvsGgkMiT", "reviewer": "Reviewer_2yqc", "summary": "In this work they propose importance sampling estimation of the score function needed for minimising the KL divergence between q_z and p_z in SIVI. To do this they use a CNF proposal.\n\nThey compare their methods (one which uses the importance sampling estimator and one which doesn't) against a kernel stein discrepancy based method (KSIVI) and a particle approximation to a Euclidean-Wasserstein gradient flow (PVI). They show that their methods perform comparably to PVI and KSIVI on a Bayesian Logistic Regression task, and that their importance sampling based method performs favourably (both in terms of approximation quality and clock-time) on a Conditioned Diffusion Process task.\n\n## update after rebuttal\nI maintain my recommendation.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your constructive feedback. We have made the following revisions in response to your comments:\n\n1. **Computational Cost Comparison**: Following your suggestion to investigate computational costs, we have now performed comparisons based on the Conditioned Diffusion Process task for all methods. To facilitate a fair comparison, we use the total iterations of KSIVI (outer loop iterations × inner loop iterations) instead of just the outer iterations as done in Sec 5.3. This comparison highlights that AISIVI learns much faster per iteration than KSIVI (as shown in the [table](https://figshare.com/s/f45697639e8908df84dd?file=53348411)), but also emphasizes the extraordinarily fast implementation of KSIVI when comparing run times. Overall, the comparison confirms that only the gold standard KSIVI and our AISIVI reach log marginal likelihood values > 70,000, while all other methods notably lag behind. This result aligns with the visual comparison in [Fig. 4](https://figshare.com/s/f45697639e8908df84dd?file=53348420) and demonstrates that it is possible to achieve such performance without relying on a kernel-based approach. \n\n\n2. **Explanation for Consistency**: Regarding your question \n> On line 250 in column 2: \"This alternating training is possible since sIS,k(z) is a consistent estimator of the score gradient for any τε|z with supp(qε|z) ⊂ supp(τε|z).\"\n\nabout the consistency of alternating training: This statement means that as long as K (the number of importance samples) is large enough, we can draw from any proposal distribution that satisfies the support assumption. Since the score gradient estimator ($s_{\\text{IS}}$) is consistent, it converges almost surely. This allows us to keep the proposal distribution fixed while updating the SIVI model, and update the flow afterwards with our unbiased forward KL gradient, which moves the current CNF closer to the optimal proposal distribution (the target) for the new SIVI model. This enables convergence to the global solution. We will make this more clear in an updated paper version and thank the reviewer for this question. \n\n3. **Typo and Reference Update**: We have corrected the typo and greatly appreciate the reference to Andrade (2024). We will replace the current citation with Andrade (2024), as it provides a more relevant theorem on variance underestimation.\n\n\n4. **Algorithm Clarification**: We updated the algorithm description to make it clear that θ refers to the parameters of τ (the normalizing flow). This change should clarify the notation and its role in the model.\n\n\nWe hope these revisions address your concerns. Thank you again for your detailed feedback, which has been incredibly helpful in improving the clarity and rigor of our work.", "claims_and_evidence": "I believe the authors show that their proposed method works and is comparable to other existing methods, while saving on computation thanks to the replacement of the costly MCMC step with importance sampling.", "methods_and_evaluation_criteria": "Yes, the datasets and models evaluated on seem adequate.", "theoretical_claims": "I looked over the proof in the main text which seemed correct to me. I also looked at some of the proofs of consistency in the appendix and could see no issues.", "experimental_designs_or_analyses": "As previously mentioned the experiments seemed sensible.", "relation_to_broader_scientific_literature": "The work builds on Titsias & Ruiz (2019) which introduces Unbiased Implicit Variational Inference, and in particular includes the path gradient estimator which is needed to make their importance weighted estimator of the score function work.\n\nThis work also relates to Yin & Zhou (2018) in that they both use a semi-implicit distribution as a proposal.", "essential_references_not_discussed": "Not that I'm aware of.", "other_strengths_and_weaknesses": "I thought this was a well written paper, using the background section to both cover the state of the field but also motivate the development of the presented method was a nice touch.\n\nSection 3.2 was interesting and I appreciated that some thought was given to efficient implementation.\n\nIn the results I would have liked to have seen iterations per second for all the methods used in the final example.", "other_comments_or_suggestions": "\"principle manner\" line 029\n\nFrom line 55 to 62 in the second column, I wonder if Andrade (2024) might not be a better reference as they have a theorem aimed at this very question, whereas VI: A review for statisticians seems to suggest that underestimation of the variance is \"a consequence of its objective function\"?\n\nIn algorithm 2 it isn't clear what theta is for (I'm guessing the parameters of tau).", "questions_for_authors": "On line 250 in column 2: \"This alternating training is possible since sIS,k(z) is a consistent estimator of the score gradient for any τε|z with supp(qε|z) ⊂ supp(τε|z).\"\n\nCould you explain this further?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The authors develop a new approach for semi-implicit variational inference, where the variational distribution marginalizes an implicit distribution with known likelihood. Optimizing the variational bound with pathwise estimators requires the gradient of log of q. The fresh direction is to explore how to estimate this gradient with importance sampling. A through analysis with consideration to memory usage is also carried out.
Decision: Accept (poster)