Semi-Implicit Variational Inference via Score Matching
Authors: Longlin Yu
Cheng Zhang
Venue: ICLR 2023 notable top 25% (2023)
Keywords: Semi-implicit variational inference
denoising score matching
minimax optimization

Abstract: Semi-implicit variational inference (SIVI) greatly enriches the expressiveness of variational families by considering implicit variational distributions defined in a hierarchical manner. However, due to the intractable densities of variational distributions, current SIVI approaches often use surrogate evidence lower bounds (ELBOs) or employ expensive inner-loop MCMC runs for unbiased ELBOs for training. In this paper, we propose SIVI-SM, a new method for SIVI based on an alternative training objective via score matching. Leveraging the hierarchical structure of semi-implicit variational families, the score matching objective allows a minimax formulation where the intractable variational densities can be naturally handled with denoising score matching.  We show that SIVI-SM closely matches the accuracy of MCMC and outperforms ELBO-based SIVI methods in a variety of Bayesian inference tasks. 

Published as a conference paper at ICLR 2023
SEMI-IMPLICIT VARIATIONAL INFERENCE VIA SCORE MATCHING
Longlin Yu School of Mathematical Sciences Peking University, Beijing, China 11yu@pku. edu.cn cn
Cheng Zhang
School of Mathematical Sciences and Center for Statistical Science Peking University, Beijing, China
chengzhangemath pku. edu. cn
ABSTRACT
Semi- impiiiiplicit variational inference (SIVI) greatly enriches the expressiveness of variational families by considering implicit variational distributions defined in a hierarchical manner. However, due to the intractable densities of variational distri- butions, current SIVI approaches often use surrogate evidence lower bounds (EL- BOs) or employ expensive inner-loop MCMC runs for direct ELBO maximization for training. In this paper, we propose SIVI-SM, a new method for SIVI based on an alternative training objective via score matching. Leveraging the hierarchical structure of semn--impe comi-implicit variational families, the score matching objective allows a minimax formulation where the intractable variational densities can be naturally handled with denoising score matching. We show that SIVI-SM closely matches the accuracy of MCMC and outperforms ELBO-based SIVI methods a variety of Bayesian inference tasks.
INTRODUCTION
Variational inference(VI (VT) is an approximate Bayesian inference approach where the inference prob- lem is transformed into an optimization problem (Jordan et al. 1999, Wainwright & Jordan 2008 Blei et al. (017). starts by introducing a family of variational distributions over the model param- eters (or latent variables) to approximate the posterior. The goal then is to find the closest member from this family of distributions to the target posterior, where the closeness is usually measured by the Kullback-Leible (KL) divergence from the posterior to the variational approximation. In prac- tice, this is often achieved by maximizing the evidence lower bound (ELBO), which is equivalent to minimizing the KL divergence (Jordan al. 1999..
One of the classical VI methods is mean-field VI (Bishop & Tipping 2000). where the variational distributions are assumed to be factorized over the parameters (or latent variables). When combined with conditional conjugacy, this often leads to simple optimization schemes with closed-form update rules (Blei et al. 2017p. While popular, the factorizable assumption and conjugacy condition greatly restrict the flexibility and applicability of variational posteriors, especially for complicated models with high dimensional parameter space. Recent years have witnessed much progress in the field of VI that extends it to more complicated settings. For example, the conjugacy condition has been removed by the black-box VI methods which allow a broad class of models via Monte carlo gradient
estimators (Nott et al.] 2012; Paisley et al.) 2012 Ranganath et al.) 2014: Rezende al.) 2014 Kingma& & Welling 2014). On the other hand, more flexible variational familics have been proposed that either explicitly incorporate more complicated structures among the parameters (Jaakkola &
Jordan 1998 Saul & Jordan 1996 Giordano et al. 2015] Tran et al., (015) or borrow ideas from
invertible transformation of probability distributions (Rezende & Mohamed| [2015) DDinh et al. 2017]
Kingma et al.) 2016 Papamakarios et al. 2019). All these methods require tractable densities for the variational distributions.
turns out that the variational family can be further expanded by allowing implicit models that have intractable densities but are easy to sample from (Huszár 2017). One way to construct these implicit models is to transform simple base distribution via a deterministic map, i.c., a deep neural
"Corresponding author.

---

Published as a conference paper at ICLR 2023
2018a| b Song al.) 2019). Due the intractable densities of implicit models, when evaluating the ELBO during training. one often resorts to density ratio estimation which is known to be challenging in high-dimg -ith-iminniioall settings (Sugiyamal et al. 2012). To avoid density ratio estimation, semi-implicit variational inference (SIVI) has been proposed where the variational distributions are formed through semi-implicit hierarchical con- struction and surrogate ELBOs (asymptotically unbiased) are employed for training (Yin & Zhou 2018 Moens al. 2021). Instead of surrogate ELBOs, an unbiased gradient estimator of the ex- act ELBO has been derived based on MCMC samples from a reverse conditional (itsias & Ruiz, 2019). However, the computation for the inner-loop MCMC runs can easily become expensive in high-dimensional regimes. There are also approaches that estimate the gradients instead of the
network (Tran et al. 2017 Mescheder et al. 2017: Shi et al.
objective (Li & Turner 2018
Shi al. 2018b: Song et al. 2019).
Besides KI. divergence, score-based distance measures have also been introduced in various statisti- cal tasks (Hyvärinen, | et al.
2018) and have shown advantages in complicated nonlinear models (Song & Ermon 2019; Ding et al. 2019 Elkhalil et al.] 2021). Recently, there are also some studies that use score matching for variational inference (Yang et al. 2019 Hu et al. 2018). How- ever, these methods exploit the hierarchical structure of semi-implicit variational distributions. In this paper, we propose SIVI-SM, a new method for SIVI using alternative training objective via score matching. We show that the score matching objective and the semi-implicit hierarchical construction of variational posteriors can be combined rally handled with denoising score matching. We demonstrate the effectiveness and efficiency of our method on both synthetic distributions and a variety of real data Bayesian inference tasks.
not designed for SIVI and hence either do not apply to or not fully
minimax formulation where the intractability of densities is natu-
2 BACKGROUND
Semi-Implicit Variational Inference Semi-implicit variational inference (SIVI) (Yin & Zhou 2018) posits a flexible variational family defined hierarchically using mixing parameter as follows
(p(x²) = /
(=(2))) z ~
where p = (ob } are the variational parameters. This variational distribution is called semi-implicit as the conditional layer 4a (a) (a]z) is required to be explicit but the mixing layer ge (z) can be implicit, and (x) is often implicit unless ge x)) is conjugate to (z)(z)) Compared to standard VI, the above hierarchical construction allows a much richer variational family that is able to capture complicated
dependencies between parameters (Yin & Zhou 2018).
Similar to standard VI, current SIVI methods fit the model parameters by maximizing the evidence lower bound (ELBO) derived as follows
logp(D) > ELBO := (m)gs(() [logp(D, (D x)) log (z)],
where D is the observed data. As q, is no longer tractable, Yin & Zhou (2018) considered sequence of lower bounds of ELBO
((.x))
ELBO > Cr :=
log
2(1),. gar ad
(=(=) L+1 (0(x)))) + El_ ( (1)
Note that L, is an asymptotically exact surrogate ELBO as L + co. An increasing sequence of { therefore, is often suggested, with L2, being optimized at the t-th iteration.
Instead of maximizing surrogate ELBOs, Titsias& & Ruiz (2019) proposed unbiased implicit varia- tional inference (UIVI) which is based unbiased gradient estimator of the exact ELBO. More
specifically, consider a fixed mixing distribution (2)x) = g(2) and a reparameterizable conditional
(=/=)) such that TT(2, e), (e(E) < x (=(=))) then
ELBO = (()((() logg(D xx) - log go (2) ( (g.(e)((s) (0 <) + g (2, {)]
(2)
2

---

Published as a conference paper at ICLR 2023
where
N6 F0 (2,c) = Va log PD, a)) (z.c) VoTo(z,e).
(3)
g6 ent (2, == Eys(=' x) V log (z) (=,,(z.c)) (= V Volo(z,e).
(4)
The gradient term in 4 involves an expectation w.r.t. the reverse conditional 4s x)) which can be estimated using MCMC sampler (e.g., Hamiltonian Monte Carlo (Neal 2011)). However, the inner-loop MCMC runs can easily become computationally expensive in high dimensional regimes. See more detailed discussion on the derivation and computation issues of UIVI in Appendix and and
Score Matching Score matching is first introduced by HHyärinen (0055) to learn un-normalized
statistical models given i..d. samples from unknown data distribution p(æ). Instead of estimating p(æ) directly, score matching trains a score network S(x) to estimate the score the data distribu- tion, i.e. V log p(æ), by minimizing the score matching objective Ep(z) I [[(()) Va log p(x)
Using the trick of partial integration, Hyvärinen (2005) shows that the score matching objective is equivalent to the following up to constant
E B (TVV) + 1[
(5)
The expectation in Eq. can be quickly estimated using data samples. However, it is often challeng- ing scale up score matching high dimensional data due the computation of IIr V (S(x)).
A commonly used variant of score matching that can scale up to high dimensional data is denoising score matching (DSM) (Vincent 2011). The first step of DSM is to perturb the data with a known
noise distribution Ja (ø) æ), which leads to perturbed data distribution 4o (z) = f go (æ) x p()) dæ. The score matching objective for qo (ā)) turns out to be equivalent to the following up to a constant
1
2 Ee (#)m))(())
[[))())
V a log go ( (ā(x))])
Unlike Eq. Eq. 6 does not involve the trace term and be computed efficiently, long as the score of the noise distribution V log da (i|z) x>) easy to compute. Note that the optimal score network here estimates the score of the perturbed data distribution rather than that of the true data distribution. small noise, therefore, is required for accurate approximation of the true data score v log p(æ). Despite this, DSM is widely used in learning energy based models (Saremi al. 2018) and score based generative models (Song & Ermon 2019).
3 PROPOSED METHOD
While ELBO-based training objectives prove effective for semi-implicit variational inference, cur- rent approaches either rely on surrogates of the exact ELBO expensive inner-loop MCMC runs for unbiased gradient estimates due to the intractable variational posteriors. In this section, we in- troduce an alternative training objective for SIVI based on score matching. We show that the score matching objective be reformulated in minimax fashion such that the semi-implicn hicrarchical construction of variational posteriors can be efficiently exploited using denoising score matching. Throughout this section, we assume the conditional layer 4s (x) z) to be reparamcterizable and its score function V log (x) 2z) is easy to evaluate T
3.1 A MINIMAX REFORMULATION
Rather than maximizing the ELBO as in previous semi-implicit variational inference methods, we can instead minimize the following Fisher divergence that compares the score functions the target and the variational distribution
min Exzge (a)) [(()) - V log g (x)
'This assumption is quite general and holds for many classical distributions that are commonly used as conditionals, such Gaussian and other exponential family distributions.
3

---

Published as a conference paper at ICLR 2023
Here S(x) V loggp((=)D) V (DD, x)) is the score of the target posterior distribution, and the variational distribution 9p(x) is defined in Eq. Due to the semi-implicit construction in Eq. [ the score of variational distribution, i.c. v log (x)x), is intractable, making the Fisher divergence in Eq. 7 not readily computable. Although the hierarchical structure of (x) allows estimate its score function via denoising score matching, the estimated score function would break the dependency on the variational parameter yp, leading to biased gradient estimates (see an illustration in Appendix L). Fortunately, this issue can be remedied by reformulating 7 as minimax problem. The key observation is that the squared norm of S(x)) V log 9p (x) can be viewed as the maximum value of the following nested optimization problem
[[ Ve log q (x)] - max (()
2f(x) (S(x)) V log (x)] - ll
Væ.
where (ax) is arbitrary function of and the unique optimal solution is
(x) == S(x) & log (p
Based on this observation, we can rewrite the optimization problem in 7a
min max
(f)=
V loggp(x)) - (x)
(8)
Now we can take advantage of the hierarchical structure of (ar) ride of the intractable score term V log (x) in Eq. similarly as done in DSM. More specifically, note that
V x logg
1 (x(x))
/
(=(=))o(<(22) log ( (z)z)dz.
We have
a ()) (0(2) /
(x (o)
as (z))oo(z=2)V loggo(rlz)dzzaz
|| qe (e(2)a) (z) (
(a) V log (z)
Substituting Eq. into Eq. 1qq.&comme completes reformulation.
Theorem 1. Assume the variational distribution q. is a semi-implicit distribution defined by Eq. 7 then the optimization problem in is equivalent to the following minimax problem
min max
Exgrag (()))) (ax)) [S(x) V log qol (a<(=))) (I. 1b
(10)
Where == [ Moreover, assume that f can represent any function. (1*', ff) defines Nash-equilibriam of Eq. TO then ,"' is given by
f (x) S(x) V log qu- (x).
4² arg min E (m) S(x) Va log ( (x))1()
(11)
See detailed proof Theorem T Appendix Note that the objective equation X can also be derived via Stein discrepancy minimization (Liu et al. 2016 Gorham & Mackey 2015 Ranganath ctt al. 2016, Grathwohl et al. 2020). However, our reformulation in equation| I0 takes a further step by utilizing the hicrarchical structure of (x) and hence can easily scale up to high dimensions. See Appendix K for a more detailed discussion.
3.2 PRACTICAL ALGORITHMS
In practice, parameterize with neural network (x)... According the above minimax reformulation, the Monte Carlo estimation of the objective function in Eq. [0]aan be easily obtained by sampling from the hierarchical variational distribution z re(z), go (z(x))). Furthermore, using the reparameterization trick (Kingma & Welling 2013). i.c. x = Tz(z;)),
(he(r), where
ge (c), y ~ (7), we can rewrite Eq. 10 as tollows
1
min max
Eg (r)-
(T)(
Va logg) 5 f((x)]
x=To(*.e), ==)((())))==
4

---

Published as a conference paper at ICLR 2023
Algorithm 1 SIVI-SM with multivariate Gaussian conditional layer
Input: Score of target posterior distribution S(z). Total iteration number N. Number of gradient steps K for the inner optimization on f (z)).
Output: Variational parameter yp and the neural network parameter t/.
Initialize PP. 170
for 0 to N do
Sample {-((), 723,
, (m)) from prior (7()) and let x(1) = he(()()), 1,..., m.
Sample {e(1), 6 6(2) 6 ) y from No, I).
Compute (+))) = (z(2(t) + (2()()) @ c())
Update yp by descending its stochastic gradient:
(()(())))))))))))+++))))))) + 1 (
oel
for j 1 do
Sample {7(1), (22)
(mm) from prior (-(7)) and let x0)) = he(r = =1, 1...., m
Sample {e(l) (2))
(m) from (0, I).
Compute (0)) = (z) + (z(2 @ c(i)
Update & by ascending its stochastic gradient:
1 ( -
end for end for
This allows to directly optimize the parameters p, with gradient based optimization meth- ods (Goodfellow et 2014). Optimizing f to completion the inner loop of training com- putational prohibitive. Therefore, alternate between K steps of optimizing f and one step of optimizing de- As mentioned before, we use the multivariate Gaussian distribution with diagonal covariance matrix for the conditional (x((2))) ~ (p)(2z) (oaag("))))) which can be reparame- terized as follows
(z) a (z) e, where 6~ NN(0 I),
where O means the element-wise product. The corresponding score function Va log qs (z/2))) (z) c. The complete procedure SIVI-SM formally presented in Algorithm
3.3 THEORETICAL RESULTS REGARDING NEURAL NETWORK APPROXIMATION
The inexact lower-level optimization for neural networks introduces approximation errors. Also, neural networks themselves may introduce approximation gaps due to their approximation capaci- ties. These numerical errors may affect the approximation quality of variational distribution q. (x)), which we analyze the following proposition.
Proposition 1. Let Ωi be the feasible domain of Vip E Ωt say that fre()) is c-accurate, if
Barq (m) (x))1 e, where R.((x)= = S(x) V log log fecey
(12)
Let 4º' be one of the optimal variational parameters defined in Eq. [I and &p be the one obtained using neural network approximation defined as follows with f(s(e) being e-accurate
c = arg min Margmin(Eaaag,aa (x) 25
- Va (og))(()) 11 (a: [12].
PER
Then have
Exrgy [(() V log (x))11 <
V log ap-
(x) + e.
(13)
See detailed proof of Proposition I Appendix C From proposition] I we see that the approxima- tion error of our numerical solution to the minimax problem in 10 can be controlled by two terms.
5

---

Published as a conference paper at ICLR 2023
The first term (a)) V log 96- (z)1l measures the approximation ability of the varia- tional distribution, and the second term measures the approxmaaioonoppimzzaiii network (a).. As long the approximation error of fa((z) is small, the minimax formulation in 10 can provide variational posteriors with similar approximation accuracy to those of the original problem in in in terms of Fisher divergence the target posterior.
error of the neural
Remark, When the variational parameter is fixed, the lower-level optimization problem on (x) equivalent to the following
min Rerg (a)) ||((2)) V (o))) - J (2))
and we use this objective as measure of approximation accuracy of (ax))) given yp in Eq. 12
EXPERIMENTS
In this section, we compare SIVI-SM ELBO-based methods including the original SIVI and UIVI on a range of inference tasks. We first show the effectiveness of our method and illustrate the role of the auxiliary network approximation fg on several wwo-dimensional toy examples. The KL divergence from the target distributions to different variational approximations was also provided for direct comparison. We also compare the performance of SIVI-SM with both baseline methods on several Bayesian inference tasks, including multidimensional Bayesian logistic regression problem and high dimensional Bayesian multinomial logistic regression problem. Following Titsias & Ruiz 2211 set the conditional layer to be (x(2) . (x))))(()) diag ( and fix the mixing layer as q(>) = (0, I). The variational parameters therefore are ip = for o All experiments were implemented in Pytorch (Paszke et al. 2019). If not otherwise specified, we use the Adam optimizer for training (Kingma & Ba 2014).
4.1 TOY EXAMPLES
We first apply SIVI-SM to approximate three synthetic distributions defined on two-dimensional space: a banana-shaped distribution, a multimodal Gaussian, and an X-shaped mixture of Gaussian. The densities of these distributions comparison, we used the same configuration of semi-implicit distribution family as in UIVI (Titsias) & Ruiz, 2019). The (=(=2) is a multilayer perceptron (MLP) with layer widths [3, 50, 50, 2). The network approximation f (a)) is parameterized by a layers MLP with layer widths [2. 128, 128, 2]. For SIVI-SM, we set the number of inner-loop gradient steps K 1. For SIVI, we set L 50 for the surrogate ELBO defined in Eq. [ For UIVI, we used 10 iterations for every inner-loop HMC sampling. To facilitate exploration, for all methods, we used the annealing trick (Rezende & Mo- hamed, 2015) during training for the multimodal and X-shaped Gaussian distributions. Variational approximations from all methods were obtained after 50,000 variational parameter updates.
given in Table 2 in Appendix [ For the convenience of
Figure 5 in Appendix F shows the contour plots of the synthetic distributions, together with 1000 samples from the trained variational distributions. We see that SIVI-SM produces samples that match the target distributions well. We also report the KL divergence from the target distributions to the variational posteriors (estimated via the ITE package (Szabó) 2014) using 100,000 samples from each distribution) given by different methods in Table 3 in Appendix G We see that SIVI-SM performs better for more challenging target distributions. To better understand the role the network the
approximation (x) played during the training process, we visualize its training dynamics X-shaped distribution in Figure [ We see that during training, (x) automatically detected where the current approximation insufficient and guided the variational posterior towards these areas. Note the Yash-equilibrium of (z) in Eq. is the difference between the score functions of the target distribution p(x) and (x). As the variational posterior gets closer to the target, the signal provided by (x) becomes weaker and would converge to zero in the perfect case. More details on the convergence of f (a:) can be found in Appendix H
HHeee a is vector with the same dimension

---

Published as a conference paper at ICLR 2023
1000
2000
3000
4000
Figure I: The quiver plots of (a) and samples from the variational posteriors during the training process the X-shaped distribution.
SIVI-SM
SIVI
UIVI
Figure 2: Comparison of the marginal and pairwise joint posteriors. The contours of the marginal and pairwise empirical densities trained by the three semi-implicit variational inference algorithms, i.c. SIVI-SM (orange), SIVI (blue) and UIVI (green), are plotted against the ground truth (black).
4.2 BAYESIAN LOGISTIC REGRESSION
Our second example is Bayesian logistic regression problem where the log- likelihhood function takes the following form
log logp(ula%,, 3) = ys * 33'22 log(1 + exp((8" +)), E 10, [[.]]
Here are covariates, and E 10.1) are binary response variables. Following Yin & Zhou 2018)), we set the prior as G NNO0, 'I), where a = 0.01. We consider the waveform dataset where the dimension of æ; is 21 which leads to parameter space of 22 dimensions. We used standard 10-dimensional Gaussian prior for the q(2). For Ha(22), we used a 4 layer MLP with layer widths [10, 100, 100, 22). The network approximation fo (B) is also a 4 layer MLP with layer width [22, 256, 256, 22]. Similarly as in section 4.1 we set the number of inner-loop gradient steps K in SIVI-SM. For SIVI, we /. 100 and used the same training method in Yin & Zhou (2018). For UIVI, we set the length of inner-loop HMC iterations to be 10 with the first 5 iterations discarded as burn-in, with 5 leapfrog steps in each iteration. The results of all methods were collected after 20,000 variational parameter updates.
We collected 1000 samples of / to represent the approximated posterior distributions for all three SIVI variants. The ground truth was formed from a long MCMC run of 400,000 iterations using parallel stochastic gradient Langevin dynamics (SGLD) (Welling & Teh, 2011) with 1000 indepen- dent particles, and small stepsize of 10-1 Figure 2 shows the posterior estimates provided by different SIVI variants in contrast to the ground truth MCMC results. We see that SIVI and UIVI tend to slightly underestimate the variance for both univariate marginal and pairwise joint posteriors (especially for 3,, 35), while SIVI-SM agreed with MCMC well. Furthermore, we also examined the covariance estimates of & and the results were presented in Figure 3
We see that SIVI-SM
https::/archiis
uci edu/mm ht/mmm//maaaa /tt.haachhccaaa ne 1ear- htw.a/iiiiiiaaaaaaaaaaaeeeeeeee htt.::/e/eeaaea.eeeeeeeeeeeaaaae

---

Published as a conference paper at ICLR 2023
RMSE: 0.0184
RMSE: 0.0255
RMSE: 0.0313
0.3
0.3
0.2
0.2
0.2
SIVI-SM
0.1
0.1
0.1
SIVI
0.0
0.0
-0.2
-022
-0.
Covariance
Covariance
Covariance
-0.3
-.3
0.1 SGLD
0.0
0.2
0.3
-0.3
0,33-0220011
0.0 0.1 0.2
0.3
-0.3
-0.3-0.-00
0.2 0.3 SGLD
0.0
0.1
SGLD
Figure 3: Scatter plot comparison of the sample covariances of the posterior. The X-axis and Y- axis represent the estimates from the ground truth MCMC runs and the corresponding SIVT variants respectively. The red lines are the regression lines.
provides the best overall approximation to the posterior which achieved the smallest rooted mean square error (RMSH) the ground truth 0.0184.
4.3 BAYESIAN MULTINOMIAL LOGISTIC REGRESSION
Our next example is a Bayesian multinomial logistic regression problem. For a data set of N co- variate and label pairs {(Ti, v1) 1, .,..., N}. where y/ {I,..., R}. the categorical likelihood is
p(: = r|æ;)) oc exp([1, xx] B,), {1,2, .1, T
where B = (BT BT. 3T is the model parameter and follows standard Gaussian prior. Fol- lowing Titsias & Ruiz (019). we used two data sets: MNIST and HAPT MNIST is commonly
used dataset in machine learning that contains 60,000 training and 10,000 test instances of 28x28
images of aund-written digits which has R 10 classes. HAPT (Reyes-Ortis et al.) (016) is human activity recognition dataset. contains 7.767 training and 3,162 test data points, and each of them contains features of 561-dimensional measurements captured by inertial sensors, which correspond to R = 12 classes of static postures, dynamic activities and postural transitions. The dimensions of the posterior distributions are 7,850 (MNIST) and 6,744 (HAPT) respectively.
We used the same variational family as before, with a 100-dimensional standard Gaussian prior for q(=). We used MLPs with two hidden layers for the mean network (z)(2) of the Gaussian condi- tional and the network approximation (3). with 200 hidden neurons for (=(=)) and 256 hidden neurons for (3) for each of the hidden layers respectively. We used the same initialization of variational parameters for all methods. Following Titsias & Ruiz (2019), we used a minibatch size of 2,000 for MNIST and 863 for HAPT. As before, we set the number of inner-loop gradient steps K = 1 in SIVI-SM. For SIVI, we set I. 200 as previously done by (Titsias & Ruiz (2019). For UIVI, we set the number of inner-loop HMC iterations to be 10 and discarded the first iterations as burn-in, with 5 leapfrog steps in each iteration. As done in UIVI, we used the RMSProp op- timizer (Tieleman & Hinton 2012) for training. We used different batch sizes during training to investigate its effect the quality of variational approximations for different methods. These batch sizes were selected in such a way that the corresponding computational times are comparable be- tween different methods. See detailed comparison experiments were run on RTX2080 GPU). As the gradient computation for the inner-loop HMMC sampling required by UIVI not S the batch size for UIVI is set as m 1 which was also used by Titsias & Ruiz (2019). For all methods, the results were collected after 90,000 variational parameter updates for MNIST and 40,000 variational parameter updates for HAPT.
the computation times in Appendix [ (the
Figure 4 shows the predictive log- likelioel on the test data as function of the number of iterations for both data sets, where the estimates were formed based on 8,000 samples from the variational
http:://anm Lecun. www.w//ww..//w///////ooeaoa
http: /tw.c//acchicco ve. ics . uci www.uceiin eedu/mm /tcc/aahhiiiac ne- l ttt.annngggataaaaaeeeeee303
"Se detailed explanation Appendix
8

---

Published as a conference paper at ICLR 2023
MNIST
HAPT
-0.25
-0.15
-0.30
-0.20
00.55
-0.35
-0.30
Test LL
Test LL
-0.40
-020
-0.35
SIVI SM(200)
-0.40
-0.45
SMM/20))
-0.30
SMM/101
0.50
SMVI-0M2200
SIVW10)
-0.50
SIV111)
SIVI-SMI100)
SM/11
UIVI(1)
0.55
SIV122)
UIVI(1)
-0.55
20000
40000
60000
B0000
-066
10000
20000
30000
40000
Iteration
Iteration
Figure 4: Estimates of the test log-likelihood for the Bayesian multinomial logistic regression model. The number in parentheses specifies the batch sizes used for training.
distributions fitted using different methods, done in UIVI Titsias & Ruiz 2019). Although SIVI-SM converges slower at the beginning due to the slow training of the network approximation (3). it eventually surpasses the other ELBO-based variants and achieves better prediction on both datasets. Compared to ELBO--aase methods, SIVI-SM would benefit more from large batch sizes.
4.4 BAYESIAN NEURAL NETWORKS
Lastly, compare method with SIVI, UIVI and SGLD sampling the posterior of Bayesian neural network on the UCI datasets. We conduct the two-layer network with 50 hidden units and ReLU activation function. The datasets are all randomly partitioned into 90% for training and 10% for testing. We use the variational family as before, with 3-dimensional standard Gaussian prior for ((2), and 10 hidden neurons for (z)(2) and 16 hidden neurons for (a)). The results are averaged over 10 random trials. We refer the reader to Appendix J for hyper-parameter tuning and other experiment details. T the average test RMSE and NLL and their standard deviation. We see that SIVI-SM can achieve par or better results than SIVI and UIVI. Although SGLD performs better some datasets, requires a long run to generate samples.
Table 1: Averaged test RMSE and test negative log-likelihood of Bayesian Neural Networks on seven UCI datasets. The results were averaged from 10 independent runs.
AVG. TEST RMSE
AVG. TEST NLL
Dataset
UIVI
SGLD
SIVI-SM
SIVI
SGLD
SIVI-SM
BOSTON
2.714gn.0s
2.81720.11
3.186_o.09
2.62110.10
2.420 09
2.397 10.07
3.161ga.08
2.396.20.15
CONCRETE
6.205+0.12 6.049+0.10
6.512=0.08
5.392+a.00
3.247
3.221+0.11
3.978+a0.0
3.156=20.0
PROTEIN
1.818go.os 4.908
4.768
1.903
2. 2.901go.02
3.01620.02
2.979
3.015
POWER
4.13-0.02 4.134+0.03
4.067=20.1
4.086=o.05
2.853
2.85220.06
2.965+a0.08
2.854
WINEWHITE
0.6404o.os 0.646+0.04
0.639
0.64644.o0
0.975
0.983
0.976
0.985
WINERED
0.577 0.675+0.06
0.651
0.560+a.02
0.871+0.11
1.0744..07
0.983
0.836
YACHT
1.902#0.21
2.1407
2.377
1.559+o.15
2.147
2.309+0.00
2.631=9.08
1.683++0.18
CONCLUSION
We proposed SIVI-SM, a new method for semi-implicit variational inference based on alternative training objective via score matching. Unlike the ELBO-based objectives, we showed that the score matching objective allows minimax formulation where the hierarchical structure of semi-implicit variational families can be more efficiently exploited as the corresponding intractable variational densities be naturally handled with denoising score matching. In experiments, demonstrated that SIVI-SM closely matches the accuracy of MCMC in posterior estimation and outperforms two typical ELBO--based methods (SIVI and UTVT) in variety of Bayesian inference tasks.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGMENTS
This work was supported by National Natural Science Foundation of China (grant no. 12201014). The research of Cheng Zhang was support in part by the Key Laboratory of Mathematics and Its Ap- plications (LMAM) and the Key Laboratory of Mathematical Economics and Quantitative Finance (LMEQF) of Peking University. The authors are grateful for the computational resources provided by the Megvii institute. The authors appreciate the anonymous ICLR reviewers for their constructive feedback.
REFERENCES
Christopher M. Bishop and Michael E. Tipping. Variational relevance vector machines. Proceed- ings of the Sixteenth Conference on Uncertainty in Artificial Intelligence, UAI'00, pp. 46-53, San Francisco, CA, USA, 2000. Morgan Kaufmann Publishers Inc. ISBN 1558607099.
D. M Blei, A. Kucukelbir, and J. McAuliffe. Variational inference: A review for statisticians. Journal of the American statistical Association, 1126118):59--87, 2017.
J. Ding, R. Calderbank, and V. Tarokh. Gradient information for representation and modeling. In Advances in Neural Information Processing Systems 32, pp. 124-130, 2019.
Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real nvp. In International Conference Learning Representations, 2017.
Khalil Elkhalil, Ali Hasan, Jie Ding, Sina Farsiu, and Vahid Tarokh. Fisher auto-encoders. In Arindam Banerjee and Kenji Fukumizu (eds.), Proceedings The 24th International Conference on Artificial Intelligence and Statistics, volume 130 of Proceedings of Machine Learning Re- search, pp. 352-360. PMLR, 13-15 Apr 2021. URL v130/elkha www.hhaliilaa....m html
/tt.ttpps//ppoeeeiiniie /ttp:/eeooceeeeseeee/eeee/.e mlr. press/
R. J. Giordano, T. Broderick, and M. I. Jordan. Linear response methods for accurate covariance estimates from mean field variational bayes. In Advances in Neural Information Processing Sys- tems, 2015.
Goodfellow, Pouget-Abadie, M. Mirza, Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. Generative adversarial nets. In Advances in Neural Information Processing System 27 (NIPS), 2014.
Jackson Gorham and Lester Mackey, Measuring sample quality with stein's method. In Advances in Neural Information Processing Systems, pp. 226 234, 2015.
Will Grathwohl, Kuan-Chieh Wang, Jörn-Henrik Jacobsen, David Duvenaud, and Richard Zemel. Learning the stein discrepancy for training and evaluating energy- energy-based -bssed models without sampling. In International Conference on Machine Learning, pp. 3732-3747. PMLR, 2020.
Tianyang Hu, Zixiang Chen, Hanxi Sun, Jincheng Bai, Mao Ye, and Guang Cheng. Stein neural sampler. arXiv preprint arXiv:1810.03544 2018
Ferene Huszár. Variational inference using implicit distributions. arXiv preprint arXiv: 1702.08235, 2017.
Aapo Hyvärinen. Estimation of non-normalized statistical models by score matching. Journal of Machine Learning Research, 6(4):695-709, 2005. URL http: jmlr.
hyvarinenUSa.
Jaakkola and Jordan. Improving the mean field approximation via the use of mixture distributions. In Learning in Graphical Models, pp. 173-173, 1998.
M. I. Jordan, Z. Ghahramani, T. Jaakkola, and L. Saul. Introduction variational methods for graphical models. Machine Learning, 37:183-233, 1999.
D. Kingma and M. Welling, Auto-encoding variational bayes. In ICLR, 2014.
10

---

Published as a conference paper at ICLR 2023
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6990, 2014.
Diederik P Kingma and Max Welling, Auto-encoding variational bayes, 2013,
URL https:
//arxiv.org abs/ 1312..2. 6114
Diederik P. Kingma, Tim Salimans, Rafal Jozefowicz, Xi Chen, Ilya Sutskever, and Max Welling- Improved variational inference with inverse autoregressive flow. In Advances in Neural Informa- tion Processing Systems, pp. 4743-4751, 2016.
Y. Li and R. E. Turner. Gradient estimators for implicit models. In International Conference on Learning Representations, 2018.
Qiang Liu, Jason Lee, and Michael Jordan. A kernelized stein discrepancy for goodness-of-ft tests. In International conference on machine learning, pp. 276-284. PMLR, 2016.
L. M. Mescheder, S. Nowozin, and A. Geiger. Adversarial variational bayes: Unifying variational autoencoders and generative adversarial networks. In Proceedings of the 34th International Con- ference Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, 2017.
V. Moens, H. Ren, A. Maraval, R. Tutunov, J. Wang, and H. Ammar. Efficient semi-implicit varia- tional inference. arXiv preprint (rxXiv:2101000077 2021.
Radford Neal. MCMC using hamiltonian dynamics. In S Brooks, A Gelman, G Jones, and XL Meng (eds.), Handbook of Markov Chuin Monte Carlo, Chapman & Hall/CRC Handbooks of Modern Statistical Methods. Taylor & Francis, 2011. ISBN 9781420079425. URL hittp://book
google. com www.books/bbo...oooooooooo...... woo.s/ookksoooooo http:/oooos..ddfffAAA.......
D. J. Nott, S. L. Tan, M. Villani, and R. Kohn. Regression density estimation with variational methods and stochastic approximation. Journal of Computational and Graphical Statistics, 21 (3):797-820, 2012.
J. W. Paisley, D. M. Blei, and M. I. Jordan. Variational bayesian inference with stochastic search. In Proceedings of the 29th International Conference on Machine Learning ICML, 2012.
G. Papamakarios, E. Nalisnick. D. Rezende, Mohamed, and B. Lakshminarayanan. Normalizing flows for probabilistic modeling and inference. ArXiv Preprint arXiv:/9/2.02762, 2019.
A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, 7. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf. E. Yang, Z. Vito,o M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala. PyTorch: An imperative style. high-performance deep learning library. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
R. Rangamath, S. Gerrish, and D. M. Blei. Black box variational inference. In AISTASS, pp. 814-822, 2014.
Rajesh Ranganath, Dustin Tran, Jaan Altosaar, and David Blei. Operator variational inference. Advances in Neural Information Processing Systems, 29, 2016.
Jorge-L Reyes- Ortiz, Luca Oneto, Albert Samà, Xavier Parra, and Davide Anguita. Transition-awe aware human activity recognition using smartphones. Neurocomputing, 767,
D. Rezende and S. Mohamed. Variational inference with normalizing flows. In Proceedings of The 32nd International Conference on Machine Learning, pp. 1530-1538, 2015.
D. J. Rezende, S. Mohamed, and D. Wierstra. Stochastic backpropagation and approximate infer- ence deep generative models. In International Conference on Machine Learning, 2014.
Saeed Saremi, Arash Mehrjou, Bernhard Schölkopf, and Aapo Hyvärinen. Deep energy estimator networks. arXiv preprint arXiv: 180500836, 2018. doi: 1044850/ARXIV..88550088 URL.
https: //apsi///rxivv org/ alss www.////..... 18011000000 08306
L. K. Saul and M. I. Jordan. Exploiting tractable substructures intractable networks, In Advances in Neural Information Processing Systems, 1996.

---

Published as a conference paper at ICLR 2023
J. Shi, S. Sun, and J. Zhu. Kerne implicit variational inference. In International Conference Learning Representations, 2018a.
J. Shi, S. Sun, and J. Zhu. A spectral approach to gradient estimation for implicit distributions. In International Conference Machine Learning, 2018b.
Y. Song, S. Garg, J. Shi, and S. Ermon. Sliced score matching: scalable approach density and score estimation. In Proceedings of the Thirty-Fifth Conference on Uncertainty in Artificial Intelligence, 2019.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. In Advances in Neural Information Processing Systems, pp. 11895-11907 2019.
Charles Stein. A bound for the error in the normal approximation to the distribution of sum of dependent random variables. In Proc. Sixth Berkeley Symp. Math. Stat. Prob., pp. 583-602, 1972.
Masashi Sugiyama. Taiji Suzuki, and Takafumi Kanamori. Density ratio estimation in machine learning. Cambridge University Press, 2012.
Zoltán Szabó. Information theoretical estimators toolbox. Journal of Machine Learning Research, 15:283-287, 2014.
T. Tieleman and G. Hinton. Lecture 6.5-rmsprop: Divide the gradient by running average of its recent magnitude. Coursera: Neural Networks for Machine Learning, 4, 2012.
Michalis K. Titsias and Francisco J. R. Ruiz. Unbiased implicit variational inference. In The 22nd International Conference on Artificial Intelligence and Statistics, pp. 167-176. PMLR, 2019.
D. Tran, M. Blei, and E. M. Airoldi. Copula variational inference. Advances in Neural Information Processing Systems, 2015.
D. Tran, R. Ranganath, and D. M. Blei. Hierarchical implicit models and likelihood-free variational inference. In Advances in Neural Information Processing Systems, 2017.
Pascal Vincent. A connection between score matching and denoising autoencoders. Neural Compu- tation, (177))166111674, 2011. doi: (/.////C.....00.........
M. J. Wainwright and M. Jordan. Graphical models, exponential families, and variational infer- ence. Foundations and Trends in Maching Learning, 1(1-2):1-305, 2008.
Max Welling and Yee Whye Teh. Bayesian learning via stochastic gradient langevin dynamics. In Proceedings of the 28th International Conference on International Conference on Machine Learning, ICML'11, pp. 681-688, Madison, WI, USA, 2011. Omnipress. ISBN 9781450306199
Yue Yang, Ryan Martin, and Howard Bondell. Variational approximations using fisher divergence, 2019. URL arwiiv /RL/attps//aaxiivco wttp
abs 90t.bb//190.0.0 / 05284
Mingzhang Yin and Mingyuan Zhou. Semi-implicit variational inference. In International Confer- ence on Machine Learning, pp. 5646-5655, 2018
C. Zhang, B. Shahbaba, and H. Zhao. Variational hamiltonian monte carlo via score matching. Bayesian Analysis, 13(2):485-500 2018.
12

---

Published as a conference paper at ICLR 2023
A DERIVATION OF EQ. 24
The gradient of EL.BO that
ELBO
V (())(() logp((, (D, logg TT((1))))
[((()((=)
v loggog D, (2, (
[g(e)((=)
[9% modd (2, -
- 9 2z E)]
(14)
where
g mad (z, == V log 2= VoTo(z,c),
g6 ent (2, == V log 20 (x) x=)e(z,)) V VoTo(z,c).
Note that the property of margin score function,
f (ogg
Va log (0(2))
(15)
Then the gradient ged can be representd as
Jo (2
(x) V log (0)) (Tg((a.) VoTa(z,e))
B PROOF OF THEOREM
Proof. As discussed in section by introducing the vector-valued function rewrite the optimization objective in Eq. as
Exng,t max (2)( (=))((((()))))) ( V log (x)) - (1
(16)
f())
Compute the score of the semi-implicit distribution q,p(x) defined in Eq. we have
1
V (x))
Vee (z)))
(x)
2E( (=)
log (2/=)dz.
46 (x)
Bring the above score of (xx) into Eq. 16 we have
Ex)gg(a) max (2((=)"(S(>)) f(m)
V log (x:) (x=)
(x())) / a (ett)oo(a (1f()))
-Eang,
max (xx(2
f(x)
max - (x) (2f(x) S(x) [ 4e (e( TV log (a
f(m)
-max f(()
E - (z) (S(2)) Va lo o =)) fl [1)
(17)
Therefor, let p {E. Φ). rewrite the original score matching problem Eq. 7 as
min max E 1 (2=_)((())))___(()))211331
[S(:) V log qo (x|=)) (z)
(4", defines Nash-equilibrium of the above problem, fixing the parameters = s², the optimal vector- valued function fz is invariant in the derivation of Eq. 17 So easily deduce f by Eq. 16
(x) S(x) Va log (æ)).
Bring f* f'(x) (x) into Eq. 10 have the unbiased approximation of 4²
E arg min
- V (21g) -1f"
E arg min (ain Va log (x: 112)-
13

---

Published as a conference paper at ICLR 2023
C PROOF OF PROPOSITION 1
Proof. Consider the score matching problem with the well e-trained (ipee) we have
9 = arg min ( (x)) (8 (x:) (S(x)) - a (x))] (1 (2)) [2],
E
arg min [ S (2)) (5(x) V log qo (z) [|S(x) V log go(()) (
(z))))
PEΩ
arg PEQ
min (()))((()) [5(x) V log (x) (z))1
(18)
Therefore, we can estimate the upper bound of Fisher divergence between S(x) and V log (x)))
E g (a) ||5(2)) V log qu (z)
E gp (ar ||5(z) v log qo (<)113
E g - (z) |[((x)
V log ao- (z)ll2
|| (z)11 + 11
E q (z) S((x) - Va log ox (x) +e,
(19)
where 10* = arg migminennte la) ([((x) Va log qu (1) And the last inequality is due to the fact that SK((P) well c-trained and |R (a) 3 is non-negative.
Li
D COMPUTATIONAL ISSUES ON UIVI
Unlike SIVI that samples from the prior (=)), UIVI samples from the posterior distribution g( [a)), which can provide unbiased gradient estimate to the exact ELBO (Titsias & Ruiz (019). However, UIVI requires computing the gradient of log (=((ar) (æ)) during the iterations of HMC sampling proce- dures. If UIVI uses a minibatch of m data points 02, In in the training process, it needs compute the Jacobian matrix [V log q( (=(2x1) Vz log (=)22)).--- ..... V, log ( [m)), which is not scalable for automatic differentiation using backpropagation. Therefore, we set the batch size m for UIVI as done in Titsias Ruiz 2019
E DENSITIES FOR THE TOY EXAMPLES
Table 2: Synthetic target distributions used in the toy experiments.
Name
pl a
Parameters
Banana-shaped
x = (V1, of + 12 + 1), v N)
Σ = 0.9 0,9]
Multimodal
Na I) + NN (2)
pi = #1 = [2.
X-shaped
Wo(1 Σn) |2(x000 Ex)
Σ 2s 1,8
-1.8
-1.8
14

---

Published as a conference paper at ICLR 2023
Banana-shaped
Multimodal
X-shaped
7.5
fnet's norm
fnet's norm
fnet's norm
0.4
5.0
SM loss
SM loss
SM loss
0.4
0.2
2.5
0.2
0.0
-2.5
0.0
-0.2
-0.2
-0.4
20000
40000
20000
40000
20000
40000
Iteration
Iteration
Iteration
Figure 6: The training loss and L2-norm of (x). f (x) (fnet's norm) and the training objective loss (SM loss) Eq. 10 are both estimated using 500 samples.
SIVI SAMPLES ON THE TOY EXAMPLES
Banana-shaped
Multimodal
X-shaped
2
-2
0-
-3
-2
-2
-4
-4
Figure 5: Performance toy examples. The samples from the variational posteriors fitted with SIVI-SM (orange) match the shape of target distributions (blue).
G KL DIVERGENCE RESULTS ON THE TOY EXAMPLES
Table 3: KL divergence from the target to the variational posteriors. The results were averaged from 5 independent runs with one standard deviation in the parentheses.
Name
SIVI
UIVI
SIVI-SM(ours) 0.1936(0.0169)
-hhapa--hhapdd
0.1876(0.0230)
0.3602 1106)
Multimodal X-shaped 00.041)0.0118)
0.1823(0.0025)
0.0611(0.0192)
0.0005
0.0236(0.008)
0.0046(0.0038)
H CONVERGENCE PERFORMANCE OF SIVI-SM
Here, demonstrate the convergence behavior of SIVI-SM in experiments. For the topy amples in section 4.1 we use 500 samples from qup (æ) to form the Monte Carlo estimates of the loss (w) (SM loss) in Eq. 10 and the L2-norm (wr) f (z) 112 of the fø function(fnet's norm) during the training process. F the estimated SM loss and fnet's norm function of the number of iterations for the three synthetic toy distributions.
Similarly, Figure shows the convergence traces of the SM loss and fnet's norm in the experiments in section 4.3 Note that although the dimensions of the posterior distributions are high, i... 7850 for MNIST and 6744 for HAPT, the corresponding fnet's norms can be quite low (58.510 for MNIST
15

---

Published as a conference paper at ICLR 2023
MNIST
НАРT
10a
Inet's norm(200)
1030
10a
fnee's norm(200)
10
SM loss(200)
fnet's norm(100)
10a
SM loss(200)
fnet'' norm(100)
10c
SM loss(100)
loss(100)
10a
10a
104
10a
10
10a
20000
40000
60000
B0000
10000
20000
30000
40000
Iteration
Iteration
Figure 7: Loss convergence for MNIST and HAPT. The loss trace has been smoothed with rolling window of size 5.
and 8.267 for HAPT), indicating the effectiveness of our methods even in high dimensional spaces.
I SECONDS PER ITERATION IN FIGURE 4
The following table shows the run times of different methods per iteration on a RTX2080 GPU. We that the run times for SIVI and SIVI-SM comparable with the chosen pairs of batch sizes (i.e., 10 vs 100 and 20 vs 200). As discussed before, the inner-loop HMC iterations make UIVI slower than other methods.
Table 4: Seconds per iteration for MNIST and НАРТ.
MNIST
HAPT
Method
S/it
Method
S/it
Method
S/it
Method
S/it
SIVI(20)
0.0107
SIVI-SM(200)
0.0088
SIVI(20)
0.0048
SIVI-SM(200)
0.0059
SIVI(10)
0.0065
SIVI-SM(100)
0.0058
SIVI(10)
0.0045
SIVI-SM(100)
0.0057
SIVI(1)
0.0042
UIVI(1)
0.0507
SIVI(1)
0.0042
UIVI(1)
0.0493
EXPERIMENT SETTING FOR BAYESIAN NEURAL NETWORKS
For SIVI, we L 100 the batch size m 10 training process. For UIVI, the setting of HMC inner loop is similar with section 4.3. For SGLD, choose the step size from {10-4,10' 10 1000-,000110000 10-63 and iteration number in (50000, 100000} by validation training process with 100 particles. For SIVI-SM, we set inner-loop gradient steps K 1,3 by validation and run 20,000 iterations for training.
K RELATED METHODS
Consider test functions class F, the Stein discrepancy (Gorham Mackey 2015) measure between p and g is defined follows
S(g. p)
sup
gx))))
V log) f(<) + TT(V=.(()))
(20)
JEF
This measure based on the following Stein's identity (Stein 1972)
By(a) [V log gog"
Tr (T
0.
(21)
An early example in variational inference used Stein discrepancy is operator variational inference (OPVT) (Ranganath et al. 2016), which constructs variational operator (e.g. Langevin-Stein Op-
16

---

Published as a conference paper at ICLR 2023
erator Oss) objective
Lig, OL
sup fer
V [
Then OPVI solves the minmax optimization problem simultaneously with q and f. Unlike OPVI, learned Stein discrepancy (LSD) (Grathwhhl et al. 2020) utilizes the L2 regularization term to substitute the constraint of
LLSD sup L g())
V (olgp(t))"
-
(22)
fact, the variational objective of SIVI-SM in Eq can be viewed as LL.D. Bring Eq into Eq 22 and let À 2. we have
LLSD sup E() f(x)² (V. logp(x) V logg(>)) -22-211660 [1
2 Ra) V Vlogp(>) (o))(
However, OPVI and LSD both involve the Tr(V (x)) term which not easy to compute for high dimensional problems. Our method takes further step by utilizing the hierarchical structure of ((x) in Eq | Using a mathematical trick that is similar to denoising score matching, we arrive at formulation that easily scales up high dimensions.
L ON THE BIASENESS OF THE MC GRADIENT ESTIMATE OF THE FISHER DIVERGENCE IN EQ. 7
=
5000
10000
2
t=
5000
t= 10000
2
-4
2
4
2
Figure 8: The quiver plots of (x) and samples from the variational posteriors during the training process of the multimodal Gaussian distribution. Up: SIVI-SM. Bottom: Biased gradient estimates.
TThe objective is similar with the definition of Stein measure in
(Lu al. 2016
17


Reviews:
[{"review_id": "wf581_xDLx2", "reviewer": "Reviewer_sufy", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new approach to semi-implicit variational inference (SIVI). SIVI-type formulations greatly increase the expressiveness of variational posteriors but introduce intractabilities to their inference. Existing methods rely on additional lower bounds on the evidence lower bound (ELBO), or require an expensive Markov chain Monte Carlo (MCMC) estimate of an inner gradient expectation. This work proposes the use of an alternative objective (the Fisher divergence), which, combined with the hierarchical nature of semi-implicit variational distributions can be approximated as the solution (Nash equilibrium) of saddle point optimization problem. This insight leverages the established link between denoising and score matching.", "summary_of_the_review": "While I am not entirely convinced about some fundamental aspects of this work (as discussed in \"Weaknesses\") I am leaning toward recommending acceptance of this paper. I am looking forward to discussions of the points raised.", "strength_and_weaknesses": "\nThe clarity of the writing and the soundness of the proposed technique are definite strengths of this work, as is the novelty. These are discussed in further detail in the subsequent section. \n\nOne significant weakness I wish to highlight is the conceptual motivation behind this approach at a higher level. Presumably, the appeal of SIVI-type methods is that one is able to circumvent the need to solve these minimax/saddle point optimization problems, the instability and difficulty of which have long plagued GANs and their derivative methods. SIVI-type enables one to steer clear of this problematic paradigm altogether, but the proposed method brings us right back to where we started. Although the experimental results indicate benefits over traditional SIVI-type methods, it's difficult to assess how this approach will generalize beyond the three relatively simple problems considered here. I'm interested to hear what the authors have to say and would like to see this discussed in the manuscript. Is there something about the formulation (Fisher divergence, score matching objective, denoising) that fundamentally changes the problematic nature of this minimax game?\n\nSecondly, at a more fundamental level, this paper adopts an unconventional paradigm of variational inference, namely, minimizing the Fisher divergence. This is not a well-studied case (unlike other divergences in the f-divergence family beyond the reverse KL divergence, such as the alpha-divergence). The only other case of this that I've seen is Yang et al. 2019 (https://arxiv.org/abs/1905.05284) which has been cited in this manuscript but is not widely recognized by the community (has it been accepted by a peer-reviewed publication yet?) This facet should be given more attention. Particularly, in this sense, it is difficult to directly draw comparisons between SIVI/UIVI and this method, since it is solving a fundamentally different problem. To what extent are the improvements derived from using this different objective? It seems this manuscript could benefit from some ablation studies.\n\n### Misc questions and issues\n\n\nRegarding footnote 2, why not just write the covariance as the scale-identity matrix \"\\sigma \\mathbf{I}\"?\n\nIn figure 3, in the SIVI-SM pane, the colour of the true and approximate densities are too similar (red and orange, respectively) which makes it far more difficult to discern their differences.  In contrast, the colours representing the other methods in the other panes are quite different to that of the ground truth, which makes the presentation slightly unfair.\n\nIn Section 4.2, regarding the covariance estimates of \\beta (shown in Figure 4) is this simply \\sigma? Please clarify.\n\nTypos: \n- \"log-likeilhood\"\n- \"in term of\"\n- \"get ride of\"\n- \"ground truth _were_ formed\"\n- \"all _the_ three SIVI variants\"\n- number of missing commas all throughout the text. The manuscript could generally benefit from a few more passes of copy-editing.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "KVhUSclmOT", "reviewer": "Reviewer_cW7H", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a variational inference method with fisher divergence as the objective function to enable very flexible implicit variational posterior. The implicit variational posterior is defined as a hierarchical model that can be viewed as an infinite mixture (i.e., the semi-implicit variational posterior). The motivation for using fisher divergence is that one can get rid of computing the density of the hierarchical variational posterior. To achieve this, the paper proposes to rewrite the fisher divergence with its dual norm formulation. The final objective then has a minimax formulation, which results in an adversarial training algorithm. ", "summary_of_the_review": "Due to the weak points I raised, I currently lean towards rejection. However, I am happy to revise my opinion if these are sufficiently addressed. ", "strength_and_weaknesses": "## Strengths\n\n* The idea of rewriting score matching loss using its dual norm is an interesting one---it leads to a nice form that does not require computing the density/score of the hierarchical variational posterior. \n\n* The authors provide theoretical justification of not solving the inner max problem to the optimum. \n\n## Weaknesses\n\n* Although directly using score matching loss as the variational objective is relatively under explored, its close relatives---Stein discrepancies----have been studied extensively. \n\nThere is a known connection between score matching loss and Stein discrepancies, see, e.g., \nLiu, Q., Lee, J., & Jordan, M. (2016, June). A kernelized Stein discrepancy for goodness-of-fit tests. In International conference on machine learning (pp. 276-284). PMLR.\nBarp, A., Briol, F. X., Duncan, A., Girolami, M., & Mackey, L. (2019). Minimum stein discrepancy estimators. Advances in Neural Information Processing Systems, 32.\n\nIt would be nice to contrast the dual form in this paper with Stein discrepancies derived from the same function class (neural networks). There is also work that does this:\n\nRanganath, R., Tran, D., Altosaar, J., & Blei, D. (2016). Operator variational inference. Advances in Neural Information Processing Systems, 29.\n\nGrathwohl, Will, et al. \"Learning the stein discrepancy for training and evaluating energy-based models without sampling.\" International Conference on Machine Learning. PMLR, 2020.\n\n* The connection I mentioned above seems to suggest that the proposed method can be recovered with minimizing the Stein discrepancy (with neural network function class) between variational posterior and true posterior. Could the authors comment on this?\n\n* The literature review in section 1 is missing the work on implicit variational inference that uses gradient/score estimation techniques. See, e.g., \n\nShi, J., Sun, S., & Zhu, J. (2018, July). A spectral approach to gradient estimation for implicit distributions. In International Conference on Machine Learning (pp. 4644-4653). PMLR.\n\nSong, Yang, et al. \"Sliced score matching: A scalable approach to density and score estimation.\" Uncertainty in Artificial Intelligence. PMLR, 2020. \n\n* Appendix D suggests that UIVI is implemented with batch size 1, which means a substantially weaker baseline. I doubt if this is an unavoidable limitation of UIVI. \n\n* Experiments do not include multi-modal target distributions. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "J4AKTJFyVA", "reviewer": "Reviewer_egpL", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a novel score-matching-based method to learn semi-implicit variational distributions. In semi-implicit models, the score of the variational distribution is challenging to estimate directly and Monte Carlo estimates of the score matching objective are biased due to the loss being a non-linear function of the scores. The authors instead re-write the squared distance between the score of the distribution being approximated and the score of the variational distribution as the inner product of the score difference with the output of a learnable function. The authors put forth a ridge regression style learning objective for this auxiliary function which is optimised when the function matches the score difference and thus the exact squared distance between scores is recovered. At the same time, this expression is now only linear in the score of the variational distribution, allowing for learning with unbiased MonteCarlo estimates of the gradients of variational parameters. This results in a min-max objective, where the auxiliary function is trained to approximate the error in the variational distribution’s score and the variational distribution is optimised such that its score matches that of the desired distribution. \n\nThe authors validate their method on some toy 2d densities, 22-dimensional logistic regression and multinomial logistic regression on the MNIST (7850 dimensions) and HAPT (6744 dimensions) datasets. As baselines, Semi-implicit variational inference (SIVI) and unbiased-implicit variational inference (UIVI) are used. The proposed method performs on par or better than baselines on all tasks. Another advantage of the proposed method is that it admits minibatching, unlike UIVI, which relies on HMC updates.  ", "summary_of_the_review": "I like this paper and would like to see it accepted at ICLR. However, in their current state, the experiments seem to be a bit weak for what I would expect from an ICLR paper. If the authors address the issues described above in this regard I would be happy to raise my score to an 8. \n", "strength_and_weaknesses": "**Strengths**\n\n* The paper presents a neat idea that makes sense and seems to work reasonably well\n* The proposed method is clear and easy to follow. \n\n\n**Weaknesses**\n\n\n The experiments are relatively simple and do not cover a very diverse range of cases.\n* All of the non-toy posteriors considered are convex. This does not allow the reader to get an idea of how well the proposed method would fare under multimodal posteriors. \n \n* The UIVI paper validates their approach using VAEs. These models can present a multimodal posterior distribution over the latent space. Perhaps it could be good to replicate this experiment.\n\n\n* The authors motivate their method by observing the biasedness of the MC estimate of the Fisher divergence in equation 7. I think that a small experiment showing the unsuitability of this estimator (even in the appendix if there is no space for it in the main text) would make the argument much more convincing.\n\n* As a practitioner, I always struggle to imagine in which types of situations it would be best to apply a more exotic inference method, like the one described in this paper, instead of HMC. I would suggest that the authors include HMC with a compute allowance matching that of the implicit VI methods as a baseline.\n\t \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-25%