On importance-weighted autoencoders
Authors: Axel Finke
Alexandre H. Thiery
Venue:  (2020)
Keywords: variational inference
autoencoders
importance sampling

Abstract: The importance weighted autoencoder (IWAE) (Burda et al., 2016) is a popular variational-inference method which achieves a tighter evidence bound (and hence a lower bias) than standard variational autoencoders by optimising a multi-sample objective, i.e. an objective that is expressible as an integral over $K > 1$ Monte Carlo samples. Unfortunately, IWAE crucially relies on the availability of reparametrisations and even if these exist, the multi-sample objective leads to inference-network gradients which break down as $K$ is increased (Rainforth et al., 2018). This breakdown can only be circumvented by removing high-variance score-function terms, either by heuristically ignoring them (which yields the 'sticking-the-landing' IWAE (IWAE-STL) gradient from Roeder et al. (2017)) or through an identity from Tucker et al. (2019) (which yields the 'doubly-reparametrised' IWAE (IWAE-DREG) gradient). In this work, we argue that directly optimising the proposal distribution in importance sampling as in the reweighted wake-sleep (RWS) algorithm from Bornschein & Bengio (2015) is preferable to optimising IWAE-type multi-sample objectives. To formalise this argument, we introduce an adaptive-importance sampling framework termed adaptive importance sampling for learning (AISLE) which slightly generalises the RWS algorithm. We then show that AISLE admits IWAE-STL and IWAE-DREG (i.e. the IWAE-gradients which avoid breakdown) as special cases.

Under review as a conference paper at ICLR 2020
ON IMPORTANCE-WEIGHTE AUTOENCODERS
A nonymouu authors
Paper under double-blind review
ABSTRACT
The importance weighted autoencoder (IWAE) Burda et al. 2016 is a popular variational--mlerenee, method which achieves a lighter evidence bound (and hence a lower bias) than standard variational autoencoders by optimising a multi-sample objective, ie. an objective that is expressible as integral over K > Monte Carlo samples. Unfortunately, the [WAE multi-sample objective leads to inference-network gradients which break down as K is increases (Rainforth et al., 2018). This breakdown be circumvented by removing high-variance scoro-function terms, cit by heuristically ignoring them (which yields the "sticking- the-landing WWAE TWAE-STL gradient from Roeder et al. (2017)) ) or through an identity from IWAE (IWAE) DREG] gradient). In this work, we develop an encompassing tramework which directly optimises the proposal distribution in importance sampling in the reweighted wake-sleep (RWS) algorithm from Bornschein & Bengio (2015). From this unified framework, most of the previously proposed gradi- ent estimators can be naturally derived. This permits better understanding the assumptions and trade-offs that are at play. Importantly, the derived gradient estimators guaranteed to not degenerate K cc.
only
Tucker al. (2019) (which yields the
INTRODUCTION
Let æ be some obsorvation and let be some latent variable taking values in some space Z. These are modeled via the generative model po(2, = (0(2)po(z11)) which gives rise to the marginal likelihood pa(x) = S2 po(z, da of the model parameters 0. In this work, analyse algorithms for variational inference, i.e. algorithms which aim to
1. learn the generative model, i.e. find value 0* which is approximately equal to the mazimum--lkklliione estimate (MLE) (0L = arg max Po(x);
2. construct tractable variational approrimation (2) of po(2|x) po(2, x) (p) i.e. find the value o² such that (o)=(z) is as close as possible to p=(2 in some suitable sense.
A few comments about this setting are in order. Firstly, as is common in the literature, we restrict our presentation to single latent representation observation pair (2, to avoid notational clutter the extension to multiple independent observations is straightforward. Secondly, we assume that no parameters are shared between the generative model po (2, and the variational approxination (z.= This is common in neural-network applications but could be relaxed. Thirdly, our setting is general enough to cover amortised inference. For this reason, often refer to as the parameters of an inference network.
Two main classes of stochastic gradient-ascent algorithms for optimising & = (0,0) which employ K Monte Carlo samples ('particles") to reduce errors have been proposed.
The importance weighted autoencoder (TWAE) Burda et al. 2016 max- imizes joint lower bound 23 < p=(z) whose bias decreases as K + 3. The
IWAE
gradients of this objective can be unbiasedly approximated via the Monte-Cane method. Unfortunately, the signal-to-noise ratio of the IWAE d-gradient vanishes as K grows (Rainforth et al. 2018). Two modified [WAE ø-gradients avoid this breakdown by removing high-variance 'score-function" terms:

---

Under review as a conference paper at ICLR 2020
IWAE-STL
The Betcking-the-landiin IWAE (WWAE-STL ø-gradient
Roeder et a.l.
2017
heuristically drops the problematic score- function terms -rggdadeennt. This induces bias for the IWAE objective.
from the [WAE
IWAE-DRE
The 'doubly-repoomeerrseed IWAE (IWAE-DREG o-gradient
Tucker et al.
2019
unbiasedly removes the problematic score-tunction terms
from the IWAE ø-gradient using a formal identily.
RWS. The reweighted wake-sleep (RWS) algorithm (Bornschein & Bengio 2015
optimises two separate objectives for f and Φ. Its gradients are approximated by self-normalised importance sampling with K particles: this induces a bias which vanishes as K + 00. IRWS] can be viewed as an adaptive importanco-sampling approach which iteratively improves its proposal distribution while simultancously optimising 0 via stochastic approximation. Crucially, the RWS d-gradients do not degenerate K-
00.
Of these two methods, the IWAE is the most popular and Tucker et al. 2019) demonstrated empirically that RWS can break down, conjecturing that this is due to the fact that RWS does not optimise joint objective (for 0 and Φ). Meanwhile, the IWAE-STL gradient performed consistently well despite lacking firm thcoretical footing. [WAE suffers from
the above-mentioned o-gradient breakdown and exhibited inferior empirical performance to
RWS Le et al., 2019). Thus, it is not clear whether the multi-sample objective approach of IWAL or the adaptive importancc-sammplnng approach RWS is preferable.
In this work, we show that directly optimising the proposal distribution, e.g. as done by RWWS preferable to optimising the IWAE multi-sample objective because (a) the sample objective typically relies on reparametrisstions and, even if these are available, leads to the d-gradiont breakdown, (b) modifications of the IWAE ø-gradient which avoid this breakdown (i.c. IWAE-STL and IWAE-DREG] can be justified in more principled manner by taking view. This conclusion was already reached by Le al. (2019) based on numerical experiments. They demonstt that the need for reparametrisations can make [WAE] inferrior to RWS) e.g. for discrete latent variables. Our work complements theirs by formalising this argument. To this end, we slightly generalise the RWS algorithm to obtain generic adaptive importance-sampling framework for variational inference which we term ladaptive importance sampling for learning (ATSLE) for ease of reference. We then show that [aaT|AASEEJJaam adSSttsammtts admits not only RWS but also the IWAF-DREG and IWAE-TL gradients as special cases.
an RWS type adaptive importance-samplinn
Contributions. Novel material is presented in Section 3 where we introduce the AISLE framework. From this, most of the previously proposed gradient estimators can be naturally derived in a principled manner. Importantly, the derived gradient estimators are guarantced to not degenerate K- xo. Specifically, we establish the following connections.
We prove that the IWAF-STI gradient can be recovered as a special case of AISLE via a principled and novel application of the tbouble-egarametristtio
identity from Tucker et al. (2019). This indicates that the breakdown of RWS) observed in Tucker al. (2019) may not be due to its lack of a joint objective as previously conjectured [since WAW-STT Lavaoidddddd avoided this breakdown despite having the same idealised objective
as RWS). Our work also provides a thcorctical foundation for IWAE-STL which was hitherto only heuristically justified as biased IWAE gradient.
We prove that ATSLE also admits the [WAE-DREG] gradient a special case. Our derivation also makes it clear that the learning rate should be scaled as O(K) for the IWAE ø-gradient (and its modified version IWAE-DREG] unless the gradients are normalisod ised as implicitly done by popular optimisers such as ADAM (Kingma & Ba 2015). In contrast, the learning rate for AISLE need not be scaled up with of K.
AISLE leads to new family of gradient When applied to the family of a-divergences, estimators that generalises some previously derived in the literature.
In the supplementary materials, we provide insights into the impact of the self- normalisation bias some of the importance-sampling based gradient approxima-
2

---

Under review as a conference paper at ICLR 2020
tions (Appendix A and empirically compare the main algorithms discussed in this
work (Appendix
We stress that the focus of work is not necossarily to derive new algorithms nor to establish which of the various special cases of AISLE is preferable. Indeed, while we compare all algorithms discussed in this work empirically on Gaussian models in the supplementary materials, we refer the reader to Tucker et al. (2019); Le et al. (2019) for an extensive empirical comparisons of all the algorithms discussed this work.
Notation. We repeatedly employ the shorthand p(f) f(2)p(2) dz for the integral of some p-integrable test function f: thus, p(f) = =Ez=plf(2)) if p is probability measure. Furthermore, q² g = II= ((="). To keep the notation concise, we hereafter suppress dependence the observation 2; i.e. write (0(z) (0,x(=) as well
no(2) = Po(2))) = po(2, (0(=))
po(x) Zo
where 70(2) = po(z, x) and where Zg == po(x)= = 700(2) dz.
BACKGROUND
IMPORTANCE SAMPLING
The expectation qs of a test function f: Z + I can be unbiasedly estimated by the quantity [f(=*)++.... + )]/K using set of K particles, z = (z)...,z%)) ~ & which are independent and identically distributed (IID) according qs- Similarly, expectations of the type ne(F) can be approximated by the self nelfansedl importance sampling estimate
K
Wip
and mp(z) = 70(2)
to 2)) = Σ f(*h)
with a E (xt)
g g0(2)
k=1
The notation (0, z) stresses the dependence of this estimator on & and z. The quantity (2+) are called the kth importance weight and w its self-normalised version. For readability, we have dropped the dependence of w on z Z Z from the notation.
Remark 1. Since Zn (ww), the quantity Zo(0.z) z) = [w)(2²) .... ( (2) is an unbiased (fimportance-sampinngg estimator of Zg.
Remark 2. The self normaliedd estimate fio (ff is typically not unbiased. Under mild assumptions (e-g. if sup wa < 00), its bias vanishes at rate O(K-1), its standard deviation vanishes at Monto-Carlo rate OKK1//2) and to (f) + To(f) almost surely as K + co.
2.2 IMPORTANCE WEIGHTED AUTOENCODER (TWAE)
Objective. The importance weighted autoencoder (IWAE) introduced by Burda al.) (2016). seeks to find a value //" of the generalive-model parameters 0 which maximises lower bound CK on the og-marginal likelihood ('evidence'). This bound depends on the inference-network parameters & and the number of samples, K 2 1:
(0", o²) arg max. C
c =5 Zo z)].
(1)
where the expectation is w.r.t. L @K For any finite K, optimisation of the inference- network parameters tightens the evidence bound. Burda et al. (2016) prove that for any
have that LK t log Zg as K 00. If K = 1, the IWAb reduces to the variational aatoennoder (VAE) from Kingma & Welling (2014). However, for K 1, as pointed out in
Cremer al. [2017 1: Domke & Sheldon (2018), the IWAE also constitutes another VAE on an extended space based on an auxiliary-varii variaabe construction developed in Andrieu &
Roberts (209); Andrieu et al. (2010): Lee (201 (see, e.g. Finke 2015 for review).
3

---

Under review as a conference paper at ICLR 2020
Standard reparametrisstion gradient. The gradient of the IWAE AE objective from Vy = V log (o z) + (z), with Gy(z) = log (0, z) _x=1 Vu log (2 The intractable quantity E B (z)) can be approximated unbiasedly via a vanilla Monte Carlo approach using single (K- (Kdimmnsional) sample point (2) 22
go @k Unfortunately, this approximation typically has such a large variance that it becomes impracticably noisy Paislee et al. 2012). To remove this high-variance term, the well known reperamctrisation trick Kingmaa & Welling 2014) is usually employed. It requires the following assumption.
(R1)
There exists distribution qe some space E and family of differentiable mappings h E + Z such that if c ~ 9e we have that 2 = 2(c) = hT(c) ~ 9ø-
Under R1
with e....,,th
e nn
ge and 2 = =((x)) ha(c"), the gradient can be expressed
Vo vo ( V log (o(()))))
V CK
E
V log w wau(th) E W
(2)
with #
(2)). Here, the notation x indicates that one does not differentiate we w.r.t. v. The IWAE then uses a vanilla Monte Carlo estimate of
V Vollog War hollur (
VINAE (0, z) TIWA (0,2)
K Σ
V log no (2(*) V log [x(")))))))))
(3)
Before proceeding, we state the following lemma, proved in Tucker et al. 2019 Section 8.1), which generalises of the well-known identity go
gp(V. (Vo log q6) = O.
Lemma (Tucker et al. (2019)). Under R1 for suitably integrable fo Z - R, we have
go (fv V log q=) qe (V of o holl go (V holle'= h
We now exclusively focus the é-portion of the IWAE gradient, GINIA (8, 2).
Remark 3 (drawbacks of the IWAE (-gradientt.. The gradient @INAA (0, 2) has three drawbacks. The last two of these are attributable the score-function terms V log (2)) in the e-gradient portion of (3).
Reliance on reparametrisations. A reparametrisation à la R1 is necessary to remove the high- term (z(z). For, e.g. discrete, models that violate R1 control-variate approaches (Muih & Rezende 2016) or continuous relaxations have been proposed but these incur additional implementation, tuning and computation costs whilst not necessarily reducing the variance (Le et al. 2019
Vanishing signal-to-nois ratio. The o-gradient breaks down in the sense that
its signal-to-noise ratio vanishes as E[VINEE (0, z)1/ var[Vrwa (0, 2)/ O(K-1/2) [Rainforth al., 2018). This is because PIwAE (0, z) constitutes a self-normalised importance-sampar approximation of To(
V a log 9s) = 0,, an identity which
directly follows from Lemma 1 with fo w
Inability to achieve zcro variance. As pointed out Rooder et al. 201 vanlii (0, 2)] > O even in the ideal scenario where ds na despite the fact that in this case, We is constant and hence varllog (0 z)] 00
Two modifications of SIWAR (B, z) have been proposed which (under R11 avoid the score- function terms in 3 and hence (a) exhibit a stable signal-to-noise ratio as K - co and (b) can achieve zero variance if go = ne (because then = since wy is constant).
IWAE-STL The 'sticking-the-landinn IWAE (IWAE-STL) gradient proposed by Roeder et al. 2017 heuristically ignores the score function terms,
K
IWAE-STL (0,z)
=
(4)
1=1
As shown in Tucker et al. 2019 ) this introduces an additional bias whenever K 1.
4

---

Under review as a conference paper at ICLR 2020
IWAE-DREG The 'doudllyrrprrmmchhisdd" IWAE (IWAE-DREG) gradient pro-
posed by Tucker et al. 2019) removes the score-function terms through Lemma
DRAGEDREGG DREG (A, 2)
(2*).
(5)
The quantities DIWAE-DRES (0. z) and FiNA z) are equal in expectation.
2.3
REWEIGITED WAKE-SLEEP (RWS)
The reweighted wake-sleep (RWS)
algorithm was proposed in Bornschein & Bongio (0055 ) dæ is the Kullback- (KL) divergence from to the RWS algorithm seeks to optimise vi = (0,0) as
Letting g = Iz lo Ip(())
l == arg max, log Zo, {
5 = arg min KT (1
The 0- and o d-gradients read
V 9 log Zo V (L(TTlloo))
Vo log 70 V log qá
(6)
These quantities are usually intractable and therefore approximated by replacing ma by the solf-normalised importance sampling approximation to (é) z) (this does not require R1
E Rows (o. z) RWS
K
V log (2())) log 90 (2*))
(0, =
(7)
Since relies on self- normmnord importance sampling, Remark shows that its bias relative to 6 of order O(1/K). Appendix discusses the impact of this bias on the é-gradient in more detail. The optimisation of both e and & is carried out simultancously, allowing both gradients to share the same particles and weights. Nonetheless, the lack of joint objective (for both 0 and Φ) is often viewed as the main drawback of RWS.
RWS-DREG Under R1 Tucker et al. (201 proposed the following 'doubly
reperametrised RWS (RWS- DREG) gradient which equal to VIEws z) in expectation and is derived by applying Lemma to the latter. reads
BWS- (0, 2)
Σ
(=*))
(8)
where the function f(w)==((1 w) used to transform the self-normalised importance weights wh In high-dimensional settings, it is typically the case that the ordered self- normalised imporlance weights (x()) <... < w (1) are such that i R - पr & (kk
(2) and
(2))) ws < W. for & > 3. The transformed weights (F( ) J== are then mainly supported the two particles with the largest self- normalsddd weights.
AISLE: A UNIFIED ADAPTIVE IMPORTANCE-SAMPIIN FRAMEWORK
3.1 OBJECTIVE
If 0 is fixed, the RWS algorithm reduces to an adaptive importance-sanne sampceinaaaiiinn scheme which optimises the proposal distribution by minimising the 'inclusive' KT.divergence from the target distribution mo to the proposal 4e (sce, c.g., Douc et al. Copp/ et al. 2008). If instead é is fixed, the RWS algorithm reduces to a crrchassii-apppoxiiin
algorithm for estimating the MLE of the generative-model parameters 0. The advantage of optimising
Following Tucker et al. 2019 (based empirical results in Le et al. 2019 only use the
'wake-phase' p-updates for RWS
5

---

Under review as a conference paper at ICLR 2020
and o simultancously is that (a) Monte Carlo samples used to approximate the 8-gradient can be re-used to approximate the Ø-gradient and (b) optimising 6 typically reduces the error (both in terms of bias and variance) of the I-gradient approximation.
However, adapting the proposal distribution qs in importance-sampliinn schemes need not
necessarily be based minimising the (inchusive) KL diveerggee Numerous other techniques
cxist in the litcrature (c.g. Goweke 1989 Evans
1991 Oh & Berger 1992 Richard & Zhang
2007 Cornebise al., 2008 and may sometimes be preterable. Indeed, another popular approach with strong theoretical support is based on minimising the X--divergence (see, e.g., Deniz Akyildiz & Miguez RWS objective as
2019). Based on this insight, we slightly generalise the
00 arg maxg log Zg:
(9)
6 = arg ming Dr( (19o).
Here, Dr(pll4) == (((e))))==( da is some f-divergence from p to 9. We reiterate that alternative approaches for optimising Φ (which do not minimise -divergences) could be used. However, we state (9) for concreteness as it suffices for the remainder of this work; we call the resulting algorithm adaptive importance sampling for learning (AISLE) As will become clear below, this unified framework permits a straightiorward and principled derivation of robust d-gradient estimators that do not degenerate as 00.
3.2 9-GRADIENT
Optimisation is again performed via stochastic gradient- The intractable 0-gradient Vo log Za To (V log is approximatod as in RWS i.c. for ~ 9
SK
K
AISLE (0, 2) = G (o, 2) = SIWAR (0, 2)
W Vo Vologno(=")
The 0-gradient is thus the same for all algorithms discussed in this work although the IWAE paradigm views as an unbiased gradient of (biased) lower-bound to the evidence, while anoorancc-sampllin (and consequently biased approximation the gradient V log Ze for the 'exact' objective.
AISLE (and RWS) interpret it a self-normaliss
3.3
Ó-GRADIENT
3.3.1
GENERAL DERIVATION
In the derivations to follow, integrals of the form (e a wyl V log qe) naturally appear. These can also be expressed as 2 ( w V log go) with (y) Fly)y. By Lemma
ne (LF wyy V log
Exras ww (w))))) (z)]
Zg
Approximating the expectation as well as the normalising constant 2g on the r.h.s. with the vanilla Monte Carlo method with K samples 2!..., ~ 4s
fib SK yields the estimator
K
(o was V gs)
Σ
T H'(ww(=*)) (==))
(10)
Remark 2 shows that this approximation has a bias of order O(K-1) and a standard- deviation of order OKK21/)). Now, most of the f-divergences used for variational in- ference in intractable models are such that there exists a function f:R. R + satisfying Di(Tellgg) 2% Sz (1w)())los))ddz
C(0) for exponent pendent Φ. In other words, for a given value of f, the optimization of the f-divergence as function of á be carried out without relying the knowledge of Z,. Writing g(y) = (v) f(v)/v, simple algebra then directly shows that
R and constant C(0) inde-
Vo Di( = 2 / glws log (=()))o((2)
(11)
6

---

Under review as a conference paper at ICLR 2020
Since the integral in [11 is an expectation with respect to nT, it can be approximated with self- importance sampling, possibly multiplied an additional importance-sampingg approximation Zo z) of Zy raised to some power. This leads to,
K Di ~ L0( + Σ
" @ log (o(").
(12)
Indeed, Equation
10
applies to
11 leading to the reparametrised estimator
K
Dr(To (lo)) ) (o, 2)
Ty 11' h'mp(()))) (=*)),
(13)
where h(y) = g(u)y and g= IR given immediately above particular cases.
11 We now describe several
3.3.2 SPECIAL CASE 'INCLUSIVE' KL-DIVERGENCE RWS AND IWAE-STL
We have (LTT|llg>) zg fz (2) + C(0) with -1 and fly) y log(y). In that case, with the notations of Section 3.3.1 we have g(y) = 1 and (1())= = 1.
AISLE-KI-NOREP RWS. Without relying on any reparamctrisation, Equation
12 yields the following gradient, which clearly equals (0,
V Dr(Fallga)
SAILLEKLL-OORE (0, z)
V log)
(14)
AISLE-KL Using reparametrisation, Equation
13 yields the gradient:
a Dr((Folas) AILLBBK (0,
(2*))
(15)
We thus arrive at the following result which demonstrates that IWAE-STL can be derived in a principled manner from AISLE i.c without the need for a multi-sample objective.
Proposition 1. For any (0, o z), AAISLEKI (0, z) DiwAsssm (0, z).
Proposition is notable because it shows that [WAE-TTL (which avoids the breakdown highlighted in Rainforth et al. (2018) and which can also achiove zero variance) can be derived in a principled manner from AISLE i.c. without relying on a multi-sample objective. Proposition thus provides a theoret basis for IWAE-STL which was previously viewed as an alternative gradient for [WAE for which it is biased and only heuristically justified. Furthermore, the fact that IWAE-STL. exhibited good empirical performance in Tucker et al. (2019) even in example in which RWS broke down, suggests that this breakdown may not be due to RWS lack of optimising a joint objective as previously conjectured.
obtained an alternative 'doubly-roparametriscl RWS d-gradient Via (#, z) given in [8 by first replacing the exact (but intractable) é-gradient by the self-normalised importance-sampling approximation (8,0)(8,2) z) and then applying the identily from Lemma I Note that this may result in a variance reduction but does not change the bias of the gradient estimator. In contrast, AISLE-KT, is derived by first applying Lemma to the exact (RWS) ø-gradient and then approximating the resulting expression. This can potentially reduce both bias and variance.
Finally, recall that Tucker al. (019
3.3.3 SPECIAL CASE a-DIVERGENCE: IWAE-DREG
Up to some irrelevant additive constant, the a-divergence between two distributions and g is given by (=(p())/(<)))) dz for some a > 1. This can also expressed as 25 f (2))()))sss)) dz with = aa and f(y)= y In this case, with the notation from Section 3.3.1 we have g = (a 1)y"~1 and h'(g) = a(a - 1) -- Note that the case a = 2 is equivalent, up to an irrelevant additive constant, to standard x² divergeenee Minimising this divergence natural in importance sampling since x²( (9,) = var=~go ( Zo] I is the variance of the importance weights.

---

Under review as a conference paper at ICLR 2020
AISLE-a-NOREP,
Without relying any reparametrisation, Equation
13
yields
- (0, (a: 1)K-
log go()),
(16)
with the following special case which also proportional to the 'score gradient' from
Dieng et al. [2017 Appendix G): VAISLE- -NOREP
(f. ==
K
< (w
log (a(2").
AISLE-a. Using reparametrisation Equation
12
becomes
A (0, z) = aa(x 1)K
(17)
A
again with the special case AISLEXX x² (0,z) = 2K > kmll
(2*).
This demonstrates that IWAE-DREG can be derived (up the proportionality factor 2K)
in a principled manner from AISLE i.c. without the need for a multi-sample objective.
Proposition 2. For any (0, 00 z), V
AIst (0,z) 2Λ RIWAAARRDDRDS (0,z)
Note that if the implementatiin normalises the gradients, e.g, as ellectively done by ADAM (Kngga & Ba 2015), the constant factor cancels out and AISLE- x² becomes equivalent
to WREG Otherwise (e.g. in plain stochastic gradient-ascent this shows that the learning rate needs be scaled as ((K) for the IWAE or [WAF-DREG d-gradients.
3.3.4 SPECIAL CASE 'EXCLUSIVE' KL-DIVERGENCE.
For the 'exclusive' diverggee we have (T) f(wa()))q dz C(0) with f(y) = log(y). In that case, with the notation from Section 3.3.1 we have h'(y) = 1/y. This directly leads to the following approximation,
K (z*)).
V Dr(Tollls)
AISLE-RREVKK (0, =
This can be recognised as a simple average over K independent replicates of the 'sticking- the-landing" estimator for VAEs proposed in Roeder et al. (2017) Equation 8). As we discuss in Appendix [A] optimising this "exclusive" diverreeee can sometimes lead to faster convergence of ø than optimising the 'inclusive' KL diverreee must be taken because minimising the exclusive divergence does not necossarily lead to well behaved or even well-defined importance weights and thus can negatively affect learning of (whose gradient is a self-normaliser importance-samplnng approximation which makes use of those weights).
(L((Tolqo)) However,
4 CONCLUSION
We have shown that the adlative-impprrancce sampling paradigm of the frewiggted sleke-seepp (RWS) [Bornschein & Bengio 2015) is preferable to the multi-sample objective paradigm of importance weighted autoencoders (IWAEs) Burda et al.) 2016) because the former achieves all the goals of the latter whilst avoiding its drawbacks. To formalise this argument, we have introduced simple, unified iesaptiv-immpotance--ueee
framework termed ladaptive importance sampling for learning (AISLE) (which slightly gencralisos the RWS algorithm and have proved that AISLE allows us to derive the -sstickingghe-llnddltei IWAE (IWAE-STL gradient from Roeder et al. (2117) and the reprrdlg-eprrameeriiedl IWAE (IWAE-DREG)
gradient from Tucker et al.
2019 as special cases.
We hope that this work highlights the potential for further improving variational techniques by drawing upon the vast body of research on (adaptive) importance sampling in the computational statistics literature. Conversely, the methodological connections established in this work may also serve to emphasise the utility of the reparamctrisation trick from Kingma & Welling (2014); TTucker et al. (2019) to computational statisticians.
In companion article, are extending the present work to the variational sequentia) Monte Carlo methods from Maddison et al. (2017); Le et al. (2018): Naesseth et al. (2018)
and to the lensor Monte Carlo approach from Aitchison (2018

---

Under review as a conference paper at ICLR 2020
REFERENCES
Laurence Aitchison. Tensor Monte Carlo: particle methods for the GPU era. ar e-prints, art. (XXi:1880.00893,, Jun 2018.
Christophe Andrieu and Gareth Roberts. The pseudo-marginal approach for elficient Monte Carlo computations. The Annals of Statistics, 37(2):697-725) 2009.
Christophe Andrieu, Arnaud Doucet, and Roman Holenstein. Particle Markov chain Monte Carlo methods. Journal of the Royal Statistical Sociely: Series B (Statistical Melhodooogy), 72(3):269 342, 2010. With discussion.
Robert Bamler, Cheng Zhang, Manfred Opper, and Stephan Mandt. Perturbative black box varialional inference. Advances in Neural Information Processing Systems (NewrIPS), pp. 5079 5088, 2017.
Jörg Bornschein and Yoshua Bengio. Reweighted wake-sleep. In 3rd International Conference on Learning Represenlations (ICLR), 2015.
Yuri Burda, Roger Grosse, and Ruslan Salakhutdinov. Importance weighted autoencoders. In 4th International Conference on Learning Representations (ICLR), 2016.
Olivier Cappé, Randal Douc, Arnaud Guillin, Jean-Michel Marin, and Christian P Robert. Adaptive importance sampling in general mixture classes. Statistics and Computing, 18 (4):447 459, 2008
Julien Cornebise, Éric Moulines, and Jimmy Olsson. Adaptive methods for sequential importance sampling with application to state space models. Statistics and Computing, 18 (4):461-480, 2008
Chris Cremer, Quaid Morris, and David Duvenaud. Reinterpreting importaance-wiighte autoencoders. In 5th International Conference on Learning Representations (ICLR), 2017.
Ömer Deniz Akyildiz and Joaquín Míguez. Convergence rates for optimised adaptive importance samplers. arxi e-prints, art. arXiv:1903.1204: 2019.
Adji Bousso Dieng, Dustin Tran, Rajesh Ranganath, John Paisley, and David Blei. Variational inference via X upper bound minimization. Advances in Neural Information Processing Systems (NeurIPS), pp. 2732 2741, 2017.
Justán Domke and Daniel R Sheldon. Importance weighting and variational inference. Advances in Neural Information Processing Systems (NeurIPS), pp. 4475 4455,,444, 2018
Randal Douc, Arnaud Guillin, Jean-Michel Marin, and Christian P Robert. Convergence of adaptive mixtures of importance sampling schemes. The Annals of Slatistics, 35(1): 420 420-448, 2007.
Michael Evans. Adaptive importance sampling and chaining, Statistical Numerical Inlegration, Contemporary Mathematics, 115:137-143, 1991.
Axel Finke. On exlended state-space constructions for Monte Carlo methods. PhD thesis, Department of Statistics, University of Warwick, UK, 2015.
John Geweke. Bayesian inference in econometric models using Monte Carlo integration. Econometrica, 57(6))1317 1339, 1989.
Edward L Tonides. Truncated importance sampling. Journal of Computational and Graphical Statistics, 17(2):295 311, 2008.
Diederik P Kingma and Jimmy Lei Ba. ADAM: A method for stochastic optimization. In 3rd International Conference on Learning Representations (ICLR), 2015.
Diederik P Kingma and Max Welling, Auto-encoding variational Bayes In 2nd International
Conference on Learning Represenlations (ICL.), 2014.
9

---

Under review as a conference paper at ICLR 2020
Augustine Kong, Jun S. Liu, and Wing Hung Wong. Sequential imputations and Bayesian missing data problems. Journal of the American Statistical Association, 89(425) (0(425):278 1994.
Tuan Anh Le, Maximilian Igl, Tom Rainforth, Tom Jin, and Frank Wood. Auto-cncoding sequential Monte Carlo. In 6th International Conference on Learning Representations (ICLR), 2018.
Than Anh Le, Adam Kosiorck, N Siddharth, Yee Whye Tch, and Frank Wood. Revisiting rewcighted wake-sleep for models with stochastic control flow. In Proceedings of the 35th Conference on Uncertainty in Artificial Intelligence (UAI), 2019.
Anthony Lee. On auciliary variables and many-core architectures computational statistics.
PhD thosis, Department of Statistics, University of Oxford, UK, 2011.
Jun Liu. Metropolized independent sampling with comparisons to rejection sampling and importance sampling. Statistics and Computing, 6(2):113-119, 1996.
Jun S. Liu. Monte Carlo Strategies in Scientific Computing. Springer Series in Statistics. Springer, 2001.
Chris Maddison, John Lawson, George Tucker, Nicolas Heess, Mohammad Norouzi, Andriy Mnih, Arnaud Doucet, and Yee Whye Tch. Filtering variational objectives. Advances in Neural Information Processing Systems (NeurIPS), pp. 6573-6583, 2017.
A. Mnih and D. J. Rezende. Variational inference for Monte Carlo objectives. In 33rd International Conference on Machine Learning (ICML), 2016.
Christian A Naesseth, Scott W Linderman, Rajesh Ranganath, and David M Blei. Variational sequential Monte Carlo. In 21st International Conference on Artificial Intelligence and Statistics (ATSTATS), 2018.
Man-Suk Oh and James O Berger. Adaptive importance sampling in Monte Carlo integration. Journal of Statislical Compulation and Simvulation, 41(3-)):143 168, 1992.
John Paisley, David Blei, and Michael Jordan. Variational Bayesian inference with stochastic search. In 29th International Conference on Machine Learning (ICML), 2012.
Tom Rainforth, Adam R Kosiorek, Tuan Anh Le, Chris Maddison, Maximilian Igl, Frank Wood, and Yee Whye Tch. Tighter variational bounds are not necessarily better. In Bayesion Deep Learning (NeurIPS 2018 workshop), 2018.
Jean- Franiis Richard and Wei Zhang. Efficient high-dimensional importance sampling. Journal of Econometrics, 121(5):1385-141) 2007.
Geoffrey Roeder, Yuhuai Wu, and David K Duvenaud. Sticking the landing: Simple, lower- variance gradient estimators for variational inference. Advances in Neural Information Processing Systems (NewrIPS), pp. 6925-6920 2017.
George Tucker, Dieterich Lawson, Shixiang Gu, and Chris J Maddison. Doubly reparameter- ized gradient estimators for Monte Carlo objectives. In 7th International Conference on Learning Represendations (ICLR), 2019.
Ming Xu, Matias Quiroz, Robert Kohn, and Scott A Sisson. Variance reduction propertics of the reparameterization trick. In The 22nd International Conference on Artificial Intelligence and Statislics (AISTATS), pp. 2711 2720, 2019.
10

---

Under review as a conference paper at ICLR 2020
A ON THE RÔLE OF THE SELI--NRRAALISTTION BIAS WITHIN
RWS AISLE
A.1 THE WALI-NORMAAISAAIIE BIAS
Within the self-normalised importance-sampling approximalion, the number of particles, K, interpolates between two extremes:
As K oo, ()) becomes an increasingly accurate approximation of (f()).
For K = however, to ()) f(=*) reduces to vanilla. Monte Carlo approx- imation of (s( (because the single self-normalised importance weight always equal to 1).
This leads to the following insight about the estimators A AIILLE.KL (0, and A (0,z).
As K t 0o, these two estimators become incrcasingly accurate approxi- mations of the 'inclusive -divergence gradients ~ KL(Tollgs) = no a and ~V (Tellgs) = ( T)), respectively.
For K = 1, however, these two estimators reduce to vanilla Monte Carlo ap- proximations of the 'erclusive' -ddieergeee gradients - V KL [ma) 4s and -2V (L(go = 20x( s), respectively.
This similar to the standard approximation of V (g Ta) if K
TWAE d-gradient which also represents a vanilla Monte Carlo
IWAE reduces to a VAE in this case.
Characterising the small-K K self self-normalisation bias of the reparametriistiinnfres AISLE gradients, AISLE-KL-NOREL andooeaaad/aa. AISLE- X--LEXX is more difficult because if K 1, they constitute vanilla Monte Carlo approximations (2019 Figure 5) lends some support to the hypothesis that the small-K nolf-normalisation bias of these gradients also favours a minimisation of the exclusive KL. divergence.
and
go (Vo log qø) = O. Nonetheless, Le et al.
A.2 INCLUSIVE VS EXCLUSIVE KL DIVERGENCE MINIMISATION
Recall that the main motivation for use of IWAEs (instead of VAEsh was the idea that we could use self-normalised importance-sampp ling approximations with K > particles to reduce the bias of the f gradddinnt relative to Vo log Zg. The error of such (self-normalised) importance-sampling approximations can be controlled by ensuring that q is close to To (in some suitable sense) in any part of the space Z in which vy has positive probability mass. For instance, it is well known that the error will be small if the 'inclusive' K divergeee (T((mellgs) is small as this implies well-behaved importance weights. Tn contrast, a small "exclusive" KL divergence KL(gol RT) is not sufficient for well-behaved importance weights because the latter only ensures that qs is close to πe in those parts of the space in which Is has positive probability mass.
Let Q = }qo} (which is indexed by d) be the family of proposal distributions variational family. Then can distinguish two scenarios.
1. Sufficiently expressive Q. For the moment, assume that the family Q is flexible ('expressive") enough the sense that it contains a distribution gø- which is (at least approximately) equal to ne and that our optimiser can reach the value ø of &. In this case, minimising the exclusive divergeec can still yield well-behaved importance weights because in this case, o= arg min, KL(TT) [gs) is (at least approxinately) equal to arg min KL(qsl]
2. Insufficiently expressive Q. In general, the family Q is not flexible enough in the sense that all of its members are 'far away' from ne e.g. if the D components 2D. of 2 = Z1:D are highly correlated under Ty whilst (=(2)= I[ (2,) is fully factorised. In this case, minimising the exclusive KL could lead to poorly-behaved importance weights and we should optimise @² = arg min, KT (gg)) discussed above.
11

---

Under review as a conference paper at ICLR 2020
Remark 4. In Scenario above, i.c. for sufficiently flexible Q, using gradient-deseent algorithm which seeks to minimise the exclusive divergence can sometimes be preferable to a gradient-descent algorithm which seeks to minimise the inclusive divergence. This is because both find (approximately) the same optimum but the latter may exhibit faster convergence in some applications. In such scenarios, the discussion in Subsection A.] indicates that smaller number of particles, K, could then be preferable for some of the 0-gradients because (a) the O(K-²) self-normaiisationn bias outweighs the 0(K-1/2) standard deviation and (b) the direction of this bias may favour faster convergence.
Unfortumately, simply setting K for the approximation of the ø-gradients is not necessarily optimal because
even in the somewhat idealised scenario [ above and even the direction of the self-normalisation bias encourages faster convergence, increasing K is still desirable to reduce the variance of the gradient approximations and furthermore, even in this scenario, sceking to optimise the exclusive diveeeegeec could lead to poorly behaved importanco-sampling approximations the 8-gradient whenever Φ is still far away from optimal;
using the information contained in all K particles and weights (which have already been
to approximate the 0-gradient) seemns wasteful;
if K = the reparametrisation-fres AISLE d-gradients, AISLE-KL- and AISLE- X are simply vanilla Monte Carlo estimates of 0 and the RWS DREG ø-gradient is then equal to 0.
B EMPIRICAL ILLUSTRATION
B.1 ALGORITHMS
In these supplementary materials, we illustrate the different ø-gradient estimators (recall that all algorithms discussed in this work share the same O 0-gradient estimator). Specifically, we compare the following approximations.
The gradient for AISLE based on the KL divergence without any further reparametrisation from [14] i.e. this coincides with the standard IRWS gradient from 77) This gradient does not require R1 but does not achieve zero
AISLE-KI-NOREP
variance even if 9 mg.
AISLE-KL
The gradient for AISLE based on the KL divergence after reparametris- ing and exploiting the identity from Lemma it is given by (15) and coincides with the [WAF-STT gradient. from
-FE-STT-ggadienn
AISLE-;-NOORI! The gradient for AISLE based the >d-vivereeece without any reparametrisation given in 16 This gradient again does not require R1 but
does not achieve zero variance even if go mg.
AISLE-
The gradient for AISLE based on the x >dddiveree after reparametris-
ing and exploiting the identity from Lemma 1 it is given by 17) and is alsow proportional to IWAE-DREG from Tucker et al. (2019) which was stated in (5) When normalising the gradients (as, e.g. implicibly done by optimisers such as ADAM Kingma & Ba 2015) the proportionality constant cancels out so that both these gradient approximations lead computationally the same algorithm.
IWAE,
The gradient for IWAE employing the reparametrisation trick from Kingma & Welling (2014). Its sampling approximation is given in (3). Recall that this is the d-gradient whose signal-to-noise ratio degenerates with K as pointed out in Rainforth et al. (2018) (and which also cannot achieve zero variance even if 4o mo).
Withhin the IWAE paradigm, using different numbers of particles for the 0 and g-gradients was recently proposed in Rainforth et al. (2018); Le et al. (2018) who termed this approach 'alternating cvidence lower boundy, albeit their aim was to circumvent the signal-to-noise ratio brea breakdown of the IWAE d-gradient which distinct from the phenomenon discussed here.
12

---

gradient from 15 which was Under review as a conference paper at ICLR 2020
IWAE-DREG
The 'doubly-roparametiisd
IWAE
proposed in Tucker et al. (2019). It is proportional to AISLE-x
RWS-DREG
The "doubly-eearramel rised RWS d-gradient from which was Tucker et al. (2019) who derived it by applying the identity from
proposed in Lemma
to the RWS o-gradient.
B.2 MODEL
Generative model. We have N D-dimensional observations r (N) E RD and N D-dimensional latent variables z(1), ,..., (()) RRP. Unless otherwise stated, any vector E RDD is to be viewed as D x column vector.
Hercafter, wherever necessary, we add an additional subscript to make the dependence on the observations explicit. The joint law (the 'generative model"), parametrised by 8, of the observations and latent variables then factorises
II (o (2(()) )po (o(()())))
II Yo, (2(1)). a=1
We model each latent variable observation pair (z, x)) as
Po(z) = N(2; p, 2)),
pe(2|2) == N(x; 2; 1),
where 0: μ MIDD RD, where E (0a,0 (d.dd) (1...))) DxD is assumed to be known and where denotes the D x D-identity matrix. For any 0,
Zz.z = po(x) = N( 1I Σ),
(18)
(z) = pp(z)zz) N(2; vo.x, P).
(19)
with P P= P= and vo.5 P( 1 4 x). In particular, @ML 1
18 implics that
Σ Ex=1z())) x(1)
N
Proposal/variatioaal vvariational approximation. We take the proposal distributions as fully- factored Gaussian:
Proposal/
(0.x = N(2; + bbC),.
(20)
where = (ad,d) DD. E DxD b bIDD E RDD and, for CLD c. E R.D. C
diag(e20: (e.gee............ e2cD) The parameters optimise are thus
Φ = (a. T.bT.cT)
where 0a == [ad,1- .d. | E Dx1 denotes the column vector formed by the elements in the dth row of A. Furthermore, for the reparametrisation trick, we take q(e) N(e;O, where E gD is a vector whose elements are all O, so that
ho. = Ar + C2/2e,
which mcans that (2) 1-2 -x b).
Note that the mean of the proposal in 20 coincides with the mean of the posterior in 19 if A P and b= P5-1p.
This model similar to the one used as a benchmark in Rainforth al. (2018 Section 4) and also in Tucker et al. (019 Section 6.1) who specified both the generative model and the variational approximation to be isotropic Gaussians. Specifically, their setting can be recovered by taking Σ := and fixing
1og(2/3)/2 that C' = I1 throughout. Here, in order to investigate a slightly more realistic scenario, we also allow for the components of the latent vectors to be correlated prrelltedd/ependeent under the generative model. However, as the variational approximation remains restricted to being fully factored, it may fail to fully capture the uncertainty about the latent. variables.
13

---

Under review as a conference paper at ICLR 2020
Gradient calculations. We end this subsection by stating the expressions needed to calculate the gradients in the Gaussian example presented above. Throughout, we use the denominator-layont notation for vector and matrix calculus and sometimes write e = 61:D = h to simplify the notation. Thus,
(z (2-11) E V log n0 (z) = Σ -m)
Vo o log n0 (z) Σ-
(21)
to go (z) -c- b)
= -C TR.
(22)
Let 0g = [dd, Od.2.-.., dd Dxl denote the column vector formed by the elements in the dth row of A. Then, letting @ denote clementwise multiplication and using the convention that addition or subtraction the scalar is to be done elementwise,
V ea log a (2) =
a. ba)a
cxp(-ca)cax E R., de D),
V. log a (z) (z - 1 Ar
C--1/2 RD,
Ve log As. (z) C (/2 Az - 3 C CC2/ Ar-b) bb) -1
= E TR.
Furthermore, write hø.a = [ho.r.1...., hD i.e.
d = 2d = a + ba + exp(ca)<a,
and let ((d) = 10.... O, ....., 0J E E be the vector whose entries are all 0 except for the dth entry which is Then, for d E (1....,)) D),
.,,
V a (c) = ((d d' d' (1....., D),
(23)
= (d) E ERD
(24)
V a))e) exp(ca)ead)) ER
(25)
Again writing == h (2) implies that
Vollog war he.==le==<()) = Wooha......... Voho,..Di(c) Seheeet....Vohh....lee..... log Wa (2),
so that, letting [Vz log (=)a denote the dth element of the vector Vz log (2);
aa [log uaya o ( ( = [V == log wi. (z))asr,
V, llog way o hogall (p))) = V log w (2).
Vellog way he (v(
/ log (2).
From this, since
Vollog o wao he. V Vollog wayz ho V log (z)
we have that
aa [log wae hs (E) = ([V, log w - --11/2
V. flog a Wayr o ha = V. log u (2) - 1/1/2,
V Vollog Waar (s.)((e)
/ log (z) - -E
Impact of the reparametrisation. We end this subsection by briefly illustrating the impact of the reparametrisation trick combined with the identity from Tucker et al. (019 which was given in Lemmal) [ Recall that this approach yields ø-gradients that are expressible as integrals of path-derivative functions there exists a value & such that le.z = Tø. then Wx.r x Tg. /0.z//0.x 1 = is constant so that we obtain varoanci d-gradients (sce, c.g., Roeder ct al. 2017 for discussion this).
v.r Vollog Wwyx ha h Thus, if
For simplicity, assume that Σ = I and recall that we then have gotr = Np.z if the values (A, b, C)) implied by o² are (A (A,b,C")) C*) G 20, 1T)
14

---

Under review as a conference paper at ICLR 2020
By 21 and 22 and with the usual convention (2): we then have
V log Wijn (2) (xr - C
= =2 1 b*) (Az + C-122(C-- c)e].
(26)
Note that the only source of randomness in this expression the multivariate normal random variable e, Thus, by (23 and (24), for any values of and and any K 1. the variance of the A- and b-gradient portion of AISLE-KKL (WAE-STT and AISLE-X IWAE-DREG goes to zero as C + C* = I. In other words, in this model, these sscore-fucctitt function free Φ g-gaadients achieve (near) zero variance for the parameters governing the proposal mean as soon as the variance-parameters fall within a neighbourhood of their optimal values. Furthermore, 25 combined with (26) shows that for any K 2 1, the variance of the C-gradient portion also goes to zero (A,6CC) C*) (A*,b,C* A more thorough analysis of the benefits of reparametrisation-tiic. gradients in Gaussian settings is carried out in Xu et al. (2019).
B.3 SIMULATIONS
Setup. We end this section by empirically comparing the algorithms from Subsection B.1 We run each of these algorithms for a varying number of particles, K E {1, 10, 100), and varying model dimensions, D E {2, 5, 10). Each of these configurations is repeated indopendently 100 times. Each time using a new synthetic data sot consisting of N = 25 observations sampled from the generative model after generating new 'true' prior mean vector as μ ~ N(O,I), I). Since all the algorithums share the same 0-gradient, we focus only on the optimisation of and thus simply fix gth throughout. We show results for the following model settings.
Figure [ The generative model is specified via = In this case, there exists a value o² of such that (2) (z)) Note that this corresponds to Scenario in Subsection A.2
Figure 2 The generative model is specified via = (0.95ld-d'++1) (ed"))
Note that in this case, the fully-factored variational approximation cannot fully mimic the dependence structure the latent variables undor the generative model. That is, in this case, (0.=(2) + (2)) for any values of &. Note that this corresponds to Scenario in Subsection
To initialise the gradient-aseo algorithm, we draw each component of the initial values Φn of & [II according lo standard normal distribution. We use both plain stochastic gradient-ascent with the gradients normalised to have unit Lı-norm (Figures la 2a and ADAM KKngmaa & Ba 2015) with default parameter values (Figures [b] The total number of iterations 10, 000; in each case, the learning-rate parameters at the ith step are 11/2
scenarios with fixed values of cas c.g. as Rainforth et al. (2018): Tucker et al. (0019). However, we omit the results as this did not significantly change the relative performance of the different algorithms. For the
We also the algorithms in each of the
same reason, we omit results related to the optimisation of A and C.
15

---

1
KE3
K
10"
18
10-
A-1
--10
R-
AISE MOBKP
10
18-14
AESLE
AISLEAT
XBRL
BWS-DREG
1871
18
Eteration
Treeation
a. Gradient ascent.
b.
ADAM
Figure 1. Average Li-error of the estimates of the parameters b bI.D governing the mean of the Gaussian variational family. The average taken over the components of and the figure displays the median error at each iteration over 100 independent runs of each a algorithm, each using a different data set consisting of 25 observations sampled from the model. Note the logarithmic scaling on the second axis. Here, the covariance malrix Σ I is diagonal.
3874.
0073
1077
00-14
K: 230
10-
AISEK MOREP
AISERSI
38-1-
TTAILLA VIMIET
ASSLE
107
38"9
IWAE-DRBG
IRABBORE BWS-DREG
0031
300
2000
1091
6300
5000
2003 (30)
93.
2001
690)
500
2000
20.11
08:0
230
JEC
5001
5301
a. Gradient ascent.
ADAM
Figure 2. The same setting in Figure 1 except that here, the covariance matrix Σ (0-95) (d.ete( DY2 is not diagonal matria. Again, note the logarithmic scaling on the second axis.

---

Under review as a conference paper at ICLR 2020
Summary of results. Below, outline what we believe to be the main takeaways from these simulation results for this particular model. However, further theoretical analysis is required to determine whether these hold in more general scenarios.
1. The Sscore-function free' Aldivvereence based A a algorithms typically performed somewhat better than their X² -ddivergence based counterparts, i.e. ATSLE-KT, outper- formed AASLLE-X We conjecture that this is due to the fact that the X diveergeeee based variants square the (self normaiie)) importance weights which increases the variance of the d-gradients.
2.
The performance of the ø-gradients AISLE-KI-NORED and AISLE-X (which do not use any reparamctrisation) typically benefited strongly from moderate (relative to the dimension of the latent variables) increases in the number of particles. In the scenario shown in Figure 2 for larger K, these gradients almost attained the performance of the 'score-fure function free' ø-gradient AISLE- KL IWAE-STI and outperformed AISLE- x TIAA- DREG We conjecture that this is due to the fact that in the scenario shown in Figure the variational family does not include the target distribution, i.e. ds * Ty for any Φ, and as a result, the main advantage of the 'score-function free' gradients - i.c. the fact that they can potentially achieve zero variance cannot be realised.
'score-function
3. The standard IWAE d-gradient performed worse than the other methods in any of the sccnarios considered (except in the trivial case K 1 in which IWAE reduces to the VAE). Indeed, as expected, the performance of the standard IWAE d-gradient consistently worsened with increasing K. This can be attributed
the issue highlighted in Rainforth et al. (2018) (see Subsection 2.2 i.e. to the fact that the signal-to-noise ratio of this gradient vanishes as O(K (2) (as this gradient
constitutes a self-normalised importance-samplinn approximation of an integral which is equal to zero).
4. More surprisingly, the 'score-function free' d-gradients AISLE-KL, IWAE-STL
AISLE-x x [WAF-DREG did not necessarily improve with increasing K. Indeed, their performance sometimes became worse as can be seen most clearly in Figure We note that this cannot be explained by the signal-to-noise ratio decay (which Rainforth et al. (2018) highlighted for the standard IWAE é-gradient) because the 'score-function free' ó-gradients do not constitute self-normalised importance- sampling approximations of integrals which are equal to zero. Instead, we conjecture
that as discussed in Remark 4 in the scenario shown Figure [ the O(K-1) self-normalisation bias of these gradients happens to be beneficial and outweighs the O(K-1/2)
decrease obtained from increasing K.
5. The reabalrrpparaeetriseed [WWS RWS-DREG from Tucker et al. (2019)
and given in (8) performed well for a moderate to large number of particles. Though
it crucially requires K (note that for K 1 this gradient is simply a vector of zeros).
17


Reviews:
[{"review_id": "ryllMzxatr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "UPDATE: bumping up my score after the revisions\n\n---\n\n\nNice connections but novelty and practical takeaways unclear\n\nSUMMARY OF THE PAPER:\n\nThis paper views recent IWAE-based [1] methods (IWAE-STL [2], IWAE-DREG [3], RWS [4, 5]) for training generative models p and inference networks q under a common framework, AISLE.\nThis heavily relies on the \"double-reparameterization\" property by [2] and is restated in Lemma 1.\nThis framework makes it explicit that we're interested in\n1) maximizing the (log) marginal likelihood wrt p parameters, and\n2) minimizing some divergence between the posterior in the learned model to q.\n\nIn AISLE, IWAE-STL's q-gradient is viewed as a doubly-reparameterized self-normalized importance sampling (SNIS) estimate of KL(p || q).\nThis is in contrast to viewing it as a biased estimator of the IWAE's q-gradient.\nThis can potentially explain why it performs well when number of SNIS samples are increased.\nIt is also some evidence against the fact that having no unified objective is bad (because there isn't evidence of IWAE-STL diverging despite there being no unified objective).\n\nIWAE-DREG's q-gradient is viewed as a doubly-reparameterized SNIS estimate of X-divergence(p || q) (up to multiplicative constant of the number of SNIS samples).\nThis is in contrast to viewing it as an unbiased estimator of the IWAE's q-gradient.\n\nThe view on RWS is unchanged: the q-gradient is a SNIS estimate of KL(p || q).\n\nFor me, the main contribution is viewing IWAE-STL and IWAE-DREG q-gradient estimators as biased gradients of an explicit divergence rather than of the IWAE objective.\nI also found the observation that the signal-to-noise (SNR) decrease in IWAE's q-gradient can be proved by noting that it is a SNIS estimator of a zero vector nice.\n\nSTRUCTURE:\nThe article is well-written and easy to understand.\n\nNOVELTY:\nA different view on IWAE-STL and IWAE-DREG is interesting and novel (as mentioned above).\nThis means that IWAE-STL and IWAE-DREG are good not only because they reduce gradient variance (as previously understood) but also potentially because they directly target a divergence.\nViewing generalization of RWS as a main contribution (first bulletpoint of Section 1.2: \"...we show that AISLE admits RWS as a special case.\") is a bit of a stretch since this generalization is very straightforward from the way RWS is formulated.\nThe recommendation of using RWS-style algorithms over IWAE as given in the abstract (\"we argue that directly optimising the proposal distribution in importance sampling as in the RWS algorithm is preferable to optimising IWAE-type multi-sample objectives) is also not novel since this is also advocated by [5] (section 3.2: \"This makes RWS a preferable option to IWAE for learning inference networks because the phi updates in RWS directly target minimization of the expected KL divergences from the true to approximate posterior\").\nThe recommendation as a method for non-reparameterisable latent variables at the end of section 1.2 (\"as well as further algorithms which do not require reparameterisations\") is also given in [5].\nAre there different adaptive importance sampling algorithms that could be used within AISLE that would improve on IWAE-STL/IWAE-DREG/RWS?\n\nEXPERIMENTS:\nThere are no experiments in the main paper.\nHowever, experiments that would support/falsify the following points could be good:\n- RWS and IWAE-STL don't suffer from non-unified objectives because IWAE-STL has non-unified objectives but doesn't diverge,\n- [targeting direct divergence] is more useful than (or as useful as) [lower variance gradient estimators].\n\nCONCLUSION:\nWhile I really like the presentation and connections made in the paper, I'm not sure what the practical takeaways are (other than use IWAE-STL, IWAE-DREG, RWS over IWAE which is advocated by [2], [3], [5]).\nI'm giving this a weak accept due to the former.\nI'm willing to bump up my score if\n- the paper is modified to more accurately reflect the contributions or\n- there are experiments that provide additional support for the [targeting direct divergence] view in addition to [2, 3, 4, 5], or\n- there is a new practical algorithm that the AISLE generalization would suggest that is better than IWAE-STL, IWAE-DREG, RWS in some respects.\n\n[1] Importance Weighted Autoencoders. https://arxiv.org/abs/1509.00519\n[2] Sticking the Landing: Simple, Lower-Variance Gradient Estimators for Variational Inference. https://arxiv.org/abs/1703.09194\n[3] Doubly Reparameterized Gradient Estimators for Monte Carlo Objectives. https://arxiv.org/abs/1810.04152\n[4] Reweighted Wake-Sleep. https://arxiv.org/abs/1406.2751\n[5] Revisiting Reweighted Wake-Sleep for Models with Stochastic Control Flow. https://arxiv.org/abs/1805.10469\n[6] Variational Inference via χ-Upper Bound Minimization. https://arxiv.org/abs/1611.00328\n[7] Tighter Variational Bounds are Not Necessarily Better. https://arxiv.org/abs/1802.04537", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "N/A", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "S1euSLkyqS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThe authors review recent developments in gradient estimators for the IWAE bound and use them to develop a new theoretical justification for the Sticking the Landing (STL) estimator.\n\nUnfortunately, the sole novelty in this paper is a new justification for the STL estimator. The presentation, while thorough, is not novel or particularly clear. There are no experiments. These factors combine to lead me to suggest a reject.\n\nSpecific points:\n* The \"AISLE framework\" is simply used to point out that IWAE and RWS optimize KLs in different directions for the parameters of q. This is well-known in the literature and is discussed in several of the papers cited by the authors. A new framework is not needed to point this out.\n* There seems to be an overall misunderstanding of the difficulties associated with multi-sample objectives. The difficulties with IWAE do not come because it is multi-sample, but because the KL direction optimized for the approximate posterior (q) is from q to p. Thus to compute gradients of the KL we must take gradients back through latent variables sampled from q. RWS avoids this by optimizing the other KL direction, and thus does not need to take gradients through the sampling operation. Many of the issues with IWAE mentioned in the paper also appear with the standard ELBO, which is a single-sample bound. \n* Furthermore, IWAE does not necessarily require reparameterizations to deal with the high variance of its terms. Control variates can be used when latent variables are discrete, e.g. Mnih et. al 2016 \"Variational inference for Monte Carlo objectives\" and Tucker et al. 2017 \"REBAR: Low-variance, unbiased gradient estimates for discrete latent variable models\". \n* RWS can still be thought of as a multi-sample objective in the sense that you use multiple samples of z to estimate the gradient for one data point x. The true difference is that the RWS gradient estimator is an asymptotically consistent estimator of the gradient of the marginal likelihood, while the IWAE gradient estimator is an unbiased estimator of the gradient of an objective (IWAE lower bound) that becomes the marginal likelihood in the limit of infinite samples.\n* As such, your claim to \"have shown that the adaptive-importance sampling paradigm of the reweighted wake-sleep is preferable to the multi-sample objective paradigm of importance weighted autoencoders\" is far too strong, especially considering the fact that experimental evidence in Tucker et al. 2018 shows there are situations where either one is preferable. Additionally, the DReGS estimator avoids 2 of the 3 issues you present in remark 1.\n* A smaller point: I found it hard to follow your derivations when compared with Tucker et al. 2018 because their identities use expectations over standard gaussian noise (epsilons) while expectations in your paper are all written with respect to q (e.g. the right-hand side of lemma 1). It would be helpful to go more in-depth about why that is.\n\nTo change my mind the authors would have to include experimental evaluation of some kind and demonstrate more novelty.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "BygWq_BCFr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: This paper presents a unifying framework through which much of the recent work on maximum likelihood learning in latent variable models (variational autoencoders) via multisample variational approaches and importance weighted approaches can be understood. It is a relatively clean framework that shows how many of the popular approaches can be described a distinct gradient-based approaches for a single underlying framework with two separate objectives for the generative model weights and the variational posterior weights.\n\nStrengths: \n- The framework is elegant and the derivations are simple. \n-  It clarifies the connection between distinct algorithms and shows the consistency of ones that were previously poorly understood (IWAE-STL).\n- It has the potential of generating new interesting algorithms.\n\nWeaknesses:\n- It is not clear to me why the first bullet of Remark 1 is so crucial. IWAE can be applied in discrete settings (see, e.g., Mnih & Rezende, 2016) and, as you show, reparameterizations can be applied in the adaptive importance sampling type algorithms to potential improve the variance.\n- The paper currently struggles with some organizational issues. It reads as if the paper was written as a 15 page paper and split in half to satisfy the length requirements. In particular, I would recommend shortening the introduction, cutting out as much of Sec 2 as possible, and moving experiments into the main draft. Derivations are fine left in the Appendix. \n- I appreciate that it wasn't the primary aim of the paper to introduce new algorithms, but I think it could strengthen the contribution to consider at least a few. Are there any other interesting divergences to consider for the proposal distribution objective?\n- The experiments are quite lacking. In tandem with the above point (consider novel algorithms), the ICLR community might rightfully expect some experiments on large scale models.  I appreciate that there might not be much consistency in terms of which methods outperform others, but large scale experiments would at least present evidence of this point.\n\nCitations:\nMnih & Rezende, 2016. https://arxiv.org/abs/1602.06725", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors argue that directly optimizing the IS proposal distribution as in RWS is preferable to optimizing the IWAE multi-sample objective. They formalize this with an adaptive IS framework, AISLE, that generalizes RWS, IWAE-STL and IWAE-DREG. 

Generally reviewers found the paper to be well-written and the connections drawn in this paper interesting. However, all reviewers raised concerns about the lack of experiments (Reviewer 3 suggested several experiments that could be done to clarify remaining questions) and practical takeaways. 

The authors responded by explaining that "the main "practical" takeaway from our work is the following: If one is interested in the bias-reduction potential offered by IWAEs over plain VAEs then the adaptive importance-sampling framework appears to be a better starting point for designing new algorithms than the specific multi-sample objective used by IWAE. This is because the former retains all of the benefits of the latter without inheriting its drawbacks." I did not find this argument convincing as a primary advantage of variational approaches over WS is that the variational approach optimizes a unified objective. At least in principle, this is a serious drawback of the WS approaches. Experiments and/or a discussion of this is warranted.

This paper is borderline, and unfortunately, due to the high number of quality submissions this year, I have to recommend rejection at this point.

Decision: Reject