Nested Variational Inference
Authors: Heiko Zimmermann
Hao Wu
Babak Esmaeili
Jan-Willem van de Meent
Venue: NeurIPS 2021 Poster (2021)
Keywords: Variational Inference
Importance Sampling
Monte Carlo methods

Abstract: We develop nested variational inference (NVI), a family of methods that learn proposals for nested importance samplers by minimizing an forward or reverse KL divergence at each level of nesting. NVI is applicable to many commonly-used importance sampling strategies and provides a mechanism for learning intermediate densities, which can serve as heuristics to guide the sampler. Our experiments apply NVI to (a) sample from a multimodal distribution using a learned annealing path (b) learn heuristics that approximate the likelihood of future observations in a hidden Markov model and (c) to perform amortized inference in hierarchical deep generative models. We observe that optimizing nested objectives leads to improved sample quality in terms of log average weight and effective sample size.

Nested Variational Inference
Heiko Zimmermann' w.z.mermannnuua.a... ml
Hao Wu"
wu ww..haooonorrthhastern edu
Babak Esmaeili b. e.eemaeill Mi (uui@.va.... ml
Jan Willem van de Meent |*
vandemeent@uva. nl
Institute of Informatics, University of Amsterdam KKoorry College of Computer Sciences, Northeastern University
Abstract
We develop nested variational inference (NVI), a family of methods that learn proposals for nested importance samplers by minimizing an forward or reverse KL divergence at each level of nesting. NVI is applicable to many commonly-un importance sampling strategies and provides mechanism for learning intermediate densities, which can serve as heuristics to guide the sampler. Our experiments apply NVI (a) sample from multimodal distribution using learned annealing path (b) learn heuristics that approximate the likelihood of future observations in hidden Markov model and (c) to perform amortized inference in hierarchical deep generative models. We observe that optimizing nested objectives leads to improved sample quality in terms log average weight and effective sample size.
Introduction
Deep generative models provide a mechanism for incorporating priors into methods for unsupervised representation learning. This particularly useful in settings where the prior defines an inductive bias that reflects the structure of the underlying domain. Training models with structured priors, however, poses some challenges. standard strategy for training deep generative models is to maximize reparameterized evidence lower bound with respect to both the generative and an inference model [Kingma, Welling, 2013; Rezende et al., 2014]. This approach works well variational autoencoders with isotropic Gaussian priors, but often fails for models with more structured priors or likelihoods.
In recent years, range of strategies for improving upon standard reparameterized variational inference have been put forward. These include wake-sleep style variational methods that minimize the forward KI--divergence [Bornschein, Bengio, 2015; et al., 2019] as well as sampling schemes that incorporate annealing [Huang et al., 2018]. Sequential Monte Carlo [Le al., 2018; Nacsseth et al., 2017; Maddison et al., 2017], Gibbs sampling [Wu et al., 2019; Wang et al., 2018], and MCMC updates [Salimans et al., 2015; Hoffiman, 2017; Li et al., 2017]. While these methods offer flexible inference, typically resulting in better approximations to the posterior compared to traditional variational inference methods, they either model-specifie, requiring specialized sampling schemes and gradient estimators, or can not be easily composed with other techniques.
In this paper, we propose nested variational inference, framework for combining nested importance sampling and variational inference. Nested importance sampling formalizes the construction of proposals by way of calls to other importance samplers [Naesseth et al., 2015; Naesseth et al., 2019]. Many existing importance samplers are instances of nested samplers, including methods based on annealed importance sampling [Neal, 2001] and sequential Monte Carlo [Del Moral al., 2006].
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

NVI learns proposals by optimizing a divergence at each level of nesting. Additionally, we combine nested variational objectives with importance resampling to further improve sampling quality, without the need to undergo extra steps to maintain differentiability due to the local nature of the objective. Resampling also allows to compute gradient estimates based on incremental weights, which depend only on variables that are sampled locally, rather than on all variables in the model. Doing yields lower variance weights, whilst maintaining a high sample diversity relative to existing methods.
2 Background
2.1 Stochastic Variational Inference.
Stochastic variational methods approximate target density T(;; = = with parameters using variational density {(2; o) with parameters Φ. Two common variational objectives the forward and reverse Kullback-Leibe (KL) divergence, which are both instances of f-divergences
((((::2) B q
D, (TT 11 ()
with fww) = w log w and f(w) = log w respectively. We are typically interested in the setting of model with latent variables and observations x. this case, 7 0) = pa(a, 2) is the joint density of the model, and Z = =p( is the marginal likelihood.
where π (2; 0) is the posterior (= æ))
Reverse KL-divergence. When optimizing KL [))). known as the reverse or exclusive KL divergence, is common practice to maximize a lower bound [ = (log(l////// = log - x) with respect to 0 and é. The gradient of L can be approximated using reparameterized samples [Kingma, Welling, 2013; Rezende et al., 2014], likelihood-ratte estimators [Wingate, Weber, 2013; Ranganath et al., 2014], combination of the two [Schulman et al., 2015; Ritchie et al., 2016a].
Forward KL-divergence. In the case of the forward divergence KL (z || q). also known as the inclusive KL divergence, stochastic variational methods typically optimize separate objectives for the inference and generative model. A common strategy is to train the generative model by either maximizing the ELBO or likelihood on the data. Optimizing the inference model requires samples from म, which itself requires approximate inference. A common strategy, which popularized in the context reweighted wake-sleep (RWS) methods [Bornschein, Bengio, 2015; Le et al., 2019], is to use q proposal in importance sampler.
2.2 Importance Sampling
Self-Normalized Importance Samplers. An expectation [o])]] with respect to π can be rewritten with respect to a proposal density g by introducing an unnormalized importance weight w,
1 Z
72;0) 0)
E
[=]=
E w 9())],
q(2;
Self-normalizer estimators weighted samples with respect to g, and to compute an estimate Z of the normalizing constant,
both approximate the expectation
$ (c) S
=
1 2 Σ (o(e) g(2(0)),
2
200 ~ g(:
(3)
The resulting estimator is consistent, i.e. 9 و/ E,z [g(2)) the number of samples S increases. but it is not unbiased, since follows from Jensen's inequality that 1/Z = 1/ Eg < E, If the discrepancy between and π is large, the importance weights will have large variance, which also has an impact on the bias. When g = म, the importance weight w = has zero variance, and the Jensen's inequality is tight. In the context of stochastic variational inference, this means that gradient estimates might initially strongly biased, since there will typically be large discrepancy between q and However, the variance will typically decrease as the quality of the approximation improves.

---

Ti (21)
T2 (2()
ii' (2x)
Ta (24))
(22) 23)
Is Rs x2)
2a (ze kgg
rı (21 (22)
r2(22 (za)
ra(23 =24)
Dy (#2r1 (92m1)
Dy (mar (3T2))
D (mar's (4mT))
Figure Nested variational inference minimizes an f-divergence at each step in a sequence of densities to learn forward proposals yk: reverse kernels 1 11, and intermediate densities we-
Nested importance sampling and proper weighting Nested Importance sampling formalizes the construction of proposals by way of calls to other importance samplers. Extending the example above, we now assume is some unnormalized density itself that we cannot directly generates samples from. We can employ another importance sampler with proposal 7) to simulate samples from q.
9(=");)
v),
w
2, T. Σ w
,)("));)))
can subsequently used compute the normalizing constant Zg of q and to generate samples that are approximately distributed w.r.t q via resampling. This allows us to use q as a proposal in a subsequent importance sampling step. whose weighted samples again be used to compute consistent estimates w.r.t. the target density of interest This can be generalized, by introducing the notion of proper weighting.
The resulting weighted samples (20)) )
Definition 2.1 (Proper weighting). Let be a probability density. For some constant 0, random pair (w, z) ~ is properly weighted (p. w.) for an unnormalized probubility density 7 = ZT w and all measurable functions g it holds that
= c f dz ((2) g(2) cZ [g(z))
E
[w g(2)]
01,2 -1
2
As long as we can generate properly weighted samples for the target density x from some sampler II, we compute consistent estimates
Ex w(+)
g(2(e))
o TR E2aT [g(2)]
3
[g(2)]-
(4)
E w(e)
e
2009
3 Nested Variational Inference
widely used strategy in importance sampling is to decompose difficult sampling problem into a series of easier problems. A common approach is to define a sequence of unnormalized densities {nx)Æ= that interpolate between an initial density тТ = 71/Z1, for which sampling is easy, and the final target density тк YK/ Tk/ZKr ZZ.. At each step, samples from the preceding density serve to construct proposals for the next density, which is typically combined with importance resampling or application of Markov chain Monte Carlo (MCMC) operator improve the average sample quality.
Nested variational objectives. NVI defines objectives for optimizing importance samplers that target a sequence of densities. At every step, it minimizes the discrepancy between a forward density i = 5x/ /Zk -1 which acts as the proposal, and reverse density मं 5 which defines an intermediate target. We define the forward density by combining the preceding target 7 with forward kernel qk. and the reverse density by combining the next target 7/. with a reverse kernel rk-
(Z =k 1) = (2x) Tk 1
in (Zz, 1x-1) qu (=x|=x-1) 7h- (z).
(5)
Our goal learn pairs of densities मe and सt. that are similar as possible. To do so, we minimize a variational objective 'D that comprises an f-divergence for each step in the sequence, along with divergence between the first intermediate target कi and an initial proposal Q1. Concretely, the NVI objective can be formulated as,
K
D = Dy(TI 91) + Σ
Dy mx).
(6)
k=
3

---

Since each intermediate density Tr occurs in both t. and in + this defines a trade-off between maximizing the similarity T and the similarity to TT-1- To optimize this objective, we need to be able to simulate samples from the intermediate densities.
Sampling from intermediate densities. Given pair (#k-1; that is properly weighted for the previous target density 7k-1, we can use sequential Monte Carlo sampling [Del Moral et al., 2006] construction define a pair (wa, 2x) that properly weighted for 7/k.
3is (2k #k-
2k q.( zk-1),
wk Uj Wk-1,
Uk
Fn (Fx: 2k
We refer to the ratio Us as the incremental weight. In this construction, (wk; 2k-1:k) is properly weighted for ty which implies that (wk, 2x.) is also properly weighted for Yks since
f d (2ks 2k-1) - / -dzz- 7/ (2.
dzx-1
(2x -1 (x,) = (zx))
(8)
Sequential importance sampling can be combined with other operations that preserve proper weight- ing, including rejection sampling, application of an MCMC transition operator, and importance resampling. This defines a class of samplers that admits many popular methods special cases, including sequential Monte Carlo (SMC) [Doucet et al., 2001], annealed importance sampling (AIS) [Neal, 2001], and SMC samplers [Del Moral et al., 2006]. In this work we only consider variational transition kernels and importance resampling. These samplers vary in the sequences of densities they define. For example, AIS and SMC samplers both define intermediate density over fixed common domain. Here, common strategy is to define an annealing path YYk >x<=z) 71 (==)*=Pp y (zx) )0x for == 31 < 82 < ... 3x = 1. In contrast, when using SMC for spte-space models we can define the intermediate target T to be the the filtering distribution on the first k states. In this setting, the
dimensionality the support increases at each step and we can omit the reverse kernel rk-1- We are exploring both of these cases in Section 4.
3.1 Computing Gradient Estimates
The NVI objective can be optimized with respect to three sets of densities. We will use 6x. dz and oR to denote the parameters of the densities Tk. 9k, and n respectively. For notational convenience, we use Oæ: = {As, Φ } to refer to the parameters of the reverse density # and PJ. ( - to refer to the parameters of the forward density Rk.
Gradients of the Forward KL divergence. When we employ the forward KL as the objective, the derivative with respect Pu can be expressed (see Appendix E.3),
KL TR ml 0pp
#x)
E 3x
log 7n (Ek: px)1; Opc
IE Wk-1
2 Opk
log 7h- (zx 1: 1 00-
(9)
This case is the nested analogue of RWS-style variational inference. We can move the derivative into the expectation, since 7t. does not depend on Ω:- We then decompose log मि = log 24 log - and use the identity from Equation 21 to express the gradient log Z as expectation with respect to -- The resulting expectations can be approximated using norf-normalized estimators based on the outgoing weights wt and incoming weights w respectively.
The gradient of the forward KL with respect to Oe is more difficult to approximate, since the expectation is computed with respect to t₁. which depends the parameters Pv. The gradient of this expectation has the form (see Appendix E.3)
Opk
KL (TT
#K)
=
E #k
log Uh
Opk
log th (2k, 2k 2k Pk)
(10)
a
a
E 22
log
UK
log in (Zk, 1i Op..
log 2kk
log 7M (zxi 0x)
A2
Opk
In principle, we can approximate this gradient using self-normalized estimators based on the outgoing weight wi- We experimented extensively with this estimator, but unfortunately we found it to be unstable, particularly for the gradient with respect to the parameters of the reverse kernel r For this reason, our experiments employ the reverse KL when learning reverse kernels.

---

Our hypothesis is that the instability in this estimator arises because the gradient decreases the probability of high-weight samples and increases the probability of low-weight samples, rather than the other way around. This could lead to problems during early stages of training, when the estimator will underrepresent low-weight samples, for which the probability should increase.
Gradients of the Reverse KL divergence. When computing the gradient of the reverse KL with respect Pk. we obtain the nested analogue of methods that maximize a lower bound. Here we can either use reparameterized samples [Kingma, Welling, 2013; Rezende et al., 2014] or likelihood-ratio estimators [Wingate, Weber, 2013; Ranganath al., 2014]. We will follow Ritchie et (2016b) and define a unified estimator in which proposals are generated using construction
Wk U/ 11 Fk g( Φk), ms (Ex Φu), Wk-1, 2k-1
11g-1-
(11)
This construction recovers reparameterized samplers the special case when ÿj does not depend on parameters, and recovers non non-reparameterized samplers when 2j; .μ. This means it is applicable to models with continuous variables, discrete variables, or a combination of the two. The gradient the reverse KL for proposals that are constructed in this manner becomes (see Appendix E.2)
a
Ozk
KL (tx 0pk
|| #x)
= E
log 7K (2k, 2* 1: 0pp
azk
a
+
E
log Opk
log 7k (2k 1k Ph)
(12)
8 Opp
E
log Uk
E
log 1 (2-11(2-
In this gradient, the first term represents the pathwise derivative with respect to reparameterized samples. The second term defines a likelihood-ratio estimator in terms of the unnormalized density Ie. and the third term computes the contribution of the gradient the log normalizer log -
Computing the gradient of the reverse KL with respect to 5 is once again straightforward, since we are computing an expectation with respect to 7.. which does not depend on pk- This means we can move the derivative into the expectation, which yields gradient analogous to that in Equation 9,
a KL 0p
a
a Opp
#k
=
log 4ne Tu pp
fh (Zk, 1k-
E si
log 7k (Exi 02)
(13)
Variance Reduction. To reduce the variance of the gradient estimates use the expected log- incremental weight as baseline for score function terms and employ the sticking-the-landing trick [Roeder et al., 2017] when reparameterizing the forward kemne as described in Appendix E.
3.2 Relationship to Setertance-Weighttee and Self-Normalized Estimators
There exists large body of work on methods that combine variational inference with MCMC and importance sampling. We refer to Appendix A for comprehensive discussion of related and indirectly related approaches. To position NVI in the context of the most directly related work, we here focus on commonly used importance-weighter and self- normalized estimators.
NVI differs from existing methods in that it defines objective for pairs of variables (zk, # each level of nesting, rather than a single objective for the entire sequence of variables (21, 2,, z A standard approach for combining importance sampling and variational inference is to define "importance-weigheee lower bound Lx log 2x [Burda et al., 2016]. By Jensen's inequality, E[Ck] log k| log Zx. which implies that we can define stochastic lower bound using any properry-weightee sampler for 7k. including samplers based on SMC [Le et al., 2018; Naesseth al., 2018; Maddison al., 20. For purposes of learning the target density 7K, this approach is equivalent to computing an RWS-style estimator of the gradient,
1
11
ΣK =
w a0
log 7K (2)))xx))
(14)
00
ZR
S
However, these two approaches are not equivalent for purposes of learning the proposals. We can maximize stochastic lower bound to learn gk. but this requires roubbe-reparamettriieed estimators
5

---

SIS
Pg Fig Vic-1
SMC
Mar-1
Weight contributions in the self-normalizer gradient estimators for the forward KL- divergence for VI and NVI using SIS (no resampling) and SMC (resampling). computes gradient estimates using the final weights (SIS), which simplify to the final incremental weight when resam- pling is performed (SMC). NVI computes gradient estimates based on the intermediate weights (SIS), which simplify to the intermediate incremental weights when resampling is performed (SMC).
Figure 2:
[Tucker et al., 2018] in order to avoid problems with the signal-to-noise ratio in this estimator, which can paradoxically deteriorate with the number of samples [Rainforth et al., 2018]. The estimators in NVI do not suffer from this problem, since we do not compute the logarithm of an average weight.
NVI is also not equivalent learning proposals with RWS-style estimators. If we use sequential importance sampling (SIS) to generate samples, a self- normalized gradient for the parameters of gk that is analogous to the one in Equation 24 has the form
E
log (=k 2k-1: 26)
K
log 9k (2k 2 Φx). (15)
KK-KK
ZK
Note that this expression depends on the final weight WK. By contrast, a NVI objective based on the forward KL yields self-normalized estimator that is defined in terms of the intermediate weight wa
E Th
log qr (=x - 2x) Oib
a
1 2x
Σ
w a log qu (=f. = dz)-
(16)
Oáv
If instead of SIS we employ seguential importance resampling (i.e. SMC), then the incoming weight w identical for all samples. This means that can express this estimator in terms of the incremental weight rather than the intermediate weight Wk
2 2 U 0Φp
log q (2 1 2x)
log (x 1 2x).
(17)
see that NVI allows us to compute gradient estimates that are localized to a specific level of the sampler. In practice, this can lead to lower-varianee gradient estimates.
Having localized gradient computations also offers potential memory advantages. Existing methods typically perform reverse-mod mode automatic differentiation on an objective that is computed from the final weights (e.g. the stochastic lower bound). This means that memory requirements scale as OSS)) since the system needs to keep the entire computation graph in memory. In NVI, gradient estimates level do not require differentiation of the incoming weights w This means that is possible to perform automatic differentiation on a locally-defined objective before proceeding to the next level of nesting, which means that memory requirements would scale as OSS). should therefore in principle be possible to employ a large number of levels of nesting K in NVI, although we do not evaluate the stability NVI at large K our experiments.
4 Experiments
We evaluate NVI on three tasks, (1) learning to sample form an unnormalized target density where intermediate densities are generated using annealing, (2) learning heuristic factors to approximate the marginal likelihood of future observations in state-space models, and finally (3) inferring distributions over classes from small numbers of examples in deep generative Bayesian mixtures.
4.1 Sampling from Multimodal Densities via Annealing
common strategy when sampling from densities with multiple isolated modes is to anneal from an initial density 71, which is typically unimodal distribution that sample from easily, to the

---

AVO
NVIR
a(aa()))
bg ~ (((l)
(1(-(1))))) su (((1)))))))
(((1)))))
Re ~ 9et-1zs)
by sx(-
(x(-11)))
Figure 3: Samples from forward kernels trained with AVO, and NVIR*.
(a)
(b)
1.0
Limorr
NVI
SVI
BSS/LL
0.0
NVIR
VSMC
10a
KT(mk || Tk
Linear NVIS NVIR
ESS /L.
10
2
3
5
10k 20k
10k:
20h
Figure 4: (a) ESS relative to the number of samples 36 during training for different methods using pairs of transition kernels (sequence length K 8) averaged across 10 independent Error bars indicate +2 standard deviations; mean and standard deviation are computed based rolling average with window size 100. (b) (Top) Annealing paths learned by NVT" and NIVR* * and the linearly spaced geometric annealing schedule (Linear) used by AVO, NVI, and NVIR. Results averaged over 10 restarts; error bars indicate two standard deviations. (Bottom) Numerically computed KL-divergences between consecutive intermediate distributions for different schedules.
target density 7K. which is multimodal [Neal, 2001]. Recent work on annealed variational objectives (AVOs) learns forward kernels qu and reverse kernels Fk-1 for an annealing sequence by optimizing a variational lower bound at each level of nesting [Huang et al., 2018].
L TT
LAVO
E Q1.-..Akk
log vR): (x(=) (<x))"=*>x())*> k= 1, K. (18)
NVI allows us to improve upon AVO in two ways. First, at every step to optimize an SMC sampler rather than an annealed importance sampler. Second, we can learn the annealing schedule ($I...,, 3x) ) as part of the intermediate densities % such that .1....,By2
perform importance resampling
intermediate densities are scheduled more equidistantly in terms of KL-divergence.
We illustrate the effect of these two modifications in Figure 3, in which we compare AVO to NVI with resampling and learned path, which we refer to NVIR". Both methods minimize the reverse KL at each step The learned annealing path in NVIR* results in smoother interpolation between the initial and final density. We also see that AVO does not assign equal mass to modes, whereas NVIR" yields more even distribution.
In Figure 4a we compute the reverse KL between targets each step the sequence. For the standard linear annealing path, the KL decreases with kt suggesting that later intermediate densities are increasingly redundant. By constrast, NVIR* we see that the KL approximately constant across the path, which is what we would expect when minimizing D with respect to Bg. This is also the case an ablation without resampling, which we refer to as NVI*.
As noted in Section 3.1, 1, we found optimization of the forward KL to be unstable when learning Px-

---

log 2
(log 2.08)
ESS
Seq. length
K=6
K=8
K=2
K=4
K=6
GT
SVI
1.86
1.89
1,92
1,72
51
47
32
25
-lw
2.06
55
AVO
2.01
2,05
NVI
1.86
1.97
2,03
2,06
51
45
45
41
NVIR
1.86
1.98
2.04
2,06
51
99
NVIR*
NVI"
1.86
2.06
2,07
2.07
51
NVIR
1.86
2.06
2,07
2,08
51
94
97
97
AVO-Bow
2.07
2,08
-VIo-flow
2.05
2,08
2,08
2,08
28
81
79
70
Table 1: Sample efficiency for NVI variants and baselines for K - annealing steps and L samples per step for fixed budget of K L 288 samples. Metries are computed for 100 batches of 100 samples per model across 10 restarts.
Figure 5: Ground truth densities (GT) and samples from final target density for NVIR* with 2 interme- diate densities (K=4).
Figure 4b shows rolling average of the ESS and its variance during training. We compare NVI-based methods to SVI and a variational SMC sampler [Le et al., 2018; Maddison et al., 2017: Naesseth et al., 2017]. NVIR* * has consistently higher ESS and significantly lower variance compared to baselines. These plots also provides insight into the role of resampling in training dynamics. In NVI", observe cascading convergence, which is absent in NVIR". We hypothesize that resampling reduces the reliance on high-quality proposals from step 1 when estimating gradients at step k.
Annealed NVI has similar cases as normalizing flows [Rezende, Mohamed, 2015]. Inspired by concurrent work of Arbel et al. (2021), which explores a similar combination of SMC samplers and normalizing flows, we compare flow-based versions of NVI to planar normalizing flows, which maximize a standard lower bound (SVI-flow). We find that a normalizing flows can be effectively trained with NVI, in that samplers produce better estimates of the normalizing constant and higher ESS compared to SVI-flow. We also find that flow based models are able to produce high-quality samples with fewer intermediate densities (Figure 8). Moreover, we that combining a flow-based proposal with learned 3n values (NVI'-flow) results in a more accurate approximation of the target than in ablation with linear interpolation path (AVO-flow).
In Table I we report sample quality in terms of the stochastic lower bound 2x log 2x and the effective sample size ESS = EE, wx )2 E, (wx))² ) The first metric can be interpreted as a measure of the average sample quality, whereas the second metric quantifies sample diversity. We compare NVI with and without resampling (NVIR* and its NVI*) to ablations with a linear annealing path (NVIR and NVT), AVO, and standard SVI baseline in which there no intermediate densities. We additionally compare against AVO-flow and -fo'-flow, which employ flows. We observe that NVIR and NVI*-flow outperform ablations and baselines in terms of log Z, and are competitive in terms of ESS. We show qualitative results for two additional target densities in Figure 5. For more details we refer to Appendix G.1.
4.2 Learning Heuristic Factors for State-space Models
Sequential Monte Carlo methods are commonly used in state-space models to generate samples by proposing one variable at a time. To do so, they define a sequence of densities F = 7x/2, /2g on the first & time points in a model, which are also known filtering distributions,
Th (2 m) = p( Zi.k, 7) = p(r) p( =1 m) II p(r,, 21 21-1,7), Zx = pp=(()))) (19)
Here 2i:k and Fisk are sequences of hidden states and observations, and is a set of global variables of the model. The densities " differ from those in the annealing task in Section 4.1 in that the dimensionality of the support increases at each step. We define forward density 9y that combines the preceding target - with proposal qu for the time point zg, and define a reverse density tn = 7K that is equal to the next intermediate density (which means that we omit F -k- 1),
(z 11) = 7 (z1:k 11),
(z 11) Or. (z Tki A1:k-1, 7k- (21:k-1.7)
(20)
A limitation of this construction that the filtering distribution --- is not always a good proposal, since it does not incorporate knowledge of future observations. Ideally, we would like to define
AVO- flow is itself novel combination of AVO and flows, albeit incremental one.
8

---

No fi wride
Newua COUSE te wide (fed Cuerl
Ymeel ClrSs tie ctrie (Pertal Grad
Kossi Sexved the stice (Ported Coud!
4
eslee 2/
26
125 b
255 2
355 2
Figure 6: (Top) qualitative results of an instance with K = 200 time steps (x-axis). Observations color-coded based on the inferred assignments. Each colored band corresponds to the inferred cluster mean and standard deviation; grey bands indicate the ground truth of the clusters. (Bottom) We compute log Z and the ESS using 1000 samples and report average values over 2000 test instances.
intermediate densities "rk (Z1:k; 7)) = p(11:K 2i:k; 1i) that correspond to the smoothing distribution, but this requires computation of the marginal likelihood of future observations p( =k,, 71), which is intractable. This is particularly problematic when sampling 1) as part of the SMC construction. The first density Ti (21, 11) p(2 17 x1) will be similar to the prior, which will result in poor sample efficiency, since the smoothing distribution p(z1, 7 T1:K) will typically be much more concentrated.
To overcome this problem, will use NVI to learn heuristic factors the that approximate the marginal likelihood of future observations. We define a sequence of densities (ro..... -*k),
(0 =(()) vo(21.x)1), (Z = p( Risk; 17) Vh (o( | n)) k == 1, 1,2,.... K.
Our goal is to learn parameters 0 of the heuristic factor such that that intermediate densities approx- imate the smoothing distribution. This approach is similar to recently proposed work on twisted variational SMC [Lawson et al., 2018], which maximized a stochastic lower bound.
To evaluate the this approach, will learn heuristic factors for a hidden Markov model (HMM). While HMMs are a well-understood model class, they are a good test case, in that they give rise to significant sample degeneracy in SMC and allow to compute tractable heuristics. We optimize an NVI objective based on the forward KL with respect to the heuristic factor t'g. an initial proposal (u(n) x11K: σ) and a forward kernel qu [2 Zk, 21, )x).. Figure shows qualitative and quantita- tive results. We compare NVIR with neural GMM-heuristic to a baseline without a heuristic and a basclinn that uses a hand-coded GMM-heuriste convex update operator proposed in the Automatic Structured Variational Inference (ASVI) [Ambro- gioni et al., 2021]. Training ASVI by optimizing the ELBO failed due to the high variance of the score function estimator, which is needed for the discrete latent variables. However, we were able to train ASVI- based models with NVIR". We refer this novel combination as NASVI. Because NVI let us treat each step as separate optimization problem, we also compare partial optimization with respect to ftx only to full optimization with respect to both ftx and ÿt. While full optimization yields poor results, partial optimization learns a neural heuristic whose performance is similar to the GMM heuristic, which is a strong baseline in this context. For more details see Appendix G.2.
In addition, we train models which employ the
4.3 Meta Learning with Deep Generative models
In this experiment, we evaluate NVI in the context of deep generative models with hierarchically- structured priors. Concretely, we consider the task of inferring class weights from a mini-batch of images in fully unsupervised manner. For this purpose, we employ a variational autoencoder with prior in the form of Bayesian Gaussian mixture model. Unlike the previous experiments. here we both train generative and an inference model. The forward kernels in this setting act as the generative model and the reverse kernels are the inference model. Moreover, in this model sampling simplifies because we design the intermediate densities to be tractable, hence no importance sampling construction is needed. Regardless, we still can construct an NVI objective to train hierarchical deep generative models more efficiently and with more accurate posterior inference.
Model Description. We define a hierarchical deep generative model for batches of N images of the form (see Appendix G.3 for a graphical model and architecture description)
A ~ Dir(-; a)
Cn 1
Zn ~ N- 1 1/Tc.) I, ~ p(-1 2):0) for n = 1...N.
Here X, Cn. Ans and Xn. refer to the cluster probabilities, cluster assignments, latent codes, and observed images respectively. In the likelihood p(x[2;0m), we use a convolutional network to parameterize a continuous Bernoulli distribution [Loaiza- Ganiza, Cunningham, 2019]. We define

---

0
0.5
RWS
/
/
7 6
5
F
0.25
0
T
7
4
0.00
Original
Simulated
01234580
56789
Original
Simulated
0123 2
4
56
788
0.5
NVI
0
0 0 5
6
F
0
0.0
0.00
Original
Simulated
01 1123456789
Original
Simulated
012345677
Figure 7: BGMM-VAE trained MNIST & FashionMNIST with the RWS objective (Top) and the NVI objective (Bottom). (Left) Samples from a test mini-batch of size N = 300. (Middle) Samples from the generative model, generated from the À inferred from the test mini-batch. (Right) Comparison of ground truth X and the expected inferred value.
proposals ((= 2z; )2), ((c]2; de), and ((}ee; Φx). which are also parameterized by neural networks. We refer this model as Bayesian Gaussian mixture model VAE (BGMM-VAE).
Objective. To construct an NVI objective, we define intermediate densities for c and z. Unlike in previous experiments, we employ tractable densities in the form of a categorical (e 0o) for cluster assignments and a 8-layer planar flow T= (2; 0.) for the latent codes. Subsequently, we can define the forward and reverse densities at each step
4 c) = =+ (A)p(c(A), #t(c, 2) = (c)p(2)c), *a(2,z) <=))((a|2)
* c) (A)g()))c) #g(c, (22)(())2, #4(2,z) (2))()))) where # (x:) is empirical distribution over mini-batches of training data. We minimize the forward KL for the first two steps and the reverse KL at the final step.
Since the intermediate densities are tractable in this model, no nested importance sampling is required to optimize this nested objective; we can compute gradient estimates based on (single) sample from p( (C))((((A) in the first term, (e(e) in the second, and ((=)q(=(xx) in the final term. To learn the parameters {u, T, 0z } of the generative model, we maximize a single-sampli approximation of a F,g log (p(x,2, c, >) q(2, c, {/
lower bound
Results. We evaluate NVI for the BGMM-VAL using the following procedure. We generate mini- batches with a sampled X (for which make use of class labels that are not provided the model). We then compute the expectation of A under g(), c, x/xr by sampling from the inference model, and compare this value against X* Additionally, we generate a new mini-batch given the inferred by running the generative model forward. We compare NVI against RWS, where we use 10 samples to estimate the gradients at each iteration. The results are shown Figure The cluster indices are rearranged based on the association of clusters to true classes. We observe that for RWS, even though the sample quality is reasonable, the posterior approximation is quite poor. When we train with NVI however, both the inferred À and the generated samples match the test mini-batch reasonably well.
Conclusion
We developed NVI, framework that combines nested importance sampling and variational inference by optimizing variational objective every level of nesting. This formulation allows us to learn proposals and intermediate densities for general class of samplers, which admit most commonly used importance sampling strategies special cases. Our experiments demonstrate that samplers trained with NVI are able to outperform basclines when sampling from multimodal densities, Bayesian state-space models, and hierarchical deep generative models. Moreover, experiments show that learning intermediate distributions results in better samplers.
NVI is particularly useful in the context of deep probabilistic programming systems. Because NVI can be applied to learn proposals for a wide variety of importance samplers, it can be combined with methods for inference programming that allow users to tailor sampling strategies to particular probabilistic program. Concretely, NVI can be applied to nested importance samplers that are defined using a grammar of composable inference combinators [Stites et al., 2021], functions that implement primitive operations which preserve proper weighting, such as using samples from one program as a proposal to another program, sequential composition of programs, and importance resampling.
10

---

Acknowledgements
This work was supported by the Intel Corporation, the 3M Corporation, NSF awards 1835309 and 2047253, startup funds from Northeastern University, the Air Force Research Laboratory (AFRL), and DARPA.
References
Ambrogioni, Luca, Lin, Kate, Fertig, Emily, Vikram, Sharad, Hinne, Max, Moore, Dave, Gerven, Marcel. "Automatic structured variational inference". International Conference on Artificial Intelli- gence and Statistics. PMLR. 2021, pp. 676-684.
inffrencce.IImteraatioond
Arbel, Michael, Matthews, Alexander GDG, Doucet, Amaud. "Annealed Flow Transport Monte Carlo", arXiv preprint arXiv:2102.07500 (2021).
Bornschein, Jörg, Bengio, Yoshua. "Reweighted Wake-Sleep". International Conference Learning Representations (2015). arXiv: 1406. 2751.
Burda, Yuri, Grosse, Roger, Salakhutdinov, Ruslan. "Importance Weighted Autoencoders". Interna- fional Conference on Reprexentations, 2016. arXiv: 1509. 00519.
Del Moral, Pierre, Doucet, Arnaud, Jasra, Ajay. "Sequential Monte Carlo Samplers". Journal of the Royal Statistical Society: Series (Statistical Methodology) 68.3 (June 2006), pp. 411-436. DOL: 10. 1111/j 1868-9868.... 2006. 00553.055... X.
Doucet, Arnaud, Freitas, Nando, Gordon, Neil, eds. Sequential Monte Carlo Methods in Practice. New York, NY: Springer New York, 2001. ISBN: 978-1 778-1--441--288--01 97877-477777-447777333777777 DOI:
10. 1007 /978 4777--3377779
Hoffman, Matthew D. "Learning deep latent Gaussian models with Markov chain Monte Carlo". Proceedings of the 34th International Conference on Machine Learning-Volume Volume 70. JMLR. org. 2017, pp. 1510-1519.
Huang, Chin- Weii Tan, Shawn, Lacoste, Alexandre, Courville, Aaron "Improving explorability in variational inference with annealed variational objectives". Advances in Neural Information Processing Systems. 2018, pp. 9701-9711.
Kingma, Diederik P. Welling, Max. "Auto-encoding variational bayes". 13Xiv:1312.6114 (2013).
arXiv preprint
Lawson, Dieterich, Tucker, George, Naesseth, Christian A, Maddison, Chris, Adams, Ryan P, Teh, Yee Whye. "Twisted variational sequential monte carlo". Third workshop on Bayesian Deep Learning (NeurIPS). 2018.
Le, Tuan Anh, Igl, Maximilian, Rainforth, Tom, Jin, Tom, Wood, Frank. "Auto-Encoding Sequential Monte Carlo". International Conference on Learning Representations. 2018, arXiv: 1705. 10306.
Le, Tuan Anh, Kosiorek, A, Siddharth, N. Teh, Yee Whye, Wood, Frank. "Revisiting reweighted wake-sleep for models with stochastic control flow" (2019).
Li, Yingzhen, Turner, Richard E, Liu, Qiang. "Approximate inference with amortised meme". arXiv preprint
(2017).
Seaiza-Ganem. Gabriel, Cunningham, John P. "The continuous Bernoull:: fixing a pervasive error in variational autoencoders". Advances Neural Information Processing Systems. Ed. by Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Ban Buc, E. Fox, R. Garnett, Vol. 32, Curran Associates, Inc., 2019.

---

Maddison, Chris J. Lawson, Dieterich, Tucker, George, Heess, Nicolas. Norouzi, Mohammad, Mnih, Andriy, Doucet, Arnaud, Teh, Yee Whye. "Filtering variational objectives". Proceedings of the 31st International Conference Neural Information Processing Systems. 2017, pp. 6576-6586.
Naesseth, Christian, Linderman, Scott, Ranganath, Rajesh, Blei, David. "Variational sequential monte carlo". International Conference on Artificial Intelligence and Statistics. PMLR. 2018, pp. 968- 977.
Naesseth, Christian, Lindsten, Fredrik, Schon, Thomas. "Nested Sequential Monte Carlo Methods". International Conference on Machine Learning. 2015, pp. 1292-1301.
Naesseth, Christian A, Linderman, Scott W, Ranganath, Rajesh, Blei, David M. "Variational sequen- tial monte carlo". arXiv preprint arXiv: //14 (2017).
Naesseth, Christian A., Lindsten, Fredrik, Schön, Thomas B. "Elements of Sequential Monte Carlo". arXiv:1903.04797 [cs, stat] (Mar. 2019). arXiv: 1903.04797. (Visited on 12/16/2019).
Neal, Radford M. "Annealed importance sampling". Statistics and computing 1 11.2 (2001), pp. 125- 139.
Rainforth, Tom, Kosiorek, Adam, Le, Tuan Anh, Maddison, Chris, Igl. Maximilian, Wood, Frank, Teh, Yee Whye. "Tighter Variational Bounds Are Not Necessarily Better". en. International Conference on Machine Learning. July 2018, pp. 4277-4285
Ranganath, Rajesh, Gerrish, Sean, Blei, David. "Black Box Variational Inference". en. Artificial Intelligence and Statistics. Apr. 2014, pp. 814-822.
Rezende, Danilo, Mohamed, Shakir. "Variational Inference with Normalizing Flows". International Conference on Machine Learning. 2015, pp. 1530-1538.
Rezende, Danilo Jimenez, Mohamed, Shakir, Wierstra, Daan. "Stochastic Backpropagation and Approximate Inference in Deep Generative Models". Proceedings of the 3/ International Confer- ence on Machine Learning. Ed. by Eric Xing, Tony Jebara. Vol. 32. Proceedings of Machine Learning Research 2. Bejing, China: PMLR, June 2014, pp. 1278-1286.
Ritchie, Daniel, Horsfall, Paul, Goodman, Noah D. "Deep Amortized Inference for Probabilistic Programs". en. arXiv: [cs, stat] (Oct. 2016). arXiv:
[cs, stat].
Ritchie, Daniel, Thomas. Anna, Hanrahan, Pat, Goodman, Noah D. "Neurally-guide" procedural models: learning to guide procedural models with deep neural networks". arXiv preprint arXiv: 1603.06143 (2016).
Roeder, Geoffrey, Wu, Yuhuai, Duvenaud, David K. "Sticking the Landing: Simple, Lower- Variance Gradient Estimators for Variational Inference". Advances Neural Information Processing Systems. Ed. by I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, R. Gamett. Vol. 30. Curran Associates, Inc., 2017. URL: https: https::/ /ptp///eooece....e//////// wwwrips cc/ cc/paper .df.
Salimans, Tim, Kingma, Diederik, Welling. Max. "Markov chain monte carlo and variational in- ference: Bridging the gap". International Conference on Machine Learning. 2015, pp. 1218- 1226.
Schulman, John, Heess, Nicolas, Weber, Theophane, Abbeel, Pieter. "Gradient Estimation Using Stochastic Computation Graphs". Advances in Neural Information Processing Systems 28. Ed. by C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, R. Garnett. Curran Associates, Inc., 2015,
pp. 3528-3536.
Stites, Sam, Zimmermann, Heiko, Wu, Hao, Sennesh, Eli, van de Meent, Jil-We. "Learning Proposals for Probabilistic Programs with Inference Combinators". Proceedings of the 37th
12

---

Conference on Uncertainty in Artificial Intelligence (UAI 2021) (Mar. 2021). arXiv: 00668 [cs, stat].
Tucker, George, Lawson, Dieterich, Gu, Shixiang, Maddison, Chris "Doubly reparameterized gradient estimators for monte carlo objectives". arXiv preprint arXiv:/810.04//: (2018).
Wang,
Tongzhou, Wu, Yi, Moore, Dave, Russell, Stuart J. "Meta-lcarning MCMC proposals". Advances in Neural Information Processing Systems. 2018, pp. 4146-4156.
Wingate, David, Weber, Theo. "Automated Variational Inference in Probabilistic Programming". arXiv preprint arXiv: 11X1v1130..1999 (2013), pp. 1-7. arXiv: 1301. 1299.
Wu, Hao, Zimmermann, Heiko, Semnesh, Eli, Le, Tuan Anh, Meent, Jan-Willen van de. "Amortized Population Gibbs Samplers with Neural Sufficient Statistics". arXiv preprint arXiv: 19/1,01382 (2019).
13


Reviews:
[{"review_id": "lizjaXddO0", "reviewer": "Reviewer_FQ8n", "summary": "This paper proposes an inference procedure that combines ideas from variational and nested inference, resulting in a sequence of forward and reverse proposal distributions that are optimized with local objectives that encourage the forward and backward edge marginals to be similar. This allows the objective and therefore gradients to decouple. The addition of resampling, afforded by the connection to nested inference, additionally improves sample diversity.", "questions": "", "limitations": "", "rating": 7, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "*Originality*\n* This work proposes a novel combination of prior techniques, namely the local marginal objectives from AVO and the addition of resampling for intermediate distributions.\n\n*Quality*\n* The submission is technically sound, and its claims are well-supported by the experimental results. Experiment 1, multimodal sampling, demonstrates quantitative improvements over flow and AVO baselines, both in accuracy as well as sample diversity. Experiment 2, on posterior inference in a small HMM, demonstrates NVI’s use in a problem where intermediate densities are not of constant dimension. Experiment 3 uses NVI on a mixture model on (fashion) MNIST, and contains a favorable qualitative comparison to the same model trained with RWS.\n\n*Clarity*\n* The submission is clearly written, although I am hopeful the presentation can be greatly simplified. I would also have liked to see discussion of / relation to AVO earlier than the experiments section.\n* The notation might be a bit redundant, with different variable names for (edge) marginals, unnormalized marginals, and proposal distributions. In addition to the different directions, it took a while to get used to the notation. While it does not detract from the paper, any simplifications would be welcome.\n* In addition to the appendix, I believe an expert reader could reproduce the results.\n* Will the in-text derivation in lines 116-118 be replaced by equations given more space?\n\n*Significance*\n* I found the results convincing, although I am not familiar with the SMC literature.\nAdditionally, the experiments were performed on small data, which leads to concerns regarding scale.\n\n[AVO] Huang, Chin-Wei, Tan, Shawn, Lacoste, Alexandre, Courville, Aaron C. “Improving explorability in variational inference with annealed variational objectives”. Advances in Neural Information Processing Systems. 2018, pp. 9701–9711.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I would like to see speed comparisons between NVI and baselines where appropriate, as that could be a limiting factor in its applicability. For example in the HMM, how fast would exact inference be relative to NVI? This also causes me to be concerned about whether this technique can scale to models with larger state spaces, such as HMMs with thousands of states.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "McKjVSS_4Dp", "reviewer": "Reviewer_3PA6", "summary": "The author proposes a nested variational inference (NVI) framework that combines nested importance sampling and variational inference. This framework allows the training of the important sampling proposal by minimizing KL divergences at each level of nesting. This also provides us a way to learn intermediate densities, which serves as a heuristics for guiding the importance sampler. Empirically, the author evaluates NVI by (1) sampling from a multimodal distribution with a learned annealing path; (2) learn heuristics for future observations in the state-space model; (3) perform amortized inference in the hierarchical deep generative model.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Original\nThe NVI framework combines variational inference with the nested importance sampler. The idea of combining importance sampling with variational inference is not new, but the novelty of NVI is that it minimizes the KL divergence at each level of nesting and some of the existing work are special cases of NVI. To the best of my knowledge, this seems to be new. In appendix A, the author also discusses the related work. \n\n### Quality and Clarity\nThe overall idea of NVI is easy to understand. However, I still think some parts are not explained clearly, and I found them quite confusing when I first read them. I have checked the derivations for the gradient estimate, they seem to be correct. Empirically, the author conducts extensive experiments to confirm the effectiveness of the proposed method.\n\n### Significance\nCombining variational inference and importance sampling has recently attracted many research interests. This NVI framework seems to include many existing methods as special cases. Empirically, it seems to outperform the baselines in terms of ESS.\n\n### Questions\n1. In the beginning of section 3, I understand why you define a sequence that interpolates between initial $\\pi_0$ and target $\\pi_K$, and why you define the forward proposal $q_k(z_k|z_{k-1})$. I am not sure I fully understand the role of the reverse kernel $r_k(z_{k-1}|z_k)$. Why do you introduce this? In practice, how do you use this reverse kernel?\n\n2. A follow-up for question 1: why do you want the forward density $\\hat{\\pi_k}$ to be as close as possible to the reverse density $\\check{\\pi_k}$ at each level k? What does this objective actually mean? My intuition: If am the one that designs the objective, I will make sure the samples after the proposal $q_k$ should be as close as possible to the intermediate target $\\pi_k$, namely the KL divergence between the density of $z_k$ and $\\pi_k$. What is the problem of this alternative objective?\n What do you mean \n3. I am not sure I understand the meaning of line 127-130. Why it increases the probability of low-weight samples?\n4. Why AVO cannot learn the annealing parameter $\\beta$? I thought AVO also minimize KL at each level of nesting.\n5. Maybe the author can consider adding more background in the appendix to make the paper more self-contained? For example,  the reweight wake-sleep estimator. \n**Minor**:\n1. Better to add a paragraph to summarize the contributions of this paper. \n2. How do you obtain the third equality from the second equality in line 540? Why it becomes $\\frac{\\partial q_k}{\\partial \\hat{\\phi_k}}$?\n3. Maybe consider adding an overall algorithm for easier understanding. \n\n---\nI appreciate the author's response. They managed to clarify most of my concerns, especially about the KL objective. Hence, I raise my score. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "This paper should not have any negative societal impact as this is mainly a methodology paper. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "GiZ7-1MyL2D", "reviewer": "Reviewer_Ae4v", "summary": "The paper describes a framework for learning and inference in probabilistic models with structured latent spaces. It operates in the tradition of IWAE and VSMC-like methods that use importance sampling to tighten a variational bound (or equivalently, uses a variational bound as an objective for learning importance sampler proposals). Unlike methods that optimize a single global bound, NVI is defined in terms of local pairwise divergences between a sequence of target distributions, which acts to decouple the stochastic optimization, reducing gradient variance and potentially offering a strategy for parallel inference.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I struggled with this paper. On the one hand, I think it likely contains an interesting and valuable technical contribution. On the other hand, the experiments are not particularly compelling, and while that's not a dealbreaker if the idea itself is obviously worthwhile *a priori* (as I sense that the authors understand it to be), I don't think the paper as written is effective at communicating that understanding.\n\nStarting with the abstract and intro, I didn't feel that they conveyed a very clear motivation for the work, beyond 'combine nested importance sampling and variational inference', which is not by itself a goal that anyone knew they had. What useful properties do each of those ideas (NIS and VI) brings to the table, and what do we hope to get by combining them? What kind of tasks, that existing methods are poorly suited for, do we hope to be able to solve? (inference in temporal or spatiotemporal models? multimodal posteriors? high-dimensional state spaces? fitting really deep VAEs? all of the above?) Who should consider using this method? There is a lot of technical detail to get through in this paper, and the introduction could do more to give the reader a sense of why it will be worth it. \n\nI was perplexed by the choice of background material to focus on. The contribution is framed as VI + nested importance sampling, and the paper provides a nice intro to stochastic VI, but no clear definition of nested importance sampling beyond referring to Naesseth et al. 2015/2019. This seems backwards to me. The VAE paper (Kingma and Welling 2013) has 14870 Google Scholar citations, while Naesseth et al. 2015 has just 68; I would suggest that most readers (myself included) are much more likely to need an intro to the latter than the former. \n\nThe technical presentation itself is rather abstract and not straightforward to follow. There is reference to an 'NVI objective', but not a clear description of what this objective is --- is it intended to be eqn (6)? I would have expected that an importance sampling objective would at some point ground out to a procedure involving weighted samples, but the concept of weights isn't even introduced until the next section. It also wasn't clear to me how the approach relates to the concept of nested sampling, which I understand from the Naesseth papers to refer to using an 'inner' sampler to generate proposals for an 'outer' sampler. Could this work have been developed in a simpler formalism, like SMC, or is the sampler-within-a-sampler aspect crucial?\n\nOverall I think the presentation tries too hard to be general, at the expense of concreteness and intuition. It might help to frame things in terms of a specific running example, like the multimodal annealing setup from section 4.1 (introducing this earlier would also help give a sense of what kind of problems the approach is hoping to tackle). This would allow you to walk the reader concretely through the steps of applying NVI, and hopefully in the process give some intuition for why it works and where it differs from other methods.\n\nIt's not immediately obvious whether the NVI objective is a (stochastic) lower bound on the marginal likelihood, or some similar such property. Is it? If so, I think that'd be a useful argument to include.\n\nI would have liked to see some discussion of implementation, and its ease or lack thereof. As I understand it, the 'juice' of the method comes from estimating gradients using only locally random variables, and an entire page of the paper is about explicitly computing these gradients. This seems to cut against the dominant paradigm of 'just write down the computation graph and let autodiff do its thing'. How straightforward is it to implement all of this in modern DL frameworks? Are there any tricks? You could limit this discussion to the case where all gradients are reparameterizable, since handling discrete variables seems like a somewhat orthogonal (and well-studied) issue.\n\nThere's a lot of ink spilled about handling both forward and reverse KL (it's in the abstract even!), but it wasn't clear to me why this is worth focusing on. Does forward KL have any nice properties in this setting that would make up for how much of a pain it is to estimate?\n\nAs mentioned above, I didn't find the experiments compelling enough to carry the paper of their own accord. The application to deep generative modeling doesn't contain any quantitative results, or compare to any baselines (except for a reference to RWS in a 'Figure 9' that doesn't seem to have made it into the paper). The multimodal sampling problem is quite toy (though an effective illustration), and the gains over AVO are not super striking. I do appreciate that the experiments seem quite carefully done and are nicely presented, especially figures 3 and 4. In the state-space model experiment, is there any explanation for why optimizing the reverse kernel made things worse?\n\nnits:\nline 41: typo 'revserse'\nlines 101-103: SMC is listed twice as a special case.\n\n*********************************\nUpdated: thanks to the authors for your thorough response. I'm convinced that the technical contribution of the paper is solid, and in light of this am raising my score by a point. I encourage the authors to include their clarifications in a revised version of the paper---I think there's likely a much stronger paper here if well explained. The surrogate objective in particular seems worth mentioning and possibly presenting in the supplement (would it make sense to describe it as an extension/generalization of DiCE (https://arxiv.org/abs/1802.05098)?).", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "There are no direct societal impact concerns for this work, since the contribution is purely technical and not focused on a particular application.\n\nI do wish there were a bit more discussion of limitations. Have we finally solved probabilistic inference? Is this framework appropriate for all problems? (assuming not, then why?) What improvements or future directions would be worth exploring?", "review_text": "", "extra_scores": {"ethical_concerns": "n/a", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "DDVdWsrSapv", "reviewer": "Reviewer_nv8j", "summary": "In this paper, authors present Nested Variational Inference (NVI)--a framework to create increasingly better variational approximations. NVI allows for several existing sampling techniques to be employed in a single framework and is well supported by empirical results. ", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Recent years have seen several papers attempting to combing the Monte Carlo sampling methods and variational inference. I think this paper does a decent job of putting several of those work in a nice single framework. I found the paper to be very well written and easy to follow for the most part. The experiments and carefully carried out and visualizations are very informative. Overall, this is one of the better papers I have reviewed this season--a clear accept.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The proposed methods are still pretty involved and I believe it might need more work for these advanced inference techniques to find there way into every practitioners toolbox. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2-3 hours"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper learns proposals for a sequence of importance samplers by minimizing a KL divergence for each step (or level) of the sequence. The reviewers consider the work represents a solid and novel technical contribution and is above the acceptance bar for NeurIPS.

The authors are encouraged to improve the presentation and clarity by incorporating the feedback from the reviewers FQ8n, Ae4v, and 3PA6. In particular, consider adding a paragraph/list with contributions, an algorithm box, adding NIS to the background section, and discussing the limitations of NVI. The paper would also benefit from including other technical clarifications, such as the relationship to the ELBO or the intuition on why the forward KL estimator is insatiable.
Decision: Accept (Poster)