Nested Variational Inference
Authors: Heiko Zimmermann
Hao Wu
Babak Esmaeili
Jan-Willem van de Meent
Venue: NeurIPS 2021 Poster (2021)
Keywords: Variational Inference
Importance Sampling
Monte Carlo methods

Abstract: We develop nested variational inference (NVI), a family of methods that learn proposals for nested importance samplers by minimizing an forward or reverse KL divergence at each level of nesting. NVI is applicable to many commonly-used importance sampling strategies and provides a mechanism for learning intermediate densities, which can serve as heuristics to guide the sampler. Our experiments apply NVI to (a) sample from a multimodal distribution using a learned annealing path (b) learn heuristics that approximate the likelihood of future observations in a hidden Markov model and (c) to perform amortized inference in hierarchical deep generative models. We observe that optimizing nested objectives leads to improved sample quality in terms of log average weight and effective sample size.

Nested Variational Inference
Heiko Zimmermann' w.z.mermannnuua.a... ml
Hao Wu"
wu ww..haooonorrthhastern edu
Babak Esmaeili b. e.eemaeill Mi (uui@.va.... ml
Jan Willem van de Meent |*
vandemeent@uva. nl
Institute of Informatics, University of Amsterdam KKoorry College of Computer Sciences, Northeastern University
Abstract
We develop nested variational inference (NVI), a family of methods that learn proposals for nested importance samplers by minimizing an forward or reverse KL divergence at each level of nesting. NVI is applicable to many commonly-un importance sampling strategies and provides mechanism for learning intermediate densities, which can serve as heuristics to guide the sampler. Our experiments apply NVI (a) sample from multimodal distribution using learned annealing path (b) learn heuristics that approximate the likelihood of future observations in hidden Markov model and (c) to perform amortized inference in hierarchical deep generative models. We observe that optimizing nested objectives leads to improved sample quality in terms log average weight and effective sample size.
Introduction
Deep generative models provide a mechanism for incorporating priors into methods for unsupervised representation learning. This particularly useful in settings where the prior defines an inductive bias that reflects the structure of the underlying domain. Training models with structured priors, however, poses some challenges. standard strategy for training deep generative models is to maximize reparameterized evidence lower bound with respect to both the generative and an inference model [Kingma, Welling, 2013; Rezende et al., 2014]. This approach works well variational autoencoders with isotropic Gaussian priors, but often fails for models with more structured priors or likelihoods.
In recent years, range of strategies for improving upon standard reparameterized variational inference have been put forward. These include wake-sleep style variational methods that minimize the forward KI--divergence [Bornschein, Bengio, 2015; et al., 2019] as well as sampling schemes that incorporate annealing [Huang et al., 2018]. Sequential Monte Carlo [Le al., 2018; Nacsseth et al., 2017; Maddison et al., 2017], Gibbs sampling [Wu et al., 2019; Wang et al., 2018], and MCMC updates [Salimans et al., 2015; Hoffiman, 2017; Li et al., 2017]. While these methods offer flexible inference, typically resulting in better approximations to the posterior compared to traditional variational inference methods, they either model-specifie, requiring specialized sampling schemes and gradient estimators, or can not be easily composed with other techniques.
In this paper, we propose nested variational inference, framework for combining nested importance sampling and variational inference. Nested importance sampling formalizes the construction of proposals by way of calls to other importance samplers [Naesseth et al., 2015; Naesseth et al., 2019]. Many existing importance samplers are instances of nested samplers, including methods based on annealed importance sampling [Neal, 2001] and sequential Monte Carlo [Del Moral al., 2006].
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

NVI learns proposals by optimizing a divergence at each level of nesting. Additionally, we combine nested variational objectives with importance resampling to further improve sampling quality, without the need to undergo extra steps to maintain differentiability due to the local nature of the objective. Resampling also allows to compute gradient estimates based on incremental weights, which depend only on variables that are sampled locally, rather than on all variables in the model. Doing yields lower variance weights, whilst maintaining a high sample diversity relative to existing methods.
2 Background
2.1 Stochastic Variational Inference.
Stochastic variational methods approximate target density T(;; = = with parameters using variational density {(2; o) with parameters Φ. Two common variational objectives the forward and reverse Kullback-Leibe (KL) divergence, which are both instances of f-divergences
((((::2) B q
D, (TT 11 ()
with fww) = w log w and f(w) = log w respectively. We are typically interested in the setting of model with latent variables and observations x. this case, 7 0) = pa(a, 2) is the joint density of the model, and Z = =p( is the marginal likelihood.
where π (2; 0) is the posterior (= æ))
Reverse KL-divergence. When optimizing KL [))). known as the reverse or exclusive KL divergence, is common practice to maximize a lower bound [ = (log(l////// = log - x) with respect to 0 and é. The gradient of L can be approximated using reparameterized samples [Kingma, Welling, 2013; Rezende et al., 2014], likelihood-ratte estimators [Wingate, Weber, 2013; Ranganath et al., 2014], combination of the two [Schulman et al., 2015; Ritchie et al., 2016a].
Forward KL-divergence. In the case of the forward divergence KL (z || q). also known as the inclusive KL divergence, stochastic variational methods typically optimize separate objectives for the inference and generative model. A common strategy is to train the generative model by either maximizing the ELBO or likelihood on the data. Optimizing the inference model requires samples from म, which itself requires approximate inference. A common strategy, which popularized in the context reweighted wake-sleep (RWS) methods [Bornschein, Bengio, 2015; Le et al., 2019], is to use q proposal in importance sampler.
2.2 Importance Sampling
Self-Normalized Importance Samplers. An expectation [o])]] with respect to π can be rewritten with respect to a proposal density g by introducing an unnormalized importance weight w,
1 Z
7=;0) 0)
E
[=]=
E w 9())],
q(2;
Self-normalizer estimators weighted samples with respect to g, and to compute an estimate Z of the normalizing constant,
both approximate the expectation
$ (c) S
=
1 2 Σ (o(e) g(2(0)),
2
200 ~ g(:
(3)
The resulting estimator is consistent, i.e. 9 و/ E,z [g(2)) the number of samples S increases. but it is not unbiased, since follows from Jensen's inequality that 1/Z = 1/ Eg < E, If the discrepancy between and π is large, the importance weights will have large variance, which also has an impact on the bias. When g = म, the importance weight w = has zero variance, and the Jensen's inequality is tight. In the context of stochastic variational inference, this means that gradient estimates might initially strongly biased, since there will typically be large discrepancy between q and However, the variance will typically decrease as the quality of the approximation improves.

---

Ti (21)
T2 (2()
ii' (2x)
Ta (24))
(22) 23)
Is Rs x2)
2a (ze kgg
rı (21 (22)
r2(22 (za)
ra(23 =24)
Dy (#2r1 (92m1)
Dy (mar (3T2))
D (Tar's (4mT))
Figure Nested variational inference minimizes an f-divergence at each step in a sequence of densities to learn forward proposals yk: reverse kernels 1 11, and intermediate densities we-
Nested importance sampling and proper weighting Nested Importance sampling formalizes the construction of proposals by way of calls to other importance samplers. Extending the example above, we now assume is some unnormalized density itself that we cannot directly generates samples from. We can employ another importance sampler with proposal 7) to simulate samples from q.
9(=")
v),
w
2, T. Σ w
,)("));)))
can subsequently used compute the normalizing constant Zg of q and to generate samples that are approximately distributed w.r.t q via resampling. This allows us to use q as a proposal in a subsequent importance sampling step. whose weighted samples again be used to compute consistent estimates w.r.t. the target density of interest This can be generalized, by introducing the notion of proper weighting.
The resulting weighted samples (20)) )
Definition 2.1 (Proper weighting). Let be a probability density. For some constant 0, random pair (w, z) ~ is properly weighted (p. w.) for an unnormalized probubility density 7 = ZT w and all measurable functions g it holds that
= c f dz ((2) g(2) cZ [g(z))
E
[w g(2)]
01,2 -1
2
As long as we can generate properly weighted samples for the target density x from some sampler II, we compute consistent estimates
Ex w(+)
g(2(e))
o TR E2aT [g(2)]
3
[g(2)]-
(4)
E w(e)
e
2009
3 Nested Variational Inference
widely used strategy in importance sampling is to decompose difficult sampling problem into a series of easier problems. A common approach is to define a sequence of unnormalized densities {nx)Æ= that interpolate between an initial density тТ = 71/Z1, for which sampling is easy, and the final target density тк YK/ Tk/ZKr ZZ.. At each step, samples from the preceding density serve to construct proposals for the next density, which is typically combined with importance resampling or application of Markov chain Monte Carlo (MCMC) operator improve the average sample quality.
Nested variational objectives. NVI defines objectives for optimizing importance samplers that target a sequence of densities. At every step, it minimizes the discrepancy between a forward density i = 5x/ /Zk -1 which acts as the proposal, and reverse density मं 5 which defines an intermediate target. We define the forward density by combining the preceding target 7 with forward kernel qk. and the reverse density by combining the next target 7/. with a reverse kernel rk-
(Z =k 1) = (2x) Tk 1
in (Zz, 1x-1) qu (=x|=x-1) 7h- (z).
(5)
Our goal learn pairs of densities मe and सt. that are similar as possible. To do so, we minimize a variational objective 'D that comprises an f-divergence for each step in the sequence, along with divergence between the first intermediate target कi and an initial proposal Q1. Concretely, the NVI objective can be formulated as,
K
D = Dy(TI 91) + Σ
Dy mx).
(6)
k=
3

---

Since each intermediate density Tr occurs in both t. and in + this defines a trade-off between maximizing the similarity T and the similarity to TT-1- To optimize this objective, we need to be able to simulate samples from the intermediate densities.
Sampling from intermediate densities. Given pair (#k-1; that is properly weighted for the previous target density 7k-1, we can use sequential Monte Carlo sampling [Del Moral et al., 2006] construction define a pair (wa, 2x) that properly weighted for 7/k.
3is (2k #k-
2k q.( zk-1),
wk Uj Wk-1,
Uk
Fn (Fx: 2k
We refer to the ratio Us as the incremental weight. In this construction, (wk; 2k-1:k) is properly weighted for ty which implies that (wk, 2x.) is also properly weighted for Yks since
f d (2ks 2k-1) - / -dzz- 7/ (2.
dzx-1
(2x -1 (x,) = (zx))
(8)
Sequential importance sampling can be combined with other operations that preserve proper weight- ing, including rejection sampling, application of an MCMC transition operator, and importance resampling. This defines a class of samplers that admits many popular methods special cases, including sequential Monte Carlo (SMC) [Doucet et al., 2001], annealed importance sampling (AIS) [Neal, 2001], and SMC samplers [Del Moral et al., 2006]. In this work we only consider variational transition kernels and importance resampling. These samplers vary in the sequences of densities they define. For example, AIS and SMC samplers both define intermediate density over fixed common domain. Here, common strategy is to define an annealing path YYk >x<=z) 71 (==)*=Pp y (zx) )0x for == 31 < 82 < ... 3x = 1. In contrast, when using SMC for spte-space models we can define the intermediate target T to be the the filtering distribution on the first k states. In this setting, the
dimensionality the support increases at each step and we can omit the reverse kernel rk-1- We are exploring both of these cases in Section 4.
3.1 Computing Gradient Estimates
The NVI objective can be optimized with respect to three sets of densities. We will use 6x. dz and oR to denote the parameters of the densities Tk. 9k, and n respectively. For notational convenience, we use Oæ: = {As, Φ } to refer to the parameters of the reverse density # and PJ. ( - to refer to the parameters of the forward density Rk.
Gradients of the Forward KL divergence. When we employ the forward KL as the objective, the derivative with respect Pu can be expressed (see Appendix E.3),
KL TR ml 0pp
#x)
E 3x
log 7n (Ek: px)1; Opc
IE Wk-1
2 Opk
log 7h- (zx 1: 1 00-
(9)
This case is the nested analogue of RWS-style variational inference. We can move the derivative into the expectation, since 7t. does not depend on Ω:- We then decompose log मि = log 24 log - and use the identity from Equation 21 to express the gradient log Z as expectation with respect to -- The resulting expectations can be approximated using norf-normalized estimators based on the outgoing weights wt and incoming weights w respectively.
The gradient of the forward KL with respect to Oe is more difficult to approximate, since the expectation is computed with respect to t₁. which depends the parameters Pv. The gradient of this expectation has the form (see Appendix E.3)
Opk
KL (TK
#K)
=
E #k
log Uh
Opk
log th (2k, 2k 2k Pk)
(10)
a
a
E 22
log
UK
log in (Zk, 1i Op..
log 2kk
log 7M (zxi 0x)
A2
Opk
In principle, we can approximate this gradient using self-normalized estimators based on the outgoing weight wi- We experimented extensively with this estimator, but unfortunately we found it to be unstable, particularly for the gradient with respect to the parameters of the reverse kernel r For this reason, our experiments employ the reverse KL when learning reverse kernels.

---

Our hypothesis is that the instability in this estimator arises because the gradient decreases the probability of high-weight samples and increases the probability of low-weight samples, rather than the other way around. This could lead to problems during early stages of training, when the estimator will underrepresent low-weight samples, for which the probability should increase.
Gradients of the Reverse KL divergence. When computing the gradient of the reverse KL with respect Pk. we obtain the nested analogue of methods that maximize a lower bound. Here we can either use reparameterized samples [Kingma, Welling, 2013; Rezende et al., 2014] or likelihood-ratio estimators [Wingate, Weber, 2013; Ranganath al., 2014]. We will follow Ritchie et (2016b) and define a unified estimator in which proposals are generated using construction
Wk U/ 11 Fk g( Φk), ms (Ex Φu), Wk-1, 2k-1
11g-1-
(11)
This construction recovers reparameterized samplers the special case when ÿj does not depend on parameters, and recovers non non-reparameterized samplers when 2j; .μ. This means it is applicable to models with continuous variables, discrete variables, or a combination of the two. The gradient the reverse KL for proposals that are constructed in this manner becomes (see Appendix E.2)
a
Ozk
KL (tx 0pk
|| #x)
= E
log 7K (2k, 2* 1: 0pp
azk
a
+
E
log Opk
log 7k (2k 1k Ph)
(12)
8 Opp
E
log Uk
E
log 1 (2-11(2-
In this gradient, the first term represents the pathwise derivative with respect to reparameterized samples. The second term defines a likelihood-ratio estimator in terms of the unnormalized density Ie. and the third term computes the contribution of the gradient the log normalizer log -
Computing the gradient of the reverse KL with respect to 5 is once again straightforward, since we are computing an expectation with respect to 7.. which does not depend on pk- This means we can move the derivative into the expectation, which yields gradient analogous to that in Equation 9,
a KL 0p
a
a Opp
#k
=
log 4ne Tu pp
fh (Zk, 1k-
E si
log 7k (Exi 02)
(13)
Variance Reduction. To reduce the variance of the gradient estimates use the expected log- incremental weight as baseline for score function terms and employ the sticking-the-landing trick [Roeder et al., 2017] when reparameterizing the forward kemne as described in Appendix E.
3.2 Relationship to Setertance-Weighttee and Self-Normalized Estimators
There exists large body of work on methods that combine variational inference with MCMC and importance sampling. We refer to Appendix A for comprehensive discussion of related and indirectly related approaches. To position NVI in the context of the most directly related work, we here focus on commonly used importance-weighter and self- normalized estimators.
NVI differs from existing methods in that it defines objective for pairs of variables (zk, # each level of nesting, rather than a single objective for the entire sequence of variables (21, 2,, z A standard approach for combining importance sampling and variational inference is to define "importance-weigheee lower bound Lx log 2x [Burda et al., 2016]. By Jensen's inequality, E[Ck] log k| log Zx. which implies that we can define stochastic lower bound using any properry-weightee sampler for 7k. including samplers based on SMC [Le et al., 2018; Naesseth al., 2018; Maddison al., 20. For purposes of learning the target density 7K, this approach is equivalent to computing an RWS-style estimator of the gradient,
1
11
ΣK =
w a0
log 7K (2)))xx))
(14)
00
ZR
S
However, these two approaches are not equivalent for purposes of learning the proposals. We can maximize stochastic lower bound to learn gk. but this requires roubbe-reparamettriieed estimators
5

---

SIS
Pg Fig Vic-1
SMC
Mar-1
Weight contributions in the self-normalizer gradient estimators for the forward KL- divergence for VI and NVI using SIS (no resampling) and SMC (resampling). computes gradient estimates using the final weights (SIS), which simplify to the final incremental weight when resam- pling is performed (SMC). NVI computes gradient estimates based on the intermediate weights (SIS), which simplify to the intermediate incremental weights when resampling is performed (SMC).
Figure 2:
[Tucker et al., 2018] in order to avoid problems with the signal-to-noise ratio in this estimator, which can paradoxically deteriorate with the number of samples [Rainforth et al., 2018]. The estimators in NVI do not suffer from this problem, since we do not compute the logarithm of an average weight.
NVI is also not equivalent learning proposals with RWS-style estimators. If we use sequential importance sampling (SIS) to generate samples, a self- normalized gradient for the parameters of gk that is analogous to the one in Equation 24 has the form
E
log (=k 2k-1: 26)
K
log 9k (2k 2 Φx). (15)
KK-KK
ZK
Note that this expression depends on the final weight WK. By contrast, a NVI objective based on the forward KL yields self-normalized estimator that is defined in terms of the intermediate weight wa
E Th
log qr (=x - 2x) Oib
a
1 2x
Σ
w a log qu (=f. = dz)-
(16)
Oáv
If instead of SIS we employ seguential importance resampling (i.e. SMC), then the incoming weight w identical for all samples. This means that can express this estimator in terms of the incremental weight rather than the intermediate weight Wk
2 2 U 0Φp
log q (2 1 2x)
log (x 14- 2x).
(17)
see that NVI allows us to compute gradient estimates that are localized to a specific level of the sampler. In practice, this can lead to lower-varianee gradient estimates.
Having localized gradient computations also offers potential memory advantages. Existing methods typically perform reverse-mod mode automatic differentiation on an objective that is computed from the final weights (e.g. the stochastic lower bound). This means that memory requirements scale as OSS)) since the system needs to keep the entire computation graph in memory. In NVI, gradient estimates level do not require differentiation of the incoming weights w This means that is possible to perform automatic differentiation on a locally-defined objective before proceeding to the next level of nesting, which means that memory requirements would scale as OSS). should therefore in principle be possible to employ a large number of levels of nesting K in NVI, although we do not evaluate the stability NVI at large K our experiments.
4 Experiments
We evaluate NVI on three tasks, (1) learning to sample form an unnormalized target density where intermediate densities are generated using annealing, (2) learning heuristic factors to approximate the marginal likelihood of future observations in state-space models, and finally (3) inferring distributions over classes from small numbers of examples in deep generative Bayesian mixtures.
4.1 Sampling from Multimodal Densities via Annealing
common strategy when sampling from densities with multiple isolated modes is to anneal from an initial density 71, which is typically unimodal distribution that sample from easily, to the

---

AVO
NVIR
a(aa()))
by ~ (((l)
(1(-(1))))) su (((1)))))))
(((1)))))
Re ~ 9et-1zs)
by sx(-
(x(-11)))
Figure 3: Samples from forward kernels trained with AVO, and NVIR*.
(a)
(b)
1.0
Limorr
NVI
SVI
BSS/LL
0.0
NVIR
VSMC
10a
KT(mk || Tk
Linear NVIS NVIR
ESS /L.
10
2
3
5
10k 20k
10k:
20h
Figure 4: (a) ESS relative to the number of samples 36 during training for different methods using pairs of transition kernels (sequence length K 8) averaged across 10 independent Error bars indicate +2 standard deviations; mean and standard deviation are computed based rolling average with window size 100. (b) (Top) Annealing paths learned by NVT" and NIVR* * and the linearly spaced geometric annealing schedule (Linear) used by AVO, NVI, and NVIR. Results averaged over 10 restarts; error bars indicate two standard deviations. (Bottom) Numerically computed KL-divergences between consecutive intermediate distributions for different schedules.
target density 7K. which is multimodal [Neal, 2001]. Recent work on annealed variational objectives (AVOs) learns forward kernels qu and reverse kernels Fk-1 for an annealing sequence by optimizing a variational lower bound at each level of nesting [Huang et al., 2018].
L TT
LAVO
E Q1.-..Akk
log vk): (x(=) (<x))==*>x())*> k= 1, K. (18)
NVI allows us to improve upon AVO in two ways. First, at every step to optimize an SMC sampler rather than an annealed importance sampler. Second, we can learn the annealing schedule ($I...,, 3x) ) as part of the intermediate densities % such that .1....,By2
perform importance resampling
intermediate densities are scheduled more equidistantly in terms of KL-divergence.
We illustrate the effect of these two modifications in Figure 3, in which we compare AVO to NVI with resampling and learned path, which we refer to NVIR". Both methods minimize the reverse KL at each step The learned annealing path in NVIR* results in smoother interpolation between the initial and final density. We also see that AVO does not assign equal mass to modes, whereas NVIR" yields more even distribution.
In Figure 4a we compute the reverse KL between targets each step the sequence. For the standard linear annealing path, the KL decreases with kt suggesting that later intermediate densities are increasingly redundant. By constrast, NVIR* we see that the KL approximately constant across the path, which is what we would expect when minimizing D with respect to Bg. This is also the case an ablation without resampling, which we refer to as NVI*.
As noted in Section 3.1, 1, we found optimization of the forward KL to be unstable when learning Px-

---

log 2
(log 2.08)
ESS
Seq. length
K=6
K=8
K=2
K=4
K=6
GT
SVI
1.86
1.89
1,92
1,72
51
47
32
25
-lw
2.06
55
AVO
2.01
2,05
NVI
1.86
1.97
2,03
2,06
51
45
45
41
NVIR
1.86
1.98
2.04
2,06
51
99
NVIR*
NVI"
1.86
2.06
2,07
2.07
51
NVIR
1.86
2.06
2,07
2,08
51
94
97
97
AVO-Bow
2.07
2,08
-VIo-flow
2.05
2,08
2,08
2,08
28
81
79
70
Table 1: Sample efficiency for NVI variants and baselines for K - annealing steps and L samples per step for fixed budget of K L 288 samples. Metries are computed for 100 batches of 100 samples per model across 10 restarts.
Figure 5: Ground truth densities (GT) and samples from final target density for NVIR* with 2 interme- diate densities (K=4).
Figure 4b shows rolling average of the ESS and its variance during training. We compare NVI-based methods to SVI and a variational SMC sampler [Le et al., 2018; Maddison et al., 2017: Naesseth et al., 2017]. NVIR* * has consistently higher ESS and significantly lower variance compared to baselines. These plots also provides insight into the role of resampling in training dynamics. In NVI", observe cascading convergence, which is absent in NVIR". We hypothesize that resampling reduces the reliance on high-quality proposals from step 1 when estimating gradients at step k.
Annealed NVI has similar cases as normalizing flows [Rezende, Mohamed, 2015]. Inspired by concurrent work of Arbel et al. (2021), which explores a similar combination of SMC samplers and normalizing flows, we compare flow-based versions of NVI to planar normalizing flows, which maximize a standard lower bound (SVI-flow). We find that a normalizing flows can be effectively trained with NVI, in that samplers produce better estimates of the normalizing constant and higher ESS compared to SVI-flow. We also find that flow based models are able to produce high-quality samples with fewer intermediate densities (Figure 8). Moreover, we that combining a flow-based proposal with learned 3n values (NVI'-flow) results in a more accurate approximation of the target than in ablation with linear interpolation path (AVO-flow).
In Table I we report sample quality in terms of the stochastic lower bound 2x log 2x and the effective sample size ESS = EE, wx )2 E, (wx))² ) The first metric can be interpreted as a measure of the average sample quality, whereas the second metric quantifies sample diversity. We compare NVI with and without resampling (NVIR* and its NVI*) to ablations with a linear annealing path (NVIR and NVT), AVO, and standard SVI baseline in which there no intermediate densities. We additionally compare against AVO-flow and -fo'-flow, which employ flows. We observe that NVIR and NVI*-flow outperform ablations and baselines in terms of log Z, and are competitive in terms of ESS. We show qualitative results for two additional target densities in Figure 5. For more details we refer to Appendix G.1.
4.2 Learning Heuristic Factors for State-space Models
Sequential Monte Carlo methods are commonly used in state-space models to generate samples by proposing one variable at a time. To do so, they define a sequence of densities F = 7x/2, /2g on the first & time points in a model, which are also known filtering distributions,
Th (2 m) = p( Zi.k, 7) = p(r) p( =1 m) II p(r,, 21 21-1,7), Zx = pp=(()))) (19)
Here 2i:k and Fisk are sequences of hidden states and observations, and is a set of global variables of the model. The densities " differ from those in the annealing task in Section 4.1 in that the dimensionality of the support increases at each step. We define forward density 9y that combines the preceding target - with proposal qu for the time point zg, and define a reverse density tn- = 7K that is equal to the next intermediate density (which means that we omit F -k- 1),
(z 11) = 7 (z1:k 11),
(z 11) Or (z Tki A1:k-1, 7k- (21:k-1.7)
(20)
A limitation of this construction that the filtering distribution --- is not always a good proposal, since it does not incorporate knowledge of future observations. Ideally, we would like to define
AVO- flow is itself novel combination of AVO and flows, albeit incremental one.
8

---

No fi wride
Newua COUSE te wide (fed Cuerl
Ymeel ClrSs tie ctrie (Pertal Grad
Kossi Sexved the stice (Ported Coud!
4
eslee 2/
26
125 b
255 2
355 2
Figure 6: (Top) qualitative results of an instance with K = 200 time steps (x-axis). Observations color-coded based on the inferred assignments. Each colored band corresponds to the inferred cluster mean and standard deviation; grey bands indicate the ground truth of the clusters. (Bottom) We compute log Z and the ESS using 1000 samples and report average values over 2000 test instances.
intermediate densities "rk (Z1:k; 7)) = p(11:K 2i:k; 1i) that correspond to the smoothing distribution, but this requires computation of the marginal likelihood of future observations p( =k,, 71), which is intractable. This is particularly problematic when sampling 1) as part of the SMC construction. The first density Ti (21, 11) p(2 17 x1) will be similar to the prior, which will result in poor sample efficiency, since the smoothing distribution p(z1, 7 T1:K) will typically be much more concentrated.
To overcome this problem, will use NVI to learn heuristic factors the that approximate the marginal likelihood of future observations. We define a sequence of densities (ro..... -*k),
(0 =(()) vo(21.x)1), (Z = p( Risk; 17) Vh (o( | n)) k == 1, 1,2,... K.
Our goal is to learn parameters 0 of the heuristic factor such that that intermediate densities approx- imate the smoothing distribution. This approach is similar to recently proposed work on twisted variational SMC [Lawson et al., 2018], which maximized a stochastic lower bound.
To evaluate the this approach, will learn heuristic factors for a hidden Markov model (HMM). While HMMs are a well-understood model class, they are a good test case, in that they give rise to significant sample degeneracy in SMC and allow to compute tractable heuristics. We optimize an NVI objective based on the forward KL with respect to the heuristic factor t'g. an initial proposal (u(n) x11K: σ) and a forward kernel qu [2 Zk, 21, )x).. Figure shows qualitative and quantita- tive results. We compare NVIR with neural GMM-heuristic to a baseline without a heuristic and a basclinn that uses a hand-coded GMM-heuriste convex update operator proposed in the Automatic Structured Variational Inference (ASVI) [Ambro- gioni et al., 2021]. Training ASVI by optimizing the ELBO failed due to the high variance of the score function estimator, which is needed for the discrete latent variables. However, we were able to train ASVI- based models with NVIR". We refer this novel combination as NASVI. Because NVI let us treat each step as separate optimization problem, we also compare partial optimization with respect to ftx only to full optimization with respect to both ftx and ÿt. While full optimization yields poor results, partial optimization learns a neural heuristic whose performance is similar to the GMM heuristic, which is a strong baseline in this context. For more details see Appendix G.2.
In addition, we train models which employ the
4.3 Meta Learning with Deep Generative models
In this experiment, we evaluate NVI in the context of deep generative models with hierarchically- structured priors. Concretely, we consider the task of inferring class weights from a mini-batch of images in fully unsupervised manner. For this purpose, we employ a variational autoencoder with prior in the form of Bayesian Gaussian mixture model. Unlike the previous experiments. here we both train generative and an inference model. The forward kernels in this setting act as the generative model and the reverse kernels are the inference model. Moreover, in this model sampling simplifies because we design the intermediate densities to be tractable, hence no importance sampling construction is needed. Regardless, we still can construct an NVI objective to train hierarchical deep generative models more efficiently and with more accurate posterior inference.
Model Description. We define a hierarchical deep generative model for batches of N images of the form (see Appendix G.3 for a graphical model and architecture description)
A ~ Dir(-; a)
Cn 1
Zn ~ N- I 1/Tc.) I, ~ p(-1 2):0) for n = 1...N.
Here X, Cn. Ans and Xn. refer to the cluster probabilities, cluster assignments, latent codes, and observed images respectively. In the likelihood p(x[2;0m), we use a convolutional network to parameterize a continuous Bernoulli distribution [Loaiza- Ganiza, Cunningham, 2019]. We define

---

0
0.5
RWS
/
/
7 6
5
F
0.25
0
T
7
4
0.00
Original
Simulated
01234580
66789
Original
Simulated
0123 2
4
56
778
0.5
NVI
0
0 0 5
6
F
0
0.0
0.00
Original
Simulated
01 1123456789
Original
Simulated
012345677
Figure 7: BGMM-VAE trained MNIST & FashionMNIST with the RWS objective (Top) and the NVI objective (Bottom). (Left) Samples from a test mini-batch of size N = 300. (Middle) Samples from the generative model, generated from the À inferred from the test mini-batch. (Right) Comparison of ground truth X and the expected inferred value.
proposals ((= 2z; )2), [(c]2; de), and ((}ee; Φx). which are also parameterized by neural networks. We refer this model as Bayesian Gaussian mixture model VAE (BGMM-VAE).
Objective. To construct an NVI objective, we define intermediate densities for c and z. Unlike in previous experiments, we employ tractable densities in the form of a categorical (e 0o) for cluster assignments and a 8-layer planar flow T= (2; 0.) for the latent codes. Subsequently, we can define the forward and reverse densities at each step
4 c) = =+ (A)p(c(A), #t(c, 2) = (c)p(2/c), *a(2,z) <=))((a|2)
* c) (A)g()))c) #g(c, (22)(())2, #4(2,z) (2))()))) where # (x:) is empirical distribution over mini-batches of training data. We minimize the forward KL for the first two steps and the reverse KL at the final step.
Since the intermediate densities are tractable in this model, no nested importance sampling is required to optimize this nested objective; we can compute gradient estimates based on (single) sample from p( (C))((((A) in the first term, (A(e) in the second, and ((=)q(=(xx) in the final term. To learn the parameters {u, T, 0z } of the generative model, we maximize a single-sampli approximation of a F,g log (p(x,2, c, >) q(2, c, {/
lower bound
Results. We evaluate NVI for the BGMM-VAL using the following procedure. We generate mini- batches with a sampled X (for which make use of class labels that are not provided the model). We then compute the expectation of A under g(), c, x/xr by sampling from the inference model, and compare this value against X* Additionally, we generate a new mini-batch given the inferred by running the generative model forward. We compare NVI against RWS, where we use 10 samples to estimate the gradients at each iteration. The results are shown Figure The cluster indices are rearranged based on the association of clusters to true classes. We observe that for RWS, even though the sample quality is reasonable, the posterior approximation is quite poor. When we train with NVI however, both the inferred À and the generated samples match the test mini-batch reasonably well.
Conclusion
We developed NVI, framework that combines nested importance sampling and variational inference by optimizing variational objective every level of nesting. This formulation allows us to learn proposals and intermediate densities for general class of samplers, which admit most commonly used importance sampling strategies special cases. Our experiments demonstrate that samplers trained with NVI are able to outperform basclines when sampling from multimodal densities, Bayesian state-space models, and hierarchical deep generative models. Moreover, experiments show that learning intermediate distributions results in better samplers.
NVI is particularly useful in the context of deep probabilistic programming systems. Because NVI can be applied to learn proposals for a wide variety of importance samplers, it can be combined with methods for inference programming that allow users to tailor sampling strategies to particular probabilistic program. Concretely, NVI can be applied to nested importance samplers that are defined using a grammar of composable inference combinators [Stites et al., 2021], functions that implement primitive operations which preserve proper weighting, such as using samples from one program as a proposal to another program, sequential composition of programs, and importance resampling.
10

---

Acknowledgements
This work was supported by the Intel Corporation, the 3M Corporation, NSF awards 1835309 and 2047253, startup funds from Northeastern University, the Air Force Research Laboratory (AFRL), and DARPA.
References
Ambrogioni, Luca, Lin, Kate, Fertig, Emily, Vikram, Sharad, Hinne, Max, Moore, Dave, Gerven, Marcel. "Automatic structured variational inference". International Conference on Artificial Intelli- gence and Statistics. PMLR. 2021, pp. 676-684.
inffrencce.IImteraationnd
Arbel, Michael, Matthews, Alexander GDG, Doucet, Amaud. "Annealed Flow Transport Monte Carlo", arXiv preprint arXiv:2102.07500 (2021).
Bornschein, Jörg, Bengio, Yoshua. "Reweighted Wake-Sleep". International Conference Learning Representations (2015). arXiv: 1406. 2751.
Burda, Yuri, Grosse, Roger, Salakhutdinov, Ruslan. "Importance Weighted Autoencoders". Interna- fional Conference on Reprexentations, 2016. arXiv: 1509. 00519.
Del Moral, Pierre, Doucet, Arnaud, Jasra, Ajay. "Sequential Monte Carlo Samplers". Journal of the Royal Statistical Society: Series (Statistical Methodology) 68.3 (June 2006), pp. 411-436. DOL: 10. 1111/j 1868-9868.... 2006. 00553.055... X.
Doucet, Arnaud, Freitas, Nando, Gordon, Neil, eds. Sequential Monte Carlo Methods in Practice. New York, NY: Springer New York, 2001. ISBN: 978-1 778-1--441--2887-01 97877-477777744777733337777- DOI:
10. 1007 /978 4777--3377779
Hoffman, Matthew D. "Learning deep latent Gaussian models with Markov chain Monte Carlo". Proceedings of the 34th International Conference on Machine Learning-Volume Volume 70. JMLR. org. 2017, pp. 1510-1519.
Huang, Chin- Weii Tan, Shawn, Lacoste, Alexandre, Courville, Aaron "Improving explorability in variational inference with annealed variational objectives". Advances in Neural Information Processing Systems. 2018, pp. 9701-9711.
Kingma, Diederik P. Welling, Max. "Auto-encoding variational bayes". 13Xiv:1312.6114 (2013).
arXiv preprint
Lawson, Dieterich, Tucker, George, Naesseth, Christian A, Maddison, Chris, Adams, Ryan P, Teh, Yee Whye. "Twisted variational sequential monte carlo". Third workshop on Bayesian Deep Learning (NeurIPS). 2018.
Le, Tuan Anh, Igl, Maximilian, Rainforth, Tom, Jin, Tom, Wood, Frank. "Auto-Encoding Sequential Monte Carlo". International Conference on Learning Representations. 2018, arXiv: 1705. 10306.
Le, Tuan Anh, Kosiorek, A, Siddharth, N. Teh, Yee Whye, Wood, Frank. "Revisiting reweighted wake-sleep for models with stochastic control flow" (2019).
Li, Yingzhen, Turner, Richard E, Liu, Qiang. "Approximate inference with amortised meme". arXiv preprint
(2017).
Seaiza-Ganem. Gabriel, Cunningham, John P. "The continuous Bernoull:: fixing a pervasive error in variational autoencoders". Advances Neural Information Processing Systems. Ed. by Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Ban Buc, E. Fox, R. Garnett, Vol. 32, Curran Associates, Inc., 2019.

---

Maddison, Chris J. Lawson, Dieterich, Tucker, George, Heess, Nicolas. Norouzi, Mohammad, Mnih, Andriy, Doucet, Arnaud, Teh, Yee Whye. "Filtering variational objectives". Proceedings of the 31st International Conference Neural Information Processing Systems. 2017, pp. 6576-6586.
Naesseth, Christian, Linderman, Scott, Ranganath, Rajesh, Blei, David. "Variational sequential monte carlo". International Conference on Artificial Intelligence and Statistics. PMLR. 2018, pp. 968- 977.
Naesseth, Christian, Lindsten, Fredrik, Schon, Thomas. "Nested Sequential Monte Carlo Methods". International Conference on Machine Learning. 2015, pp. 1292-1301.
Naesseth, Christian A, Linderman, Scott W, Ranganath, Rajesh, Blei, David M. "Variational sequen- tial monte carlo". arXiv preprint arXiv: //14 (2017).
Naesseth, Christian A., Lindsten, Fredrik, Schön, Thomas B. "Elements of Sequential Monte Carlo". arXiv:1903.04797 [cs, stat] (Mar. 2019). arXiv: 1903.04797. (Visited on 12/16/2019).
Neal, Radford M. "Annealed importance sampling". Statistics and computing 1 11.2 (2001), pp. 125- 139.
Rainforth, Tom, Kosiorek, Adam, Le, Tuan Anh, Maddison, Chris, Igl. Maximilian, Wood, Frank, Teh, Yee Whye. "Tighter Variational Bounds Are Not Necessarily Better". en. International Conference on Machine Learning. July 2018, pp. 4277-4285
Ranganath, Rajesh, Gerrish, Sean, Blei, David. "Black Box Variational Inference". en. Artificial Intelligence and Statistics. Apr. 2014, pp. 814-822.
Rezende, Danilo, Mohamed, Shakir. "Variational Inference with Normalizing Flows". International Conference on Machine Learning. 2015, pp. 1530-1538.
Rezende, Danilo Jimenez, Mohamed, Shakir, Wierstra, Daan. "Stochastic Backpropagation and Approximate Inference in Deep Generative Models". Proceedings of the 3/ International Confer- ence on Machine Learning. Ed. by Eric Xing, Tony Jebara. Vol. 32. Proceedings of Machine Learning Research 2. Bejing, China: PMLR, June 2014, pp. 1278-1286.
Ritchie, Daniel, Horsfall, Paul, Goodman, Noah D. "Deep Amortized Inference for Probabilistic Programs". en. arXiv: [cs, stat] (Oct. 2016). arXiv:
[cs, stat].
Ritchie, Daniel, Thomas. Anna, Hanrahan, Pat, Goodman, Noah D. "Neurally-guide" procedural models: learning to guide procedural models with deep neural networks". arXiv preprint arXiv: 1603.06143 (2016).
Roeder, Geoffrey, Wu, Yuhuai, Duvenaud, David K. "Sticking the Landing: Simple, Lower- Variance Gradient Estimators for Variational Inference". Advances Neural Information Processing Systems. Ed. by I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, R. Gamett. Vol. 30. Curran Associates, Inc., 2017. URL: https: https::/ /ptp///eooece....e//////// wwwrips cc/ cc/paper .df.
Salimans, Tim, Kingma, Diederik, Welling. Max. "Markov chain monte carlo and variational in- ference: Bridging the gap". International Conference on Machine Learning. 2015, pp. 1218- 1226.
Schulman, John, Heess, Nicolas, Weber, Theophane, Abbeel, Pieter. "Gradient Estimation Using Stochastic Computation Graphs". Advances in Neural Information Processing Systems 28. Ed. by C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, R. Garnett. Curran Associates, Inc., 2015,
pp. 3528-3536.
Stites, Sam, Zimmermann, Heiko, Wu, Hao, Sennesh, Eli, van de Meent, Jil-We. "Learning Proposals for Probabilistic Programs with Inference Combinators". Proceedings of the 37th
12

---

Conference on Uncertainty in Artificial Intelligence (UAI 2021) (Mar. 2021). arXiv: 00668 [cs, stat].
Tucker, George, Lawson, Dieterich, Gu, Shixiang, Maddison, Chris "Doubly reparameterized gradient estimators for monte carlo objectives". arXiv preprint arXiv:/810.04//: (2018).
Wang,
Tongzhou, Wu, Yi, Moore, Dave, Russell, Stuart J. "Meta-lcarning MCMC proposals". Advances in Neural Information Processing Systems. 2018, pp. 4146-4156.
Wingate, David, Weber, Theo. "Automated Variational Inference in Probabilistic Programming". arXiv preprint arXiv: 11X1v1130..199 (2013), pp. 1-7. arXiv: 1301. 1299.
Wu, Hao, Zimmermann, Heiko, Semnesh, Eli, Le, Tuan Anh, Meent, Jan-Willen van de. "Amortized Population Gibbs Samplers with Neural Sufficient Statistics". arXiv preprint arXiv: 19/1,01382 (2019).
13


Reviews:
[{"review_id": "k1DCBN_buc7", "reviewer": "Reviewer_PZ8J", "summary": "The paper proposes Nested Variational Inference (NVI), a method to train (unnormalized) intermediate distributions between some simple initial approximation and an intractable target. The formulation is done in a general way, and compatible with several algorithms, such as annealed importance sampling (AIS) and sequential monte carlo (SMC).", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- Up to my knowledge the method proposed in the paper, NVI, is novel.\n- In some ways the method is close to AVO (Huang et al 2018), it proposes a sequence of K distributions bridging some simple initial approximation with the target, and proposes to learn forward and reverse kernels between bridging densities. These kernels are conditional densities differentiable in its parameters. NVI has two main advantages over AVO: it can learn parameters of the intermediate densities (e.g. annealing coefficients), and it can be combined with proper weighting preserving operations (such as MCMC kernels). I believe that the latter could possibly be used with AVO too, though the original AVO paper does not address this is any way.\n- Compared to AIS (or SMC), an advantage is that NVI allows the learning of intermediate densities' parameters, and the parameters of some additional transition kernels.\n\n\nWeaknesses:\n- As I understand it, NVI cannot tune parameters of MCMC kernels (since these usually require a metropolis-hastings correction). However, the authors show that NVI may be combined with MCMC kernels or re-sampling (without tuning their parameters) in a simple way, since these transitions preserve \"proper weighting\". However, I'm not completely clear on how much the kernels allowed by NVI (q and r in the paper) would actually help, when compared to the addition of powerful MCMC kernels, such as HMC, used to improve samples while preserving proper weighting. This of course depends on the form chosen for q and r, but could be checked for a few simple choices (e.g. Gaussians as in the first set of experiments).\n- Related to the comment above, I think it would be great if the paper included an empirical comparison against powerful methods such as SMC and AIS. Different questions could be addressed: How much tuning the bridging coefficients using NVI helps over AIS (all other things, like MCMC transitions used, kept constant)? How much does having additional learnable kernels (such as Gaussian with learnable mean and covariance) help?\n- The method uses biased gradient estimators to tune all of its parameters. I understand that this bias will decrease as the number of intermediate densities is increased, or if powerful \"proper weighting\" transitions are used. However, I think that a discussion and an analysis to understand the effect of this bias be useful. The analysis could even be a simple empirical evaluation, comparing the method's final performance when using more samples to estimate gradients (or any other way to reduce bias) in a few different settings (preferably high dimensional, self-normalized importance sampling is known to have issues in high dimensions).\n- The objective being optimized (eq. 6), while intuitive, is not really justified (can it be mapped to some bound as the typical ELBO?).\n\n\nComments on empirical evaluation:\n- As I see it, the method does three things jointly: (1) tuning intermediate densities; (2) tuning some transition kernels; (3) allows the use of proper weighting preserving transitions. Other methods achieve only subsets of these (e.g. AVO only (2), AIS and SMC (3), and there are some methods for (1)). It would be nice to have different experiments to verify how much benefits are obtained from each of (1) (2) and (3), and how this changes as more bridging densities are used.\n- While the mixture of Gaussians is a nice example and allows for nice visualizations, it is a 2 dimensional toy problem. It would be great to see comparisons in more plain inference tasks using real models in higher dimensions. I really like all the metrics and results reported for the task, but I think that adding other realistic tasks (e.g. could possibly check models in the inference gym [1], hierarchical regression, bayesian linear regression, etc) would improve this part of the empirical evaluation significantly. (Second experiment is also on synthetic data.)\n- For the BGMM-VAE, the mapping to the NVI framework was not very clear to me.\n- Overall, I think it would be nice to have an empirical evaluation on more tasks used by previous work (and with real models and data), using the typical performance measures. For instance, the method could be used to train a \"normal\" VAE, that is, using only z as latent variables (with a std Gaussian prior). Does NVI lead to better log-likelihoods than other methods typically used for the task? (e.g. a plain diagonal Gaussian approximate posterior, importance weighting, ...).\n\n\nClarity: I think there's room for improvement. Some comments (and personal suggestions) next:\n- While the whole formulation is done in a general way, I think that some instantiation of the ideas at some early point in the paper could help. For instance, right before Section 3.1 a paragraph giving a possible choice for the forward and reverse kernels could be given (e.g. Gaussian with mean and variance as learned functions of the previous samples). I think this could help, specially because the terminology forward and reverse kernels is also used in the context of AIS and SMC, where they are taken to be MCMC kernels. As I understand, in this work the kernels are required to be differentiable (MCMC kernels are not), thus the requirements on the kernels for each method is quite different. In addition, MCMC transitions may be combined with NVI, but they would not play the role of the transition kernels used in the paper. Personally, I think adding a few comments on this may help.\n- The experimental section could include a more thorough description of the tasks in the main text.\n- Could include a more explicit mapping to the NVI framework using the notation from Section 3 for the BGMM-VAE experiment, specifying clearly what are the intermediate distributions (\\hat \\pi, and \\check \\pi (inverted hat)), forward/reverse kernels (q and r), etc.\n\n[1] https://github.com/tensorflow/probability/tree/main/spinoffs/inference_gym", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "One limitation the authors mention is that the use of forward KL divergence tends to be unstable in some scenarios. Is there an additional discussion about limitations that I'm missing?", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "H4uvtWh_DxW", "reviewer": "Reviewer_4J52", "summary": "The authors introduce a new inference and approximate weighted sampling framework that combines f-divergence variational inference and nested importance sampling. An original variation sampling density is transformed through a series of weight preserving operators that map the proposal to an approximation of an unnormalized target distribution. On the other hand, reverse transformations are use to transform the target back to the sampling distribution. The transformations are trained by minimizing an f-divergence at each nesting level. ", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper integrates known techniques such as variational inference and nested importance sampling in an interesting way. The sequential nature of the transformation from a simple sampling density to a complex distribution reminds me of diffusion-based generative models and I would be interested in seeing if this kind of important sampling approach can be useful in that context.\n\nHowever, the proposed approach is rather complicated and the paper fails to show convincing use cases where this complexity is needed. The most obvious use case for the approach is to learn paths for annealed importance sampling. If this were the goal of the authors, the paper should have covered the theory of annealed importance sampling in greater details and compared the new approach with the existing alternatives. The experiment section should have accordingly been focused on comparing the relative performance of different AIS paths in a wide range of relevant inference problems. \n\nIn general, the presentation of the method is too generic. The authors do not commit to any specific transformation operators. The result is a very large family of possible methods, each with very different performances and characteristics. It is really difficult therefore to evaluate the proposed method. \n\nThe experiments are unfocused ad non-systematic. The experiment with multimodal densities is relevant. However, the set of baselines is rather idiosyncratic and it does not represent the current state-of-the-art in density estimation. The experiment with state-space models is interesting but It si in my opinion outside the main scope of the paper. Application of the method to a Bayesian filtering setting introduces several additional complexities that need to be addressed in the body of the paper. I would suggest to cover the filtering application in an independent follow up work. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors discuss the limitations of their approach only briefly. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "Gd1smAaA6VJ", "reviewer": "Reviewer_UVCx", "summary": "The authors present a method to jointly learn a series of proposals and intermediate densities for e.g. SMC, or annealed importance sampling. They use a novel objective which sums divergences between distributions over each adjacent pair of variables in the chain, and argue that this leads to lower-variance gradient estimates than prior work which uses a single objective for the entire sequence of variables. They show experimentally the benefits of this method in three different settings.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "# Novelty\nThe method introduced is novel AFAIK, and well situated with regard to prior work.\n\n# Strengths\n- The paper is well-written (modulo some minor points below).\n- The paper presents a convincing case in Section 3.3 for why NVI's gradient estimates could be expected to be low-variance.\n- The experiments are diverse and the results are convincing - particularly for the annealing experiment in 4.1.\n\n# Questions/weaknesses\n- It is counter-intuitive to me that \"full optimization yields poor results\" compared to partial optimization in Section 4.2. Do the authors have any idea why this might be?\n- In Table 1, very similar results are reported for several methods (particularly AVO-flow and NVI*-flow). It would be helpful if confidence intervals were included, or if the authors could at least comment on the significance of the differences. Similarly for $\\log \\hat{Z}$ and ESS in Figure 6.\n\n# Presentation\n- In Section 3, I think it should be made clearer than $\\gamma_k$ can be learnable. This was a point of confusion for me until reaching the experiments section.\n- In Section 3, line 82, it is confusing when you say you can optimize Equation 6 purely with respect to $q_1$ and $\\pi_k$, since Equation 6 also depends on $r_k$ and so it is unclear that is done with $r_k$ in this case. After reading Section 4.2, I am assuming that this is only done in problems where $r_k$ is omitted (as in Equation 20), but it would be helpful if this was made clearer.\n- I am a bit confused about that is in bold in Table 1 - I am not sure why nothing is in bold for the $K=2$ columns and think the $\\log \\hat{Z}$ entry for AVO-flow and $K=4$ should be bold.\n\nEDIT: Having read the other reviewer's comments, I agree with them that the paper could benefit from a significantly more focused presentation and more systematic experiments. I have therefore lowered my score to a 6.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "7"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper introduces an approach to sampling from intractable distributions (NVI) by learning sequences of proposal distributions for nested importance sampling, with the proposal at each level of nesting learned by minimizing the (local) reverse KL-divergence.

The reviewers found the method interesting and novel but felt that the presentation was too unfocused, and the evaluation, while broad, insufficiently systematic to be convincing. In particular, they found the annealed sampling study in Section 4.1 informative but thought that it was compromised by the use of a toy dataset. The paper would be strengthened by including a more realistic dataset in the study, a clearer motivation for the choice of the baselines (i.e. making it clear that the baselines correspond to ablations), and providing more detail about how exactly NVI is applied in each setting. In summary, the paper clearly has a lot of potential, but needs some more work to deliver on it.
Decision: Accept (Poster)