Slice Sampling Reparameterization Gradients
Authors: David M. Zoltowski
Diana Cai
Ryan P Adams
Venue: NeurIPS 2021 Spotlight (2021)
Keywords: Monte Carlo gradient estimation
slice sampling
energy-based models
variational inference

Abstract: Many probabilistic modeling problems in machine learning use gradient-based optimization in which the objective takes the form of an expectation. These problems can be challenging when the parameters to be optimized determine the probability distribution under which the expectation is being taken, as the na\"ive Monte Carlo procedure is not differentiable. Reparameterization gradients make it possible to efficiently perform optimization of these Monte Carlo objectives by transforming the expectation to be differentiable, but the approach is typically limited to distributions with simple forms and tractable normalization constants. Here we describe how to differentiate samples from slice sampling to compute \textit{slice sampling reparameterization gradients}, enabling a richer class of Monte Carlo objective functions to be optimized. Slice sampling is a Markov chain Monte Carlo algorithm for simulating samples from probability distributions; it only requires a density function that can be evaluated point-wise up to a normalization constant, making it applicable to a variety of inference problems and unnormalized models. Our approach is based on the observation that when the slice endpoints are known, the sampling path is a deterministic and differentiable function of the pseudo-random variables, since the algorithm is rejection-free. We evaluate the method on synthetic examples and apply it to a variety of applications with reparameterization of unnormalized probability distributions. 

Slice Sampling Reparameterization Gradients
David M. Zoltowski
Diana Cai
Princeton Neuroscience Institute
Department of Computer Science
Princeton University Princeton, 08540
Princeton University Princeton, NJ 08540 dcai@cs, www.ceton..cnncmommmmmm edu
woltowest1@prinncttoncco edu
Ryan P. Adams
Department of Computer Science
Princeton University Princeton, NJ 08540
rpalprinceton.c edu
Abstract
Many probabilistic modeling problems in machine learning gradient-based optimization in which the objective takes the form of an expectation. These problems can be challenging when the parameters be optimized determine the probability distribution under which the expectation is being taken, as the naïve differentiable, Reparameterization gradients make
Monte Carlo procedure it possible to efficiently perform optimization of these Monte Carlo objectives by transforming the expectation be differentiable, but the approach is typically limited to distributions with simple forms and tractable normalization constants.
Here we describe how to differentiate samples from slice sampling compute slice sampling reparameterization gradients, enabling richer class of Monte Carlo objective functions to optimized. Slice sampling a Markov chain Monte Carlo algorithm for simulating samples from probability distributions; it only requires density function that can evaluated point-wise up to a normalization constant, making it applicable to a variety of inference problems and unnormalized models. Our approach is based on the observation that when the slice endpoints are known, the sampling path is deterministic and differentiable function of the pseudo- random variables, since the algorithm is rejection- free. We evaluate the method on synthetic examples and apply it variety of applications with reparameterization unnormalized probability distributions.
Introduction
Probabilistic modeling is powerful approach to inferring latent structure in complex real-world processes, but often presents computational challenges for inference and further downstream tasks. In many modern probabilistic models, inference is recast optimization of probabilistic objective that takes the form of an expectation of a loss function with respect to some distribution. A salient example is variational inference (27 3), where simpler distribution is optimized to approximate posterior distribution by minimizing the Kullback-Leeibee (KI.) divergence to the truth. In particular, Monte Carlo methods for estimating the gradient of the KI. with respect the parameters have extended the use of variational inference to non-conjugate Bayesian models 43 44 and neural networks (3T). Such probabilistic objectives also appear in number of other applications in machine learning and computational science, including the generator loss in generative adversarial networks [5], computing the sensitivity of expectations under the posterior distribution prior
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

hyperparameters 221 19 12), computing the Black-Scholes delta in computational finance [13] and
choosing a design that maximizes the probability of improvement in an experiment [56].
Typically, the probabilistic objectives are comprised of expectations that cannot be computed in
closed form, gradients are often estimated via Monte Carlo sampling [39 Two popular classes of Monte Carlo gradient estimators are the score finction estimator [32
55
4 and the pathwise
(or "reparameterization gradient") estimator [23 24 45
Score function gradient estimators as they apply when the underlying density is differentiable and can be sampled
31
49
are general-purpont from, even if the cost function is not differentiable. However, they often have high variance and are used with variance reduction techniques (e.g.. Ranganath et al. [44]. In contrast, reparameterization gradients apply when the loss function is differentiable and samples from the underlying density can be generated by known deterministic, differentiable transformation of samples from simpler distribution that does not depend on the model parameters. Typically, reparameterization gradients have lower variance than score function gradients, provided the loss function is sufficiently smooth 39.. Therefore, developing effective reparameterized gradient estimators has been an active area of and stochastic backpropagation [3 45
research in sensitivity and perturbation analysis [23 24 13
However, reparameterization gradients are primarily limited distributions with tractable normalizing constants. This precludes their use for complex models of interest such as energy- -based models and non-conjugate Bayesian models (although alternative training methods exist, see e.g. Lawson et al. B3) ). Thus, generalizing reparameterizatioo with MCMC to unnormalized distributions is an important direction for developing effective gradient estimators for complicated models. Indeed, some recent work has focused on reparameterization gradients for unnormalized distributions in specialized problems, including Gibbs samplers with reparameterizable conditional sampling steps [5 and dynamics-based MCMC without accept/reject steps (e.g., Salimans et al. [50. Dai et al. [8]. However, developing estimators for general unnormalized distributions using reparameterized gradients and MCMC is not straightforward with existing approaches. For instance, not all Gibbs sampling steps are reparameterizable using current methods, due to, e.g., rejection sampling M or "Metropolis- within-Gibbs" sampling (7]. Next, dynamics-based MCMC samplers without accept/reject steps are approximate samplers with asymptotic bias. Finally, a key obstacle in applying MCMC methods with accept/reject steps, such as the Metropolis- Hastinggs (MH) (22). Metropolis- Adjustee Langevin (MALA) [16], and Hamiltonian Monte Carlo (HMC) algorithms, is that they do not have differentiable sample paths.
An appealing alternative to MCMC methods currently used for reparameterized gradients slice sampling [1], auxiliary-variable MCMC method that can be applied to unnormalized probability distributions and does not require an accept/reject step or sensitive tuning parameters. Crucially, the lack of an accept/reject step leads to the key observation that for a fixed pseudo-random sequence, the realized slice sampling Markov chain is differentiable with respect to the model parameters. In this work, we develop reparameterization gradients for samples generated from slice sampling that apply to distributions known only up to a normalizing constant. Slice sampling reparameterizztioo gradicnts are broadly applicable to complicated multivariate distributions, such energy based models (HBMs) (55) and non-conjugate Bayesian models. While the generated samples are correlated and the gradient estimates are biased because simulate from a finite Markov chain, demonstrate the efficacy of slice sampling reparameterization gradients in simulations, investigating the bias and variance properties of reparameterized slice sampling in comparison with existing gradient estimators. We then show applications of slice sampling reparameterized gradients to a number of problems in machine learning and statistics in the areas of deep generative modeling, approximate inference, and Bayesian sensitivity analysis.
2 Background
In this work concerned with probabilistic objectives that take the form of expectation of real-valued function l with respect to a base density Po (z),, i...
((0) = [ [((=)]
(1)
Typically the expectation cannot be computed in closed form, so computing the gradient of this objective with respect to the distributional parameters 0 requires stochastic estimates of the gradient of the expectation. Two popular classes of stochastic gradient estimators are score function gradients and reparameterization (or pathwise) gradients; see Mohamed et al. [39] for review.

---

2.1 Continuous reparameterization gradients
Reparameterization gradients are used for problems with a differentiable loss function & and contin- uous density Pg- They apply when samples from pa can be generated by a deterministic transfor- mation fe of samples from base distribution p(e) that does not depend on 0. That is, if ~ p(e) then I = fe( ~ pe (z).. Applying this transformation, the gradient of the objective with respect to I can then be expressed as an expectation with respect to a distribution that does not depend on 0. Assuming that the derivative and integral can be interchanged [39 Section 4.3.1]:
VoL(O)= VolEpte)
I [Vo(ffe())]
Monte Carlo estimates of the gradient are then computed using samples from p(e).
Many examples of reparameterization gradients for continuous distributions exist. "One-liner" reparameterization gradients use simple function to transform the base distribution into the desired parameterized distribution [3] 45 39] The inverse CDF (quantile function) can be used to transform uniform random variables into arbitrary 1D distributions, and gradients can be computed when evaluating and differentiating the inverse CDF are numerically tractable. Additional examples are implicit reparameterization gradients [10), doubly reparameterized gradients [53, reparameterization
of accept/reject sampling [40], and generalized reparameterization gradients [48].
2.2 Markov chain Monte Carlo via slice sampling
For complicated target distributions, direct sampling via, e.g., the inverse CDF, is often not feasible and moreover it is often the case that the target density function can only be evaluated up to an unknown normalization constant; this commonly occurs Bayesian posterior inference where the marginal likelihood is unavailable. Markov chain Monte Carlo (MCMC) algorithms address both of these challenges. An MCMC algorithm is a recipe for constructing a Markov chain that is easy to simulate and that converges in distribution to the target, gaining computational tractability at the cost of initialization bias and correlation between the samples.
The most common recipe for constructing an MCMC transition operator is the Metropolis-Hastimee algorithm (88) which combines easy-to-simulate proposal distribution with randomized ac- cept/reject decision that depends only on a ratio of densities, thereby avoiding the need to know the normalization constant. Variations on this theme include Gibbs sampling, in which the proposal is chosen to be the conditional distribution for a subset of the variables, and Hamiltonian Monte Carlo (HMC) [42) in which the proposal is constructed from fictive dynamical system.
Slice sampling 40 is an alternative approach to MCMC based on the observation that samples taken uniformly from the volume beneath the target PDF have the correct marginal distribution. Slice sampling defines a Markov transition operator that leaves this uniform distribution invariant, and is example of an auxiliary variable MCMC method (along with HMC, Swendsen- Wang [5]], pseudo-marginal MCMC []. and others). It is appealing because such transition operators can be constructed without having to identify a proposal distribution and the updates never get "stuck" as can happen with Metropolis-
it is also generally robust to choices of tuning parameters.
Consider distribution with density pa(x) To where the normalizing constant may not be known. Slice sampling typically proceeds in two steps. Starting from point Wn, height y is sampled uniformly beneath the density at vn such that yo+1 ~ (d,,, m (z,)). The height yn+1 defines "slice" through the probability density given by S {æ y/+1 (x)))) The next point #1 is then sampled uniformly from the set S. There are different methods for sampling uniformly from S. including component-wise updates and construction of hyper-rectangles: here we use random-direction slice sampling to sample from the set S (37) Chapter 29.7). A random direction is generated from uniform distribution over directions, and the direction defines a line segment through the slice with endpoints x and x Finally, a value 2n+1 is sampled uniformly between the endpoints. The most common procedure for finding the endpoints, as proposed by Neal [1), is to use a reversible "stepping-out" out" procedure followed by "interval shrinking" to determine +1 1. In this work we will not use this stepping out procedure and will instead perform a direct search for the slice boundaries. This direct search is key to the present work, as it eliminates the oi-differrnntable accept/reject step that otherwise appears in most implementations of slice sampling.
3

---

T8 (3n)
forward pass
- reverse-mode gradients of x(xN)
y - No (
(x))))
61
E2
a
3
2
NNN
E (xN)
2: 0n Zn x
p((n)) x To TT(
1n+1 x
+ u2 u2 x
Figure 1: Slice sampling. Left: One step of ID slice sampler. Right: Reparameterized slice sampling computational graph with reverse mode gradients shown for a loss computed on the final sample.
3 Slice sampling reparameterization gradients
3.1 Random-direction slice sampling with numerical slice endpoints
Here we describe the steps of random-direction slice sampling in more detail. We consider continuous, unnormalized density Ty (2) with parameters 0 and start from a point Un E Rd.. We sample three random quantities: two uniform random numbers #1, U2 E [0, 1] and uniform random unit vector d IR.. The value #1 determines the height of the slice (x,,) The direction d induces a line through the slice. This line intersects with the density at points where To (æ,, + ad) = 41 to (x., for scalar values a. The intersecting points on the slice S induce the set of a values
{ar R (a + od) un no (xra))
Define the slice endpoints representing the closest intersecting locations on the slice S in the positive and negative directions where Ty (In + ad) = U1 (
as
= + aaatd d. = Im + a d,
where a+ A, == max A. a>0
(4)
a<D
The points a and a are identified using numerical root finding (Appendix A). The next sam- ple #441 is then taken uniformly between the endpoints x and #
In+1 x + (x x In + uza d (1 u2)a d.
(5)
Importantly, a and a are implicit functions wn. d, un and 0, denoted by at (xn, d, un 0). and a (wn, d, u1, 0). Via Equation 5 the next sample iterate #n-1 is then a deterministic function of the previous iterate Vn the parameters and the random variables u1, u2 and
In+1 In + U2 a (In, d, U1, 0) + (1 u1- a (æn, d, u1 0) d.
(6)
In Figure [ and later sections, we use to refer the random draws u1, U2, and d. Notably, the per- iteration cost of identifying the slice endpoints is similar for one-dimensional and higher-dimensioonll distributions, as the algorithm searches for slice endpoints along a single direction per-iteration regardless of the dimensionality of the sampling space.
3.2 Differentiating the slice sampling path
Recall that the goal is to estimate the gradient of an expectation taken with respect to pe(x) x We (x( with unknown normalization constant. Consider the following reparameterization of the proba- bilistic objective [(0) and its Monte Carlo estimator T(()), respectively:
(( = Ep, [[(x)] = Emc)(((x(c;
(7)
((<(sn 0)) =: L(0), where ((n) idd p(c),

---

and where (x;;0) is deterministic function of e and & and p(c) is the distribution of the uniform random variables i, and ui and the uniform random direction d. Given an initial sample 2g, the
slice sampling algorithm generates the samples 21:N == (#n) where V, x<(("); 0) is the
value of the function established in Equation ( evaluated at the previous point In-1 and the
sample (() (uf") U2 d After the forward pass of generating the samples, evaluate the objective function and use reverse-mode automatic differentiation (AD) to compute gradients [17] Section 3.2]. In what follows, we derive the reverse-mode gradients, and for notational simplicity, we will drop the explicit dependence of 2n on and 0. Our derivation uses implicit differentiation to efficiently compute gradients of the slice endpoints, avoiding the need to compute gradients through the numerical root finding algorithm.
Applying the chain rule, the gradient of L(0) with respect to is given by recurrence relation backwards through the samples nn
VoL(O) = [e(o(2)) Va.
(8)
n=l
where Ja(a))) RRdx is the Jacobian of æ,, with respect to 0 E R. Here we observe that the gradient VoL(O) depends on the gradients of the loss with respect to each sample, Væ, L R. For
the final sample x this value is simply VZN L = + ( Due to the sequential dependency, for earlier samples n, where 1,.... N the loss gradients need be backpropagated via
Va,L L
N Va, ((xx) + [Ja,, (In+1) )1" VEn+IL. L.
The computational graph demonstrates how each sample xx+1 depends on the previous samples Zin and the parameters 0 (Figure [T). Therefore, to compute gradients with respect to the paramcters and/or the initial condition, the Jacobians Ja.. (In+1) and Io(xn+1) are needed. Using Equation (5) for 2,+1, these Jacobians can be computed indirectly via adjoints, i.e.,
(xn+)))1) = Jæn, (2n + duza+ + d(l u2)a Is + undV 2,, [a+]]+++++ (1 u2 )VV æ,, [a_] Tol (x = To (x:, + duza+ + + d(1 u2)a = u2dVo + (1 - (VolVoll=
(10)
We use implicit differentiation to compute Va, [at]], V V Vo(at] Vola The values are solutions to
f d, a, 0) =lnTo(o, + ad) In In In O. Applying implicit differentiation 117 Chap. 15) results in:
(11)
V F
Va, In To ( + ad) Va, In To (xx)
V a. a
(12)
05/0ce
d' x, In (m + ad)
Vol
Va ln no (xn, + ad) V (x)
Vea =
0f/0a
d m.. In TT + ad)
After computing (V,, (, the gradient VoL is then formed by computing by Equa- tion (8). Importantly, we compute V. without ever fully representing either of the Jacobians by using vector-Jacobian products (Appendix D). We implemented the forward sampling and reverse mode AD in JAX A.
3.3 Gradient of expected log likelihood
In applications, we often require the gradient of the expected log likelihood Eg,( [log qs (x)) with respect to ø. For example, this arises in variational inference when optimizing the ELBO or KL divergence. For a distribution (x) = zt elo(a), the gradient of the reparameterized objective
Bo Dog go (x)) V Mo
(13)
R Vol V log 7
(14)
'Our implementation is available http://ggthabcomm//ttccoolllllls
5

---

standard reparam
scor
slice S-10
slice S-20
Langevin am.5
Langevin a-1
Langevin amy2
1D Gaussian
5D Gaussian (1)
5D Gaussian (2)
5D Laplace
Gradient Bias
slice Se5
2.5
-0.5
-8
0
-3
10
10'
Gradient Variance
10
10
10'
10"
101
-3
0
3
Figure 2: Gradient estimator bias and variance for various Monte Carlo gradient estimators across four different distributions (columns). Slice sampling reparameterization gradients are green.
This appears require the gradient of the unknown log normalizing constant. However, applying the total derivative [46] yields
p(e) V log go (a(())))))
Ppte) v (ogg
V (o
(15)
Ep(e) (pncllaaf((((eee))))ooo Eptc) (V log (x())
(16)
Epp())
(17)
where we dropped the dependence of æ on ø to indicate evaluation at a value æ. We drop the second (16) because is the expected score function and is therefore equal zero. Notably, this gradient of the expected log likelihood does not require the gradient of the log normalizer and we can compute the necessary quantities construct Monte Carlo gradient estimates. In our experiments, we use this estimator when optimizing the ELBO and KL divergence with unnormalized models.
Experiments
Here we apply slice sampling reparameterization gradients to several problems, demonstrating the generality of the approach and the potential appeal of reparameterization gradients for unnormalized distributions. additional experimental details see the supplementary material (Section C).
4.1 Bias and variance
Following Mohamed et al. 39] we quantify the empirical bias and variance of slice sampling repa- rameterization gradients comparison standard reparameterization gradients, score function gra- dients, and reparameterized Langevin dynamics (Figure Z We estimate Volly( Σ, (2; - k) with D for multiple distributions q and various values of k. These experiments show that slice sampling gradient estimates have initialization bias for finite sample sizes, which is expected. Importantly, however, also find that slice sampling reparameterization gradients can be lower variance than score function gradients computed from exact samples from the distribution.
The first example is me-dimmnsional Gaussian with <æx; 8) = Nxxx; f, 1) and 0 = 1. In this case, slice sampling mixes well and the resulting reparameterization gradients are nearly unbiased with variance similar to the standard reparameterization estimator. We next focus on comparing slice sampling with Langevin dynamics on multi-dimensional examples. We consider two 5-dimensional multivariate Gaussian distributions g(æ; 0) (x; 01, V.)) with diagonal covariances V,, and V and Laplace distribution g(æ; 0) II, Laplace(ı;; 0, bz). While Langevin with tuned step size may outperform slice sampling in some cases, we find that the performance of Langevin dynamics

---

target
fit Gaussian D = 0.704
fit y EBM DE (gl 0.030
0
42
-8
21
-2
X1
-2
21
Figure 3: The target (left) and fit mean-field Gaussian (middle) and EBM (righ() approximating distributions. Each panel is shown with the same color scale.
varies considerably for the same set of step sizes across the two multivariate Gaussian distributions. On the other hand, slice sampling performance is consistent across each of the examples and improves with more samples. The versatility of slice sampling is highlighted by the Laplace example. In this case the gradient of the mean from reparameterized Langevin dynamics is zero. On the other hand, slice sampling reparameterization gradients have low bias and notably have substantially lower variance than the score function gradients computed from exact samples from the distribution.
4.2 Minimizing KL-divergence
Minimizing the KL-divergence (KL(Helpp) w.r.t. between a parameterized distribution qe and target p core problem of approximate inference. Here optimize this objective using slice sampling reparameterization gradients on a number of synthetic examples. Our goal for these experiments simply to demonstrate successful optimization using slice sampling reparameterization gradients, and we expect other methods such as normalizing flows or alternative gradient estimators would also perform well (see Section C.2 for comparison to a normalizing flow model). We present one experiment in the main text with additional experiments in Section C.7
To demonstrate the potential utility of unnormalized approximate distributions, we choose 21) "banana" distribution that has a complicated dependency between the two dimensions as the tar- get (Figure 3). We optimize EBM approximate distribution
1
(z(2)
(18)
Z(ő)
where fo is feed-forward neural network mapping from the two-dimensional input space to scalar energy. The Gaussian term is included in the model to ensure integrability. We optimized by
minimizing Dzl(gelln using slice sampling reparameterization gradients and Adam (30].
The fit and target distributions are shown in Figure 3 along with comparison to fitting mean-field Gaussian to the same objective. Qualitatively, the fit EBM appears remarkably similar to the target. We estimated the KL divergence between the target p and the approximate distributions q using numerical integration. We find that the KL divergence between the fit EBM and the target approaches zero and is much lower than the KL divergence between the target and the mean-fiels Gaussian posterior (Figure 3
4.3 Variational contrastive divergence with fully reparameterized gradients
Combining variational inference with MCMC methods is active area of research 50 26
47
One approach is to use samples from a variational posterior to initialize an MCMC chain targeting the true posterior. The distribution of the resulting samples will be closer in KL divergence the true posterior than the variational posterior, making the ELBO tighter. However, this approach can be challenging because the dependence on the variational parameters decreases as the length of the MCMC chain increases and the distribution of the generated samples is not available in closed form.
Ruiz and Titsias [47] propose the variational contrastive divergence (VCD) objective for combining variational inference with MCMC to address these two problems. Let the approximate posterior
be qe (z), and let 4g (x) be the distribution given by sampling initial value z ge and then
(0)
7

---

MNIST
example MCMC sample steps
test log los-llo--llkeee -ll-lllllhooo
zo]
E
94.3
5
5
5
5
5
5
5
5
0
46.0
1
1
1
1
1
1
1
1
1
Fashion MNIST
9
9
9
9
9
9
9
9
9
7
test log-Ikklllhooo
116
33
3
3
3
3
3
3
3
3
120
7
1
1
7
7
2
5
15
20
MCMC iterations
Figure 4: (Left) held out log-likelihood for MNIST and Fashion MNIST as function of the number of MCMC iterations. (Right) The mean observation conditioned on sample z; for z sampled from the variational posterior and for T samples targeting the true posterior.
running 1. steps of an MCMC chain targeting the true posterior p(z æ)). Then the VCD objective LVOD (0) (=) p(x, 2) log go {0 (60 [logp(z, 2) log go (z)]. (19)
(*) and instead only requires sampling from it. However, the second term of the VCD objective is an expectation with respect to MCMC samples, and Ruiz and Titsias 47
Notably, this does not require evaluating dg
score function estimator estimate this gradient. With reparameterized slice sampling we can use fully reparameterized gradients to optimize the VCD objective. Notably, this is an example that requires differential MCMC algorithm obtain reparameterization gradients.
We fit a variational autoencoder (VAE) 31 using the VCD objective to MNIST 34 and Fashion MNIST [58] Our network architectures matched those used in Ruiz and Titsias [7]] the latent dimensionality was 10 and both the encoder and decoder had two hidden layers with 200 units and ReLU nonlinearities. The model was optimized for 400K iterations using ADAM with batch sizes of 100. We also estimated the held-out log-likelihood via the importance sampling approach used in Ruiz and Titsias (47.
45
Table 1: Estimated heldout log likelihoods.
method
MNIST
Fashion MNIST
VCD 47
-95,86
-117.65 -11.002
VCD w/ Reparam (Ours)
-93.99
In [able compare the held-out log likelihoods on MNIST and Fashion MNIST for our method with slice sampling reparameterization gradients and the results reported Ruiz and Titsias [47] for S = 8 samples. We find that slice sampling reparameterizztion gradients perform comparably or better than the original VCD results, with each method having a higher test log likelihood on one of the datasets. As a function of the number of MCMC iterations (Figure the test log likelihood for slice sampling reparameterization gradients appears to scale better than the results obtained in Ruiz and Titsias 47 (Figure 2 in that paper). Finally, for samples generated from the slice sampler, optimization using reparameterization gradients largely outperformed optimization using score function gradients (Appendix C
We also demonstrate that the learned model and slice sampling procedure provide meaningful result. In Figure 4 show the expected value of the observation given a sample from the variational posterior and how steps of slice sampling towards the target posterior improve the latent sample. The resulting conditional mean estimate for 2r appears more similar to the true observation æ than the estimate for 2p. For example, in the bottom row the initial sample appears more like a nine but is transformed to one, which is the true observation.
4.4 Conditional EBM approximate posterior
We next used slice sampling reparameterization gradients to VAE with conditional EBM approximate posterior:

---

(< x))
(xp(fo(x)" (0(z) + log po(2))
(20)
Z(0.z)
The observations are æ, the latent variables are 2 E RD, and log po) is prior. The functions fa(x) and (2(2) map to vectors with embedding dimensionality D.. This differs from the standard VAE in that the encoder / does not map to the mean of the latent space, but rather a separate embedding that nonlinearly combined by g(z) give the energy. The form of the conditional EBM has been studied previously in Khemakhem et al. (28 29). We fit the model to MNIST data (Figure(5) with D 20 and D. 50. Unconditional samples from the generative model look reasonable. We visualized the embedding of held out test images by the function f using t-SNE [36]. Interestingly, we see clusters the embedding space corresponding Finally, baseline comparison we also fit the model using reparameterized Langevin dynamics. The resulting model provided a qualitatively similar embedding of the digits (Appendix C.4
digits, with some overlap for similar digits.
unconditionel samples
visualization of ambedding fg(#)
4
9
4
2 9
5 6
8
4
6
3
9
Dipht
5
3
3
0
3
8 ② 1
SSNNE aade
2
0
3
4
4
8
3
3
3
2
8
8
3
9
8
4
4
3
7
0
4
3
9
3
8
6
9
8
0
7
4
8
8
1-SNE axis
Figure 5: Unconditional samples (left) and embedding visualization (right) of a VAE fit to MNIST with an EBM approximate posterior.
4.5 Adversarial training of EBMs
We next apply our method to training generative adversarial networks (GANs) 15] with EBM generators. Let the generator distribution be pz(x)) efo(*), Standard GANs define an implicit distribution over the data points, whereas an EBM defines an unnormalized log- -igllikel in the data
space. We optimize the Wasserstein GAN 2 objective with the gradient penalty from [8]
min max maxEE.(a)) Ep.(2)) DED
[(x)] - Ep (=) [D(z)] + [E V
D(t)ll2 - 1)
(21)
where are sampled uniformly between pairs of real and generated data points 18 D is the discriminator, and 'D is the set of 1-Lipschitz functions. We trained two dimensional EBMs using this approach on three synthetic datasets, and we find that the distribution of generated samples from the learned energy qualitatively match the training distribution (Figure leff). Our network architecture and optimizer were based on those in [52] 6) and future work we are interested exploring applications of reparameterizztiion gradients for unnormalized distributions to disciminator-driven latent sampling (6) and latent optimization for GANs 57]
4.6 Bayesian sensitivity analysis
The sensitivity of expectations under Bayesian posteriors to prior hyperparameters is a common quantity of interest Bayesian statisties [21] 19]. One formalization of this sensitivity to the prior is as the gradient of the expectation with respect to the prior hyperparameters. Formally, the local sensitivity of an expectation Ep., [g)) at a point au is V.Ep, (+))]la==a) where Pa is the posterior Pa p(0 DD a) 20). Typically the normalizing constant of the posterior is unknown and score function gradient approaches are needed estimate this gradient. However, slice sampling reparameterization gradients may provide lower variance estimator of the sensitivity.
Baseball example
We consider non-conjugate hierarchical model of baseball hit probabilities 5 The model of the number of successes (hits) for player j in K2 attempts (at bats) is
Φ ~ U[ 1]
Parcto(1, 0, Beta( K, (1 - Φ) R)) yj Binomial(K; K,,0;), (22)

---

Adversariialy-trained EBMS
Bayesian sensitivity analysis
SS
estimated value linear approximation
train
gen
energy
1.5
Figure 6: Left: Training and learned generative samples (top) and learned energy function (bottom) for the Swiss roll, two circles, and spirals datasets. Right: Linear approximation to posterior mean hierarchical hit probability based on the estimated sensitivity at a = 1.5.
where determines the mean and 6. roughly determines the variance of the hierarchical mean hit probability across players. The hyperparameter of interest, a, determines the shape of the prior distri- bution on K. We estimated the local sensitivity ValEp. [o] using slice sampling reparameterization gradients. We find that the estimated local sensitivity provides a good linear approximation to how the value of determines the hierarchical mean hit probability (Figure right). We present additional results in the supplementary material (Appendix C
5 Discussion
The slice sampler used in this paper samples random directions from the uniform distribution over directions for identifying one-dimensional slices in higher dimensions. For complex distributions, the dimensionality increases it may be decreasingly likely that the sampled random direction helps to efficiently explore the distribution. This can reduce the sampling efficiency of the algorithm and limit the performance in higher dimensions. In future work, we interested in exploring non-uniform distributions over random directions.
Limitations
Next, the algorithm relies on identifying the closest points that satisfy the slice boundary condition, which are the slice endpoints. To identify these points, we first "step out" from the current point until a bracketing point is found before rumning the root finder algorithm (Appendix AA. Ideally, this bracket contains only one point that satisfies the boundary condition, which is the slice endpoint. However, for multimodal distributions this is not guaranteed, and therefore finding the slice boundaries can be inexact for multimodal distributions. In particular, this more challenging when modes are closely connected such that there multiple nearby points that satisfy the slice endpoint condition. We can check if the next sampled point is on the slice to verify if this is an issue, and modify the stepping out procedure probe more finely spaced locations if needed.
Conclusion In this work we developed slice sampling reparameterized gradients and demonstrated the method variety of applications involving approximate inference, energy- based models, GANs, and Bayesian statistics. Among many possible directions, in future work we are interested in combining reparameterized slice sampling with normalizing flows to improve the geometry of the sampling space (25), developing reparameterization gradients for other classes of slice samplers, and continuing explore applications of reparameterization gradients of unnormalized distributions.
Societal Impact The potential negative societal impacts of our work are broadly the harm that can be done using reparameterization gradient estimators optimize models. One example would be optimizing GANs for harmful applications such as "DeepFakes."
Acknowledgments and Disclosure of Funding
We thank Dougal Maclaurin for valuable discussions. Additionally, we thank members of the Princeton Laboratory for Intelligent Probabilistic Systems and the anonymous reviewers for valuable feedback. This work was partially funded by NSF IIS-2007278. D. Zoltowski was supported in part by NIH grant T32MH065214. Cai was supported in part by a Google Ph.D. Fellowship in Machine Learning.
10

---

References
[T] C. Andrieu and G. Roberts. The pseudo-marginal approach for efficient Monte Carlo computations. The Annals of Statistics, 37(2):69--255 2009.
[2] M. Arjovsky, S. Chintala, and L.. Bottou. Wasserstein GAN. arXiv preprint arXiv: 1701.0787,
2017.
[3] D. M. Blei, A. Kucukelbir, and J. D. McAuliffe. Variational inference: A review for statisticians. Journal of the American Statistical Association, (12(518):899-877 2017.
[4] J. Bradbury, R. Frostig, P. Hawkins, M. J. Johnson, C. Leary, D. Maclaurin, and S. Wanderman-
Milne. JAX: composable transformations of Python+NumPy programs, 2018. URL h //gihub. wwt.eocccooooooee goon//googe/jaar
[5] B. Carpenter, J. Gabry, and B. Goodrich. Hierarchical partial pooling for repeated binary trials. [Accessed 16
URL lutpss:/ccan..--ree hwtp://wwwwwooo.ccccccc/c/////// June 2017], 2017.
[6] Che, R. Zhang, Sohl-Dickstein, H. Larochelle, L. Paull, Y. Cao, and Bengio. Your GAN secretly mergy-basse model and you should use discriminator driven latent sampling. In Advances in Neural Information Processing Systems, volume 33, pages 1225--12287, 2020.
[7] S. Chib and E. Greenberg. Understanding the Metropolis-Hasting algorithm. The American Statistician, 49(4):327-333 1995.
[8] B. Dai, Z. Liu, H. Dai, He, A. Gretton, L. Song, and D. Schuurmans. Exponential family estimation via adversarial dynamics embedding. In Advances in Neural Information Processing Systems, pages 10979-10990, 2019.
[9] B. Efron and C. Morris. Data analysis using Stein's 's estimator and its generalizations. Journal of the American Stafistical Association, 70(350):31 1-330) 1-319, 1975.
[10] M. Figurnov, S. Mohamed, and Mnih. Implicit reparameterization gradients. In Advances in Neural Information Processing Systems, pages 441-452, 2018.
[11] W. R. Gilks, N. G. Best, and K. Tan. Adaptive rejection Metropolis sampling within Gibbs sampling. Journal of the Royal Statistical Society: Series C (Applied Statisties), 44(4):455 472, 1995.
[12] R. Giordano, Broderick, and M. Jordan. Covariances, robustness and variational Bayes Journal of Machine Learning Research, 19(51), 2018.
[13] Glasserman. Monte Carlo methods in financial engineering, volume 53, Springer Science Business Media, 2013.
[14] P. W. Glynn. Likelihood ratio gradient estimation for stochastic systems. Communications of the ACM, 33(10)775-84, 1990.
[15] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Bengio. Generative adversarial nets. In Advances in Neural Information Processing Systems, volume 27, pages 2672-2680. 2014.
[16] U. Grenander and M. Miller. Representations of knowledge in complex systems. Journal of the Royal Statistical Society: Series B (Methodological), 56(4):549-581, 1994.
[17) A. Griewank and A. Walther. Evaluating derivatives: principles and techniques of algorithmic differentiation. SIAM, 2008.
[18) I. Gulrajani, F. Ahmed, M. Arjovsky, V. Dumoulin, and A. C. Courville. Improved training of Wasserstein GANs. In Advances in Neural Information Processing Systems, pages 5767-5777, 2017.
[19] P. Gustafson. Local sensitivity of posterior expectations. Annals of Statistics, 24(1):174-195,
1996.
[20] P. Gustafson. Local Robustness in Bayesian Analysis, pages 71-88. Springer New York, New York, NY, 2000.
[21] Gustafson. Local robustness in Bayesian analysis. In Robust Bayesian Analysis, pages 71-88. Springer, 2000.
[22] W. Hastings. Monte Carlo sampling methods using Markov chains and their applications.
1970.

---

[23] Y. Ho and X. Cao. Optimization and perturbation analysis of queueing networks. Journal of Optimization Theory and Applications. 40(4):559-582, 1983.
[24] Y.-C. L.. Ho and X.-R. Cao. Perturbation analysis of discrete event dynamic systems, volume
145. Springer Science & Business Media, 2012.
[25] M. Hoffman, P. Sountsov, V. Dillon, I. Langmore, D. Tran, and S. Vasudevan. Neutra- lizing bad geometry in Hamiltonian Monte Carlo using neural transport. arXiv preprint arXiv: 2019.
[26] M. Hoffmman Learning deep latent Gaussian models with Markov chain Monte Carlo. In International Conference on Machine Learning, pages 1510-1519, 2017.
[27] M. I. Jordan, Z. Ghahramani, T. S. Jaakkola, and L. K. Saul. An introduction to variational methods for graphical models. Machine Learning, 37(2):183-233, 1999.
[28] I. Khemakhem, D. Kingma, R. Monti, and A. Hyvarinen. Variational autoencoders and nonlinear ICA: A unifying framework. In International Conference Artificial Intelligence and Statistics, pages 2207-2217. PMIR, 2020.
[29] I. Khemakhem, R. P. Monti, D. P. Kingma, and A. Hyvärinen. ICE-BecM: Identifiable condi- tional energy-based deep models. arXiv preprint arXiv:2002.11537 2020.
[30] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv preprint a41i:: 6980, 2014.
[31] D. P. Kingma and M. Welling. Auto-encoding variational Bayes.
arXiv preprint
arXiv:1312.6114 14, 2013.
[32] J. P. Kleijnen and R. Y. Rubinstein. Optimization and sensitivity analysis of computer simulation models by the score function method. European Journal of Operational Research, 88(3):413- 427, 1996.
[33] D. Lawson, G. Tucker, B. Dai, and R. Ranganath. Energy-inspired models: Learning with sampler-induces distributions. In Advances in Neural Information Processing Systems, vol- ume 32, 2019.
[34] Y. LeCun and C. Cortes. MNIST handwritten digit database. 2010. URL http:h/yann......./////// lecun com/exdb/mmt mxxdb/mnist/
[35] Y. LeCun, S. Chopra, R. Hadsell, M. Ranzato, and F. Huang. A tutorial on energy-based learning. Predicting Structured Data, 1(0), 2006.
[36] L. v. d. Maaten and G. Hinton. Visualizing data using I-SNE, Journal of Machine Learning Research, (Nov): 2605, 2008.
[37] J. C. MacKay. Information Theory, Inference & Learning Algorithms. Cambridge University Press, 2002. ISBN 0521642981.
[38] N. Metropolis, A. W. Rosenbluth, M. N. Rosenbluth, A. H. Teller, and E. Teller. Equation of state calculations by fast computing machines. The Journal of Chemical Physics, 21(6): 1087-1092, 1953.
[39] S. Mohamed, M. Rosca, M. Figurnov, and A. Mnih. Monte Carlo gradient estimation in machine learning. Journal of Machine Learning Research, 21(132):1-62, 2020.
[40] C. Naesseth, F. Ruiz, S. Linderman, and D. Blei. Reparameterization gradients through acceptance-rejection sampling algorithms. In Artificial Intelligence and Statistics, pages 489- 498. PMLR, 2017.
[41] R. M. Neal. Slice sampling. Annals of Statistics, pages 705-741, 2003.
[42]
R. M. Neal. MCMC using Hamiltonian dynamics. Handbook of Markov Chain Monte Carlo, (11):2, 2011.
[43] Paisley, D. Blei, and M. Jordan. Variational Bayesian inference with stochastic search. arXiv preprint arXiv: (206. 6430, 2012.
[44] R. Ranganath, S. Gerrish, and D. Blei. Black box variational inference. In Proceedings of the Seventeenth International Conference on Artificial Intelligence and Statistics, pages 81 814- 2014.
12

---

[45] D. J. Rezende, S. Mohamed, and D. Wierstra. Stochastic backpropagation and approximate inference deep generative models. In International Conference on Machine Learning, pages 1278-1286. PMLR, 2014.
[46] G. Roeder, Y. Wu, and D. K. Duvenaud. Sticking the landing: Simple, lower-variance gradient estimators for variational inference. In Advances in Neural Information Processing Systems, pages 1925-69344 2017.
[47] Ruiz and M. Titsias. A contrastive divergence for combining variational inference and MCMC. In International Conference on Machine Learning, pages 5537-5545, 2019.
[48] F. J. Ruiz, M. K. Titsias, and D. M. Blei. The generalized reparameterization gradient. In Advances Neural Information Processing Systems, pages 460-468, 2016.
[49] T. Salimans, D. A. Knowles, al. Fixed-form variational posterior approximation through stochastic linear regression. Bayesian Analysis, 8(4):837-882, 2013.
[50] T. Salimans, D. Kingma, and M. Welling. Markov chain Monte Carlo and variational inference: Bridging the gap. In International Conference Machine Learning, pages 2218-1226, 2015.
[51] R. H. Swendsen and J.-S. Wang. Nonuniversal critical dynamics in Monte Carlo simulations. Physical Review Letters, 58(2)88, 1987.
[52] A. Tanaka. Discriminator optimal transport. In Advances in Neural Information Processing Systems, pages 6813-6823, 2019.
[53] G. Tucker, D. Lawson, S. Gu, and C. J. Maddison. Doubly reparameterized gradient estimators for Monte Carlo objectives. In International Conference Learning Representations, 2018.
[54] A. Vahdat, E. Andriyash, and W. G. Macready. Undirected graphical models as approximate posteriors. International Conference on Machine Learning, 2020.
[55] R. J. Williams. Simple statistical grudient-following algorithms for connectionist reinforcement learning. Machine Learning. 8(3-4):229-255 1992.
[56] Wilson, F. Hutter, and M. Deisenroth. Maximizing acquisition functions for Bayesian optimization. In Advances in Neural Information Processing Systems, pages 9884-9895, 2018.
[57] Y. Wu, J. Donahue, D. Balduzzi, K. Simonyan, and T. Lillicrap. LOGAN: Latent optimisation for generative adversarial networks. arXiv preprint arXiv: a9Xiv: 19/0,0095,2 2019.
[58] H. Xiao, K. Rasul, and R. Vollgraf. Fashion-MNIST: novel image dataset for benchmarking machine learning algorithms. arXiv preprint 17X17:1708.00747, 2017.
13


Reviews:
[{"review_id": "q2NsC2-_CdN", "reviewer": "Reviewer_dDr8", "summary": "The authors present a slice sampling technique for computing reparameterized gradients for unnormalized distributions. They do this by using a deterministic version of slice sampling by computing the end-points of the slice via a root finding procedure that makes the sampling bath deterministic and differentiable. The authors provide comprehensive experimental results in support of the technique on a variety of problems.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I found the general idea proposed in the paper by the authors well motivated and interesting. The main idea has been presented with much clarity by the authors and the intuitive explanations around the procedure followed by mathematical details is most welcome (and refreshing; I wish more papers follow such practice). The idea itself is sufficiently novel and empirically back-up by the experiments by the authors. The experiments are nicely designed to give nice proofs-of-concept for the method. \n\nA small nit I'll add wrt to the experiment in section 4.3 and Fig 4: If possible, it will be nice if the left figure in Fig 4 has the baseline for VCD with architecture similar to that in [45] as well (instead of being refered to another paper). This will make the comparison immediately obvious and I believe the authors can generate that plot easily considering they use the same setup as in [45].\n\nI am also curious about the cost of finding the end-points in general. On surface, it seems this might become trickier (especially in higher dimensions) and wanted to know if the authors encountered any problems around it in their experiments. I am also curious how this whole pipeline of finding the exact endpoints etc effects the running time in general. \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors haven't included a discussion around the limitations of the proposed method. At the least I would consider including a discussion around the *possible* limitations for finding the exact end-points and the computational cost especially when longer chains are need to simulate samples from the target density.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "~6"}, "extra_text": {}}, {"review_id": "l1oW0vXP9J", "reviewer": "Reviewer_Pwj1", "summary": "The authors propose a technique to compute reparameterized gradients w.r.t. parameters of an unnormalized variational density based on slice sampling.\nInstead of performing slice sampling with the common \"stepping out & shrinkage\" procedure the authors use a numeric root finding procedure to determine the slice endpoints, which does not require accept/reject steps and is therefore differentiable.\n", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Summary:**\nThe authors propose a technique to compute reparameterized gradients w.r.t. parameters of an unnormalized variational density based on slice sampling.\nInstead of performing slice sampling with the common \"stepping out & shrinkage\" procedure the authors use a numeric root finding procedure to determine the slice endpoints, which does not require accept/reject steps and is therefore differentiable.\n\n**Originality:**\nThe presented work is novel and presents an interesting new combination of (auxiliary-variable) MCMC and (stochastic) optimization/variational inference.\n\n**Clarity:**\nThe paper is clearly written and easy to follow given it's technical depth. The authors do a good job in explaining the relevant techniques along the way without getting side-tracked.\n\n**Quality:**\nThis is a high quality paper which is reflected in the presentation, writing.\nHowever, with the exception of experiment 4.3, the authors do not evaluate against other suitable state-of-the-art methods nor baseline. Hence, while the authors demonstrate that their method can be applied and provides useful results for a variety of tasks, it is impossible to assess how good these results are compared to existing methods in practice.\nFor example, the value of the KL-divergence in experiment 4.2. seems small and the density seems to be very similar to the target density. But it is unclear if a simple flow-based model or SMC(S)-based model would not have achieved similar or better results. Based on the limited evaluation (only on rather simple target density) it's unclear what the benefits of training neural density approximation with the proposed method are.\nIn my opinion, the paper would benefit from focusing on fewer but more substantial experiments, i.e. experiments with a clear aim and conclusion/takeaway which demonstrate the usefulness of the proposed method compared to existing methods.\nMy takeaway after reading the experiments section was that the method is usable in a variety of settings, but I was not able to clearly see the benefits from using the proposed method over other, possibly simpler, existing methods.\n\n**Significance:**\nTraining variational densities/energies is an important task in reinforcement learning, probabilistic machine learning and beyond. I think the paper is methodologically interesting and makes a significant technical contribution to the field. As such it is of interest to the broader NeurIPS community.\nHowever, as described above, without a more thorough evaluation of the performance of their methods compared to other state-of-the-art methods it's hard to assess the impact/usefulness of the presented contribution in practice.\n\n**Additional Questions and Comments:**\n- I wonder if there is a case where discarding initial samples from the slice sampler, i.e. a burn in period, is useful? Is it the case that even early samples provide a useful contribution to the gradient signal and hence should not be discarded, or are there settings, e.g. highly complicated densities, where the initial samples introduce an unacceptable bias to the gradient signal.\n- Typo on line 215: variance *than* the score function gradients\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "adequate", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "dOUAbkyCO03", "reviewer": "Reviewer_915a", "summary": "This paper introduces an implementation of slice sampling that is compatible with automatic differentiation. This allows differentiation through slice sampling trajectories in probabilistic loss functions when learning generative models. The authors implement their method for different generative learning tasks to show its utility.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The main contributions of this work are 1) introduction of an efficient slice sampling implementation that is compatible with automatic differentiation and 2) a reparameterization method for evaluating gradients through probabilistic loss functions directly (i.e. without the need for MCMC samples). \n\nThe slice sampling method that the authors base their work on is classical and the central innovation for making it differentiable is ensuring the differentiability of the process of finding the local \"width\" of a given slice. This is a nice idea. One major drawback of their sampling method is the lack of information about the landscape geometry when choosing the direction for MCMC updates, since this is chosen uniformly. The standard method Langevin sampling uses the gradient term to guide is trajectories, which can be helpful for exploring high dimensional spaces. I am not sure MCMC methods based on random search directions can scale as well, and it might be interesting for the authors to consider improving this part of their algorithm.\n\nThe reparameterization gradient method introduced in this paper is the more interesting contribution from my perspective. Instead of simply using differentiable slice sampling as a tool for evolving MCMC samples, the authors find an efficient method for differentiating through the slice sampling update process to get loss gradients directly. One part of the method raises some questions. It appears that in the explanation of (7), the authors are using a sampling trajectory $(x_0, x_1, ... , x_N)$ as samples for approximating an expectation with the law of large numbers. This does not seem appropriate, because the samples will be highly correlated since they belong to the same trajectory. It would make more sense to me to use the last sample $x_N$ from independent chains. Unless I am mistaken, it appears that slice reparameterization uses a single chain to approximate the expectation. Further clarification of this would be helpful.\n\nThe experimental results demonstrate a reasonably wide variety of applications for the proposed method. While the scale and complexity of the experiments is somewhat limited, the proposed method seems to work well in different situations. Nonetheless, there doesn't seem to be any significant benefit to the slice sampling as opposed to Langevin sampling except for more straightforward tuning.\n\nI feel like too much space is devoted to experimental results and not enough space is devoted to demonstrating how the reparameterization method works in practice. The authors give a brief derivation of the ELBO reparamterization in the paper and put the rest in the appendix. As far as I can see, there is no derivation of reparameterization of the KL loss for EBM learning, and it is not immediately clear to me how this is done (I would assume the reparameterization trick is used on the term $E_{q_\\phi} [ \\nabla_\\phi f_\\phi (X) ]$ term from the derivative of the log normalizer). Since the experimental results travel well-established ground in terms of the kinds of experiments presented, the paper would be strengthened by focusing more on the details of the new method.\n\n**Overall**: This paper presents some nice ideas for introducing a new slice sampling method and for absorbing sampling trajectories directly into the loss function in generative modeling. More complex experiments (color images perhaps) could help strengthen the paper. There are a few points where further discussion would help strengthen my understanding of the the proposed method.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have adequately address the limitations and societal impact of their work.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5 hours"}, "extra_text": {}}, {"review_id": "Xl_VBLE_pz", "reviewer": "Reviewer_SVGs", "summary": "This paper provides a method based on automatically differentiable slice-sampling to compute gradients of an expectation with respect to parameters of the distribution, even when the normalization constants of the respective distribution are unknown. In such cases, the proposed method provides a low-variance (as opposed to score-gradients) alternative to reparameterization gradients. ", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, the paper is well written and the presentation is clear. The proposed method is clearly useful and the experiments support this further since you 1) compare to other gradient estimators and 2) present experiments with EBMs with unavailable normalization constants. The solution with slice sampling and the observation that the slice sample path is differentiable is very elegant.\n\n### Comments and questions\n- are there cases where finding the slice boundaries is very inefficient or inexact? What happens in this case? Certainly the samples are biased (i.e. not from the target distribution). Is this ever a problem in practice?\n- Figure 2 is quite hard to parse. For example, the caption could mention that the figure titles correspond to the distribution $q(x;\\theta)$ and the parameter $\\theta$ at which the gradient is estimated. This is made clear in 4.1 which is significantly more verbose. Another suggestion is to highlight the proposed method *slice $S=10$* and *slice $S=20$*, e.g. by using a different linestyle or thickness for the corresponding lines. This allows to quickly gauge the performance of the approach and understand the impact of $S$.\n- Section 4.2 (toy example) suggests at first that the proposed method combined with EBMs is the only method that could achieve such a fit but I suppose methods like normalizing flows could achieve a similar distribution, right? It would be good to mention something like this, or that the goal is just to show what's possible when using the method with EBMs that would be hard otherwise.\n- Section 4.4: which other methods would enable this application? If there is no other method that could compete here, state and explain it, otherwise I would suggest to have some sort of baseline here. Surely, the application is interesting and impressive but hard to relate to existing methods.\n- I would be interested see the comparison of runtime for the experiment in Figure 2 to understand how expensive slice samples are. From what I expect, the root finding could take long, especially in higher dimensions. Do you have some results on this?\n- Overall, the experiments section shows the proposed method in a variety of settings and also on a breadth of problems which is nice. However, I think it is important to explain why the proposed method should be used instead of other possible approaches and how these alternative approaches would perform in comparison to your method. If no other method is applicable in these cases, this can be stated and explained. Maybe you could elaborate a bit on the lack of baselines/methods you compare to?\n\n### Typos\n- line 215 *..variance the score..* -> *..variance than score..*?\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The limitations of the work are unfortunately not discussed. For example, the runtime in comparison to other methods is not investigated. Discussion of limitations should be extended. There is no immediate societal impact.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper introduces a differentiable slice sampler and considered its application to variational inference. The reviewers unanimously agreed that this was an important contribution. One reviewer was concerned that the use of uniform search directions were a draw back of the method. Many reviewers felt that the paper could benefit from fewer, more focused experiments. 

After discussion, the reviewers unanimously agreed that the paper should be accepted, as long as the authors make the changes that they propose (including additional discussions on the method and reducing the density of the experiments). I am happy to recommend acceptance. 
Decision: Accept (Spotlight)