Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: The authors propose a technique to compute reparameterized gradients w.r.t. parameters of an unnormalized variational density based on slice sampling.
Instead of performing slice sampling with the common "stepping out & shrinkage" procedure the authors use a numeric root finding procedure to determine the slice endpoints, which does not require accept/reject steps and is therefore differentiable.

main_review: **Summary:**
The authors propose a technique to compute reparameterized gradients w.r.t. parameters of an unnormalized variational density based on slice sampling.
Instead of performing slice sampling with the common "stepping out & shrinkage" procedure the authors use a numeric root finding procedure to determine the slice endpoints, which does not require accept/reject steps and is therefore differentiable.

**Originality:**
The presented work is novel and presents an interesting new combination of (auxiliary-variable) MCMC and (stochastic) optimization/variational inference.

**Clarity:**
The paper is clearly written and easy to follow given it's technical depth. The authors do a good job in explaining the relevant techniques along the way without getting side-tracked.

**Quality:**
This is a high quality paper which is reflected in the presentation, writing.
However, with the exception of experiment 4.3, the authors do not evaluate against other suitable state-of-the-art methods nor baseline. Hence, while the authors demonstrate that their method can be applied and provides useful results for a variety of tasks, it is impossible to assess how good these results are compared to existing methods in practice.
For example, the value of the KL-divergence in experiment 4.2. seems small and the density seems to be very similar to the target density. But it is unclear if a simple flow-based model or SMC(S)-based model would not have achieved similar or better results. Based on the limited evaluation (only on rather simple target density) it's unclear what the benefits of training neural density approximation with the proposed method are.
In my opinion, the paper would benefit from focusing on fewer but more substantial experiments, i.e. experiments with a clear aim and conclusion/takeaway which demonstrate the usefulness of the proposed method compared to existing methods.
My takeaway after reading the experiments section was that the method is usable in a variety of settings, but I was not able to clearly see the benefits from using the proposed method over other, possibly simpler, existing methods.

**Significance:**
Training variational densities/energies is an important task in reinforcement learning, probabilistic machine learning and beyond. I think the paper is methodologically interesting and makes a significant technical contribution to the field. As such it is of interest to the broader NeurIPS community.
However, as described above, without a more thorough evaluation of the performance of their methods compared to other state-of-the-art methods it's hard to assess the impact/usefulness of the presented contribution in practice.

**Additional Questions and Comments:**
- I wonder if there is a case where discarding initial samples from the slice sampler, i.e. a burn in period, is useful? Is it the case that even early samples provide a useful contribution to the gradient signal and hence should not be discarded, or are there settings, e.g. highly complicated densities, where the initial samples introduce an unacceptable bias to the gradient signal.
- Typo on line 215: variance *than* the score function gradients


limitations_and_societal_impact: adequate
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper provides a method based on automatically differentiable slice-sampling to compute gradients of an expectation with respect to parameters of the distribution, even when the normalization constants of the respective distribution are unknown. In such cases, the proposed method provides a low-variance (as opposed to score-gradients) alternative to reparameterization gradients. 
main_review: Overall, the paper is well written and the presentation is clear. The proposed method is clearly useful and the experiments support this further since you 1) compare to other gradient estimators and 2) present experiments with EBMs with unavailable normalization constants. The solution with slice sampling and the observation that the slice sample path is differentiable is very elegant.

### Comments and questions
- are there cases where finding the slice boundaries is very inefficient or inexact? What happens in this case? Certainly the samples are biased (i.e. not from the target distribution). Is this ever a problem in practice?
- Figure 2 is quite hard to parse. For example, the caption could mention that the figure titles correspond to the distribution $q(x;\theta)$ and the parameter $\theta$ at which the gradient is estimated. This is made clear in 4.1 which is significantly more verbose. Another suggestion is to highlight the proposed method *slice $S=10$* and *slice $S=20$*, e.g. by using a different linestyle or thickness for the corresponding lines. This allows to quickly gauge the performance of the approach and understand the impact of $S$.
- Section 4.2 (toy example) suggests at first that the proposed method combined with EBMs is the only method that could achieve such a fit but I suppose methods like normalizing flows could achieve a similar distribution, right? It would be good to mention something like this, or that the goal is just to show what's possible when using the method with EBMs that would be hard otherwise.
- Section 4.4: which other methods would enable this application? If there is no other method that could compete here, state and explain it, otherwise I would suggest to have some sort of baseline here. Surely, the application is interesting and impressive but hard to relate to existing methods.
- I would be interested see the comparison of runtime for the experiment in Figure 2 to understand how expensive slice samples are. From what I expect, the root finding could take long, especially in higher dimensions. Do you have some results on this?
- Overall, the experiments section shows the proposed method in a variety of settings and also on a breadth of problems which is nice. However, I think it is important to explain why the proposed method should be used instead of other possible approaches and how these alternative approaches would perform in comparison to your method. If no other method is applicable in these cases, this can be stated and explained. Maybe you could elaborate a bit on the lack of baselines/methods you compare to?

### Typos
- line 215 *..variance the score..* -> *..variance than score..*?


limitations_and_societal_impact: The limitations of the work are unfortunately not discussed. For example, the runtime in comparison to other methods is not investigated. Discussion of limitations should be extended. There is no immediate societal impact.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper introduces an implementation of slice sampling that is compatible with automatic differentiation. This allows differentiation through slice sampling trajectories in probabilistic loss functions when learning generative models. The authors implement their method for different generative learning tasks to show its utility.
main_review: The main contributions of this work are 1) introduction of an efficient slice sampling implementation that is compatible with automatic differentiation and 2) a reparameterization method for evaluating gradients through probabilistic loss functions directly (i.e. without the need for MCMC samples). 

The slice sampling method that the authors base their work on is classical and the central innovation for making it differentiable is ensuring the differentiability of the process of finding the local "width" of a given slice. This is a nice idea. One major drawback of their sampling method is the lack of information about the landscape geometry when choosing the direction for MCMC updates, since this is chosen uniformly. The standard method Langevin sampling uses the gradient term to guide is trajectories, which can be helpful for exploring high dimensional spaces. I am not sure MCMC methods based on random search directions can scale as well, and it might be interesting for the authors to consider improving this part of their algorithm.

The reparameterization gradient method introduced in this paper is the more interesting contribution from my perspective. Instead of simply using differentiable slice sampling as a tool for evolving MCMC samples, the authors find an efficient method for differentiating through the slice sampling update process to get loss gradients directly. One part of the method raises some questions. It appears that in the explanation of (7), the authors are using a sampling trajectory $(x_0, x_1, ... , x_N)$ as samples for approximating an expectation with the law of large numbers. This does not seem appropriate, because the samples will be highly correlated since they belong to the same trajectory. It would make more sense to me to use the last sample $x_N$ from independent chains. Unless I am mistaken, it appears that slice reparameterization uses a single chain to approximate the expectation. Further clarification of this would be helpful.

The experimental results demonstrate a reasonably wide variety of applications for the proposed method. While the scale and complexity of the experiments is somewhat limited, the proposed method seems to work well in different situations. Nonetheless, there doesn't seem to be any significant benefit to the slice sampling as opposed to Langevin sampling except for more straightforward tuning.

I feel like too much space is devoted to experimental results and not enough space is devoted to demonstrating how the reparameterization method works in practice. The authors give a brief derivation of the ELBO reparamterization in the paper and put the rest in the appendix. As far as I can see, there is no derivation of reparameterization of the KL loss for EBM learning, and it is not immediately clear to me how this is done (I would assume the reparameterization trick is used on the term $E_{q_\phi} [ \nabla_\phi f_\phi (X) ]$ term from the derivative of the log normalizer). Since the experimental results travel well-established ground in terms of the kinds of experiments presented, the paper would be strengthened by focusing more on the details of the new method.

**Overall**: This paper presents some nice ideas for introducing a new slice sampling method and for absorbing sampling trajectories directly into the loss function in generative modeling. More complex experiments (color images perhaps) could help strengthen the paper. There are a few points where further discussion would help strengthen my understanding of the the proposed method.
limitations_and_societal_impact: The authors have adequately address the limitations and societal impact of their work.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The authors present a slice sampling technique for computing reparameterized gradients for unnormalized distributions. They do this by using a deterministic version of slice sampling by computing the end-points of the slice via a root finding procedure that makes the sampling bath deterministic and differentiable. The authors provide comprehensive experimental results in support of the technique on a variety of problems.
main_review: I found the general idea proposed in the paper by the authors well motivated and interesting. The main idea has been presented with much clarity by the authors and the intuitive explanations around the procedure followed by mathematical details is most welcome (and refreshing; I wish more papers follow such practice). The idea itself is sufficiently novel and empirically back-up by the experiments by the authors. The experiments are nicely designed to give nice proofs-of-concept for the method. 

A small nit I'll add wrt to the experiment in section 4.3 and Fig 4: If possible, it will be nice if the left figure in Fig 4 has the baseline for VCD with architecture similar to that in [45] as well (instead of being refered to another paper). This will make the comparison immediately obvious and I believe the authors can generate that plot easily considering they use the same setup as in [45].

I am also curious about the cost of finding the end-points in general. On surface, it seems this might become trickier (especially in higher dimensions) and wanted to know if the authors encountered any problems around it in their experiments. I am also curious how this whole pipeline of finding the exact endpoints etc effects the running time in general. 

limitations_and_societal_impact: The authors haven't included a discussion around the limitations of the proposed method. At the least I would consider including a discussion around the *possible* limitations for finding the exact end-points and the computational cost especially when longer chains are need to simulate samples from the target density.
review_title: null

Meta-review: This paper introduces a differentiable slice sampler and considered its application to variational inference. The reviewers unanimously agreed that this was an important contribution. One reviewer was concerned that the use of uniform search directions were a draw back of the method. Many reviewers felt that the paper could benefit from fewer, more focused experiments. 

After discussion, the reviewers unanimously agreed that the paper should be accepted, as long as the authors make the changes that they propose (including additional discussions on the method and reducing the density of the experiments). I am happy to recommend acceptance. 
Decision: Accept (Spotlight)