Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes a new adaptive multi-stage density ratio estimation framework for learning latent space energy-based model (EBM) in order to overcome the density-chasm problem. Namely, whenever the gap between p (prior distribution) and q (posterior data distribution), the classifier can obtain almost perfect accuracy with a relative poor estimate of the density ratio. The adaptive multi-stage density ratio estimation follow the telescope density-ratio estimation (TRE) approach, learns the latent space EBM through multiple stages. They claim that their work shares similar root as the Product-of-Expert (PoE), each intermediate ratio is treated as individual expert model and it has the potential to produce much sharper distribution than a single model. In training their proposed work, instead of using MLE, they resort to 
multi-stage NCE for obtaining prior models (also avoid using expensive MCMC sampling) and short-run Langevin Dynamics (LD) to infer the posterior. Their empirical results show the effectiveness of the proposed work. They also do some ablation studies to show how to determine stage numbers and the importance of adaptiveness. 
strengths_and_weaknesses: Strengths:
1. The idea of learning EBM via multi-stage density estimation is somewhat novel and also workable for overcoming the density-chasm problem. 
2. This work has done intensive ablation studies to show the importance of stage numbers and whether to use adaptive or non-adaptive. 
3. They have done experiments in SVHN, CelebA and CIFAR-10 to demonstrate the effectiveness of their proposal. 
4. They also provide the workable code and this make reproduction and extensions more feasible.
Weaknesses:
1. This work can still be regarded as an incremental research from the Telescoping density-ratio estimation. 
2. Compared to TRE, it seems that it is much less efficient since each divided component of TRE is done simultaneously and can be trained in a distributional fashion while this work is done sequentially and later stage depends on previous stages. 
3. This work lacks theoretical analysis and proofs to understand the effectiveness and efficiency of proposed approaches. 
questions: 1. Following the sequential nature of this work, we wonder if this work is really much more efficient than MCMC. 
2. The selection of stage number as four based on FID scores is not so convincing. As we mentioned before, this lacks theoretical guarantee and looks not generic either. 
3. Further, they still resort to short-run LD for posterior inferences while current standard EBM usually also make use of short-run LD for inferences. So, one question we have is when the number of stages is quite many, can is it more efficient than the standard EBM models, 
limitations: 1. Please add theoretical analysis or proofs on the stage choices or sample efficiencies if possible 
2. Looks that the constraint of adaptive to sequential learning is not convincing. Is it possible to combine sequential and distributional? 
3. They did not run experiments with TRE, which is the most closely related work. So, we do not have ideas whether their work can really work better than related works.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The authors propose a new method for unsupervised learning of latent models that consist of an energy based prior model (EBPM) and a (stochastic) deep decoder.  For the EBPM estimation, the authors propose to extend the approach of Telescoping density-Ratio Estimation (TRE) (Rhodes et al 2020).  The utility of the approach is evaluated on several image modeling benchmarks (SHVN, CelebA /CelebA-HD, CIFAR-10) and on an outlier detection task on MNIST.  
strengths_and_weaknesses: Strengths:
+ The proposed approach is clearly described, well-motivated, and appears technically correct
+ While the approach uses the same telescoping/recursive approximation idea as TRE, the idea is applied in a sufficiently novel manner to EBPM + generative model estimation
+ All results point to clear benefits of the proposed approach; competing approaches constitute a reasonable selection of SoTAs for unsupervised latent model estimation + most related approach (LEBM)
+ authors provide insightful analyses of ablation wrt #stages and ratio estimation "hardness"

Weaknesses:
- while I do not see any major downsides to the paper, it would be useful to comment on the increased complexity (during learning and inference) introduced by the telescoping ratio-based EBPM

questions: * what is the tradeoff between the increased complexity of EBPM vs a more complex decoder?  Current generative SoTA for CelebA-HD is certainly not based on the simple decoder model architectures used here (c.f., StyleGAN type of models).
* Please comment on the increase in computational complexity of learning (time-complexity) compared to eg simple Vamp prior

limitations: Yes
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes using EBMs in the latent space before being pushed through a latent variable model. The paper proposes learning this EBM using NCE instead of via MCMC sampling. The difference between the prior and the latent posterior is estimated using short-run langevin instead of a recognition network (variational inference). This technique is applied repeatedly to obtain several “stages” of density ratio estimation. 

The paper demonstrates their approach on image generation and reconstruction, as well as on anomaly detection. 

strengths_and_weaknesses: The experiment in Section 5.4 on analyzing langevin dynamics is very interesting. It uniquely shows the benefit of using a latent variable model, with the prior learned by an EBM, and how this results in stable, well-mixed markov chains.

questions: Based on figure 4, it sounds like there are decreasing marginal returns to using additional stages. Is this a correct assessment? How do you propose to choose the number of stages?

limitations: I see no potential for negative societal impact.
review_title: null

Meta-review: metareview: Reviewers unanimously agree that this submission is of good technical quality and well-written. The author proposed an unsupervised learning paradigm for latent variable models, based on EBMs and NCE, by extending the prior work on telescoping density-ratio estimation


This paper proposes using EBMs in the latent space before being pushed through a latent variable model. The paper proposes learning this EBM using NCE instead of via MCMC sampling. The difference between the prior and the latent posterior is estimated using short-run langevin instead of a recognition network (variational inference). This technique is applied repeatedly to obtain several “stages” of density ratio estimation. Reviewers find the adaptive multi-stage method interesting and are convinced that it is effective. Authors rebuttal also helped Reviewers' understanding on this paper. 

confidence: Certain
Decision: Accept