Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: This paper studies the conditions under which two conditional densities are compatible, with a particular focus on cases relevant to generative modeling, establishing conditions for compatibility and uniqueness. Based on the theoretical results, an objective for regularizing towards compatibility is proposed, implemented, and fit to data. 
main_review: In my opinion, this paper has been missing from the VAE literature for ~7 years. It fills an important hole in the literature. Importantly, the paper operationalizes the theory. 

That being said, the proofs in the appendix are quite involved and time constraints did not permit me to go through them properly. Nothing obvious seems off, but I didn't check them carefully.
limitations_and_societal_impact: Yes.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: Notes: 
  -Model the joint p(x,z) by modeling p(x|z) and q(z|x) to form a cycle.  
  -Many existing techniques use an uninformative prior p(z).  
  -Compatibility and determinacy criterions.  
  -Use parameterized density models p(x|z) and q(z|x).  
  -Define r(x,z) = log(p(x|z) / q(z|x)) and make the loss the norm of the jacobian, computed over samples from p(x)*q(z|x) (real data / inferred latents).   
  -Jacobian is intractable, so instead approximate by taking gradient wrt. x, then multiplying by random vector eta, then taking gradient wrt z.  
  -Sample using stochastic gradient langevin dynamics.  
main_review: Post-rebuttal: I appreciate the clarifications from the authors and keep my score.  

--

This paper proposes a novel criteria for training a joint distribution described by p(x | z) and q(z | x), without the constrain that the z have a simple prior.  The technique clearly works in the sense of producing decent samples and decent latent variables, but results seem unlikely to be state of the art.  At the same time, the analysis and discussion of these cyclical generative models seems like a useful contribution, and gives a more theoretical grounding for this type of algorithm than previous papers in this general area.  

Comments: 
  -Overall model and objective seems reasonably easy to implement and is cleanly explained in the paper.  

  -The experimental evaluation seems rather limited, and the classification accuracies are about the same as GibbsNet and DAE.  The generation quality seems decent, but is not quantitatively evaluated.  

  -Is my understanding correct that the langevin dynamics sampling procedure (section 3.3) is only used at test time and does not need to be used at train time?  

  -Out of curiosity, if you run a chain of z -> x -> z -> x -> ..., does this chain have good mixing properties in theory?  I.e. will the chain eventually be able to produce samples uniformly from p(x).  
limitations_and_societal_impact: Limitation is potential lack of disentanglement, but whether we want this is of course open to discussion.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The work answers an interesting question: Given two cyclic conditional distributions, under which condition there exists a compatible joint distribution, and when the compatible joint distribution is unique. 

From the theory the authors propose a generative framework called CyGen. CyGen is trained using maximum likelihood based on modelings of the conditional probabilities. Sampling from the marginal distribution modeled by the two conditional distributions is also not straightforward. Luckily this has been investigated before and there are practical MCMC based procedures that converge to the marginal distribution.

The authors also show some experimental results that the proposed CyGen is able to generate more realistic samples compared to VAE or DAE. 
main_review: Problem setting of this paper is interesting. The compatibility and determinacy analysis are very clean. The proposed CyGen algorithm, so far as I know, is novel. 

I am a little worried about how useful these cyclic condition settings are. After all, most of the models simply assume some priors instead and they can still generate very realistic-looking samples. Besides the generative settings such as CyGen, is there any other use case for the Cyclic Conditionals?

Another concern is about the practicality of CyGen. Yeah the MCMC based methods are fine but compared to most of the generative models nowadays, it is not as efficient. In the experiment the authors first pretrain the model from VAE, and then use CyGen more like a “finetuning”. I am curious if we run a cold start from CyGen, will we get meaningful results? Will we run into issues such as mode collapse or some types of "bad" local optimum?  

limitations_and_societal_impact:  the authors adequately addressed the limitations and potential negative societal impact of their work
review_title: null

Meta-review: The paper starts with the likelihood and inference model used in variational auto-encoders and studies if and when a joint distribution can be modeled using these two conditionals. The paper is a nice to read and brings to light many of the results on conditionally specified models to the VAE community. I also appreciated a practical implementation with cross-derivatives to encourage the two conditional distributions to be compatible with a joint distribution.  A couple of comments

- I don't think the full Arnold book was cited. It has more results and should be integrated into the narrative 

@book{arnold2012conditionally,
  title={Conditionally specified distributions},
  author={Arnold, Barry C and Castillo, Enrique and Alegria, Jose-Maria Sarabia},
  volume={73},
  year={2012},
  publisher={Springer Science \& Business Media}
}

- Edit the paper to frame and highlight the practical implications of the results 
Decision: Accept (Poster)