Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: Summary of the paper: The authors propose a latent variable model RaDOGAGA, a generative autoencoding model. The model is trained via a tradeoff between distortion (the reconstruction error) and the rate (the capacity of the latent space, measured by entropy). The paper provides an analysis of theoretical properties of their approach, and presents supporting experimental results.

Review tl;dr: weak reject, for three main reasons:
(i) While the existing literature around VAEs, beta-VAEs,  and Rate-Distortion theory is mentioned in the related work, the connections are not nearly discussed sufficiently. 
(ii) On top of (i), the derivation of their loss function and architecture is not sufficiently motivated. This is in astonishing contrast to 1.5 pages of main text and 8 pages of (much appreciated!) analysis of properties.
(iii) Given the paper is clearly related to existing approaches in the literature, the experiments would require a much more careful comparison to existing models. It remains unclear why an interested user should favor your model over conceptually simpler generative models with fewer hyperparameters.

Detailed review:

Nota bene: This review is a late reassignment. While I reviewed the paper to the best of my ability, time constraints did not allow me to review parts of the paper in depth.  I am open to reassess my review during the second stage.

Connection to prior art: As a probabilistic, neural autoencoding model, the connections to the family of VAE models are obvious. The loss function (eq. (4)) still looks very much like the ELBO, where the typical conditional log-likelihood was split into two distortion terms. How is this different from e.g. a beta-VAE? Particularly, what is the connection between the rate-distortion analysis of beta-VAE by Alemi et al. and yours? These things need to be discussed explicitly, with more than a sentence or two in the related work section.
A lesser, but still important omission in your discussion of prior work: The Jacobian of the generator has also been studied, even for the VAE, cf. e.g. [1]. I believe this deserves more attention in your assessment of prior art.

Motivation: You use two distortion terms: actual sample vs. undistorted reconstruction. Why is that? What is the interpretation of the multipliers? How do I choose them? Why is a large part of your architecture (the pipeline from x to \hat(x)) actually deterministic? Why are you using the entropy of the prior over the latents, rather than the KL divergence between encoder and a prior? I think an interested reader could learn much more from your paper if you discussed your model embedded in th related work rather than in isolation.

Theory: Due to aforementioned time constraints, I was not able to review the extensive theoretical analysis in depth. Still, I would strongly recommend structuring the respective sections more clearly. Separate model and architecture description from the theoretical analysis; precisely formulate your claims. In particular, state your assumptions clearly. For instance, you assume "that each function's parameter is rich enough to fit ideally" (and similar e.g. in Appendix A). Does this only mean that the true distributions are part of the parametric family? What if this is not the case? Do your parameters need to be in the optimum for your analysis to hold true? 

Given that the full 20-page manuscript spends 10 pages on theory, I think this contribution is not given appropriate space in the main text.

Experiments: There are three experiments: a simple 3D proof of concept; anomaly detection; analysis of the latent state in CelebA. As mentioned in my review of the methods section, I believe the approach to be very similar to established models. None of the experiments provides convincing evidence why I should prefer the new, arguably more complex model.
For instance, I would have much preferred that you investigate properties of your model against alternatives over the anomaly detection experiments, which did not further my understanding of the proposed model. 

Summary: The paper tackles an important problem, namely the lack of control over the latent embedding in autoencoding generative models. I believe the author's contribution can be valuable, and I particularly appreciate the effort to investigate theoretical properties. As is, the case is not sufficiently convincing to be accepted, but I encourage the authors to improve the paper.

Minor comments:
1. While I appreciate a pun, I would recommend to rename the model along with the acronym to a more concise name.
2. Please revise your notation and typsetting. Examples: x1 instead of x_1, f of f(\cdot) instead of f(), \log instead of log.
3. Introduce acronyms before using them (e.g. VAE, MSE, SSIM), even when they seem obvious to you.
4. Please carefully check the manuscript for typos, missing articles, missing spaces etc.
5. Your citations are inconsistent, in that they sometimes use first names, sometimes first name initials, and sometimes no first names.
6. To my knowledge, the term scale function does not have an obvious definition. I think you are simply referring to monotonically increasing functions. Please clarify!
7. Your figures should be understandable without too much context, they need more detailed captions.

[1] http://proceedings.mlr.press/v84/chen18e.html
review_title: Official Blind Review #4

final_score: rating: 3: Weak Reject
review_content: This paper aims to obtain latent representation of data such that probability density for the real space can be calculated correctly from that in the latent space. The authors optimize a loss function that has components related to parametric probabilistic distribution and auto encoder simultaneously. While this might be an important problem (I am not sure), the paper is not written and organized well which makes a through evaluation very difficult. I provide below some of the problems with this the paper:

Why the introduced method is better than VAE as a generative model for capturing the latest representation is not explained well. It is not also used as a baseline in most of the experiments.

The motivation for having the third term in Equation (4) needs to be explained. Also what is h() in the second term. The authors only describe briefly both terms together after they used it here but failed to describe what each term is.  Why there is an h for the second term but not for the third term. h() becomes more clear much later in the paper but when it is used the first time, it not defined. 

I believe A in Equation (5) should be also positive-definite. 

What is L(x) in Equation (8).  It needs to be defined.


Experiments:
1-	It is useful to also plot the original data in space s to see how the results in Figure 2 make sense. 
2-	Figure 3 is not clear.
3-	In the Anomaly detection experiments, the authors make two assumptions that usually do not exist in real-worlds: (1) they assume that they have access to training set that only contains normal cases. (2) They assume that they know the correct rate of anomaly. I think both these assumptions are very restrictive and unreal. While these assumptions are used for all the comparing methods, it is not obvious how different algorithms behave in real scenario. 
4-	Figure 4 and what it represents is not clear. 

Writing Problems:
1-	In the text of paragraph before Figure 1, Eq. (5) in “in the second term of Eq. (5)” is a typo and should be Eq. (4). 
2-	In the paragraph before Figure 1, the following sentence is not complete: “Then, averaging Eq. (4) according to distribution, x~P_x(x) and epsilon~ P(epsilon).”
3-	Section 4.2.1: “there is a difference is PDF → “there is a difference in PDF”
review_title: Official Blind Review #3

final_score: rating: 1: Reject
review_content: The paper propose a noisy autoencoder that considers the jacobian between data and latent spaces to match the corresponding densities. This idea has already been proposed elsewhere, and here it is applied to autoencoders. Overall I had hard time understanding the paper, the motivation, the main contribution or the claim, the model definition and the jacobian method. The paper is poorly written, with lots of issues in math notation and poor motivation and explication of what the sections are introducing, and what parts of the presentation is novel and what is already known. Lots of the math machinery is too vague to follow.

The distribution p(z) is unclear, and whether z is random variable or not. It seems that “z" is a non-random variable, and then adding noise \eps makes it stochastic. However, then p(z) without \eps does not make any sense since z is not random. It seems that p(z) is maybe a prior distribution instead (or maybe the variational posterior?), but then adding \eps noise to an already stochastic variable is strange. Overall I have hard time understanding the motivation of the two discrepancies in eq 4, what is the point of adding more noise to “z”? This seems some kind of noisy or perhaps robust AE variant, but the paper does not explicate this. I have hard time following the eqs 8-15. I am not convinced of the orthogonality argument, and I fail to see what this section tries to show or demonstrate. It seems that eq 14 is the final result, but its difficult to follow due to most terms in eq 14 being undefined. Optimizing eq 14 seems trivial since we can always match pz and pz_\varphi easily with neural networks, or similarly the two x-distributions. 

In the experiment 4.1. the proposed method seems to achieve matching densities, although the distributions are wrongly normalized. How does the density matching improve? All three methods seem to have equally good scatters. The benchmarks on table 1 show clear improvement with the method. The face experiment is unconvincing since the VAE spreads variance across all latent dimensions while RADO seems to compress them to just first 20 or so. If one would visualise the z_100 there would be no variance in RADO and possibly some variance in VAE. The paper also should compare their model to simple MNIST/VAE to highlight what problems are there in standard approaches (such as VAE), and how does the proposed method alleviate them.

Overall the paper is poorly presented and difficult to follow. Despite this the method does seem to work remarkably well, and the Jacobian idea is clearly very promising. Nevertheless in its current form the paper is badly premature for ICLR, and needs a lot more work and polish to be made understandable for wider ML audience.

Minor comments
o Px(x), x1, x2 are probably missing subscripts
o The point of eq 5 is unclear, it seems unnecessary. It also does not contain h(), which is claimed after eq6
o The log pz(z) in eq 4 is not entropy
o eq 8 is unclear, is the dx a derivative, distance or change?
o the $^t$ prefix notation is confusing, what does it mean?
o what is the \sim and line notation in eq 5?
o what are the products in eq9, are these inner products?
o in eq 13 pz, pxd or hat(pxd) have not been introduced or defined

review_title: Official Blind Review #2

Meta-review: Agreement by the reviewers: although the idea is good, the paper is very hard to read and not accurately enough formulated to merit publication.  

This can be repaired, and the authors should try again after a thorough revision and rewrite.
Decision: Reject