Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: This paper has two main contributions:
- It shows that when training (continuous gaussian) Variational Autoencoder, growing gradients are to be expected. It relates those infinite gradients to the fact that one of the goal of such models is to achieve an optimal sparse representation, and that obtaining such representations require having a variance of the observation model going to zero, which in turns imply growing gradients.
- It shows that the property of incurring such infinite gradients is not specific to Variational Autoencoder, but still holds for any kind of Autoencoder with a sparsifying regularization.
- It provides evidence that ad-hoc methods usually used to mitigate those growing gradients, such as fixing the observation model variance, have adverse effect on learning. More specifically, since the case of fixing a high variance has already been covered in the litterature, it empirically shows that fixing the variance low initially (even to a value close to the optimal one, as obtained from a run training jointly the variance, the encoder and the decoder) has an adverse effect on performance.
main_review: Overall, the paper at hand is clear, novel, and provides interesting theoretical and practical insights. The presentation of the paper is good, and notably the introduction is a very natural and well polished exposition of Variational Autoencoders.

I have two mild concerns about the paper:

- I think the way Theorem 4 is stated could be made more formal, which would ease the comprehension and avoid confusions. Currently, given the proof, I think the formal statement that holds is the following:
For any (concave, non decreasing) h with bounded gradients near 0, there exists a dataset X and a class of decoders \mu_x(z, \theta) such that, for an arbitrarily expressible encoder, the global optimum of the loss L_h(\theta, \phi) does not achieve an optimal sparse representation.
With the formulation that is given in the text, I think it could be misinterpreted that the statement holds for all class of decoders, e.g. that the statement would be:
For any (concave, non decreasing) h with bounded gradients near 0, and for all class of decoders \mu_x(z, \theta) there exists a dataset X such that, for an arbitrarily expressible encoder, the global optimum of the loss L_h(\theta, \phi) does not achieve an optimal sparse representation.

- While it is clear, given prior works, that obtaining an optimal sparse representations requires that \gamma should go to 0, it would be useful to make the relationship between \gamma going to zero and gradients becoming unbounded clearer. Notably, it might be useful to strengthen the fact that when \gamma goes to 0, not only the gradient of the loss w.r.t. to \gamma goes to \infty, but also potentially the gradient of the loss w.r.t. the parameters of the decoder, as the numerator of this gradient will typically scale like \sqrt{\gamma}, while the denominator will scale like \gamma. While it may not be possible to provide a formal statement on that, I think this would still strengthen the claim on unbounded gradients, as having unbounded gradients only on \gamma could be viewed as an easily fixable problem (e.g. by specifically normalizing this gradient), but having potentially unbounded gradients on the decoder parameters could prove much harder to deal with.

I think those two concerns are easily fixable, and given the interesting insights provided by the paper, I recommend acceptance of the paper. I will increase my ratings if my concerns are alleviated after rebuttal from the authors.

# Post rebuttal edit:
The authors have addressed my two concerns. Given their response, I am raising my score.


limitations_and_societal_impact: The limitations are, to the best of my understanding, addressed.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The author(s) challenge the conventional notion that infinite gradients are bad. Specifically, they examine infinite gradients in the context of auto-encoders and VAEs. Their theoretical findings suggest infinite gradients are necessary at global optima to achieve their definition (1) of an "optimal sparse representation."
main_review: ### Originality:
- Section 4 of this work is most similar to Dai and Wipf (2019). Dai and Wipf (2019) experimentally showed the benefit of training $\gamma$ (beyond fixing it at a low value). However, Dai and Wipf (2019) focus more heavily on ensuring a valid ancestral sampling process via a two-stage VAE. This article attempts to explain *why* optimizing  $\gamma$ is better than fixing it at an arbitrarily low value. Figure 3 is incredibly informative at showing this benefit. Thus, I feel the author(s) are well-enough distinguished from Dai and Wipf (2019). Although, I might consider moving their citation forward in this section from the last sentence since to the best of my knowledge Dai and Wipf (2019) were the first to recognize the unappreciated benefit of learning $\gamma$, which the author(s) "$\gamma$-dependent smoothing"

### Quality:
- Overall, I think the theoretical findings are well presented.

### Clarity: 
- This paper is exceptionally well written.

- Line 47: "The latter requires that we assume specific functional forms for the encoder and decoder distributions." This is only true for the encoder. The decoder distribution can be any distribution (selecting a distribution with reparameterization gradients is unnecessary since you don't differentiate any MC-integrated quantity w.r.t. decoder distribution)

- I think the author(s) need explain $\theta \in \Theta$ better in the main article. In particular, $\Theta$ is used before it is defined. Can the author(s) please elaborate what is $\Theta$ (or provide an example) somewhere before the supplement?

- Equation 13 drops $\frac{1}{n}$ between lines. This scaling reappears in equation 14.

### Significance: 

-  Theorem 4 is significant to those working on disentanglement based on flavors of the $\beta$-VAE.

- This article explains why Dai and Wipf's (2019) proposal to optimize  $\lambda$ is better than fixing it at an arbitrarily low value. While this explanation (section 4) lacks  the the theoretical scaffolding of the earlier sections, it benefits from well illustrated examples.


limitations_and_societal_impact: As the author(s) discuss in their checklist, the main limitation of their theoretical findings assumes that the global minima can be found. I agree that their work has no obvious way to negatively impact society.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This is a nice theoretical work, with direct practical implications. It could be considered as a follow-up of a recent paper appeared in ICLR 2019 (properly cited), in that it expands the preliminary considerations related to the effects of unbounded gradients (due to an appropriate consideration of the variance of the log-likelihood model adopted in traditional VAEs) on the "optimality" of sparse representations learned by variational autoencoders (VAE).

First, this paper proceeds with a definition of an optimal representation, which amounts to one inducing a zero reconstruction error, and a minimum number of informative latent variables. Here, the assumption is that the input data lies on a d-dimensional space, the learned representation $z$ on a k-dimensional space, and the data distribution on a $r$-dimensional manifold, with $r \leq k < n$.

At this point, the key contribution of the paper is a theorem, that indicates that irrespectively of the functional form of a VAE, optimal sparse representations go hand in hand with infinite gradients. This is inevitable under the modeling assumption of a Gaussian likelihood with $\gamma$ scalar variance times the identity matrix.

A second contribution is to observe that although the intuition would suggest to aim for $\gamma \to 0$, it is useful to consider an annealing schedule throughout the VAE training. This naturally arises if we consider $\gamma$ as a learnable parameter. The practical implication is that $\gamma$ should neither be prematurely set too high (as usually done in the literature) or too low (as the intuition form the aforementioned analysis would suggest): by an appropriate learning of $\gamma$ the resulting latent representation becomes optimal, with visible benefits on standard metrics.
main_review: I really like the prose and narrative of this paper, which nicely (and somehow pedagogically) explain an important phenomenon that has been studied in the literature extensively, albeit with a different set of tools.

I must confess I had troubles starting from sec.2 of this work, as my understanding of an optimal representation related to prior works that are more closely related to information theory: "Information Dropout: Learning Optimal Representations Through Noisy Computation", by Achielle, Soatto https://arxiv.org/abs/1611.01353v3, and "The information bottleneck method" by Tishby et al., https://arxiv.org/abs/physics/0004057. 
So, condition ii) in Def.1 sounds really strange to me: "the number of latent dimensions of $z$ containing no information about $X$ should be maximal". This is somehow clarified in the comments to the definitions, but honestly I find this phrasing very difficult to digest. My interpretation is: if $k > r$, then we want all of the $k-r$ components to behave as the prior.

Another point of confusion is the way that eq(3) and eq(5) are discussed, below eq(5). When you plug back eq(5) in eq(3), you do not only consider the $\log \gamma$ term, but there also is a $1/\gamma$ term to take into account. 

That being said, here is a question.

* Q1: what about situations in which the likelihood is not Gaussian? I assume similar considerations to those made in this work would somehow hold, but it would be nice to discuss this more general case.

Section 3 reads well, and did not pose any major challenges to be appreciated, especially thanks to sec 3.3 which is helpful (I did not check the full proof in the appendix).

Section 4 is again straightforward to appreciate, and the implications are clear. What would be interesting to understand is in the following question.

* Q2: what is the convergence rate of learning $\gamma$? That is, how fast does it reaches the optimal solution $\gamma \to 0$ compared to the variational parameters of the encoder, and the parameters of the decoder? Is there any potential instability in the process? Would it be possible (as illustrated in fig.3) that when $\gamma$ reaches small values, the VAE loss gets stuck into a bad local minimum?

In conclusion, I find this a good paper, which nicely extends existing considerations from the literature. It has some practical implications, but an extensive experimental campaign is already contained in previous work from ICLR 2019.
limitations_and_societal_impact: I don't think the authors clearly point out any technical limitations of their work. In my opinion, one point to address is to discuss  about a more general setting whereby the likelihood is not Gaussian.
review_title: null

Meta-review: This is a strong paper that all reviewers felt should be accepted to the conference; a viewpoint I concur with myself.  Some constructive criticisms have though been raised and I encourage the authors to incorporate these suggestions for the final version of the work.

A few extra comments from my end to help update the paper for the camera-ready version:
- I'm not sure "sparse" is the right term in Definition 1 (i.e. in the term "optimal sparse representation") and could be a bit misleading.  Sparse representations usually refer to cases where *different* sets of latent dimensions will be "activated" for different inputs, rather than the same latents always being redundant.  I would suggest using some other term like "minimal".
- It should be made clear in Eq 6 that this is now an optimization w.r.t Z.
- Even with the discussed generalization of Theorem 4, I think some care is needed in discussing the generality of the results and impossibility of AE frameworks that can satisfy Definition 1 without infinite gradients as even with this generalization, it is still a particular form of loss function rather than any possible AE framework.  As such, some of the higher level assertions made need slightly more careful quantification to avoid overclaiming / misunderstanding.
- I feel that the introduction could be tightened up a little: though nicely written at a low level, it somewhat meanders at the moment and one has to go through quite a lot of basic material before getting to the point of the paper.  In particular, it is only really in the second half of page 2 that the content becomes specific to the paper; before that it is a very generic overview of the area, which is wasteful.  In other words, though (as a reviewer noted) this is a nice introduction to VAEs, I don't think such a comprehensive introduction to them at a general level is needed and this both slightly slows down the early part of the paper and uses up space that could maybe be better utilized.
Decision: Accept (Spotlight)