Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: In this work, the authors theoretically analyze the properties of Variational Autoencoders (VAEs) and Conditional VAEs (CVAEs) when the data lies on a low-dimensional manifold. Notably, they demonstrate, under assumptions, VAEs will converge to a solution that will correspond to using the same number of latent dimensions (active dimensions) as the number of dimensions in the data manifold; therefore, allowing for the identification of the number of manifold dimensions. They also extend the analysis when the data lies on a mixture of low-dimensional manifolds. Further, the authors analyze several active practices in their framework and provide theoretical insights into why they may or may not work. Finally, the authors offer some experimental support to their claims. 
strengths_and_weaknesses: # Strengths
### Theoretical Analysis. 
In this paper, the authors propose theoretical insights into the behavior of VAE and Conditional VAE when the data lies on low dimensional manifolds. Under assumptions, the converged solutions can also be used to determine the manifold dimensions. The authors also analyze several active practices in their framework and provide theoretical insights into why some of them may or may not work. I think there are several interesting and important theoretical insights in this paper. 


# Weakness
### Theorem 1 
I am unclear about what it means to find an optimal solution under $\gamma = \gamma_0$. From definition 1, it seems that $\kappa$-simple VAE is defined under the assumption of $\gamma$ being a tunable parameter. Can the authors comment on what I am missing here? 

### Definition 4
What is $\psi(x)$ here? Is it the same as L52? Are these the first $t$ dimensions of that vector? Some $t$ dimensions? It is a bit unclear to me. 

### Choosing active dimensions 
I am a bit confused about how the active dimensions are being determined. What does $O(\gamma_0)$ mean in practice in Theorem 1 and 2 mean? Can't there be a huge constant with big-O notation? How do you decide that a dimension is active? I think this crucial to the entire paper, and I could not find any discussion on this.

### Qualitative experiments and visualizations 
Can the authors comment on the lack of visual toy problems? It is often easier to argue with a running example for manifold learning papers. However, I believe the authors can exploit this when discussing the different active practices and how they can harm the final learned model; clear visual benefits for hypothesized techniques can be compelling. 

Further, there are no qualitative experiments in the paper. It would be better if there were visual comparisons alongside the numbers. It is difficult to put the numbers by themselves in perspective without a visual yardstick. 

### Synthetic Experiments 
In L217, the authors say that the generated results are controlled by parameters $r, d, t,$ and $\kappa$. I am unsure how $\kappa$ influences the synthetic data generation process. Isn't $\kappa$ a parameter of the inference? 

Further, I am unsure if the authors report ELBO or negative ELBO under the NLL column (L226). Also, why not report it as ELBO? If you want to claim NLL, probably do other estimation techniques like Importance Weighted ELBO or as Annealed IS-based estimator. 

Also, I think the NLL values are not really informative as they cannot be compared across the rows? In each row, the data changes, and therefore there is a different target NLL value? I think a more informative experiment would be where you vary $\kappa$ and demonstrate how using more or less capacity changes the NLL, KL, and Recon for the same data? 

Furthermore, if I am correct, the last block of rows should be comparable with the second block rows. So why is the reconstruction error lower when we have less number of AD than $r$? 

### Real World experiments
In the introduction, L32, the authors mention how the different digits can lie on a different manifold. It would have been interesting if the authors analyzed their learned manifolds in this light. 
### Writing
The writing of the introduction builds a lot of suspense about what is being addressed. While this may be nice sometimes, providing a summary of the contribution can also be beneficial without forcing the reader to go through the entire section to understand what was done and how. 

The captions to tables should be more informative than they are now. For instance, in Table 3, r = 10 is implied or hidden in the text. Further, it should be mentioned which results do the tables correspond to. For example, it seems to me that Table 2 and 4 are the only real-world results and the rest are all synthetic experiments. The experimental section goes back and forth, and clear descriptions can be helpful. Another example is in Table 7. What is the true number of manifold dimensions?

# Minor things
- L32. "each could **like** on." 
- L34. "expand fully in the ambient space **but become on a** low-dimensional manifold when conditioned."
- Several other typos. Please, do a thorough re-read.

questions: Please refer to the Strengths and Weaknesses section. 
limitations: I think the authors captured the limitation reasonably well in their last comment in the paper. I agree with the authors and believe their insights must be tested thoroughly in different empirical settings. As is, the take-aways from the paper seem minimal if the experiments do not follow the results. Please refer to the Strength and Weaknesses section for more detailed comments.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper studies the behaviour of variational auto-encoders (VAEs) and conditional variational auto-encoders (CVAEs) in the presence of data supported on a low-dimensional manifold.
The authors demonstrate that VAEs are able to learn the intrinsic manifold dimension of the data at optimality.
They also show that a similar result exists for CVAEs, and that effective conditioning should reduce the loss function at optimality.

The authors then probe some common design choices in VAEs and CVAEs, noting the following:
1. Conditioned and unconditioned priors are theoretically equivalent.
2. Learning the decoder variance should result in better performance.
3. A common weight-sharing technique in autoregressive models should be avoided.

The experiments section then directly addresses their claims, particularly on the synthetic dataset with known groundtruth intrinsic manifold dimension.

### EDIT AFTER REBUTTAL

I am happy with the discussion that we've had and believe the clarity of the paper can be easily increased beyond the recent revisions which have themselves already increased the clarity. Thus, I am raising my score to a *Weak Accept*.
strengths_and_weaknesses: 
## Overall Assessment

Along the standard review dimensions of __quality__, __clarity__, and __significance__, we have *both* strengths and weaknesses, while on the dimension of __originality__ we have only strengths.
Altogether, I believe the weaknesses outweigh the strengths to some degree, and thus I would weakly recommend rejection at this time.
If the authors can adequately answer my questions about the paper overall and perhaps address some of the issues with __clarity__, I would be inclined to increase my recommendation.
I'll elaborate below.

## Strengths

1. I consider part (ii) of __Theorem 1__ to be a major strength of the paper. The utility of this result is clear: VAEs which are well-optimized have the ability to estimate intrinsic manifold dimension, under the assumption that we have a reliable method to estimate the number of active latent dimensions.
2. I found the paper overall to be a fairly original work. Of course there are quite strong connections to previous work -- namely the _Diagnosing and Enhancing VAEs_ paper -- but the analysis provided here is sufficiently novel and comes with further useful insights.
3. I found the introduction and motivation of the paper to be very clear, and the general storyline to be straightforward to follow.
4. __Theorem 2__ part (i) is quite sensible and intuitive: adding good conditioning variables can reduce the loss function.
5. I find Section 3.1 to be quite interesting.
6. I think pursuing ideas around "unions of manifolds" to be quite intuitive and worthwhile. I appreciate that this paper is bringing this idea to the table.
7. __Theorem 3__ is also an interesting result that appears to shed light on an issue with a common practical technique.
8. Table 1 is a strong result - it is nice to see that $r$ = AD in almost all cases where $\kappa \geq r$.
9. Table 3 is also a strength - we can clearly see AD = $r - t$ here.
10. Table 8 is a strong result.

## Weaknesses

There are some major weaknesses to which I would first like to devote individual sections, and then I'll comment on minor issues later on.

### On the Relationship Between Active Latent Dimensions and Manifold Dimension

Theorem 1 shows that a VAE trained to optimality will have its number of active latent dimensions exactly equal to the intrinsic manifold dimension of the data.
We would ideally also have some result that directly relates the number of active latent dimensions to the intrinsic dimensionality of the overall generative model, so that we could say samples from the generative model have dimensionality approximately equal to the true data dimension.
Yet I cannot easily see such a link between the two quantities, which reduces the significance of the result.
It is stated in the text that active latent dimensions are those "used for reconstruction"; is it then the case that the "inactive" latent dimensions do not provide any dimensionality to the generative model?

Admittedly, it is possible that I have not fully understood Definition 2 in-depth, but I would also argue that the Definition 2 (and Definition 5 by extension) is itself not very clear.
I am left lacking intuition on what active dimensions actually _mean_: simply stating that the encoder variance goes to zero with $\gamma$ does not provide much insight (and by the way, how does $\gamma_0$ factor in here? It appears that we are replacing $\gamma$ in the $\kappa$-simple VAE definition with $\gamma_0$, but then why do we need to say $\gamma_0 \leq \gamma$?).

Some clarity and further discussion would be appreciated around both of these points here.

### Significance of Theorem 2 / Corollary 2.1 and CVAE Analysis

As far as I understand, Theorem 2 part (ii) and Corollary 2.1 (by the way, I think there is an error in Corollary 2.1: shouldn't the active dimensions be $r-t_1$ and $r-t_2$, respectively?) are statements about the mechanics of the encoder in latent space.
Yet I am again left wondering how this relates to the dimensionality learned by the generative model in data space, as I see no direct correspondence between the two.
Why should changes in effective dimensionality in the latent space correspond to changes in intrinsic dimensionality in the data space?
How does this result in "different data manifolds"?

I am also a bit confused about how the proposed capacity of CVAEs to learn unions of manifolds relates to the original assumption from L61 that states the data lives on a simple Riemannian manifold with constant dimension and a single chart. 
How do we reconcile these two seemingly contradictory points?


### Minor Weakness, Issues, and Calls for Clarification

1. I appreciate that Eq. (1) is still a lower bound on the average log-likelihood of the density $p_\theta$ over the data distribution $\omega_{gt}$. However, I am curious about the authors' opinion of the validity of the maximum likelihood objective for a full-dimensional density in the presence of manifold-supported data? In particular, we can no longer interpret the maximum likelihood objective as minimizing the KL divergence from $\omega_{gt}$ to the distribution induced by $p_\theta$ over $\mathbb R^d$, as $\omega_{gt}$ is not absolutely continuous with respect to $p_\theta$'s distribution.
2. Along the same vein, it is inappropriate to compare NLL/ELBO values across data with different manifold dimensions and thus dominating measures; yet the experiment section appears to do this a few times. Some care here would be prudent.
3. I find Section 3.2 difficult to understand. How can we expect $\gamma$ to adaptively balance the two terms in the loss function when the optimal learning behaviour is simply to make $\gamma$ as small as possible? Based on this analysis, I find it remarkable that learning $\gamma$ can succeed at all.
4. The headline of Section 3.3 does not accurately represent the scope of Theorem 3, which strictly relates to conditional autoregressive models. While Theorem 3 is indeed interesting, the section purports to make a more general statement about weight sharing that does not appear.
5. What is meant by "sufficiently complex" in L70?
6. [4] and [5] are the same citation.
7. L97: It is claimed that "the ratio of the norm term in reconstruction and $\gamma$ is constant, i.e. the data dimensions $d$." Why wasn't this ever checked in practice?
8. Parentheses appear to be missing from the integrand in lines 85, 133, and 172
9. What is the role of $\kappa$ in Theorems 1 and 2 beyond simply $\kappa \geq r$? Why doesn't $\kappa$ appear in the bounds?
10. In definition 4, is $\phi$ the same $\phi$ as in L62?
11. Why does the prior variance take $x$ as an argument in definition 5?
12. L153-155 is written in a very unclear way.
13. L219-221 is a very weak and nebulous discussion.
14. In L249 it is stated that $c = u_{1:t}$, but earlier we have $c = h'(u_{1:t}) = G'u_{1:t}$. Which one is correct? Did you just take $G' = I_t$? 
15. In L239-240 it is stated that "the number of active dimensions equals to data's intrinsic dimension" and L256-257 it is stated that "This characterization of dataset complexity is consistent with their intuitive visual complexity". I would have to strongly disagree with this: in my opinion, MNIST is "less complex" than Fashion-MNIST, so I don't quite understand why it ends up with lower active dimensions and thus lower intrinsic dimension. Would the authors clarify what was meant here?
16. Why is $\kappa = 90$ so large in the Continuous Condition experiment? Is this a typo? Seems very strange.
17. I generally find the details around Table 5 and Table 6 to be lacking. How many datapoints from each class? How is the reconstruction error for each class, i.e. has the CVAE learned each of the manifolds or just the dimensionalities? Is $t$ considered to be $0$-dimensional in the case of Table 5?
18. What is the correct number of AD in Section 4.4?
19. Is there some gap between the theory (Theorem 3) and the result in Table 7 about the equivalence of $p(z)$ and $p(z \mid c)$? It appears that the conditional prior does a better job in the more difficult case of $\log \gamma = 20$ initially. Is it perhaps easier to learn with the conditional prior?
questions: These are the main questions I have of the authors. There are others listed above as well but they are more minor.
1. Can the authors provide an explanation of the link between active latent dimensions and some notion of intrinsic dimensionality of the generative model in data space?
2. How do we reconcile the simple Riemannian manifold assumption with the idea of using conditioning to learn "unions of manifolds"?
3. How can we expect to learn $\gamma$ at all in light of the discussion in Section 3.2?
4. How valid is maximizing likelihood in the presence of manifold-supported data?
5. How can we say that the results on MNIST and Fashion MNIST are consistent with intuitive visual complexity?
6. Can we get $t$ in practice for realistic conditioning variables, and if so can we get more practical understanding of intrinsic dimension from this?
limitations: I think the authors have done a reasonable job of addressing the practical limitations of their work.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper aims to understand the relationship between the true intrinsic dimension $r$ of the data, which is assumed to be smaller than the ambient space dimension $d$ on which the data lives (i.e. the paper assumes the manifold hypothesis holds, $r < d$), and the latent variable dimension $\kappa$ in Gaussian VAEs. The authors aim to prove that, when $\kappa \geq r$, the optimal ELBO achieved by a Gaussian VAE with fixed decoder variance $\gamma$ behaves like $(d-r)\log \gamma$, and that the number of activate latent dimensions (i.e. those whose posterior variance goes to 0) is equal to $r$. The authors argue that this insight can be used to estimate the intrinsic dimension $r$. The authors then aim to extend this result to the setting of conditional VAEs (CVAEs). In a final result, the authors aim to show that the practice of sharing weights between the prior and the approximate posterior, which is commonly used in CVAEs, cannot achieve the same optimal ELBO value as could be achieved without the parameter sharing from their earlier result. The authors then claim that this means the practice of sharing these parameters should not be used.

Unfortunately, the paper is very poorly written, to the point where even mathematical definitions cannot be understood in a precise manner. While I detail why I think this is the case below, I highlight that I am giving this paper a soundness score of 2 not because I necessarily think the results are wrong, but presentation is so unclear that it's very hard to judge. Similarly, I give this a contribution score of 2 not because I think the contribution itself would be bad (I actually think a properly re-written version of the paper could be quite strong), but because once again, it's very hard to judge. My confidence score of 5 aims to reflect the fact that I feel very confident that the paper has very poor exposition.

-------------------------------------------------
UPDATE AFTER REBUTTAL
-------------------------------------------------

The authors have significantly addressed the clarity issues present in their original submission. While I still believe the contribution over [4] on the active dimensions being used at optimality is somewhat oversold, this is not the only contribution made by the paper, and I believe the insights presented about conditioning and intrinsic dimension will be valuable to the community. I have thus increased my score accordingly.
strengths_and_weaknesses: ## Strengths

I believe this paper considers a relevant question in VAEs, whose answer will be of interest to the community. I also believe that the explicit consideration of the interaction between conditioning and how this affects intrinsic dimension to not only be relevant, but also understudied. The authors have correctly spotted a poorly understood area, and I think this type of research to be valuable, when properly communicated.

## Weaknesses

As I mentioned in my summary, I believe the biggest weakness of this paper is poor exposition. Below I give a list of points that are not properly explained in the mathematical exposition paper (in order of appearance):

1. In definition 1, $\gamma$ is called a "tunable scalar", this really makes it unclear whether or not it is a parameter of the VAE or not. It only becomes clear later, after continuing to read the paper, that the authors mean a fixed hyperparameter.

2. Again, in definition 1, functions being assumed to be "sufficiently complex" is not a precise mathematical assumption and should be dropped from formal definitions.

3. In definition 2, $\gamma_0$ is assumed to be "arbitrarily small" satisfying $\gamma_0 \leq \gamma$. This is again highly confusing, as $\gamma$ was previously "defined" as being part of the definition of a $\kappa$-simple VAE, so this phrasing makes it seem like the VAE does not depend on $\gamma_0$. Yet, the requirement that $\sigma_z(x)^2_j = O(f(\gamma_0))$ would be nonsensical if $\sigma_z(x)^2_j$ did not depend on $\gamma_0$. In other words, either $\sigma_z(x)^2_j$ depends on $\gamma_0$, in which case we just have a $\kappa$-simple VAE with $\gamma_0$ and the requirement that $\gamma_0 \leq \gamma$ makes no sense; or $\sigma_z(x)^2_j$ does not depend on $\gamma_0$, in which case it's $\sigma_z(x)^2_j = O(f(\gamma_0))$ that doesn't make sense. The notation from this paper is clearly inspired by that of [A], and I would recommend the authors take this one step further and make the dependence of the involved neural networks on their parameters more explicit, as in [A]. For example, changing $\sigma_z(x)^2$ to $\sigma_z(x; \phi)^2$ would make statements much more mathematically precise, e.g. $\sigma_z(x; \phi^*_{\gamma_0})^2_j = O(f(\gamma_0))$ is much clearer.

4. Again, there is yet more ambiguity in definition 2: the phrasing "where $f$ is any function that goes to $0$ when $\gamma_0$ goes to zero" does not make sense. Requiring that $\sigma_z(x)^2_j = O(f(\gamma_0))$ for any such $f$ is equivalent to requiring that $\sigma_z(x)^2_j = 0$. I believe the authors meant that there exists one such $f$, not that this holds for any $f$. Furthermore, even if I am correct and the authors meant that such an $f$ exists, I believe this would simply be equivalent to requiring that $\sigma_z(x; \phi^*_{\gamma_0})^2_j \rightarrow 0$ as $\gamma_0 \rightarrow 0^+$, which would be a much clearer way to state the requirement (big-O notation is really meant to bound convergence rates, which do not seem to be of relevance in the definition if $f$ is merely required to exist).

5. Theorem 1 says "the global optimal...", which implicitly assumes the solution is unique in $\theta$ and $\phi$, which I do not believe holds if the neural networks involved are "sufficiently complex".

6. Statement $(i)$ in Theorem one provides $\int_\mathcal{X} (d-r) \log \gamma_0 + O(1) \omega_{gt}(dx)$, an integral which does not seem to depend on $x$, and which I believe can be written simply as $ (d-r) \log \gamma_0 + O(1) $. It would also be good to be more precise as to what is considered a constant and what is not in the $O(1)$ notation. Finally, similarly to the point above, it is not clear what is meant by "can be uniquely achieved", and I do not believe this holds: for example, if a particular VAE configuration achieves the optimal ELBO, applying an orthogonal transformation to the input of the decoder would leave $p(x|z)$ unchanged, and thus achieve the exact same optimum using a different function (and changing the encoder appropriately).

7. In definition 3, the prior is missing from the definition of $\kappa$-simple CVAE. While this is not relevant for VAEs as the prior is just assumed to always be an isotropic Gaussian, this is relevant to the ensuing discussion about the prior being learned as well in the context of CVAEs.

8. Definition 4: This is where it really gets incomprehensible. What is meant by subindexing a vector with $t$? That is, $\varphi(x)$ is, by definition, a vector in $\mathbb{R}^r$. In the manuscript, subindexing with an integer has been used before to denote coordinates (e.g. $\sigma_z(x)^2_j$ is the $j^{th}$ coordinate of $\sigma_z(x)^2$, which by the way is actually defined as a matrix in definition 1, but at least this abuse is understandable), so it seems that $\varphi(x)_t$ should be the $t^{th}$ coordinate of $\varphi(x)$, but the authors go on to say that $\varphi(x)_t \in \mathbb{R}^t$ without explaining exactly what $\varphi(x)_t$ means. Is it the first $t$ coordinates of $\varphi(x)$? Furthermore, even if the previous questions were answered, what is $x$ here? The definition involves $g(c) = \varphi(x)_t$, but it is never explained what $x$ is: the left side seems to depend on $c$ but not $x$, and the right side on $x$ but not $c$.

9. Definition 5: Same issues as in definition 2.

10. Theorem 2: Same issues as in Theorem 1.

11. Corollary 2.1: $\theta^*$ and $\phi^*$ are defined as optimal solutions to a CVAE, but given the loss defined for CVAEs in equation 2, these optimal solutions should depend on $c$, and they do not. Usually the objective being minimized for CVAEs is not really equation 2, but rather an expectation of equation 2 over some prior $p(c)$, so it is not clear exactly what $\theta^*$ and $\phi^*$ mean, as they do not seem to depend on $c$.

12. Theorem 3: Once again, the notation for the loss on equation 3 is sloppy and not too clear, as the actual loss should not really depend on $l$, but the RHS does. Is there a missing summation over $l$ somewhere? Or should $x_l$ be replaced by $x_{\geq l}$?

Now I move on to other, non-mathematical, issues I have with the presentation of the paper:

13. The authors write down some issues, (i) through (iv) in the introduction and keep referring to them, and the readers keeps having to go back to see what the point was. It would be clearer to avoid this.

14. In line 161, the prior in CVAEs is defined as $p_\theta(z|c)$, which is rather confusing notation as $\theta$ is already used to denote the parameters of the decoder. This notation obfuscates the discussion about parameter sharing, and becomes even more confusing when using $q_\theta(z|x,c)$ in line 162.

15. Several experiments are carried out on simulated data, but not enough details are given about how this was generated. For example, section 4.3 should really refer to a detailed appendix explaining the setup in more detail.

Finally, the paper has several grammatical errors, typos, and minor points which are technically incorrect; all of which I would encourage the authors to try to fix. While I see these points as very minor compared to the other issues, I list some below:

- line 32: "each could like on a manifold"
- line 33: "manifold dimension of an image of 1 ...": a single image has intrinsic dimension 0, it is the manifold itself which has intrinsic dimension $>0$.
- lines 35-36: "Such an example ... as the conditioning variable" is pretty hard to understand as a sentence.
- line 128: "Theorem.1"
- line 182: "The cost of VAE balances the three modules with the two terms" is also hard to understand.
- Citations are also sloppy: [4] and [5] are the same, and at least one paper is cited as an arxiv preprint rather than as a conference paper, namely [13], which appeared in ICLR 2021. Note that [13] is simply the one I noticed, but I did not extensively check this, and would not be surprised if this happened more than once.


[A] Diagnosing and Enhancing VAE models; Dai & Wipf (2019)

questions: I have the following questions:

16. It is claimed that "this has never been rigorously proven" in the abstract, and that "this has only been proven under an assumption that the decoder is linear or approximately linear [2,4]" when talking about the number of active dimensions estimating the intrinsic dimension. Could you please explain why this is the case? I do not think [4] uses neither a linear nor an approximately linear assumption, and it is also not clear how exactly the results in [4] (particularly the discussion after equation 9 of the arxiv version of [4]) are not rigorous.

17. In Table 4, it is not clear how the AD values are obtained: were these averaged over different $c$'s?

18. How relevant is the VAE setup here? For example, [B] obtains very similar results about the value of the optimal log-likelihood to those of Theorem 1, in a general context of density estimators rather than just VAEs. Similarly, [C] extends the result of [A], which is quite related to the results in the paper being reviewed, from VAEs to density estimators in general. It feels like the ideas presented here about conditional models might be extended to the general setting of density estimators as well. Could you comment on this? To be clear, I am aware that [B] and [C] are very recent and I do not see them not being discussed or compared against in the submitted version as a problem, although I do believe some discussion should be added when the paper is updated.


[B] LIDL: Local Intrinsic Dimension Estimation Using Approximate Likelihood; Tempczyk, Michaluk, Garncarek, Spurek, Tabor & Golinski (2022)

[C] Diagnosing and Fixing Manifold Overfitting in Deep Generative Models; Loaiza-Ganem, Ross, Cresswell & Caterini (2022)
limitations: I find it hard to properly assess the limitations of this work in its current state given its poor presentation. If my doubts are clarified during the review process, particularly point 8 about definition 4, and question 16 about the differences with [A], I will update this section later on. That being said, part of the point of this paper is that intrinsic dimension can be estimated with VAEs, yet no comparison is carried out against other methods of estimating intrinsic dimension, e.g. [D].

[D] Maximum Likelihood Estimation of Intrinsic Dimension; Levina & Bickel (2004)
review_title: null

Meta-review: metareview: **Summary**: This paper studies the behavior of variational auto-encoders (VAEs) and conditional VAEs (CVAEs) when trained on data that embeds a low-dimensional manifold into a higher-dimensional space. The authors demonstrate that VAEs are able to learn the intrinsic manifold dimension of the data at optimality. They also show that a similar result exists for CVAEs, and that effective conditioning should reduce the loss function at optimality. The paper then examines some common design choices in VAEs and CVAEs, observing that conditioned and unconditioned priors are theoretically equivalent, learning the decoder variance should result in better performance, and that a common weight-sharing technique in autoregressive models should be avoided. The experiments section directly addresses their claims, particularly on the synthetic dataset with known ground truth intrinsic manifold dimension.

**Strengths**: Reviewers were in agreement that this is fairly original work that yields several interesting and important insights about the ability of VAEs to estimate the intrinsic dimension of a data manifold [R6JP,ZAHe,pxuR]. Reviewer [ZAHe] sees Theorem 1 as a major contribution; it demonstrates that well-optimized VAEs can estimate the intrinsic manifold dimension, under the assumption that we have a reliable method to estimate the number of active latent dimensions. The reviewer also notes taht Theorem 2 part (i) is sensible and intuitive; adding good conditioning variables can reduce the loss, and that Theorem 3 is also an interesting result that appears to shed light on an issue with a common practical technique. Moreover, reviewer [ZAHe] notes that empirical results are strong and that the introduction and motivations are very clear, and the overall structure is easy to follow.

**Weaknesses**: The main criticisms from reviewers focused on numerous issues with clarifty, with many examples given by each reviewer [R6JP,pxuR]. Reviewer [R6JP] notes that  the paper would be easier to follow if it included some form visualization on a toy problem and included some qualitative experiments, and that there are several aspects of writing that could be improved. The introduction takes too long to explain what the contribution is. Captions could be more informative (examples given). Reviewer [ZAHe] finds that while theoretical results link intrinsic dimension of data to activations in the latent space, there is no corresponding result that links activations in the latent space to the dimensionality of generated data. This is true in both Theorem 1 and in Theorem 2 / Corrolary 2.1. This reviewer also notes a large number of minor issues and/or addressable weakenesses (19 examples given) .


**Author Reviewer Discussion**: The authors provided clarifications on many points to reviewer [R6JP] and have updated the manuscript accordingly. In response to reviewer [ZAHe] they clarified how dimesionality of generator manifold follows from latent activations, fixed the error in corrolary 2.1 pointed out by reviewer, and provided numerous responses to other questions and comments. Reviewer [pxUR] comments that while the contribution relative to (Dai & Wipf ICML 2019) regarding active dimensions is a little  oversold, the paper makes other contributions. 

Reviewer [R6JP] updated their score 5->6. After extnsive discussion, reviewers [ZAHe] and [pxuR] also updated their scores 5->6 and 4->5.  

**Reviewer AC Discussion**: Reviewers were in consensus that author responses had improved the paper. All reviewers indicate that consider this paper above the bar for acceptance, but do think that this paper is somewhat borderline and could also be rejected.

**Overall Recommendation**: The AC is of the opinion that the evaluation and discussion that has taken place for this paper is sufficiently thorough, and will follow the recommendations by reviewers. This is a paper that is just about above the bar for acceptance, but may also need to be rejected to make room for other papers. 
confidence: Less certain
Decision: Accept