Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper proposes an algorithm to estimate the intrinsic dimension of the latent representations of variational autoencoders. The algorithm is based on observational study based on other methods.
strength_and_weaknesses: ### Strengths
+ Interesting problem will be useful to practitioners

### Weaknesses
- The organization of the paper is confusing (see comments)
- Some of the experiments and the notation in the proposed method are unclear (see comments)
- The contributions are not clearly laid out (see comments)
- As a result, it is difficult to assess the merits of the proposed work, and its contributions.


### Comments/Questions:
1. What is the motivation behind modeling a given data sample as distributed according to Poisson distribution in sec. 2.2? Why does this fit the dataset considered in the study?
2. What does the paper intend to communicate when it says "released more than 35000 IDE scores", what do these scores pertain to? does 35k refer to the number of models? Also, how are these counted?
3. Does FONDUE use the methods described in section 2.2? If yes, how does it use them, and why do the distributional assumptions apply (see #1)? If not, what does it rely on for IDE?
4. Related question to #3: What are IDE_z and IDE_mu. These appear to be scalars? How are these calculated n Algorithm 2? Do these rely on methods described in section 2.2? If so consider clearly defining these in the paper.
5. The organization is pretty confusing. While I like that the study is guided by experiments, and the authors use this to guide the proposed method. It is extremely unclear what the contributions are in terms of methodology. 
6. Also, it seems FONDUE needs to train a VAE at every iteration of its execution? If would be useful to comment on the computational complexity of the algorithm. How does it scale with size of the dataset and the sample itself?
clarity,_quality,_novelty_and_reproducibility: The paper is somewhat unclear. The method is closely related to methods that exist in the literature. The novelty seems to be a successive approximation of the intrinsic dimension based on calls to these methods (see comments). There are links to resources, but I am unsure if they guarantee reproducibility (since results related to some claims will be released upon acceptance). 
summary_of_the_review: I do find the method to be of use in practice, and I am enthusiatic about it. It is just that the paper does not seem to be in good shape at the moment. I hope the authors can reorganize the paper and can clearly identify the technical contributions.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The authors explore the intrinsic dimension estimation (IDE) of the data and latent representations learned by VAEs to measure the optimal size of the latent space. 
They define the optimal dimension as having as many latent dimensions as possible but no passive measurements. 
The authors evaluated their approach on three different datasets of increasing complexity. 
strength_and_weaknesses: ## Strengths
- Interesting and important topic 
- Well written 

## Weaknesses
- The ID calculation seems to be sensitive to the number of neighbors $k$ (as described in section 2.2). It is unclear how to set $k$ in practice for a dataset with no or limited access to ground truth data, and the proposed method does not sound too convincing either.
- No related Work in the main text. What about previous work that worked on automating the number of latent dimensions? It would also be interesting for experimental evaluation.
- I understand that the proposed method is mainly intended for VAEs as it requires the resampling operation. Are there extensions possible to go to a broader range of models? The extensions to autoencoder are a bit ad-hoc by saying we have an autoencoder as soon as we remove the resampling operation.
- Limited evaluation concerning potential downstream tasks. VAEs are often evaluated with respect to their ability to learn meaningful latent representations. It could be an interesting experiment to see the effect of a (semi)-automatic latent dimension specification on potential downstream tasks. 

clarity,_quality,_novelty_and_reproducibility: ### Clarity 

The paper is well-written and describes all the necessary steps. 

### Quality 

The paper is empirically motivated. All performed experiments are described in detail. It lacks experimental comparison to previous works. 

### Novelty 

As far as I know, the idea to use IDEs in VAEs is new. A related work section is missing, which makes assessing the novelty more difficult. 

### Reproducibility 

All the results seem to be reproducible. Code, dataset, and a lot of details are provided. 
summary_of_the_review: Interesting paper, but the lack of related work in the main text and the missing comparison to baselines let weaknesses outweigh the interesting idea.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper explores intrinsic dimension estimation of the data using VAE. They found that this estimation can be made after a few steps of training and proposed a method FONDUE to provide a more principled method for selecting latent dimensions for the VAE.
strength_and_weaknesses: Strength:
1. A thorough experiments across a lot of settings for the intrinsic dimension estimations. 
2. The experiments have averaged over random seeds which is more reproducible
3. The idea itself could be useful in determining latent dimension for VAE during training.

Weakness:
1. To make the paper more convincing, I think it should be discussed why one cannot just set the latent dimensions very big, such that it would be almost definitely contain the IDE. I understand the idea of "very big" is a bit vague, but I think to motivate the paper better, it should add discussion on if or why having significantly larger latent dimension than the IDE for VAE is less than ideal.
2. Another note, I think the word "intrinsic dimension" is often mixed up with the word "intrinsic dimension estimates", for example in conclusion. From my understanding, there should be only one intrinsic dimension for each dataset while the estimates could vary depending on the architecture.  
3. I think the polarized regime is not very well explained. In dai-wipf's i believe some of the values in the encoder covariance go to 1 some to 0, thus polarized. But i think to simply say some collapse some don't, I'm not sure if it's really easy to understand for someone who hasn't read the original paper. i also think, in Dai-wipf's analysis, the intrinsic dimension is of the output diagonal covariance from the encoder, but here it seems to be considering the intrinsic dimension of a matrix, I suppose it's the layer that produces the covariance and mean. But then I am also confused as to what is the intrinsic dimension of a sampled representation. 
clarity,_quality,_novelty_and_reproducibility: I find this paper to be interesting and potentially useful. The clarity on explaining some of the concepts, such as intrinsic dimensions, or polarized regime, could be a bit less hand-wavy. The experiments seem to be reproducible. 
summary_of_the_review: I think this is an interesting paper but the clarity of writing and explanation could have some improvements. Also some discussion on the effect of having significantly higher latent dimension than intrinsic dimension should be discussed for a more well-rounded motivation.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The work presented in this paper is of empirical nature, and attempts at determining the number of latent variables to be used in generic Variational Autoencoder (VAE) models. This is done by relying on two existing methods to estimate the intrinsic dimension (ID) of i.i.d. input samples, that use the concept of neighborhood to approximate the data intrinsic dimension.

The goal of the experimental setup is to study the intrinsic dimensions of the representations learned by VAEs to assess whether they can be used to determine the optimal number of latent dimensions of VAEs. More precisely, the proposed experimental methodology is to compute the ID for all layers’ activations of several variants of a VAE model, each with an increasing dimension of the latent space.
The ultimate goal of this empirical study is to design a new algorithm to automatically find the optimal number of latent dimensions for VAEs in an efficient and unsupervised way.

As for the results of the study, the authors first attempt at validating the two methods to estimate the data ID, which indicates that the used methods tend to overestimate the ID.

Then, the authors move to applying ID estimation to intermediate layers of a VAE, trained with different latent dimensions, and present a number of findings, that can be summarized as: if the dimension of the latent space is correctly defined (such that it is at least equal to the ID of the input data), then the behavior of the ID at each layer of the encoder/decoder architecture follows a well defined pattern, the ID of the latent space corresponds to the data ID, whereas the sampled distributions usually having a much larger ID.

Based on these findings, the authors present the algorithm called FONDUE, which builds on the key observation that the IDs of the mean and sampled representations start to diverge (from the ID of the data distribution) when (unused) passive variables appear, and this is already visible after the first epochs of training. After proving the correctness of the FONDUE algorithm in terms of its termination, a series of experiments show that the algorithm is a computationally efficient alternative to grid search to find an appropriate latent space size.

strength_and_weaknesses: The strength of this paper are:
* This work is very clear and well motivated
* The proposed approach is simple and relies on thoughtful observations from an empirical analysis of the virtues of existing IDE methods


The weaknesses of this paper are:
* I am missing an appropriate evaluation of the impact of an informed choice of the latent space dimension on performance, such as a table with likelihood values. Fig 8 and Fig 9 are difficult to digest, whereas a simple table with likelihood values for different $|z|$ would be perfect.
* Claims about optimality overstate the contribution. There is no proof of optimality in this work, so I suggest the authors to either rephrase their statements, or to try and prove that the output of the FONDUE algorithm is optimal. For example, synthetic datasets are endowed with ground truth number of factors of variation; the value for the latent dimension is larger than the ground truth (due to over-estimates caused by IDE methods used in this work), which implies that the found solution is not optimal
* The threshold $t$ is set arbitrarily and heuristically. Is there any informed principle to pick a good $t$?
* One additional observation is as follows. In practical endeavors, it is frequent to see VAEs or AEs to be used as a “pre-training” step, with a reconstruction objective. Then, decoders are disconnected, encoders are freezed, and the latent representation of the input is used as input to a task-specific architecture. Is the “ideal” latent size obtained by FONDUE still relevant, for example, in terms of downstream performance?
* Apart from Appendix G, which is commendable as it presents an alternative to FONDUE, I do not see any alternative methods, nor a related work section. Is it possible nowadays that nobody else in the huge body of work that exists on VAEs, has explored the same research question addressed in this work?

clarity,_quality,_novelty_and_reproducibility: The article reads very well, it covers the basics with a useful background section, and presents both the experimental protocol and proposed method clearly.
As noted in the previous part of the review, I am surprised to see no related work at all. This also hurts the experimental evaluation of FONDUE as it is not compared to existing alternatives (albeit Appendix G is an effort in the right direction).
Concerning the novelty, this is somehow hard to judge, since no related work is discussed, and since the method borrows known methods as the key enabler for the proposed approach. The FONDUE algorithm per-se is simple (which is not used here as a bad connotation), but it is based on heuristics and approximations (because the methods it relies on are approximations of the ID). A possible suggestion to further develop theoretically grounded bases for this work is to dig into the information-theory spin that several work has proposed to study VAEs, such that those based on the information bottleneck principle, see e.g. [1, 2].

[1] @inproceedings{
alemi2017deep,
title={Deep Variational Information Bottleneck},
author={Alexander A. Alemi and Ian Fischer and Joshua V. Dillon and Kevin Murphy},
booktitle={International Conference on Learning Representations},
year={2017},
url={https://openreview.net/forum?id=HyxQzBceg}
}

[2] @misc{https://doi.org/10.48550/arxiv.1912.00830,
  url = {https://arxiv.org/abs/1912.00830},
  author = {Voloshynovskiy, Slava and Kondah, Mouad and Rezaeifar, Shideh and Taran, Olga and Holotyak, Taras and Rezende, Danilo Jimenez},
  title = {Information bottleneck through variational glasses},
  publisher = {arXiv},
  year = {2019},
}

summary_of_the_review: This paper presents a clear empirical study and a simple heuristic to determine a good size for the latent dimension of a variational autoencoder. My main concerns relate to 1) claims of optimality, 2) lack of comparison with alternatives, 3) a somewhat limited study in terms of practical use of VAEs for learning representations that are useful for downstream tasks.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: There was a consensus among reviewers that this paper should be rejected. While reviewers thought the paper had an interesting idea, the key concerns were that there as no comparison to alternative approaches and that there were no experimental results showcasing the approaches's usefulness for downstream tasks.
justification_for_why_not_higher_score: No reviewer recommended to accept the paper.
Decision: Reject