Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This work studies the layer-wise representations of different VAEs. The similarity between the layer-wise representations is measured using the Centered Kernel Alignment metric and the Procrustes scores. Experiments are performed to study the effect of different hyperparameters on the same model, regularization, initialization, and learning objectives.
 
strength_and_weaknesses: Strengths:

+The work is well-written.
+The limitations of the approach are clearly discussed. The analysis are performed taking into account the limitations of the proposed metrics.
+The results seem like they should be reproducible given sufficient compute.

Weaknesses:
-The approach is limited to the comparison of representations from similar layers. For example, the similarity between convolutional and deconvolutional layers cannot be evaluated using the proposed approach.
- The findings in the paper, regarding posterior collapse, and encoder learning before decoder to prevent posterior collapse are already well-established in the community and it is not clear what is the main "new" finding of the paper. The metrics, especially CKA [1] have been used to study deep networks in prior work and the work applies the same tools to VAEs, thus there are no new contributions.
 - CKA is claimed to be a metric of choice for monitoring posterior collapse. What makes it a better tool? Why can we just monitor the latents?
- The hyperparameters considered for evaluation is limited to choice of regularization? There are other factors such as optimizers, lr schedulers etc which also can impact training. What are the insights here?

[1] Similarity of Neural Network Representations Revisited. ICML 2019.

clarity,_quality,_novelty_and_reproducibility: Clarity:
The paper is well-written. The advantages and limitations of the proposed metrics are clearly discussed and the experimental results are discussed in-line with the limitations. The reason to choose specific metrics can be elaborated.

Quality:
The experimental set-up is outlined clearly. The experiments are detailed. The approach and the metrics are well-explained.

Novelty:
CKA has been used in  [1] to study the representations of different deep learning frameworks (not VAE in particular). 
The paper does not provide a new streamlined procedure or any new insights over the former. Therefore, while the findings are well demonstrated through experiments, the originality is limited.
[1] Similarity of Neural Network Representations Revisited. ICML 2019.


Reproducibility:
The work should be reproducible.

summary_of_the_review: Overall, the paper is well-written and the experiments are conducted to evaluate the similarity of different representations from the VAEs. The findings in the paper such as  "encoders learn before decoders" are already well-known in the community. Moreover, the paper argues that linear CKA can be used to track posterior collapse. The motivation for this claim is not clear. Experiments are limited to certain design choices, why only choose regularization as a hyperparameter?
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper covers a very extensive empirical study of the representational similarity among different layers between auto-encoders at different stages of training and between different auto-encoders.  The study involves two different measures of similarity.
strength_and_weaknesses: The paper is clearly written and presents very thorough work.  I am generally very supportive of thorough empirical studies whatever their outcome.  From the scientific point of view there is a lot to like.

My hesitation is about the usefulness of the measure of similarity.  I find the similarity measure puzzling and I feel the paper would benefit from more interpretation of what the similarity is measuring.  For example, in an auto-encoder I would naively expect that there would be similarity between the input to the encoder and the output of the decoder.  This was not displayed at all as far as I can see.  There were interpretations being made that the encoder learnt representations far faster than the decoder.  I would like to have seen some other evidence to convince me that this was not an artefact of the measure of similarity being used.

Part of the justification of the paper was as a study of disentanglement.  Indeed the paper studied a variety of VAEs that were designed to achieve disentanglement.  I found this very unconvincing.  In particular the measures were specifically designed to be invariant to  rotations in the representations as I understand it.  However rotations in latent space are critical to disentanglement so it seemed like a strange measure to investigate disentanglement.  Apart from a discussion in the introductions I could not see that the paper threw any light on disentanglement.
clarity,_quality,_novelty_and_reproducibility: The paper was clearly written.  The data collect was clearly very extensive and done very professionally.  To the best of my knowledge this is the first time these measures have been used for studying VAEs.  In principle I don't see why the results would not be reproducible, although the computational cost of doing so means that I don't believe it is worth the effort.  I welcome the fact that the authors went to some length to make their raw data publicly available.
summary_of_the_review: Although there is much to be admired about the paper, I still need to be convinced about its utility.  It describes two related measure of similarity, however, interpreting what these measures tell us is very difficult.  Without a better explanation of what we are meant to take away from these measures I struggle to gain much insight.  I would have preferred to see a more holistic analysis that could be much more limited, but would provide more insight into how we should interpret these similarity measures and what they tell us.  It would have been helpful to see the similarity plot of a model with itself at the same point in time as this would provide a baseline to understand the results shown.  There seems to be a sudden loss of similarity as the representation hits the decoder.  Given that auto-encoders preserve a lot information it would have been really useful to understand what is going on.  Finally, the paper is built up as an exploration of disentanglement, but it is not at all obvious that the paper has anything to say about this.

Given the huge work that has gone into this, I would like to be persuaded to increase my score.  But to do this I need more help in understanding the utility of the work that has been carried out.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper investigates the behavior of disentangled representation learning in VAEs. Especially, the authors utilizes Centred Kernel Alignment to compare the internal behavior of VAEs. Procrustes score is mentioned in the main paper, but the authors did not utilize it due to its limitation on the computational complexity. As a result, they found that the encoder parameters are optimized earlier that the parameters in the decoder, regardless of hyper-parameters, objective function, and the datasests.

strength_and_weaknesses: The paper explores training procedures of VAE which is an interesting part, and the authors clearly list the observations that they have studied.

The authors argue that they have tested 300 VAEs, but I can not find the list of them. Does the 300(=4*5*5*3) VAEs implies the combinations on the [4 versions of VAEs of different objective functions] * [5 different initializations] * [5 different regularization hyper-parameters] * [3 datasets]? 

clarity,_quality,_novelty_and_reproducibility: There is not much clarity problem. And, the authors specified their experiment settings and provided a URL for the code release. Also, they promised to release the pre-trained models later on. 

My concern with this paper is the quality and novelty. While the authors argue that they have tested 300 VAEs in their experiments, but the list of VAEs is pretty limited, and this limitation makes the quality, and novelty ambiguous. The authors should investigate a broader class of VAEs which contains NVAE [1] and \delta-VAE [2]. NVAE is one of the most popular VAEs these days. I wonder how the same experimental results in NVAE come out. Also, \delta-VAE is one representative model to solve the posterior collapse from the structural perspective. I wonder if the same observation can be claimed in \delta-VAE. The works [3-5] are studies to solve the posterior (or component) collapse issue in terms of utilizing other prior distributions than the Gaussian. Since the prior selection also can be considered as a hyper-parameter selection, I wonder if the claimed contribution points still hold even in the cases where other various prior distributions are assumed. What happens if the ones monitor those VAEs [2-5] with CKA score?

[1] NVAE: A Deep Hierarchical Variational Autoencoder, https://proceedings.neurips.cc/paper/2020/file/e3b21256183cf7c2c7a66be163579d37-Paper.pdf

[2] Preventing Posterior Collapse with delta-VAEs, https://openreview.net/pdf?id=BJe0Gn0cY7

[3] Stick-Breaking Variational Autoencoders, https://openreview.net/pdf?id=S1jmAotxg

[4] Dirichlet Variational Autoencoder, https://www.sciencedirect.com/science/article/pii/S0031320320303174

[5] Neural Discrete Representation Learning, https://proceedings.neurips.cc/paper/2017/file/7a98af17e63a0ac09ce2e96d03992fbc-Paper.pdf

summary_of_the_review: The paper investigates an interesting problem, the learning procedure of VAE. While the work shows some promising observations on the VAE learning and posterior collapse problem, the contribution is somewhat limited due to the shallow experiments. Therefore, I slightly lean to the negative side.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper discussed a few similarity metrics and conducted layer-wise comparisons of VAEs using the similarity metric with different datasets, models and parameters. And conclusions are drawn on the model's disentanglement and posterior collapse based on the results. The authors found that encoder representations are learnt before decoder across all settings and they are similar across hyperparameters and learning objective within the same dataset.
strength_and_weaknesses: Strength:
1. detailed discussion on the limitations of the similarity metrics
2. code and libraries are provided
3. there are a diversity of datsets, similarity metrics and settings considered.

Weakness:
1. I think the similarity metric's connection with posterior collapse should be explained in greater details than references to the other papers. i don't think it's straightforward and since it is one of the main points of the paper, the motivation and justification of it could be a bit more detailed. The word "polarized regime" could also have a bit more details for the audience. 
For example, the paper Figure 5 mentions "the mean and sampled representations present a growing number of passive variables which, in the case of sampled representations, leads to high dissimilarity with the input in (b)." Why does the high dissimilarity necessarily means posterior collapse, it seems like the the value is in fact converging so what if that amount of dissimilarity is just right for the model? There isn't really a threshold number that determines what a posterior collapse is for the specific problem or dataset, so i think it is ill-defined. 

2. I feel that similar conclusion such as encoder learnt before decoder has been proposed or discussed or hypothesized in other literature and that should be discussed. For example, this paper "lagging inference networks and posterior collapse in variational autoencoders" is proposed based on the idea that the encoder trains slower than decoder. I think a more thorough discussion and citations in this line of work is warranted.

clarity,_quality,_novelty_and_reproducibility: The paper needs improvement on explaining and discussing the relation of the results with concepts such as posterior collapse and disentanglement. The finding of the result doesn't seem to be novel -- though the method could be -- begs the question of how significant the contribution is, considering the similarity metrics are not novel either. Since code is included, I think the paper can be considered reproducible. 
summary_of_the_review: Overall, I think using similarity metric could be a good idea, but the findings and their implications aren't very novel. The connection of the finding to the claims should be discussed in greater details and more rigorously too.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper describes an evaluation framework based on representation similarity to study and explain the way VAEs learn and diagnose some of the main problems with this type of model. One aspect that sets this work apart is that it looks at representations at every layer, not just the inputs and latent space.
strength_and_weaknesses: Strengths:
The paper analyses important aspects of VAEs with a principled experimwntal framework. It provides valuable and scientifically grounded explainations of problems, phenomena and trends which I personally encountered more then once in building VAEs and are, for the great part, left to intuition and practical experience to deal with. Experimentally or theoretically grounded interpretability insights are always greatly appriciated in deep learning in genereal, especially in representation/generative models.

Weaknesses:
My main concern is with the choice of experiments and the conclusions that come from it. 

1) The conclusions are clear and they are indeed proven by the experiments, but either some parts of the experiments are redundant, or there is some additional conclusions that are not explained in the paper. All conclusions detailed as "implications" at the end of each experimental section only depend on comparing the same layer, e.g. e5 at 25 and 1000+ iterations, with different betas etc. So to see evidence of the claims, I only need to look at the diagonals in the matrices shown. What are the off diagonal similarities showing the reader? One interesting thing is that all encoder layers seem to get stuck to just reproduce the inputs. This is very interesting, but it is never mentioned under "implications".

2) The similarity of the representations at each layers is the main novelty aspect of the evaluation (other than the conclusions themselves), but I am not sure what motivates this choice. The more intuitive and simpler thing to do would be to compare just the similarity of the encoder/decoder weights themselves. What can we see by comparing embeddings over a dataset as opposed to weights values?
clarity,_quality,_novelty_and_reproducibility: The paper is very clear, aside of the couple of points detailed above in weaknesses. 

The methods are not novel in themselves, but the important thing here is that it is providing novel valuable insights into problems with VAEs and I believe this can be greatly beneficial to the community. 

The paper is of good quality; methodologically sound, well written and easy to follow.

The results are highly reproducible, as the authors provide links to frameworks and results.
summary_of_the_review: A good paper which peovides some useful insights into the inner working of VAEs. It is definitely a valuable contribution to ICLR and I can see researches in the area could draw from these insights to push forward the state of the art.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper studies layer-wise representation in VAEs using the Centered Kernel Alignment metric and the Procrustes scores. While there were some strengths to the paper, there was two consistent concerns among the reviewers: novelty and significance. For the former, it was mentioned that the metrics and methods do not appear to be model and that some of the findings in the paper such as "encoders learn before decoders" are already well-known in the community. For the latter, it seemed unclear what utility the proposed method brings to users. I hence recommend rejecting this paper.
justification_for_why_not_higher_score: There were two consistent concerns among reviewers.
Decision: Reject