How do Variational Autoencoders Learn? Insights from Representational Similarity
Authors: Lisa Bonheme
Marek Grzes
Venue: Submitted to ICLR 2023 (2023)
Keywords: variational autoencoders
VAEs
CKA
Procrustes
representation learning
representational similarity
learning dynamics

Abstract: The ability of Variational Autoencoders (VAEs) to learn disentangled representations has made them popular for practical applications. However, their behaviour is not yet fully understood. For example, the questions of when they can provide disentangled representations, or suffer from posterior collapse are still areas of active research. Despite this, there are no layerwise comparisons of the representations learned by VAEs, which would further our understanding of these models. In this paper, we thus look into the internal behaviour of VAEs using representational similarity techniques. Specifically, using the CKA and Procrustes similarities, we found that the encoders' representations are learned long before the decoders', and this behaviour is independent of hyperparameters, learning objectives, and datasets. Moreover, the encoders' representations in all but the mean and variance layers are similar across hyperparameters and learning objectives.

Under review as a conference paper at ICLR 2023
HoW DO VARIATIONAL AUTOENCODERS LEARN? INSIGHTS FROM REPRESENTATIONAL SIMILARITY
Anonymous authors Paper under double-blind review
ABSTRACT
The ability of Variational Autoencoders (VAEs) to learn disentangled representa- tions has made them popular for practical applications. However, their behaviour is not yet fully understood. For example, the questions when they can pro- vide disentangled representations, of active research. Despite this, there are no layerwise comparisons of the rep- resentations learned by VAEs, which would further understanding of these models. In this paper, we thus look into the internal behaviour of VAEs using rep- resentational similarity techniques. Specifically, using the CKA and Procrustes similarities, we found that the encoders' representations are learned long before the decoders', and this behaviour is independent of hyperparameters, learning ob- jectives, and datasets. Moreover, the encoders' representations in all but the mean and variance layers are similar across hyperparameters and learning objectives.
suffer from posterior collapse are still areas
INTRODUCTION
Variational Autoencoders (VAEs) are considered state-of-the-arr techniques learn unsupervised disentangled representations. and have been shown to be beneficial for fairness (Locatello et al., 2019a). As result, VAEs producing disentangled representations have been extensively studied in the last few years (Locatello et al., 2019b; Mathieu et al., 2019; Rolinck al., 2019; Zietlow al., 2021), but they still suffer from poorly understood issues such as posterior collapse (Dai et al., 2020). While some work using explainability techniques has been done to shed light the behaviour of VAES (Liu et al., 2020), a comparison of the representations learned by different methods is still lacking (Zietlow et al., 2021). Moreover, the layer-by-layer similarity of the representations within models has yet to be investigated.
Fortunately, the domain of deep representational similarity is active area of research and metrics such as SVCCA (Raghu et al., 2017; Morcos et al., 2018), Procrustes distance (Schönemann. 1966), or Centred Kernel Alignment (CKA) (Kornblith et al., 2019) have proven very useful in analysing the learning dynamics of various models (Wang et al., 2019; Kudugunta et al., 2019; Raghu et al., 2019; Neyshabur al., 2020), and even helped to design UDR (Duan et al., 2020), an unsupervised metric for model selection for VAEs.
The fact that good models are more similar each other than bad ones in the context of classifica- tion (Morcos et al., 2018) generalised well to Unsupervised Disentanglement Ranking (UDR) (Ro- linek et al., 2019; Duan et al., 2020). However, such a generalisation may not always be possible, and without sufficient evidence, it would be wise to expect substantial differences between the rep- resentations learned by supervised and unsupervised models. In this paper, our aim is to take a first step toward investigating the representational similarity of generative models by analysing the simi- larity scores obtained for a variety of VAEs learning disentangled representations, and by providing some insights into why various VAE-specific methods preventing posterior collapse (Bowman al., 2016; He et al., 2019) or providing better reconstruction (Liu et al., 2021) are successful.
Our contributions are as follows:

---

Under review as a conference paper at ICLR 2023
(i) We provide the first experimental study of the representational similarity between VAEs, and have released more than 45 million similarity scores (https://t.1y/00)
iii) We have released the library created for this experiment (https: 1ttpt..y//MMini/ It can be reused with other similarity metrics or models for further research in the domain.
(ii) During our analysis, we found that (1) the encoder is learned before the decoder; (2) all the layers of the encoder, except the mean and variance layers, learn very similar represen- tations regardless of the learning objective and regularisation strength used; and (3) linear CKA could be an efficient tool track posterior collapse.
BACKGROUND
2.1 VARIATIONAL AUTOENCODERS
Variational Autoencoders (VAEs) (Kingma & Welling, 2014; Rezende & Mohamed, 2015) are deep probabilistic generative models based on variational inference. The encoder, q (z)x),, maps some input x latent representation z, which the decoder, po(x|z) z). uses attempt to reconstruct x. This can be optimised by maximising L. the evidence lower bound (ELBO)
[(O, Φ;
g woo.:looooooooooo po (x DKL (1s (z (z)x))l((2))
(1)
reconstraction
regularisation term
where p(z) is generally modelled as a multivariate Gaussian distribution M(0,, I) to permit closed form computation of the regularisation term (Doersch, 2016). We refer to the regularisation term of Equation as regularisation in the rest of the paper, and we do not tune any other forms of regu- larisation (e.g.. LI, dropout). While our goal not to study disentanglement, our experiments will focus on a range of VAEs designed to disentangle (Higgins et al., 2017; Chen et al., 2018; Burgess al., 2018; Kumar et al., 2018) because they possess useful properties: (1) posterior collapse sit- uations are easy to create; (2) these models are non non-identifiable (Khemakhem et al., 2020), but we are interested in verifying the representations learned still retain some linear relationship between models. We refer the reader to Appendix A for more details on these models.
Polarised regime and posterior collapse The polarised regime, also known as selective poste- rior collapse, is the ability of VAEs "shut down" superftuous dimensions of their sampled latent representations while providing high precision on the remaining ones (Rolinek et al., 2019; Dai al., 2020). The existence of the polarised regime a necessary condition for the VAEs provide good reconstruction (Dai & Wipf, 2018; Dai et al., 2020). However, when the weight on the reg- ularisation term of the ELBO given in Equation 1 becomes too large, the representations collapse to the prior (Lucas et al., 2019a; Dai et al., 2020). Recently, Bonheme & Grzes (2021) have also shown that the passive variables, which are "shut down" during training, are very different in mean and sampled representations (see Appendix B). This indicates that representational similarity could be a valuable tool in the study of posterior collapse.
2.2 REPRESENTATIONAL SIMILARITY METRICS
Representational similarity metrics aim to compare the geometric similarity between two repre- sentations. In the context of deep learning, these representations correspond to RR'p matrices of activations, where n is the number of data examples and p the number of neurons in a layer. Such metrics can provide various information on deep neural networks (e.g.. the training dynamics of neural networks, common and specialised layers between models).
Centred Kernel Alignment Centred Kernel Alignment (CKA) (Cortes et al., 2012: Cristianini et al., 2002) is normalised version of the Hillbert-Schmit Independence Criterion (HSIC) (Gretton et al., 2005). its name suggests, it measures the alignment between the n kernel matrices of two representations, and works well with linear kernels (Kornllih et al., 2019) for represen- tational similarity of centred layer activations. We thus focus on the linear CKA, also known as
Due to their size and to preserve anonymity, the 300 models trained for this paper will be released after the review.
2

---

Under review as a conference paper at ICLR 2023
RV-coefficient (Escoufier, 1973; Robert & Escoufier, 1976). Given the centered layer activations X E Raxm and Y E Rnxp taken over n data examples, linear CKA is defined as:
Y 1
CKA(X,Y) | YTY F
(2)
where ||-I| is the Frobenius norm. CKA is a generalisation of Pearson's correlation coefficient to higher dimensional representations (Escoufier, 1973; Robert & Escoufier, 1976) and can be seen as measuring the cosine between matrices (Josse & Holmes, 2016). takes values between (not similar) and 1 (X Y). For conciseness, will refer linear CKA as CKA in the rest of this paper.
Orthogonal Procrustes The aim of orthogonal Procrustes (Schönemann, 1966) is to align matrix Y to matrix X using orthogonal transformations Q such that
min X - YQI s.t. Q²Q=I.
The Procrustes distance, Pa. is the difference remaining between X and Y when Q optimal,
Pa Y) = | - + ||Y|| 2||"T|] x|I.
where ||-11- is the nuclear norm (see Golub & Van Loan (2013, pp. 327-328) for the full derivation from Equation 3 to Equation 4). To easily compare the results of Equation 4 with CKA, we first bound its results between O and using normalised X and Y, as detailed in Appendix C. Then, we transform the result similarity metric ranging from 0 (not similar) (X YY),
Pa
= P)
(I*1 + | 2|| x x1.)
(5)
We will refer to Equation as Procrustes similarity in the following sections.
2.3 LIMITATIONS OF CKA AND PROCRUSTES SIMILARITIES
While CKA and Procrustes lead to accurate results in practice, they suffer from some limitations that need to be taken into account in our study. Before we discuss these limitations, we should clarify that, in the rest of this paper, sim(..-) represents similarity metric in general, while CKA(-,) and Pa(s.) specifically refer to CKA and Procrustes similarities.
Sensitivity to architectures Maheswaranathan al. (2019) have shown that similarity metrics comparing the geometry of representations were overly sensitive to differences in neural architec- tures. As CKA and Procrustes belong to this metrics family, we can expect them to underestimate the similarity between activations coming from layers of different type (e.g., convolutional and de- convolutional).
Procrustes is sensitive to the number of data examples As we may have representations with high dimensional features (e.g., activations of convolutional layers), we checked the impact of the number of data examples on CKA and Procrustes. To do so, we created four increasingly different matrices A, B, C. and D with 50 features each: retains 80% of A's features, C' 50%, and D 0%. We then computed the similarity scores given by CKA and Procrustes while varying the number of data examples. As shown in Figures and l both metries agree for sim( A, B) and sim A, C)), giving scores that are close to the fraction of common features between the two matrices. However, we can see in Figure Ic that Procrustes highly overestimates sim D) while CKA scores rapidly drop.
CKA ignores small changes in representations When considering a sufficient number of data examples for both Procrustes and CKA, if two representations do not have dramatic differences (i.e.. their 10% largest principal components are the same), CKA may overestimate similarity, while Procrustes remains stable, as observed by Ding et al. (2021).
Similarity and disentanglement is important to keep in mind that CKA and Procrustes similari- ties invariant to orthogonal transformation. Thus, they will consider disentangled representation similar to a rotated (and possibly entangled) representation, as illustrated in Figure 2. Note that CKA and Procrustes are also invariant to isotropic scaling, though this does not affect disentanglement.
3

---

Under review as a conference paper at ICLR 2023
1.0
1.0
1.0
Procrustes
Procrustes
CKA
Procrustes
0.2
0.2
0.2
0.0
250
500
750
1000
0.0
250
500
750
1000
0.0
250
500
750
1000
(a) sim ( A, B)
(b) sim(A, C)
(c) sim A, D)
Figure I: We compute CKA and Procrustes similarity scores with an increasing number of data ex- amples and different similarity strength: B retains 80% of A's features, C' 50%, and D 0%. Both metrics agree in (a) and (b), but Procrustes overestimates similarity in (c).
Shape
Scale
Shape
(a) Representation A
(b) Representation B
(a) shows a disentangled representation A where the latent dimensions 21 and 22 are aligned with some ground truth factors shape and scale. (b) shows representation B which corre- sponds an orthogonal transformation of A. It is entangled z and 22 are not aligned with the ground truth factors anymore, but simm( A, B) because Procrustes and CKA are invariant under orthogonal transformations.
Figure 2:
Ensuring accurate analysis Given the limitations previously mentioned. we take three remedial actions to guarantee that our analysis is as accurate as possible. Firstly, as both metrics will likely underestimate the similarity between different layer types, will only discuss the variation of sim- ilarity when analysing such cases. For example, we will not compare sim(A, B) and sim(A, C)) if A and B convolutional layers but C is deconvolutional. We will nevertheless analyse the changes of sim( A, C) at different steps of training. Secondly, when both metrics disagree, we know that of them is likely overestimating the similarity: Procrustes if the number of data amples is not sufficient, CKA if the difference between the two representations is not large enough. Thus, we will always use the smallest of the two results for our interpretations. Lastly, when say that representations are similar, we mean that they are similar up to orthogonal rotations and isotropic scaling. Hence, two similar representations may not equally disentangled.
EXPERIMENTAL SETUP
The goal of this experiment is to study the training dynamics of VABs and the impact of initialisation. learning objectives, and regularisation on the representations learned by each layer. To do so, we measure the representational similarity of (i) One model at different epochs; (ii) Two models with the same learning objective and different regularisation strengths at the same epoch, and (iii) Two models with different learning objectives and equivalent regularisation strength at the same epoch.
Learning objectives We will focus on learning objectives whose goal is to produce disentangled representations. Specifically, we use 3-VAE (Higgins et al., 2017), B-TC VAE (Chen et al., 2018), Annealed VAE (Burgess et al., 2018), and DIP-VAE II (Kumar et al., 2018). A description of these methods can be found in Appendix A. To fairly provide complementary insights into previous observations of such models (Locatello et al., 2019b; Bonheme & Grzes, 2021), we will follow
4

---

Under review as a conference paper at ICLR 2023
the experimental design of Locatello al. (2019b) regarding the architecture, learning objectives, and regularisation used. Moreover, disent angleanngeemnt lib² will be used as a codebase for our experiment. The complete details are available in Appendix C.
Datasets We use three datasets which, based on the results of Locatello et al. (2019b), are in- creasingly difficult for VAEs in terms of reconstruction loss: dSprites' (Higgins et al., 2017), cars3D (Reed et al., 2015), and smallNorb (LeCun et al., 2004).
Training process We trained five models with different initialisations for 300,000 steps for each (learning objective, regularisation strength, dataset) triplet, and saved intermediate models to com- pare the similarity within individual models at different epochs. Appendix explains our epoch selection methodology.
Similarity measurement Given the computational complexity detailed below, for every dataset, we sampled 5,000 data examples, and we used them to compute all the similarity measurements. We compute the similarity scores between all pairs of layers of the different models following the different combinations outlined above. We will refer to the similarity scores of a group of one or more layers with itself as self-similarity. As Procrustes similarity takes significantly longer to compute compared to CKA (see below), we only used to validate CKA results, restricting its usage to one dataset: cars3D. We obtained similar results for the two metrics cars3D, thus only reported CKA results in the main paper. Procrustes results can be found in Appendix D.
Computational considerations Overall we trained 300 VAEs using 4 learning objectives, 5 dif- ferent initialisations. 5 regularisation strengths, and 3 datasets, which took around 6,000 hours on an NVIDIA A100 GPU. We then computed the CKA scores for the 15 layer activations (plus the input) of each model combinations considered above at 5 different epochs, resulting in 470 mil- lion similarity scores and approximately 7,000 hours of computation on an Intel Xeon Gold 6136 CPU. As Procrustes is slowed down by the computation of the nuclear norm for high dimensional activations, the same number of similarity scores would have been prohibitively long to compute, requiring 30,000 hours on an NVIDIA A100 GPU. We thus only computed the Procrustes similarity for one dataset, reducing the computation time to 10,000 hours. Overall, based on the estimations of Lacoste et al. (2019), the computations done for this experiment amount to 2,200 Kg of CO2, which corresponds to the CO2 produced by one person over 5 months. To mitigate the negative environmental impact of our work, released all our metric scores at https: //t. ly / OGLe3. We hope that this will help to prevent unnecessary recomputation should others wish to reuse our results.
RESULTS
4.1
HOW ARE REPRESENTATION LEARNED AS TRAINING PROGRESSES?
In this section, we will analyse the learning dynamics of VAES to answer objective (i) of Section 3. To monitor this, we will compare the representations learned by the first and last recorded snapshot of VAEs. Note that our choice of snapshots and snapshot frequency does not influence the results as verified in Appendices and J. Note that when describing the results in heatmaps, we will refer to quadrants to indicate the similarity scores between all the representations of the encoder decoder. This includes the off-diagonal CKA scores between layers of the same type.
VAE learning is bottom-up As shown in Figure 3, the encoder is learned first, and the represen- tations of its layers become similar to the input after a few epochs. The decoder then progressively learss representations that gradually become closer to the input. We observed similar trend on fully-connected architectures, as reported in Appendix F. This result can explain why, when de- coder has access to the input (as it is the case in some autoregressive VAEs), it ignores the latent representations (Bowman et al., 2016; Li et al., 2019). Indeed, in this case the decoder is not con- strained to wait for the encoder to converge before improving its reconstruction since it has direct
'tttps: / thub. wttphccooooc..ccccooooooc.o...oo wwt.cowwm///ee..o google-rooe moogleerreeeraee arrsearcchcee sentangn lenttanggemet.... _ib 'I "iceensed under Apache 2.0 licence.
5

---

Under review as a conference paper at ICLR 2023
10
4G
11
-00
-04
-1
66
(a) Trained on cars3l)
(b) Trained on dSprites
(c) Trained on smallNorb
Figure 3: (a) shows the CKA similarity scores of activations epochs 25 and 1090 of 3-TC VAE trained on cars3l) with 3 = 2. (b) shows the CKA similarity scores of activations at epochs 2 and 26 of DIP-VAI II trained on dSprites with 5. (c) shows the CKA similarity scores of activations at epochs 10 and 410 of Annealed VAE trained on smalINorb with Cmaz 5. We can that there is a high similarity between the representations learned by the encoder early in the training and after complete training (see the bright cells in the top-left quadrants in (a), (b), and (c)). However, the self-similarity between the decoder, mean and variance representations is lower, indicating some further changes in the representations after the first few epochs of training (see the dark cells in the bottom-right quadrant of parts (a). (h), and (c)). These results are averaged over seeds.
access to the input sequences. In this paper, we used standard VAEs, where the decoder is not autoregressive and must rely the latent representations as the sole source of information about the input. It thus needs to wait for the encoder to converge before being able to converge itself, effectively preventing posterior collapse when the regularisation is not too strong.
Implications The concurrent learning of encoder and decoder representations be problematic. Indeed, if the encoder learns poor representations, the decoder will not be able to provide an accurate reconstruction. As result, the encoder may struggle to provide better representations and this may lead to non-optimal results. We believe methods that facilitate the incremental learning of the encoder and decoder (e.g.. by slowly increasing the complexity of the data, as in Progressive GANs (Karras et al., 2018)) as well as methods where the encoder is explicitly learned first (He et al., 2019) are promising ways to mitigate this issue. Moreover, the early convergence of encoder can explain the success of "warm-up" methods such as annealing (Bowman et al., 2016) or aggressive inference (He et al., 2019) in the case where the encoder (quickly) converges to a bad local minima resulting in posterior collapse.
4.2 WHAT IS THE INFLUENCE OF THE HYPPERPARAMETER ON THE LEARNED REPRESENTATIONS?
So far, have compared different snapshots of the same model. In this section, we will perform fine-grained analysis of the impact of hyperparameters to answer objective (ii) of Section 3.
Impact of regularisation As discussed in (Bonheme & Grzes, 2021), passive variables have dif- ferent values in the mean and sampled representations. This phenomenon increasingly visible with higher regularisation, as more variables must become passive in order to lower the KL diver- gence (Rolinek al., 2019; Dai et al., 2020). However, little is known about how this behaviour impacts the other layers of the encoder and decoder. As shown in Figure 4, the decoder representa- tions change more than the encoder representations with the increased regularisation strength. This can be explained by Figure 5 where the sampled representations drift away from the input and mean representations, which is consistent with posterior collapse (Rolinek et al., 2019; Dai & Wipf, 2018; Dai et al., 2020). This is further confirmed by the fact that posterior collapse was already reported in (Bonheme Grzes, 2021) for these configurations. Thus, our results indicate that CKA, which quick to compute, could be a good tool to monitor the polarised regime and posterior collapse its pathological counterpart - and differentiate well the two behaviours (see Appendix G). Interest- ingly, apart from the mean and variance, the representations learned by the encoder in Figure stay

---

Under review as a conference paper at ICLR 2023
15
12
12
40,9ee Immraa22, esechm24
(a) 3-VAE
(b) DIP-VAE
Figure 4: (a) shows the CKA similarity scores between the activations of two 3-VAEs trained on dSprites with 3 = 1, and 3 = 8. (b) shows the CKA similarity scores between the activations of DIP- VAE trained on dSprites with À are taken after complete training, and all the results are averaged over 5 seeds. In both figures, we can see that the self-similarity of decoder representations low between models trained with low and high regularisation (dark cells in the bottom-right quadrants) while the encoder representations stay very similar (bright cells the top-left quadrants), except for the mean, variance and sampled representations.
and 20. For (a) and (b), the activations
04
(a) CKA (mcan, sampled)
(b) CKA (input, sampled)
Figure 5: (a) and (b) show the CKA scores of 3-VAEs trained on dSprites with 3 from to 8. (a) is the CKA between mean and sampled representations while (b) is between inputs and sampled rep- resentations. Both figures are consistent with posterior collapse caused by over-regularisation (Ro- linck et al., 2019; Bonheme & Grzes, 2021) where the mean and sampled representations present a growing number of passive variables which, in the case of sampled representations, leads to high dissimilarity with the input in (b).
(a)
(b) B
Figure 6: (a) shows the CKA scores between the inputs and the activations of the first 6 layers of the encoder of B-VAEs trained on cars3D with 3 1. (b) shows the scores between the same representations with / 8. We area corresponding to the variance interval around each line small) and the encoder layers have an equivalent similarity with the input, regardless of the regularisation strength.
that have low variance across seeds (i.e., the shaded

---

Under review as a conference paper at ICLR 2023
very similar. We can further in Figure 6 that this is consistent across seeds and very different regularisation strengths, and can also be observed with fully fulle-connected architectures in Appendix F.
Implications The polarised regime (Rolinek et al., 2019; Zietlow al., 2021; Dai et al., 2020) and posterior collapse (Dai & Wipf. 2018; Lucas et al., 2019a;;; Dai et al., 2020) do not seem to affect the representations learned by the encoder before the mean and variance layers (see Fig- ure 6). Intuitively, this would imply that the encoder learns similar representations, regardless of the regularisation strength, and then "fine-tunes" them in its mean and variance layers.
4.3 WHAT IS THE INFLUENCE OF THE LEARNING OBJECTIVES ON THE LEARNED REPRESENTATIONS?
We have seen Section 4.2 that models with the same learning objective (e.g., 3-VAE, DIP-VAE II or any other learning objective defined in Appendix A) but different regularisation were generally learning similar representations in the encoder except for the mean and variance layers. Starting from these layers and moving toward the decoder, the representational similarity decreases as the regularisation strength increases. Indeed, passive variables have different mean and variance repre- sentations than active variables (Rolinck et al., 2019; Bonheme & Grzes, 2021) and, as their number grows, so does the dissimilarity with the representations obtained at lower regularisation, which have fewer passive variables. Now objective (iii) of Section 3. For an equivalent regularisation strength but different learning objec- tives, do the models still learn similar representations in the layers of the encoder that are before the mean and variance layers?
wonder whether this pattern is also occuring in the context of
11
100
01
38
102
62
S..E lambde=1 10s1=1000
4p.wa. apodhw2s
J...J.. lanbok a0dm010
(a) Trained on cars3D
(b) Trained on dSprites
(c) Trained on smallNorb
Figure 7: (a) shows the CKA similarity scores of activations of 3-VAE and DIP-VAE trained on cars3D with 3 1. and À 1, respectively. (b) and (c) show the CKA similarity scores of the same learning objectives and regularisation strengths but trained on dSprites and smallNorb. These results averaged over seeds. We can that the representational similarity of all the layers the encoder (top-left quadrant) except mean and variance is very high (CKA stays close to 1). However, the mean, variance, sampled (center diagonal values), and decoder (bottom-right quadrants) representational similarity of different learning objectives seems to vary depending on the dataset. In (a) and (c) they have high similarity, while (b) the similarity is lower.
The representations learned by the early layers of the encoders are similar across learning objectives As seen in Section 4.2, the representations learned by the encoder are similar across hyperparameters, with mean and variance representations being less similar for high regularisa- tion. In this experiment, we show that this observation also holds across learning objectives. We can see in Figure that the representational similarity of different learning objectives with equivalent regularisation strength is quite high in all the encoder layers except mean and variance. This result is consistent across datasets, learning objectives, and well as architectures (see Appendix F), and the learned representations are also close to what is learned by classifier with equivalent architecture (see Appendix H).
The representations learned by the mean and variance layers can vary The similarity of the representations learned by the mean and variance layers (and consequently by the decoder) tends to vary across learning objectives and seems to be influenced by the dataset. Indeed, by looking at the center diagonal values of Figure we see that the similarity of the mean, variance, and
8

---

Under review as a conference paper at ICLR 2023
consequently sampled representations across different learning objectives is high for cars3D and smallNorb, but low for dSprites. This may indicate that, for a given dataset, different learning objectives can find different local optima, leading to lower similarity between mean and variance representations, and ultimately to different representations in the decoder. It seems that this phe- nomenon is very dataset-dependent, and is especially present in dSprites (see Figure 7b), which is one of the most used datasets for disentangled representation learning. The same phenomenon can be observed with fully-connected architectures in Appendix F.
The representations learned by most of the encoder layers are very similar across learning objectives, indicating that the encoder may be learning some general features from the inputs, or the only features that can be learned when the decoder performs poorly. Indeed, since the decoder initially struggles to learn, the encoder may only be able to learn very general properties. The learning of general representations in early layers is consistent with Bansal et al. (2021), who, in the context of classification, observed that neural networks were learning similar representations regardless of the initialisation, architecture or learning objective used. As such, the encoder may be viewed as feature extractor which is fine-tuned using a mean and variance layer to produce the sampled representations that will be used by the decoder. Our observations also suggest that some learning objectives may favour distinct local optima whose existence has been previously discussed (Alemi et al., 2017; Zietlow et al., 2021). Such model-specific choice of local optima may explain why some learning objectives obtained better disentanglement scores than others on specific datasets but performed worse on others (Locatello al., 2019b). Given that VAEs with non- conditional priors were shown to be non-identifiable (Khemakhem et al., 2020), it is very interesting to see that they still have similar encoders, and, for some datasets, similar decoders. Our results imply that for some datasets, different learning objectives can still provide decoder representations with strong linear relationships.
Implications
CONCLUSION
Bottom-up learning and posterior collapse As reported in Section 4.1, the encoder is learned before the decoder, which could indicate that the decoder struggles to converge before the mean and variance representations are learned. This would explain why one can observe posterior col- lapse in a setting where the decoder has access to the input, and thus can infer the mapping on its own (Bowman et al., 2016; Li et al., 2019).
Different models encode similar representations We have seen in Section 4 that the encoders, prior to their mean and variance layers, learned remarkably similar representations regardless of the initialisation, regularisation, and learning objective used to train the model. It is especially intriguing to see that even posterior collapse does not seem to affect these representations. The representational similarity of the mean, variance, and decoder representations across learning objectives generally vary depending on the dataset, indicating that different learning objectives may find different local optima for given dataset. Note that this behaviour is more visible some datasets (e.g., dSprites) than others (e.g., cars3D).
Other applications While our main focus of settings, this study also demonstrated that CKA, whose computational cost is very low, can be an efficient tool to detect posterior collapse. Indeed, one can directly compare the similarity between mean and sampled representations, which strongly decreases as the number of collapsed variables grows (Bonheme & Grzes, 2021). We believe that this could be a complementary tool to the more costly mutual information generally used for such purpose.
compare similarity of models across variety
Limitations We limited our study to similarity metries that measure difference in the geometry of the representations. While this gave us compelling insights, these metrics have some limitations, discussed in Section 2.3, and may underestimate the similarity between layers with different archi- tectures (Maheswaranathan et al., 2019). We believe that further research using dynamics-based metrics, such as fixed- fixed-point topology (Maheswaranathan et al., 2019), could provide additional in- sights into the representations learned by VAEs. It would also be beneficial to extend this study to other families of VAEs (van den Oord et al., 2017: Nalisnick & Smyth, 2017; Razavi et al., 2019; Vahdat & Kautz, 2020; Joo al., 2020) provide further insights into their various properties.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Alexander A Alemi, lan Fischer, Joshua Dillon, and Kevin Murphy. Deep Variational Information Bottleneck. In International Conference on Learning Representations, volume 5, 2017.
Yamini Bansal, Preetum Nakkiran, and Boaz Barak. Revisiting Model Stitching to Compare Neural Representations. In Advances in Neural Information Processing Systems, 2021.
Lisa Bonheme and Marek Grzes. Be More Active! Understanding the Differences between Mean and Sampled Representations of Variational Autoencoders. arXiv e-prints, 2021.
Samuel R Bowman, Luke Vilnis, Oriol Vinyals, Andrew Dai, Rafal Jozefowiez, and Samy Bengio. Generating Sentences from a Continuous Space. In Proceedings The 20th SIGNLL Conference on Computational Natural Language Learning, 2016.
Christopher Burgess, Irina Higgins, Arka Pal, Loic Matthey, Nick Watters, Guillaume Desjardins, and Alexander Lerchner. Understanding Disentangling in 3-VAE. arXiv e-prints, 2018.
Ricky T. Q. Chen, Xuechen Li, Roger B. Grosse, and David K. Duvenaud. Isolating Sources of Disentanglement in Variational Autoencoders. In Advances in Neural Information Processing Systems, volume 31, 2018.
Corinna Cortes, Mehryar Mohri, and Afshin Rostamizadeh. Algorithms for Learning Kernels Based on Centered Alignment. J. Mach. Learn. Res., 13(1), 2012.
Nello Cristianini, John Shawe-Taylor, André Elissefff, and Jaz S Kandola. On Kernel-Target Align- ment. In Advances in Neural Information Processing Systems, volume 14. 2002.
Bin Dai and David Wipf. Diagnosing and Enhancing VAE Models. In International Conference on Learning Representations, volume 6, 2018.
Bin Dai, Ziyu Wang, and David Wipf. The Usual Suspects? Reassessing Blame for VAE Posterior Collapse. In Proceedings of the 37th International Conference on Machine Learning, 2020.
Frances Ding, Stan-Stanislas Denain, and Jacob Steinhardt. Grounding Representation Similarity Through Statistical Testing. In Advances in Neural Information Processing Systems, 2021.
Carl Doersch. Tutorial on Variational Autoencoders. arXiv e-prints, 2016.
Sunny Duan, Loic Matthey, Andre Saraiva, Nick Watters, Chris Burgess, Alexander Lerchner, and
Irina Higgins. Unsupervised model selection for variational disentangled representation learning. In International Conference on Learning Representations, volume 8, 2020.
Yves Escoufier. Le traitement des variables vectorielles. Biometrics, 29(4):551-760, 1973. ISSN 0006341X, 15410420. URL http: htst:://www.. /www. hwtp::jssooooooootteeeooo wwt./rogg/lllo.ooo ettolle/2/2991 222914294 400.
Gene H. Golub and Charles F. Van Loan. Matrix computations. The Johns Hopkins University
Press, fourth edition edition, 2013. ISBN 9781421407944.
Arthur Gretton, Olivier Bousquet, Alex Smola, and Bernhard Schölkopf. Mcasuring Statistical Dependence with Hilbert-Schmid Norms. In Algorithmic Learning Theory, 2005. ISBN 978-3-
540-31696-1.
Junxian He, Daniel Spokoyny, Graham Neubig. and Taylor Kirkpatik....... Lagging Inference Networks and Posterior Collapse in Variational Autoencoders. In International Conference on Learning Representations, volume 7, 2019.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Mohamed Shakir, and Alexander Lerchner. 3-VAE: Learning Basic Visual Concepts with Con- strained Variational Framework. In International Conference on Learning Representations, vol- ume 5, 2017.
Weonyoung Joo, Wonsung Lee, Sungrae Park, and Il-Chul Moon. Dirichlet variational autoencoder. Pattern Recognition, 107:107514, 2020. ISSN 0031-3203, doi: 11.t./////.....//...p....ooo.
10

---

Under review as a conference paper at ICLR 2023
Julie Josse and Susan Holmes. Measuring multivariate association and beyond. Statistics Surveys, 10(none):132 - 167, 2016. doi: 10.1214/16-SS166. URL https: /tte:://doo 16-S$116.
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive Growing of GANs for Improved Quality, Stability, and Variation. In International Conference on Learning Representa- lions, volume 6, 2018.
llys Khemakhem, Diederik Kingma, Ricardo Monti, and Aapo Hyvarinen. Variational Autoen- coders and Nonlinear ICA: A Unifying Framework. In Proceedings of the Twenty Third Interna- tional Conference on Artificial Intelligence and Statistics, volume 108 of Proceedings of Machine Learning Research, 2020.
Diederik P. Kingma and Max Welling. Encodin Variational Bayes. In International Confer- ence on Learning Representations, volume 2, 2014.
Simon Komblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of Neu- ral Network Representations Revisited. In Proceedings of the 36th International Conference on Machine Learning. volume 97 of Proceedings of Machine Learning Research, 2019.
Sneha Kudugunta, Ankur Bapna, Isaac Caswell, and Orhan Firat. Investigating Multilingual NMT Representations at Scale. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Process- ing (EMNLP-IJCNEL) 2019.
Abhishek Kumar, Prasanna Sattigeri, and Avinash Balakrishnan. Variational Inference of Disen- tangled Latent Concepts from Unlabeled Observations. In International Conference on Learning Representationss volume 6, 2018.
Alexandre Lacoste, Alexandra Luccioni, Victor Schmidt, and Thomas Dandres. Quantifying the carbon emissions of machine learning. arXiv preprint 190.90090, 2019.
Yann LeCun, Fu Jie Huang, and Léon Bottou. Learning Methods for Generic Object Recognition with Invariance to Pose and Lighting. In Proceedings of the 2004 IEEE Computer Society Con- ference on Computer Vision and Pattern Recognition, 2004. CVPR 2004., volume 2, 2004.
Ruizhe Li. Xiao Li, Chenghua Lin, Matthew Collinson, and Rui Mao. A Stable Variational Au- toencoder for Text Modelling. In Proceedings of the 12th International Conference on Natural Language Generation, 2019.
Wenqian Liu, Runze Li, Meng Zheng, Srikrishna Karanam, Ziyan Wu, Bir Bhanu, Richard J Radke, and Octavia Camps. Towards Visually Explaining Variational Autoencoders. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, 2020.
Zhi-Song Liu, Wan-Chi Siu, and Li-Wen Wang. Variational autoencoder for reference based image super-resolution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pp. 516-525, June 2021.
Francesco Locatello, Gabriele Abbati, Thomas Rainforth, Stefan Bauer, Bernhard Schölkopf, and Olivier Bachem. On the Fairness of Disentangled Representations. In Advances in Neural Infor- mation Processing Systems, volume 32, 2019a.
Francesco Locatello, Stefan Bauer, Mario Lucie, Gunnar Raetsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. Challenging Common Assumptions in the Unsupervised Learn- ing of Disentangled Representations. In Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, 2019b.
James Lucas, George Tucker, Roger B. Grosse, and Mohammad Norouzi. Understanding Posterior Collapse in Generative Latent Variable Models. In Deep Generative Models for Highly Structured Data, ICLR 2019 Workshop, 2019a.
James Lucas, George Tucker, Roger B. Grosse, and Mohammad Norouzi. Don't Blame the ELBO! A linear VAE Perspective on Posterior Collapse. In Advances in Neural Information Processing Systems, volume 32, 2019b.
11

---

Under review as a conference paper at ICLR 2023
Niru Maheswaranathan, Alex Williams, Matthew Golub, Surya Ganguli, and David Sussillo. Uni- versality and Individuality in Neural Dynamics Across Large Populations of Recurrent Networks. In Advances in Neural Information Processing Systems, volume 32, 2019.
Emile Mathieu, Tom Rainforth, N Siddharth, and Yee Whye Teh. Disentangling Disentanglement in Variational Autoencoders. In Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, 2019.
Ari Morcos, Maithra Raghu, and Samy Bengio. Insights on Representational Similarity in Neural Networks with Canonical Correlation. In Advances in Neural Information Processing Systems, volume 31, 2018.
Eric Nalisnick and Padhraic Smyth. Stick-breaking variational autoencoders, In International Con- ference on Learning Representations, volume 5, 2017.
Behnam Neyshabur, Hanie Sedghi, and Chiyuan Zhang. What is Being Transferred in Transfer Learning? In Advances in Neural Information Processing Systems, volume 33, 2020.
Sinno Jialin Pan and Qiang Yang. A survey transfer learning. IEEE Transactions on knowledge and data engineering, 12110):3355-335) 2009.
Maithra Raghu, Justin Gilmer, Jason Yosinski, and Jaseha Sohl-Dickstein. SVCCA: Singular Vector Canonical Correlation Analysis for Deep Learning Dynamics and Interpretabilityy. In Advances in Neural Information Processing Systems, volume 30, 2017.
Maithra Raghu, Chiyuan Zhang, Jon Kleinberg, and Samy Bengio. Transfusion: Understanding Transfer Learning for Medical Imaging. In Advances Neural Information Processing Systems, volume 32, 2019.
Ali Razavi, Aaron van den Oord, Ben Poole, and Oriol Vinyals. Preventing posterior collapse with delta- VAEs. In International Conference Learning Representations, volume 7, 2019.
Scott Reed, Yi Zhang, Yuting Zhang, and Honglak Lee, Deep Visual Analogy- In Advances in Neural Information Processing Systems, volume 28, 2015.
Danilo Rezende and Shakir Mohamed. Variational Inference with Normalizing Flows. In Proceed- ings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, 2015.
Robert and Y. Escoufier. A Unifying Tool for Linear Multivariate Statistical Methods: The RV- Coefficient. Journal of the Royal Statistical Society. Series C (Applied Statistics), 25(3):257-265, 1976.
Michal Rolinek, Dominik Zietlow, and Georg Martius. Variational Autoencoders Pursue PCA Di- rections (by Accident). In Proceedings of the IEEF/CVE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
Peter H. Schönemann. A generalized solution of the Orthogonal Procrustes problem. Psychometrika, 31(1):1-10, 1966.
Arash Vahdat and Jan Kautz. Nvae: deep hierarchical variational autoencoder. In Advances in Neural Information Processing Systems, volume 33, pp. 19667-19679. Curran Associates, Inc., 2020.
Aaron van den Oord, Oriol Vinyals, and koray kavukcuoglu. Neural discrete representation learning. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
Liwei Wang. Lunjia Hu, Jiayuan Gu, Yue Wu, Zhiqiang Hu, Kun He, and John Hoperoft. Towards understanding learning representations: To what extent do different neural networks learn the same representation. Advances in Neural Information Processing Systems, volume 32, 2019.
Jason Yosinski, Jeff Clune, Anh Nguyen, Thomas Fuchs, and Hod Lipson. Understanding neural networks through deep visualization. arXiv e-prints, 2015.
Dominik Zietlow, Michal Rolinek, and Georg Martius. Demystifying Inductive Biases for (Beta-) VAE Based Architectures. Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, 18-24 Jul 2021.
12

---

Under review as a conference paper at ICLR 2023
A DISENTANGLED REPRESENTATION LEARNING
As mentioned in Section 2, are interested in the family of methods modifying the weight on the regularisation term of Equation to encourage disentanglement. In our paper, the term regularisation refers to the moderation of this parameter only. To achieve this, experiment will focus the models described below.
3-VAE The goal of this method (Higgins et al., 2017) is to penalise the regularisation term of Equation by factor 3 1, such that
C xx (x(x) Pa
(6)
Annealed VAE Burgess et al. (2018) proposed to gradually increase the encoding capacity of the network during the training process. The goal is to progressively learn latent variables by decreasing order of importance. This leads to the following objective, where C parameter that can be understood as a channel capacity and y is hyper-paramen penalising the divergence, similarly to in 3-VAE:
C(O, Φ0; x) = Ees x) [log po - DxI, (2)x))l((z)) - Cl.
(7)
As the training progresses, the channel capacity C is increased, going from zero to its maximum channel capacity Cmax and allowing higher value of the KL divergence term. VAEs that use Equa- tion 7 as learning objective are referred to Annealed VAEs in this paper.
Chen et al. (2018) argued that only the distance between the estimated latent factors and the prior should be penalised to encourage disentanglement, such that
3-TC VAE
L(O, xx (x))
(Bya (x DKL (c)(z)))]p(z))) (Ds)(1))))ll (8)
Here, DKL (9 (z)])((z) is approximated by penalising the dependencies between the dimensions of (z):
n
D II
C(O, xx ~=
(xlx)) [x; DKL (z)x))))]l( (ADr((
g0 (z())))
j=1
total
(9)
The total correlation of Equation is then approximated over a mini-batch samples ( R.g} follows:
M
M
1
B (x [log uep (z))
log
NM Σ
(=<((()))) xx,))
M
where 2(x;) is sample from 4 (z)zz; M is the number of samples in the mini-batch, and N the total number of input examples. Egg (vx [log qo (z:)) can be computed similar way.
DIP-VAE Similarly to Chen et al. (2018), Kumar et al. (2018) proposed to regularise the distance between (z(z) and p(z) using Equation 8. The main difference is that here DKI. (z))()((())) is measured by matching the moments of the learned distribution qa (z) and its prior p(z). The second moment of the learned distribution is given by
Cov [z] = Cov [x)(x)] + Ep(x) [x))]
(10)
DIP-VAE II penalises both terms of Equation 10 such that
X (2))(((()))))) Add Σ (Cov [z)) + Σ (Cov go(z) (2) - 1)²
$
where Xa and And the penalisation terms for the diagonal and off-diagonal values respectively.
13

---

Under review as a conference paper at ICLR 2023
B ADDITIONAL DETAILS ON MEAN, VARIANCE AND SAMPLED REPRESENTATIONS
This section presents concise illustration of what mean, variance and sampled representations are. As shown in Figure 8, the mean, variance and sampled representations are the last 3 layers of the encoder, where the sampled representation, 2, the input of the decoder. These representations, spe- cific to VAEs, influence the models' behaviour that can be quite different from other deep learning models, as shown by research on polarised regime and posterior collapse, for example.
Mean representation
θ
Encoder
Decoder Po (1:
x
q x)
log
log covariance representation
sampled representation
Figure The structure of a VAE
C EXPERIMENTAL SETUP
To facilitate the reproducibility of our experiment, we detail below the Procrustes normalisation process and the configuration used for model training.
Procrustes normalisation Similarly to Ding al. (2021), given an activation matrix X E R containing n samples and m features, we compute the vector & E IR" containing the mean values of the columns of X. Using the outer product &, get X 1, @ ø. where 1, E R is vector of ones and X I mm We then normalise X such that
X - X X
X
(11)
X|IF
As the Frobenius norm of X and Y is 1, and Y X is always positive (1 when x Y, smaller otherwise), Equation 4 lies in [0, 2). and Equation [0, 1).
VAE training Our implementation uses the same hyperparameters Locatello al. (2019b), and the details are listed in Table I and 2. We reimplemented Locatello et al. (2019b) code base, designed for Tensorflow 1, in Tensorflow 2 using Keras. The model architecture used is also identical, as described in Table 3. Each model is trained 5 times, seeded runs with seed values from 0 to 4. Intermediate models are saved every 1,000 steps for smallNorb. 6,000 steps for cars3D and 11,520 steps for dSprites. Every image input is normalised to have pixel values between ( and 1.
For the fully-connected models presented in Appendix F, we used the same architecture and hyper- parameters as those implemented in d sissenttngleen ement lib of Locatello et al. (2019b), and the details are presented in Table and 5.
D CONSISTENCY OF THE RESULTS WITH PROCRUSTES SIMILARITY
As mentioned in Section 3, in this section we provide a comparison between the CKA scores re- ported in the main paper, and the Procrustes scores for the cars3D dataset. We see in Figures to that Procrustes and CKA provide similar results. Figures 9 and 10 show that Procrustes tends overestimate the similarity between high-dimensional inputs, mentioned in Section 2.3 (recall
14

---

Under review as a conference paper at ICLR 2023
Table 1: Shared hyperparameters
Parameter
Value
Batch size
64
Latent space dimension
10
Optimizer
Adam
Adam: B1
0.9
Adam: $2
0.999
Adam:
lc-8
Adam: learning rate Reconstruction loss
0.0001
Bernoulli
Training steps Intermediate model saving Train/test split
300,000
every 6K steps 90/10
Table 2: Model-specific hyperparameters
Model
Parameter
Value
B-VAE
B
[1, 2, 4, 6,
B-TC VAE
5
[1, 2, 4, 6, 8]
DIP-VAE II
And
2,, 10, 20]
Aa
Aod
Annealed VAE
Cmaz
[5, 10, 25, 50, 75] 1,000
iteration threshold
100,000
Table 3: Shared architecture
Encoder
Decoder
Input: channell
R I0
Conv, kemee=444, tion=ReLU, strides=2 2
filters=32,
activa-
FC, output shape=256, activation=ReLU
Conv, kemel=4x4,
filters=32,
activa-
FC, output shape=4x4x64, activation=ReLU
tion=ReLU, strides=2
Conv, kemel=444,
filters=64,
activa-
Deconv, kemel=4x4,
filters=64,
activa-
tion=ReI.U, strides=2 2
tion=ReLUU, strides22
Conv, kernel=4x4,
filters=64,
activa-
Deconv, kemel=444,
filters=32, activa-
tion=ReLU, 2
tion=ReLU, strddes222
FC, output shape=256,
activation=ReLU,
Deconv, kemel=4x4,
filters=32, activa-
strides=2
tion=ReLU, strides=2
FC, output shape=2x10
Deconv, kemee==44, filters=chanells, activa- tion=ReLU, strides=2
Table 4: Fully-connected architecture
Encoder
Decoder
Input: chexxesx
R
FC, output shape=1200, activation=ReLU
FC, output shape=256, activation=tanb
FC, output shape=1200 1200, activation-ReLU FC, output shape=2x10
FC. output shape=1200 activation=tanh FC, output shape= activation=tanh
the example given in Figure 1). In Figure 11, we observe slightly lower similarity with Procrustes than CKA on the 5th and 6th layers of the encoder, indicating that some small changes in the rep- resentations may have been underestimated by CKA, as discussed in Section 2.3 and by Ding et al. (2021). Note that the difference between the CKA and Procrustes similarity scores in Figure 11 remains very small (around 0. indicating consistent results between both metries.
15

---

Under review as a conference paper at ICLR 2023
Table 5: Hyperparameters of fully-connected models
Model
Parameter
Value
3-VAE 3-TC VAE
[1, 16]
B
[2]
DIP-VAE II
Aod Ad
[1, 20, 50]
And
Annealed VAE
Cmaz
[5]
1,000
iteration threshold
100,000
beta, vae. betam2. epoch-1090
Input
1.0
beta, van beta=1, npoch- 25
Input
-10
on
e/1
6/2
e/2
e/
e/
e/4
e/4
e5
e/5
e/6
0.6
6/6
0.6
mean
mean
log
var
sampled
4/3
sampled
d/1
0.4
4/2
d/3
0.2
dia
0.2
d/5
d/6
Impru
e/l
WB
mean
log var
sampled
671
222
imput
e/2
e:
0/44
e5
moan
log campled
d/1
4/2
dia
d/4
d/6
beta, betam2, epoch=25
beta tc. vae, beta-1. epoch-1090
(a) CKA
(b) Procrustes
Figure 9: As reported in Figure 3a of the main paper, (a) shows the CKA similarity scores of activations at epochs 25 and 1090 of 3-TC VAE trained on cars3D with 3 = 2. (b) shows the Procrustes similarity scores of the same configuration. We observe the trend with both metrics with Procrustes slightly overestimating the similarity between high dimensional activations (bottom- right quadrants), which agrees with the properties of the Procrustes similarity reported in Section 2.3.
1. epach=1050 1090
1.0
beta beta=], epoch-1090 1090
0.0
0.8
e/4
e/6
0.6
mean
beta vse. beta
log var
0.4
sampled
d/1
0.4
d/2
4/3
d/3
d/2
0.2
9/5
d/6
d/5
9/6
e/l
0F3
e5
mean
log var samples
671
d02
605
0.0
input
e/l
0/2
e/6
255
mian
log. var campled
d/1
d/3
d/a
d/5
d/6
0.0
dip_vae vae ii, lambde epoch=1090
dip.
lambda
ep9ch=1090
(a) CKA
(b) Procrustes
As Figure 7a, (a) shows the CKA similarity scores of activations 3-VAE and DIP- VAE II trained on cars3D with / 1, and 1, respectively. (b) shows the Procrustes similarity scores using the same configuration. We observe the same trend with both metrics with Procrustes slightly overestimating the similarity between high dimensional activations (bottom-right quadrants) (c.f. Section 2.3).
Figure 10:
16

---

Under review as a conference paper at ICLR 2023
Encoder
Encoder layer
203
400
500
800
1000
200
400
600
830
1030
opoch
spoch
(a) CKA 3
(b) CKA for =
1.0
3.0
0.8
0.8
0.6
0.6
0.4
0.4
ByCf
Encoddrr layer
0.2
5.2
0.0
3.0
(c) Procrustes for 3 =
(d) Procrustes for 3 8
Figure 11: (a) shows the CKA scores between the inputs and the activations the first 6 layers of the encoder of 8-VAE trained on cars3D with 3 (b) shows the scores between the same representations with 8 = 8. (a) and (b) are discussed in Figure of the main paper. (c) and (d) are the Procrustes scores of the more variance in (c) for Procrustes with 3 1. Procrustes also displays a slightly lower similarity for layers 4 to 6 of the encoder, possibly due changes in the representation that are underestimated by CKA (c.f. Section 2.3).
configurations. We observe the same trend for both metrics with
17

---

Under review as a conference paper at ICLR 2023
E RESOURCES
As mentioned in Sections and 3, we released the code of our experiment, the pre-trained models and similarity scores:
The similarity scores can be downloaded from an anonymous Google account using the following tiny URL https: //: 3
The code can also be downloaded from an anonymous Google account using another tiny URL https: : 1y/..yyyMMn
Our pre-trained models are large (around 80 GB in total), and it was not feasible to make them available the reviewers using an anonymous link. The URL to the models will, however, be available in the non-anonymised version of this paper.
Note that the 300 VAE models released correspond to models trained with:
4 different learning objectives,
5 initialisations,
3 datasets,
regularisation strengths.
F CKA ON FULLY-CONNECTED ARCHITECTURES
In order to assess the generalisability of our findings, we have repeated our observations on the fully-comnected VAEs that described in Appendix C. We can see in Figures 12, 13, and 14 that the same general trend as for the convolutional architectures can be identified (see Figures 3c, 4a, and 7 of Sections 4.1, 4.2 and 4.3 for comparison with convolutional networks).
Learning fully-connected VAEs is also bottom-up We can see in Figure 12 that, similarly to the convolutional architectures shown in Figure 3c, the encoder is learned early in the training process. Indeed between epochs and 10, the encoder representations become highly similar to the representations of the fully trained model (see Figures 12 and 12b). The decoder is then learned with its representational similarity with the fully trained decoder raising after epoch 10 (see Figure 12c).
Impact of regularisation As in convolutional architectures shown in Figure 4a, the variance and sampled representations retain little similarity with the encoder representations the case pos- terior collapse, shown in Figure 13. Interestingly, in fully-connected architectures the decoder retains more similarity with its less regularised version than in convolutional architectures, despite suffering from poor reconstruction when heavily regularised. Thus, CKA of the representations of fully-connected decoders may not be good predictor of reconstruction quality. Despite this differ- ence, it can still used to monitor posterior collapse with fully-connected architecture by relying on the similarity scores between the encoder representations, and the mean, variance and sampled representations. This property is consistent between both architectures.
Impact of learning objective Figure 14 provides results similar to the convolutional VAEs ob- served Figure with a very high similarity between encoder layers learned from different learning objectives (see diagonal values of the upper-left quadrant). Here again, the representational similar- ity of the decoder seems vary depending on the dataset, even though this is less marked than for convolutional architectures. We can also see that the representational similarity between dif- ferent layers of the encoder vary depending on the dataset, which was less visible convolutional architectures. For example, the similarity between the first and subsequent layers of the encoder in smallNorb is much lower in fully-connecter VAEs. Given that smallNorb is hard dataset to learn for VAEs (Locatello al., 2019b),
could hypothesise that the encoder of fully-connected VAE, being less powerful, is unable to retain as much information as its convolutional counterpart, leading to lower similarity scores with the representations of the first encoder layer.
18

---

Under review as a conference paper at ICLR 2023
$2,00r
004
14
serphe
66
-02
01
-12
amakd JC CIraee apechw420
urrasbeLyALCCC C.rann spoch-420
www.ale C.nen spoche410
(a) Epoch
(b) Epoch 10
(c) Epoch 410
Figure 12: (a), (b), and (c) show CKA scores between fully-trained fully-connected Annealed VAE and a fully-connected Annealed VAE trained for 10 and 410 epochs, respectively. All the models are trained on smallNorb and the results are averaged over 5 seeds. Similarity to Figure 3c of Section 4.1, we can see that there is a high similarity between the representations learned by the encoder early in the training and after complete training (see the bright cells in the top-left quadrants (a), (b), and (c)). The mean and variance representations similarity with fully trained model increase after few more epochs (the purple line in the middle disappear between (a) and (b)), and
finally the decoder is learned (see bright cells in bottom-right quadrant (c)).
08
.22
-0
AMt.wA//C balawd,
bete PC bdte-1 4sodi=25
(a) 8
(b) 9 = 16
Figure 13: (a) and (b) show representational similarity between fully-connected 3-VAEs trained with 3 = 1, and fully-connected 3- VAVS trained with 3 = 8 and 3 = 16, respectively. All models are trained on dSprites and the scores are averaged over 5 seeds. In both figures, the encoder repre- sentations stay very similar (bright cells in the top-left quadrants), except for the mean, variance and sampled representations. While the variance representation is increasingly different as we increase 3, the decoder does not show the dramatic dissimilarity observed in convolutional architectures in Figure 4a.
19

---

Under review as a conference paper at ICLR 2023
20
4G
31
54E WwL.wwe PC biteo www.ooooooooooo...
600
-0
-1
66
PR Lmmbdae 29ach-1999
ML JLF.
00 lambdaa) esechwa20
(a) Trained on cars3l)
(b) Trained on dSprites
(c) Trained on smallNorb
Figure 14: (a) shows the CKA similarity scores of activations of 3-VAE and DIP-VAE II with fully- connected architectures trained on cars3D with / 1, and À = 1, respectively. (b) and (c) show the CKA similarity scores of the same learning objectives and regularisation strengths but trained on dSprites and smalINorb. All the results are averaged over 5 seeds. We can see that the representa- tional similarity of all the layers of the encoder (top-left quadrant) except mean and variance is high (CKA 0.8). However, in Figure 7, the mean, variance, sampled (center diagonal values), and decoder (bottom- -bottom--righ&& quadrants) representational similarity of different learning objectives seems to vary depending on the dataset. In (a) and (c) they have high similarity, while in (b) the similarity is lower. Moreover, in (c) the input and first layer of the encoder are quite distinct from the other representations, which was not the case in convolutional architectures.
G HOW WELL DOES CKA DISTINGUISH POLARISED REGIME FROM POSTERIOR COLLAPSE?
In Section 4.2, we stated that CKA could be useful tool to monitor posterior collapse. Indeed, we have seen in Section 2.1 that the polarised regime can be viewed as a "healthy" version of the posterior collapse where the only latent variables that not needed become passive, while posterior collapse will result in making most of the latent variables passive, preventing the decoder from providing good reconstruction. In both cases, passive variables are close to (V 1) in the sampled representation. For this to be possible, the mean representation of passive variables is always very close O, and the variance representation instead of 14; + azé where (N, 1). While passive variables exhibit differences between the mean and sampled representations, the active variables are very similar in mean and sampled representations. Indeed, in this case, the variance is very low, leading to 2, R (4.-
1, so that the sampling process results in 2; s
Keeping in mind that linear CKA is a generalisation of the correlation coefficient to matrices, there is any linear relationship between two representations, CKA will be high and the stronger this relationship is, the higher CKA will be.
Because mean and sampled representations have similar active variables but different passive vari- ables, the presence of active variables will result in higher CKA scores. Thus, mean and sampled representation will have higher CKA in polarised regime (where they contain many active variables) than during posterior collapse (where there are none or very few active variables). However, one wonder whether CKA can lead to false positives when VAEs contain passive variables for non- pathological reasons. For example, due to the polarised regime, if one provides more latent variables than needed by the VAEs, some of the variables will be collapsed reduce the KL divergence. As in posterior collapse, the decoder will ignore these passive variables. However, contrary to collapsed models, when passive variables are a result of the polarised regime, the decoder will still have ac- cess to meaningful information and will be able to correctly reconstruct the image, learning similar representations as a good model with fewer latent variables. We can see in Figure 15 that, oppo- sition to posterior collapse, the variance and sampled representations retain much higher similarity scores with the representations learned by other layers. Thus, one can differentiate between the two scenarios using CKA. Given that the CKA scores for the variance and sampled representations vary similarly in fully-connected architectures, CKA seems to consistently be a good predictor of poste- rior collapse across learning objectives and architectures while being robust to false positives. While one could be tempted to monitor posterior collapse using the changes of similarity scores in the de- coder, we have seen in Appendix F that the fully-connected decoders could retain relatively high
20

---

Under review as a conference paper at ICLR 2023
similarity in the of posterior collapse. Thus, recommend relying on the CKA scores of the mean, variance and sampled representations for a better robustness across architectures.
beta betam] epoch-26. 2. d.mm=20
imput
input
bota vas beta-1 epoch=26
0.8
0.8
mean
mean log var
log_ var
sampled ed
-0.4
sampled
0.4
d/1
d11
9/2
d12
4/8
d13
0.2
0/4
d14
d/5
6/5
4/6
6/6
0.0
0.0
input
eft
68
6M
e5
meean
log var sampled
dR 002
ai
imput
6/2 e22
644
log var samples 611 di di
beta_vae, betam1, epoch=26, 2.dim=10
beta
beta_vae, vae, beta mB, epoch=26
(a) =
(b) 3 =
Figure 15: (a) shows the CKA similarity scores between the activations of two 8 VAES trained on dSprites with I, the first with 10 latent variables, and the second with 20. (b) shows the CKA similarity scores between the activations of two 3-VAEs with 10 latent variables trained on dSprites with | and / the results are averaged over seeds. We can see that we retain higher similarity in the decoder representations in the case of polarised regime (bottom-right quadrant in (a)) than posterior collapse (see the darker bottom-right quadrant in (b)). The representational similarity of the mean, variance and sampled representations is also higher in the case polarised regime than posterior collapse.
For (a) and (b), the activations are taken after complete training, and all
H HOW SIMILAR ARE THE REPRESENTATIONS LEARNED BY ENCODERS AND CLASSIFIERS?
To compare VAES with classifiers, we used the convolutional architecture of an encoder for classifi- cation, replacing the mean and variance layers by the final classifier layers. As shown in Figure 16, we obtain a high representational similarity when comparing VAEs and classifiers indicating, con- sistently with the observations of Yosinski et al. (2015), that classifiers seem to learn generative features. This explains why encoders based on pre- pre-trained classifier architectures such as VGG have empirically demonstrated good performances (Liu et al., 2021) and also suggests that the weights of the pre-trained architecture could be used as-is without further updates. While using pre-trained en- coders may beneficial in the context transfer learning, domain adaptation (Pan & Yang, 2009), simply reconstruction quality (Liu et al., 2021), should not expect dramatic improvement of the training time given that the encoder is learned very early during the training (see Section 4.
REPRESENTATIONAL SIMILARITY OF VAES AT DIFFERENT EPOCHS
The results obtained in Section 4.1 have shown a high similarity between the encoders at an early stage of training and fully trained. One can wonder whether these results are influenced by the choice of epochs used in Figure 3. After explaining our epoch selection process, we show below that it does not influence our results, which are consistent over snapshots taken at different stages of training.
Epoch selection For dSprites, we took snapshots of the models at each epoch, but for cars3D and smallNorb, which both train for a higher number epochs, it was not feasible computationally to calculate the CKA between every epoch. We thus saved models trained on smallNorb every 10 epochs, and models trained on cars3D every 25 epochs. Consequently, the epochs chosen to repre- sent the early training stage in Section 4 is always the first snapshot taken for each model. Below, we preform additional experiments with a broader range of epoch numbers to show that the results consistent with findings in the main paper, and they do not depend on specific epochs.
21

---

Under review as a conference paper at ICLR 2023
bat. voc. spock-1990
1.0
-01
10
10
-0.
-.8
meg
Va
603
422
63
45
02
opoch=1090
epsch-26
dasoifier,
(a) Trained on cars3l)
(b) Trained on dSprites
(c) Trained on smallNorb
Figure 16: (a) shows the CKA similarity scores activations of a classifier and 3-VAE trained on cars3D with / 1. (b) and (c) show the CKA similarity scores of the same learning objectives and regularisation strengths but trained on dSprites and smallNorb. All the results are averaged over 5 seeds. We can see that the representational similarity between the layers of the classifiers and of the encoder (top-left quadrant) except mean and variance very high (CKA stays close to I). However, the mean, variance, sampled representations (center diagonal values) are different from the representations learned by the classifier.
Similarity changes multiple epochs In Figures 17, 18, and 19, we can observe the same trend of learning phases as in Figure 3. First, the encoder is learned, as shown by the high repre- sentational similarity of the upper-left quadrant of Figures 17a, 18a, and 19a. Then, the decoder is learned, as shown by the increased representational similarity of the bottom-right quadrant of Fig- ures 17b, 18b, and 19b. Finally, further small refinements the encoder and decoder representations take place in the remaining training time, shown by the slight increase of representational simi- larity in Figures 17c, 18c, and 19c, and Figures 17d, 18d, and 19d.
22

---

Under review as a conference paper at ICLR 2023
input
dip lambdaas spoch-2
dip. vac lambda=5, spoch=7
0.88
0.8
log.
sampled
-0.4
0.4
0.0
0.0
Innut
e/l
6/3
e/6
meen
611
072
063
d15
Ingut
6/2
014
e/6
log var led
611
612
0t3 814
616
samp
log
vip_vae_ lambda epoch-26
dip_vae_ll 1I, lambda m5 spoch-26
(a) Epoch 2
(b) Epoch 7
input
1.0
Input
1.0
dip vae lambda epoch=14
dip vae ji. lambda 5, epoch=19
e/2
e/2
e/
6/3
0.8
6/4
6/4
o5
65
e/6
0.66
6/6
0.6
mean
maan
log var
log vor
sampled
sampled
d/1
41
8/2
6/2
d/3
-0.2
6/3
0.2
814
614
025
615
4/6
4/6
Impol
e/l
6/
el5
w/6
meen
log var
sampled
6/1
672
6/4
6/6
Inpot
6/1
e/2
6/4
65
log var
sampled
621
6/2
633 d/
65
dip_vae_ ii, lambda 5
epoch=26
dip vae_va__i,
lambda
epoch=26
Epoch 14
(d) Epoch 19
Inout
o/1
e/2
e/3
0.8
w/4
dip.ve vae ji. beta=5,
6/6
0.4
0.2
Imput
8/1
e3
o5
e/6
muan
sampled
d/1
6/2 6:3
did 6.5
vee lambda -.
epoch=26
(e) Epoch 26
Figure 17: (a), (b), (c), (d), and (e) show the representational similarity between DIP-VAE after full training, and at epochs 2, 7, 14, 19, and 26, respectively. All models are trained on dSprites and the results averaged over runs.
23

---

Under review as a conference paper at ICLR 2023
input
beta te vas. beta=1, epech=22 292
bela, voe. betam. spoch-25
0.88
0.8
log.
sampled
-0.4
0.4
d55
0.0
0.0
Input
e/l
6/3
e/6
mrean
002
063
64
d/5
Ingut
6/2
014
e/6
log var sampp led
611
612
0t3 814
655
616
somp
log
peta_tc_vee, beta-1. epoch=1090
beta_tc vae,, beta=l, epoch-1090
(a) Epoch 25
(b) Epoch 292
beta, vae. betam. spoch=559
Input
1.0
beta tc vavve beta=1 spoch= 926
126ch=926
Input
1.0
e/2
e/2
e/
633
0.8
6/4
6/4
e/
65
e/6
0.66
6/6
0.6
mean
maan
log var
log vor
sampled
sampled
d/1
41
4/2
d/2
d/3
-0.2
6/3
0.2
014
d14
025
d/5
4/6
4/6
Imput
e/1
6/
6/4
6/6
meen
log var
sampled
6/1
672
oF
Inpot
6/1
e/2
6/4
6/5
log var sampled
621
6/2
6/3
65
beta beta_te_yee vae beta=1, epoch=1090
beta, tc vae, beta=1, epoch=1090
Epoch 559
(d) Epoch 826
input
beta te vae, beta 1, wwwchpppchoee
e/2
6/3
0.8
6/4
e/5
e/6
0.6
mean
log var
sampled ed
0.4
d/1
d/2
4/3
4/5
0.0
Input
e/1
a/3 2
e5
e/6
log var
sampled
d/1
d/2
6/3
d/4
dis
beta_tc vae, beta-1. epoch-1090
(c) Epoch 1090
Figure 18: (a), (b), (c), (d) and (e) show the representational similarity between 3-TC VAE after full training. and at epochs 25, 292. 559, 826, and 1090 respectively. All models are trained on cars3D and the results are averaged over 5 runs.
24

---

Under review as a conference paper at ICLR 2023
annealed vae. mmax-5, spoch=10
input
ernalle vac. maxes epoch= 110
0.8
0.8
log.
sampled
-0.4
0.4
0.0
0.0
Innut
6/1
613
es
e/6
mreen
611
002
063
64
615
Ingut
6/2
014
e/6
log var
samp led
611
612
0t3
814
616
comp
log
annealed vae, c_max=5, epoch=410
annealed vas, C_max=S, epoch=410
vas.
(a) Epoch 10
(b) Epoch 110
annealed vae. <_aa epoch-210
input
1.0
annealed vae. C_ax=5, opoch=311
input
1.0
e/2 e/
e/2
6/3
0.8
6/4
6/4
65
65
e/6
0.66
6/6
0.6
mean
maan
log var
log vor
sampled
sampled
4/1
41
4/2
d/2
d/3
-0.2
6/3
0.2
d14
614
025
6/5
4/6
4/6
Impul
e/1
63
6/4
el
6/6
meen
log var
sampled
671
672
oF3
6/4
Inpot
6/1
e/2
6/4
65
log var
sampled
621
6/2
613 d/
65
annealed vae, t_max S5 epach=410
annealed_vas vas, c_max=5, spoch=410
Epoch 210
(d) Epoch 311
annealed, vae, mama--, epoch-41c 41c
input
e/2
0/3
0.8
6/4
e/5
e/6
0.6
mean
log var
sampled
0.4
d/1
d/2
4/3
d/5
0.0
Input
6/1
8/3 8/4
e/6
log var sampled
d/1
di2
die
dis
annealed vae, c_max=5, epoch-410
(e) Epoch 311
Figure 19: (a), (b), (c), (d) and show the representational similarity between Annealed VAE after full training, and at epochs 10, 110, 210, 311, and 410 respectively. All models are trained on smallNorb and the results are averaged over runs.
25

---

Under review as a conference paper at ICLR 2023
J CONVERGENCE RATE OF DIFFERENT VAES
We can see in Figure 20 that all the models converge at the same epoch, with less regularised models reaching lower losses. While annealed VAEs start converging together with the other models, they then take longer to plateau, due to the annealing process. We can see them distinctly in the upper part of Figure 20. Overall, the epochs at which the models start converge are consistent with our choice of epoch for early training in Section 4.1.
2343
3340
260
160
190
(a) Convergence on cars3D
(b) Convergence on dSprites
(c) Convergence smallNorb
Figure 20: In (a), (b), and (c), we show the model loss of each model that converged when trained on cars3D, dSprites, and smalINorb, respectively. For each learning objective, display runs of the least and most regularised versions.
26


Reviews:
[{"review_id": "uDJspjYPCD6", "reviewer": "Reviewer_nV4t", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper covers a very extensive empirical study of the representational similarity among different layers between auto-encoders at different stages of training and between different auto-encoders.  The study involves two different measures of similarity.", "summary_of_the_review": "Although there is much to be admired about the paper, I still need to be convinced about its utility.  It describes two related measure of similarity, however, interpreting what these measures tell us is very difficult.  Without a better explanation of what we are meant to take away from these measures I struggle to gain much insight.  I would have preferred to see a more holistic analysis that could be much more limited, but would provide more insight into how we should interpret these similarity measures and what they tell us.  It would have been helpful to see the similarity plot of a model with itself at the same point in time as this would provide a baseline to understand the results shown.  There seems to be a sudden loss of similarity as the representation hits the decoder.  Given that auto-encoders preserve a lot information it would have been really useful to understand what is going on.  Finally, the paper is built up as an exploration of disentanglement, but it is not at all obvious that the paper has anything to say about this.\n\nGiven the huge work that has gone into this, I would like to be persuaded to increase my score.  But to do this I need more help in understanding the utility of the work that has been carried out.", "strength_and_weaknesses": "The paper is clearly written and presents very thorough work.  I am generally very supportive of thorough empirical studies whatever their outcome.  From the scientific point of view there is a lot to like.\n\nMy hesitation is about the usefulness of the measure of similarity.  I find the similarity measure puzzling and I feel the paper would benefit from more interpretation of what the similarity is measuring.  For example, in an auto-encoder I would naively expect that there would be similarity between the input to the encoder and the output of the decoder.  This was not displayed at all as far as I can see.  There were interpretations being made that the encoder learnt representations far faster than the decoder.  I would like to have seen some other evidence to convince me that this was not an artefact of the measure of similarity being used.\n\nPart of the justification of the paper was as a study of disentanglement.  Indeed the paper studied a variety of VAEs that were designed to achieve disentanglement.  I found this very unconvincing.  In particular the measures were specifically designed to be invariant to  rotations in the representations as I understand it.  However rotations in latent space are critical to disentanglement so it seemed like a strange measure to investigate disentanglement.  Apart from a discussion in the introductions I could not see that the paper threw any light on disentanglement.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "qpLMuTpfXeT", "reviewer": "Reviewer_2ctR", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper discussed a few similarity metrics and conducted layer-wise comparisons of VAEs using the similarity metric with different datasets, models and parameters. And conclusions are drawn on the model's disentanglement and posterior collapse based on the results. The authors found that encoder representations are learnt before decoder across all settings and they are similar across hyperparameters and learning objective within the same dataset.", "summary_of_the_review": "Overall, I think using similarity metric could be a good idea, but the findings and their implications aren't very novel. The connection of the finding to the claims should be discussed in greater details and more rigorously too. ", "strength_and_weaknesses": "Strength:\n1. detailed discussion on the limitations of the similarity metrics\n2. code and libraries are provided\n3. there are a diversity of datsets, similarity metrics and settings considered.\n\nWeakness:\n1. I think the similarity metric's connection with posterior collapse should be explained in greater details than references to the other papers. i don't think it's straightforward and since it is one of the main points of the paper, the motivation and justification of it could be a bit more detailed. The word \"polarized regime\" could also have a bit more details for the audience. \nFor example, the paper Figure 5 mentions \"the mean and sampled representations present a growing number of passive variables which, in the case of sampled representations, leads to high dissimilarity with the input in (b).\" Why does the high dissimilarity necessarily means posterior collapse, it seems like the the value is in fact converging so what if that amount of dissimilarity is just right for the model? There isn't really a threshold number that determines what a posterior collapse is for the specific problem or dataset, so i think it is ill-defined. \n\n2. I feel that similar conclusion such as encoder learnt before decoder has been proposed or discussed or hypothesized in other literature and that should be discussed. For example, this paper \"lagging inference networks and posterior collapse in variational autoencoders\" is proposed based on the idea that the encoder trains slower than decoder. I think a more thorough discussion and citations in this line of work is warranted.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "iZNMPp0wCIg", "reviewer": "Reviewer_hNQM", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work studies the layer-wise representations of different VAEs. The similarity between the layer-wise representations is measured using the Centered Kernel Alignment metric and the Procrustes scores. Experiments are performed to study the effect of different hyperparameters on the same model, regularization, initialization, and learning objectives.\n ", "summary_of_the_review": "Overall, the paper is well-written and the experiments are conducted to evaluate the similarity of different representations from the VAEs. The findings in the paper such as  \"encoders learn before decoders\" are already well-known in the community. Moreover, the paper argues that linear CKA can be used to track posterior collapse. The motivation for this claim is not clear. Experiments are limited to certain design choices, why only choose regularization as a hyperparameter?\n\n", "strength_and_weaknesses": "Strengths:\n\n+The work is well-written.\n+The limitations of the approach are clearly discussed. The analysis are performed taking into account the limitations of the proposed metrics.\n+The results seem like they should be reproducible given sufficient compute.\n\nWeaknesses:\n-The approach is limited to the comparison of representations from similar layers. For example, the similarity between convolutional and deconvolutional layers cannot be evaluated using the proposed approach.\n- The findings in the paper, regarding posterior collapse, and encoder learning before decoder to prevent posterior collapse are already well-established in the community and it is not clear what is the main \"new\" finding of the paper. The metrics, especially CKA [1] have been used to study deep networks in prior work and the work applies the same tools to VAEs, thus there are no new contributions.\n - CKA is claimed to be a metric of choice for monitoring posterior collapse. What makes it a better tool? Why can we just monitor the latents?\n- The hyperparameters considered for evaluation is limited to choice of regularization? There are other factors such as optimizers, lr schedulers etc which also can impact training. What are the insights here?\n\n[1] Similarity of Neural Network Representations Revisited. ICML 2019.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "ZiOgUkCVx-F", "reviewer": "Reviewer_rD7K", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper investigates the behavior of disentangled representation learning in VAEs. Especially, the authors utilizes Centred Kernel Alignment to compare the internal behavior of VAEs. Procrustes score is mentioned in the main paper, but the authors did not utilize it due to its limitation on the computational complexity. As a result, they found that the encoder parameters are optimized earlier that the parameters in the decoder, regardless of hyper-parameters, objective function, and the datasests.\n", "summary_of_the_review": "The paper investigates an interesting problem, the learning procedure of VAE. While the work shows some promising observations on the VAE learning and posterior collapse problem, the contribution is somewhat limited due to the shallow experiments. Therefore, I slightly lean to the negative side.\n", "strength_and_weaknesses": "The paper explores training procedures of VAE which is an interesting part, and the authors clearly list the observations that they have studied.\n\nThe authors argue that they have tested 300 VAEs, but I can not find the list of them. Does the 300(=4*5*5*3) VAEs implies the combinations on the [4 versions of VAEs of different objective functions] * [5 different initializations] * [5 different regularization hyper-parameters] * [3 datasets]? \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "QzOfCjalyI1", "reviewer": "Reviewer_dy86", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper describes an evaluation framework based on representation similarity to study and explain the way VAEs learn and diagnose some of the main problems with this type of model. One aspect that sets this work apart is that it looks at representations at every layer, not just the inputs and latent space.", "summary_of_the_review": "A good paper which peovides some useful insights into the inner working of VAEs. It is definitely a valuable contribution to ICLR and I can see researches in the area could draw from these insights to push forward the state of the art.", "strength_and_weaknesses": "Strengths:\nThe paper analyses important aspects of VAEs with a principled experimwntal framework. It provides valuable and scientifically grounded explainations of problems, phenomena and trends which I personally encountered more then once in building VAEs and are, for the great part, left to intuition and practical experience to deal with. Experimentally or theoretically grounded interpretability insights are always greatly appriciated in deep learning in genereal, especially in representation/generative models.\n\nWeaknesses:\nMy main concern is with the choice of experiments and the conclusions that come from it. \n\n1) The conclusions are clear and they are indeed proven by the experiments, but either some parts of the experiments are redundant, or there is some additional conclusions that are not explained in the paper. All conclusions detailed as \"implications\" at the end of each experimental section only depend on comparing the same layer, e.g. e5 at 25 and 1000+ iterations, with different betas etc. So to see evidence of the claims, I only need to look at the diagonals in the matrices shown. What are the off diagonal similarities showing the reader? One interesting thing is that all encoder layers seem to get stuck to just reproduce the inputs. This is very interesting, but it is never mentioned under \"implications\".\n\n2) The similarity of the representations at each layers is the main novelty aspect of the evaluation (other than the conclusions themselves), but I am not sure what motivates this choice. The more intuitive and simpler thing to do would be to compare just the similarity of the encoder/decoder weights themselves. What can we see by comparing embeddings over a dataset as opposed to weights values?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject