Unsupervised Model Selection for Variational Disentangled Representation Learning
Authors: Sunny Duan
Loic Matthey
Andre Saraiva
Nick Watters
Chris Burgess
Alexander Lerchner
Irina Higgins
Venue:  (2020)
Keywords: unsupervised disentanglement metric
disentangling
representation learning

Abstract: Disentangled representations have recently been shown to improve fairness, data efficiency and generalisation in simple supervised and reinforcement learning tasks. To extend the benefits of disentangled representations to more complex domains and practical applications, it is important to enable hyperparameter tuning and model selection of existing unsupervised approaches without requiring access to ground truth attribute labels, which are not available for most datasets. This paper addresses this problem by introducing a simple yet robust and reliable method for unsupervised disentangled model selection. We show that our approach performs comparably to the existing supervised alternatives across 5400 models from six state of the art unsupervised disentangled representation learning model classes. Furthermore, we show that the ranking produced by our approach correlates well with the final task performance on two different domains.

Published as a conference paper at ICLR 2020
UNSUpERVISED MODEL SELECTION FOR VARIATIONAL DISENTANGLED REPRESENTATION LEARNING
Sunny Duan"
Loic Matthey DeepMind
Andre Saraiva
DeepMind
DeepMind
sunnydegoogle.com
Lmatthey@googie.cmm
andreeanss@google.com
Nick Watters
Chris Burgess DeepMind
Alexander Lerchner
DeepMind
DeepMind
nwatters@yoogle.com
cpburgess@google.comm
Lerchner@google.com com
Irina Higgins
DeepMind
ir crinah@googgee.om
ABSTRACT
Disentangled representations have recently been shown to improve fairness, data efficiency and generalisation in simple supervised and reinforcement learning tasks. To extend the benefits of disentangled representations practical applications, it is important to enable hyperparameter tuning and model se- lection of existing unsupervised approaches without requiring access ground truth attribute labels, which are not available for most datasets. This paper addresses this problem by introducing a simple yet robust and reliable method for unsupervised dis- entangled model selection. Our approach, Unsupervised Disentanglement Ranking (UDR) leverages the recent theoretical results that explain why variational autoen- coders disentangle (Rolinek et al. 2019). to quantify the quality of disentanglement by performing pairwise comparisons between trained model representations, We show that our approach performs comparably to the existing supervised alternatives across 5400 models from six state the art unsupervised dissstaneeddddeeeeeetnttaioooo learning model classes. Furthermore, approach correlates well with the final task performance on two different domains.
complex domains and
show that the ranking produced by
INTRODUCTION
fampilaailice are all alike; every unhappy family is unhappy in its own way.
Leo Tolstoy, Anna Karenina
Despite the success of deep learning in the recent years (Hu et al. 2018 Espeholt et al. 2018 Silver al. 2018 Lample et al. 2018: Hessel et al. 2017 Oord et al. 2016), the majority of state of the art approaches are still missing many basic yet important properties, such faimess, data efficient learning, strong generalisation beyond the training data distribution, or the ability to transfer knowledge between tasks (Lake al. 2016 Garnelo al. 2016 Marcus 2018). The idea that good representation can help with such shortcomings is not new, and recently number of papers have demonstrated that models with disentangled representations show improvements in terms of these shortcomings (Higgins et al.] |017b 2018b| et al.) 2018 (0188Sttennie Steenbrugge et al. 2018 Nair et al.) 2018 Laversanne- Finot al. 2018 van Steenkiste et al. 2019 Locatello et al. 2019). A common
intuitive way to think about disentangled representations is that it should reflect the compositional
"Equal contribution.
We have released the code for method as part of sentangl ement_lib

---

Published as a conference paper at ICLR 2020
UDR 0.806
Seed
Seed 2
Seed
UDR 0.204
Seed
Roof haight
Car type
Colour
Rolation
Figure 1: Latent traversals for one of the best and worst ranked trained V-V models using the Unsupervised Disentanglement Ranking (UDRL) method on the 3D Cars dataset. For each seed image we fix all latents 2; to the inferred value, then vary the value of one latent at time to visualise its effect on the reconstructions. The high scoring model (left 3 blocks) appears well disentangled, since individual latents have consistent semantic meaning across seeds. The low scoring model (right block) is highly entangled, since the latent traversals are not easily interpretable.
structure of the world. For example, to describe an object we often use words pertaining to its colour, position, shape and size. We can use different words to describe these properties because they relate to independent factors of variation in our world, i.e. properties which can be compositionally recombined. Hence a disentangled representation of objects should reflect this by factorising into dimensions which correspond to those properties (Bengio et al. [013) Higgins et al. (2018)).
The ability to automatically discover the compositional factors of complex real datasets can be of great importance in many practical applications of machine learning and data science. However, it is important to be able to learn such representations in an unsupervised manner, since most interesting datasets do not have their generative factors fully labelled. For a long time scalable unsupervised disentangled representation learning was impossible. until recently new class of models based on
Variational Autoencoders (VAEs) (Kingma & Welling 22014) Rezende et al. 2014)) was developed.
These approaches (Higgins et al.) (2122] Burgess et al. 20177 al. [2088] | Kumar et al., [017 Kim & Mnih 2018) scale reasonably well and are the current state of the art in unsupervised disentangled
representation learning. However, so far the benefits of these techniques have not been widely exploited because of two major shortcomings: First, the quality of the achieved disentangling is sensitive to the choice of hyperparameters, however, model selection is currently impossible without having access to the ground truth generative process and/or attribute labels, which are required by all the currently existing disentanglement metrics (Higgins et 2017a Kim & Mnih 2018 Chen et al., 2018) Eastwood & Williams, 2018)) Ridgeway & Mozer 2018). Second, even if one could apply any of the existing disentanglement metrics for model selection, the scores produced by these metrics can vary a lot even for models with the same hy perparameterss and trained on the same data (Locatello et al. 2018). While lot of this variance explained by the actual quality of the learnt representations, some ot is introduced by the metrics themselves. In particular, all of the existing supervised disentanglement metrics assume a single "canonical" factorisation of the generative factors, any deviation from which is penalised. Such "canonical" factorisation, however, is not chosen in a principled manner. Indeed, for the majority of datasets, apart from the simplest ones, multiple equally valid disentangled representa- tions may be possible (see Higgins et al. (2018a) for discussion). For example, the intuitive way that humans reason about colour is in terms of hue and saturation. However, colour may also be represented in RGB, YUV, HSV, HSL, CIELAB. Any of the above representations are as valid as each other, yet only one of them is allowed to be "canonical" by the supervised metrics. Hence, a model that learns to represent colour in subspace aligned with HSV will be penalised by a supervised metric which assumes that the canonical disentangled representation of colour should be in RGB. This is despite the fact that both representations are equal terms of preserving the compositional property at the core of what makes disentangled representations useful (Higgins al. 2018a). Hence, the field finds itself in a predicament. From one point of view, there exists a set of approaches capable of reasonably scalable unsupervised disentangled representation learning. On the other hand, these models are hard to use in practice, because there is no easy way to do a hyperparameter search and model selection.
This paper attempts bridge this gap. We Wepropose simple yet effective method for unsupervised model selection for the class of current state-of-the-ans VAE-based unsupervised disentangled representation learning methods. Our approach, Unsupervised Disentanglement Ranking (UDR), leverages the recent
2

---

Published as a conference paper at ICLR 2020
theoretical results that explain why variational autoencoders disentangle (Rolinek et al. 2019). to quan- tify the quality of disentanglement by performing pairwise comparisons between trained model repre- sentations. Wc evaluate the validity of our unsupervised model selection metric against the four best ex- isting supervised a alternatives reported in the large scale study by Locatello et al. (2018): the 3-VAR met- ric (Higgins et al. 2017a), the Factor metric (Kim & Mnih 2018), Mutual Information Gap (MIG)
(Chen et al., 2018 8) and DCI Disentanglement scores (Eastwood & Williams, 2018). We do so for all existing state of the art disentangled representation learning approaches: 3-VAE (Higgins et al. 2017a). CCI-VAE (Burgess et al. (017). Factor VAE (Kim & Mnih, 2018), TC- (Chen et al. 2018) and two versions of DIP- VAE (Kumar et al. 2017). We validate our proposed method on two datasets with fully known generative processes commonly used to evaluate the quality of disentangled representations: dSprites (Matthey et al. 2017) and 3D Shapes (Burgess & Kim 2018), and show that our unsupervised model selection method is able to match the supervised baselines in terms of guiding hyperparameter search and picking the most disentangled trained models both quantitatively and qualitatively. We also apply our approach to the 3D Cars dataset (Reed et al. 2014). where the full set of ground truth attribute labels is not available, and confirm through visual inspection that the ranking produced by our method is meaningful (Fig. D Overall we evaluate 6 different model classes, with 6 separate hyperparameter settings and 50 seeds on 3 separate datasets, totalling 5400 models and show that our method is both ac- curate and consistent across models and datasets. Finally, we validate that the model ranking produced by our approach correlates well with the final task performance on two recently reported tasks: a classifi- cation fairness task (Locatello et al. 2019) and -oase--b-eedd reinforcement learning (RL) task (Watters) ct al. 2019). Indeed, on the former our approach outperformed the reported supervised baseline scores.
2
OPERATIONAL DEFINITION OF DISENTANGLING
Given a dataset of observations X {x (...... J. we assume that there exist a number of plausible generative processes 9i that produce the observations from a small set of corresponding K; independent generative factors c,. For each choice of gCC.c. Cn. + En. where p(c,) =II;= -I)=I((())) For example, dataset containing images of an object, which can be of a particular shape and colour, and which can be in a particular vertical and horizontal positions, may be created by generative process that assumes ground truth disentangled factorisation into shape colour x position, or shape x hue saturation x position x position Y. We operationalise a model as having learnt a disentangled representation, it learns to invert of one of the generative processes 9i and recover a latent representation E R',, , that it best explains the observed data p(z x, = (ee))=p(c,))) p(c,,2), and factorises the same way as the corresponding data generative factors e;. The choice of the generative process can be determined by the interaction between the model class and the observed data distribution p( p(æ:). as discussed next in Sec. 3.1
3 VARIATIONAL UNSUPERVISED DISENTANGLING
The current state of the art approaches unsupervised disentangled representation learning based on the Variational Autoencoder ( (VAE) framework (Rezende et al.] 2014 Kingma & Welling 2014). VAEs attempt estimate the lower bound the joint distribution of the data and the latent factors p(æ,22 by optimising the following objective:
AV = Ep(rr) Egg (L lI
where, in the usual case, the prior p(z) is chosen to be an isotropic unit Gaussian. In order to encourage disentangling, different approaches decompose the objective Eq. T intt various terms and change their relative weighting. In this paper we will consider six state of the art approaches to unsupervised disentangled representation learning that can be grouped into three broad classes based on how they
modify the objective in Eq. [ 1) 8-VAE (Higgins et al. (201 and CCI-VAE (Burgess al. 2017) upweight the KL term; 2) FactorVAE (Kim & Mnih 2018) and TC-VAE Chen et al. (0088) introduce total correlation penalty; and 3) two different implementations of DIP- VAE (-I and II) (Kumar et al.) 2017) penalise the deviation of the the marginal posterior from a factorised prior (see Sec. A.4.1l. Supplementary Material for details).
3

---

Published as a conference paper at ICLR 2020
3.1 WHY DO VAES DISENTANGLE?
In order to understand the reasoning behind our proposed unsupervised disentangled model selection method, is first important understand why VAEs disentangle. The objective in Eq. T does not in itself encourage disentangling, as discussed in Rolinek et al. (2019) and Locatello et al. (2018) Indeed, any rotationally invariant prior makes disentangled representations learnt in an unsupervised setting unidentifiable when optimising Eq. This theoretical result is not surprising and has been known for while in the ICA literature (Comon 1994), however what is surprising is that disentangling VAEs appear to work in practice. The question of what makes VAEs disentangle was answered by Rolinek et al. (2019), who were able to show that it is the peculiarities of the VAE implementation choices that allow disentangling to emerge (see also discussion in Burgess et al. (017)) Mathieu et al. (2019)). In particular, the interactions between the reconstruction objective (the first term in Eq. and the enhanced pressure match diagonal prior created by the modified objectives of the disentangling VAEs, force the decoder to approximate PCA-like behaviour locally around the data samples. (ooiine et et. (2119) demonstrated that during training VAEs often enter the so-called "polarised regime", where many of the latent dimensions of the posterior are effectively switched off' by being reduced to the prior go (z,z) (2))=p(2)) (this behaviour is often further encouraged by the extra disentangling terms added to the ELBO). When trained in such a regime, a linear approximation of the Jacobian of the decoder around a data sample Z;,
J DDec (ax)) is forced to have orthogonal columns, and hence separate the generative factors (w:) based on the amount of reconstruction variance they induce. Given that the transformations induced by different generative factors will typically have different effects the pixel space (e.g. changing the position of a sprite will typically affect more pixels than changing its size), such local orthogonalisation of the decoder induces identifiable disentangled latent space for each particular dataset. An equivalent statement is that for a well disentangled VAE, the SVD decomposition JUUVT of the Jacobian J caleulated as above, results in trivial V, which is signed permutation matrix.
UNSUPERVISED DISENTANGLED MODEL SELECTION
We now describe how the insights from Sec. motivate the development of our proposed Unsupervised Disentanglement Ranking (UDR) method. Our method relies the assumption that for a particular dataset and a VAE- based unsupervised disentangled representation learning model class, disentangled representations all alike, while every entangled representation is entangled in its own way, to rephrase Tolstoy. We justify this assumption next.
Disentangled representations are similar According to Rolinek et al. (2019) for a given
non-adversarial dataset disentangling VAE will likely keep converging the same disentangled representation (up to permutation and sign inverse). Note that this representation will correspond to
single plausible disentangled generative process di using the notation we introduced in Sec. [ This is because any two different disentangled representations 2a and 2 learnt by a VAE-based model will only differ in terms of the corresponding signed permutation matrices Va and Vo of the SVD decompositions of the locally linear approximations of the Jacobians of their decoders.
Entangled representations are different Unfortunately the field of machine learning has little theoretical understanding of the nature and learning dynamics of internal representations in neural networks. The few pieces of research that have looked into the nature of model representations (Raghu al. 2017 Li al. 2016 Wang et al. 2018 Morcos al. 2018 have been empirical rather than theoretical in nature. All of them suggest that neural networks tend to converge to different hidden representations despite being trained on the same task with the same hyperparameters and architecture and reaching similar levels of task performance. Furthermore, the theoretical analysis and the empirical demonstrations Rolinek et al. (209)) suggest that the entangled VAFs learn representations that different at least up to rotation induced by non-degenerate matrix V in the SVD decomposition of the local linear approximation of the decoder Jacobian J,
The justifications presented above rely on the theoretical work of Rolinek et al. (0019), which was empirically verified only for the 3- VAE. We have reasons to believe that the theory also holds for the other model classes presented in this paper, apart from DIP-VAE-L We empirically verify that this the case in Sec. A...i. Supplementary Materials. Furthermore, in Sec.5l show that our proposed method works well in practice across all model classes, including VAF-L.
4

---

Published as a conference paper at ICLR 2020
Unsupervised Disentanglement Ranking Our proposed UDR method consists of four steps (illustrated in Fig. Supplementary Material):
1. Train M H x S models, where H is the number of different hyperparameter settings, and S is the number of different initial model weight configurations (seeds).
2. For each trained model E Ml,....M.. sample without replacement P S S other trained models with the same hyperparameters but different seeds.
3. Perform P pairwise comparisons per trained model and calculate the respective UDR,, scores, where i E MY) is the model index, and j E (1....,P) is its unique pairwise match from Step 2. Aggregate UDR;, scores for each model i to report the final UDR, = avg, (UDR.;) scores, where avg, (.) is the median over P scores.
4.
The key part of the UDR method is Step 3, where we calculate the UDR;j score that summarises how similar the representations of the two models i and j are. As per the justifications above. two latent representations and 2, should be scored as highly similar if they axis align with each other up to permutation (the same ground truth factor C₃ may be encoded by different latent dimensions within the models, Zi,a and 2j.b where a b), sign inverse (the models may learn encode the values of the generative factor in the opposite order compared to each other, ži,a = - 2j.6), and subsetting (one model may learn a subset of the factors that the other model has learnt if the relevant disentangling heprraarameters encourage a different number of latents to be switched off in the two models). In order for the UDR to be invariant to the first scenario, we propose calculating full L L similarity matrix R.y between the individual dimensions of 2; E ER and 2, E (see Fig. 5 in Supplementary Material). In order to address the second point, we take the absolute value of the similarity matrix Reyl Finally, to address the third point, we divide the UDR score by the average number of informative latents discovered by the two models. Note that even though disentangling often happens when the VAEs enter the "polarised regime", where many of the latent dimensions switched off, the rankings produced by UDR are not affected by whether the model operates in such a regime or not.
To populate the similarity matrix Raj we calculate each matrix element as the similarity between two vectors Zí,a and zj.b, where ži,a is a response of a single latent dimension 2a model over the entire ordered dataset or fixed number of ordered mini-batches' if the former is computationally restrictive (see Sec. Supplementary Material for details). We considered two versions of the UDR score based the method used for calculating the vector similarity: the non- con-maraettri UDRs, using Spearman's 's correlation; and the parametric UDRL, using Lasso regression following past work on evaluating representations (Eastwood & Williams [0188] Li al. 2016). In practice the Lasso regression version worked slightly better, so the remainder of the paper is restricted to UDR, (we use UDR, and UDR interchangeably refer to this version), while UDRs is discussed the Supplementary Materials.
[
Given similarity matrix R.g. we want to find one-to-one correspondence between all the informative latent dimensions within the chosen pair of models. Hence, we want to see at most a single strong comelattin in each row and column of the similarity matrix. To that accord, we step through the matrix R= R.yle I. one column and row at a time, looking for the strongest correlation, and weighting by the proportional weight it has within its respective column row. We then average all such weighted scores over all the informative row and column latents to calculate the final UDR,, score:
Σ 1 IKL(b)
- re (kL(a)
(2)
dg + db
R(a.b)
E,
where r. max, b)) and 9ь max, R(a,b). IKL indicates "informative" latent within model and d is the number of such latents: d, -. I (a) and E, We define a latent dimension as "informative" if has learnt a latent posterior which diverges from the prior:
[K)((a= = f
(L(gs(zalz) |l ((=p(()))))))) 0.01
otherwise
UDR variations We explored whether doing all-to-all pairwise comparisons, with models in Step 2 sampled from the set of all M models rather than the subset of S models with the same hyperparameters, would produce more accurate results. Additionally we investigated the effect of choosing different numbers of models P for pairwise comparisons by sampling P~ U[5,45.))
5

---

Published as a conference paper at ICLR 2020
dSprites
DRAMEI
0.8
0.6
6.7
04-
Diuns
Farton
Ciums
1.0
TCMME
01.0
0.6
0.7
00
Far
GO thomn
Factor VAR
Divees
3D Shapes
1.0
n.
02
0.
03
USR
DC Dires
L000
Di stwves
Di Bamm
Hypar
at-
62
02
01
100
Do Dowm
Figure 2: Hyperparameter search results for six unsupervised disentangling model classes evaluated using the unsupervised UDR and the supervised 3. VAE, FactorVAE, MIG and DCI Disentangling metries and trained on either dSprites (top) or 3D Shapes (bottom) datasets. "Hyper" corresponds to the particular hyperparameter setting considered (see Tbl.5 5 Supplementary Materials for particular values). The box and whisker plots for each hyperparameter setting are summarising the scores for 50 different model seeds. Higher median values indivate better hyperparameters. The ranking of hyperparameters tends to be similar between the different metrics, including UDR.
UDR assumptions and limitations Note that approach is aimed at the current state of the art disentangling VAEs, for which the assumptions of our metric have been demonstrated to hold (Rolinek al. 2019). may be applied other model classes, however the following assumptions and limitations need to be considered:
1. Disentangled representations produced by two models from the same class trained the same dataset are likely to be more similar than entangled representations this holds for disentangling VAEs (Rolinek el., 20.1)) but may hold more broadly.
2. Continuous, monotonic and scalar factors- UDR assumes that these properties hold for the data generative factors and their representations. This true for the disentangling approaches described in Sec. but may not hold more generally. It is likely that UDR can be adapted to work with other kinds of generative factors (c.g. factors with special or no geometry) by exchanging the similarity calculations in Step with appropriate measure, however leave this for future work.

---

Published as a conference paper at ICLR 2020
MODEL CLASS
DSPRITES
3DSHAPES
LASSO HYPER
SPEARMAN HYPER ALL-2-ALL
LASSO HYPER
SPEARMAN HYPER
2-L-2-ALL
ALL-2-ALL
ALL-2-ALL
B-VAE 0.60
0.76
0.54
0.72
0.71
0.68
0.70
0.71
VCE 0,40
0.67
0.37
0.60
0.81
0.79
0.81
0.75
DIP-VAE 0.61
0.69
0.65
0.72
0.75
0.74
0.75
0.78
Table 1: Rank correlations between MIG and different versions of UDR across two datasets and three model classes. The performance is comparable across datasets, UDR versions and model classes. See Fig. Gl Supplementary Materials for comparisons with other supervised metrics.
3. Herd efffec since UDR detects disentangled representations through pairwise comparisons. the score it assigns to each individual model will depend on the nature of the other models involved in these comparisons. This means that UDR is unable to detect a single disentangled model within a hyperparameter sweep. It also means that when models are only compared within a single hyper- parameter setting, individual model scores may be over/under estimated as they tend to be drawn towards the mean of the scores of the other models within a hyperparameter group. Thus, it is prefer- able to perform the UDR A2A during model selection and UDR during hyperparameter selection.
4. Explicitness bias- UDR does not penalise models that learn a subset of the data generative factors. In fact, such models often score higher than those that learn the full set of generative factors, because the current state of the art disentangling approaches tend to trade-off the number of discovered factors for cleaner disentangling. As discussed in Sec. [ provide the practitioner with the ability to choose the most disentangl led model per number of factors discovered by approximating this with the score Eq.
5. Computational cost UDR requires training a number of seeds per hyperparameter setting and M x P pairwise comparisons per hyperparameter search, which may be computationally expensive. Saying this, training multiple seeds per hyperparameter setting a good research practice to produce more robust results and UDR computations are highly parallelisable
To summarise, UDR relies on number of assumptions and has certain limitations that we hope to relax in future work. However, it offers improvements over the existing supervised metrics. Apart from being the only method that does not rely on supervised attribute labels, its scores are often more representative of the true disentanglement quality (e.g. see Fig. 3nd Fig. ii Supplementary Materials), and it does not assume a single "canonical" disentangled factorisation per dataset. Hence, we believe that UDR can be a useful method for unlocking the power of unsupervised disentangled representation learning to real-life practical applications, at least in the near future.
EXPERIMENTS
Our hope was to develop method for unsupervised disentangled model selection with the following properties: it should 1) help with hyperparameter tuning by producing an aggregate score that can be used to guide evolutionary or Bayesian methods (Jaderberg et al., 2018 Snoek et al. 2012) Thornton et al. 2012 Bergstra et al.) 2011 Hutter et al.) Mikkulaneen et al. 2017]: 2) rank individual trained models based on their disentanglement quality: 3) correlate well with final task performance. In this section we evaluate our proposed UDR against these qualities. For the reported experiments we use the trained model checkpoints and supervised scores from Locatello et al. (2018) to evaluate 3-VAE, CCI-VAE, FactorVAE, TC-VAE, VAE- and dSprites (Matthey et al. 2017) and 3D Shapes (Burgess & Kim 2018) (see Sec./A.3 for details). Each model is trained with H = 6 different hyperparameter settings (detailed in A.4. Supplementary Material), with S = 50 seeds per setting, and P 50 pairwise comparisons.
VAE-II on two benchmark datasets:
UDR correlates well with the supervised metrics. To validate UDR, we calculate Spearman's correlation between its model ranking and that produced by four existing supervised disentanglement metrics found to be the most meaningful in the large scale comparison study by Locatello et al.
(2018); the original 3-VAE metric (Higgins et al. 2017a), Factor metric (Kim & Mnih | (2188). Mutual Information Gap (MIG) (Chen et al. (018)) and DCI Disentanglement (Eastwood & Williams
2018) (see A. in Supplementary Material for metric details). The average correlation for UDR
7

---

Published as a conference paper at ICLR 2020
DIP-VAE
TC-VAE
B-VAE
CCI-VAE
0.212/0.622
= 10
0.444/0,774
0.607 / 0.751
d=
0.895 / 0.616
d=3
-3
210
2
2,0
200
Figure 3: Latent traversals of the top ranked trained VIE-,, VAE, CCI-VAE and 3-VAE according to the UDR method. At the top of each plot the two presented scores UDR/FactorVAL metric. Note that the VactorVAE metric scores visually entangled models very highly. d is the number of informaaive latents. The uninformative latents are greyed out.
is 0.54 + 0.06 and for UDR-A2A is 0.60 = 0.11. This is comparable to the average Spearman's 's correlation between the model rankings produced by the different supervised metries: 0.67 + 0.2. The variance in rankings produced by the different metrics is explained by the fact that the metrics capture different aspects of disentangling (see Sec. AA.2 in Supplementary Materials for discussion of how UDR relates to other representation comparison methods). Tbl. [ provides a breakdown of correlation scores between MIG and the different versions of UDR for different model classes and datasets. It is clear that the different versions of UDR perform similarly to each other, and this holds across datasets and model classes. Note that unlike the supervised metries, UDR does not assume a "canonical" disentangled representation. Instead, it allows any one of the many equivalent possible ground truth generative processes to become the "canonical" one for each particular dataset and model class, as per the theoretical results by Rolinek et al. (219)) summarised in Sec. 3.1
UDR is useful for hyperparameter selection. Fig. compares the scores produced by UDR and the four supervised metrics for 3600 trained models, split over six model classes, two datasets and six hyperparameter settrang.mteestttinggs.. We consider the median score profiles across the six hyperparameter settings to evaluate whether particular setting is better than others. It can be seen that UDR broadly agrees with the supervised metrics on which hyperparameters are more promising for disentangling. This holds across datasets and model classes. Hence, UDR may be useful for evaluating model fitness for disentangled representation learning as part of an evolutionary algorithm or Bayesian hyperparameter tuning.
UDR is useful for model selection. Fig. 2 can also be used to examine whether a particular trained model has learnt good disentangled representation. We see that some models reach high UDR scores. For example, more models score highly as the value of the 33 hyperparameter is increased in the 3-VAE model class. This in line with the previously reported results (Higgins et al. (2017a). Note that the Oth hyperparameter setting in this case corresponds to B == 1, which is equivalent to the standard VAF objec- tive (Kingma& & Welling, 2014; Rezende et al. 2014). As expected, these models score low in terms of disentangling. We also see that for some model classes (c.g.
DIP- VAB-II and Factor on dSprites) no trained model scores highly according to UDR. This suggests that none of the hyperparam- eter choices explored were good for this particular dataset, and that no instance of the model class learnt to disentangle well. To test this, we use latent traversals to qualitatively evaluate the level of disentangle- ment achieved by the models, ranked by their UDR scores. This a common technique to qualitatively evaluate the level of disentanglement on simple visual datasets where no ground truth attribute labels are available. Such traversals involve changing the value of one latent dimension at time and evaluating its effect on the resulting reconstructions to understand whether the latent has learnt to represent anything semantically meaningful. Fig. demonstratess that the qualitative disentanglement quality reflected well in the UDR scores. The figure also highlights that the supervised metric scores can sometimes be overoptimistic. For example, compare TC- VAE and VAE traversals in Fig. 3 These are scored simi- larly by the supervised metric (0.774 and 0.751) but differently by UDR (0.444 and 0.607). Qualitative evaluation of the traversals clearly shows that 3- VAE learnt a more disentangled representation than TC- VAE, which is captured by UDR but not by the supervised metric. Fig. Fig.9in Supplementary Material pro- vides more examples. We also evaluated how well UDR ranks models trained on more complex datasets, CelebA and ImageNet, and found that it performs well (see Sec. A. Supplementary Materials).
8

---

Published as a conference paper at ICLR 2020
SAMPLE (P)
10
15
20
25
30
35
CORRELATION 0.51 1.007 0.57 0.003 0.57 10.05 0.6::0.03 0.59:10.03
0.61::0002
0.61
10.02
0.61100.01
0.61
Table 2: Rank correlations of the UDR score with the 3-VAE metric on the dSprites dataset for a 3- 3-VAE hyperparameter search as the number of pairwise comparisons P per model were changed.
UDR works well even with five pairwise comparisons. We test the effect of the number of pairwise comparisons P on the variance and accuracy of the UDR scores. Tbl. rreports the changes the rank correlation with the 3-VAE metrie on the dSprites dataset P varied between 5 and 45. We see that the correlation between the UDR and the B-VAE metric becomes higher and the variance decreases as the number of seeds increased. However, even with P P= the correlation is reasonable.
UDR generalises to a dataset with no attribute labels. We investigate whether UDR can be useful for selecting well disentangled models trained on the 3D Cars (Reed et al. 2014) dataset with poorly labelled attributes, which makes it bad fit for supervised disentanglemen metrics. Fig. shows that highly ranked model according to UDR appears disentangled, while poorly ranked one appears entangled. Fig. T0 Supplementary Material provides more examples of high and low scoring models according to the UDR method.
UDR predicts final task performance. We developed UDR to help practitioners use disentangled representations to better solve subsequent tasks. Hence, we evaluate whether the model ranking pro- duced by UDR correlates with task performance tion task introduced by Locatello et al. (2018),))) 8), and data efficiency on a clustering task for a model- based reinforcement learning agent introduced by Watters et al. (2019) (see Sec. A.88 in Supplementary Mate- rials for more details). We found that UDR had an average of 0.8 Spearman correlation with the fairness scores, which is higher than the average of 0.72 correlation between fairness and supervised scores reported by Locatello et al. (0188). We also found that UDR scores had 0.56 Spearman correlation with data effi efficiency of the COBRA agent. The difference between the best and the worst models according to UDR amounted to around 66% reduction in the number of steps to 90% success rate on the task.
two different domains: the fairness on a classifica-
6 CONCLUSION
We have introduced UDR, the first method for unsupervised model selection for variational disentangled representation learning. We have validated our approach on 5400 models covering all six state of the art VAE-based unsupervised disentangled representation learning model classes. We compared UDR to four existing supervised disentanglement metrics both quantitatively and qualitatively, and demonstrated that our approach works reliably well across three different datasets, often ranking models more accurately than the supervised alternatives. Moreover, UDR avoids one of the big shortcomings of the supervised disentangling metrics- the arbitrary choice of "canonical" disentangled factorisation, instead allowing any of the equally valid disentangled generative processes be accepted. Finally, we also demonstrated that UDR is useful for predicting final task performance using two different domains. Hence, we hope that UDR can be step towards unlocking the power of unsupervised disentangled representation learning to real-life applications.
ACKNOWLEDGEMENTS
We thank Olivier Bachem and Francesco Locatello for helping us re-use their code and model checkpoints, and Neil Rabinowitz, Avraham Ruderman and Tatjana Chavdarova for useful feedback.
REFERENCES
Alessandro Achille, Tom Eccles, Loic Mathhey Christopher P Burgess, Nick Watters, Alexander Lerchner, and Irina Higgins. Life-long disentangled representation learning with cross-domain latent homologies. NIPS, 2018.
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence, 3588):1798-1822 2013.

---

Published as a conference paper at ICLR 2020
James Bergstra, Remi Bardenet, Yoshua Bengio, and Balazs Kegl. Algorithms for hyper-parameter paraaeetoooiiiiiiittonn NIPS, 2011.
Chris Burgess and Hyunjik Kim. 3d shapes dataset. https://gtttub..om///// http://hww....eepeaiid/aaaaaaaaa
2018.
Christopher P. Burgess, Irina Higgins, Arka Pal, Loic Matthey, Nick Watters, Guillaume Desjardins, and Alexander Lerehner. Understanding disentangling 3- B-V.E NIPS Workshop of Learning Disentangled Features, 2017.
Christopher P Burgess, Loic Matthey, Nick Watters, Rishabh Kabra, Irina Higgins, Matt Botvinick, and Alexander Lerchner. MONet: Unsupervised scene decomposition and representation. arXiv preprint, January 2019.
Tian Qi Chen, Xuechen Li, Roger Grosse, and David Duvenaud. Isolating sources of disentanglement in variational autoencoders. NIPS, 2018.
Taco Cohen and Max Welling. Group equivariant convolutional networks. JCML, 2016.
Pierre Comon, Independent component analysis, a new concept? Signal Processing, 36:287-314, 1994.
Cian Eastwood and Christopher K. L. Williams. A framework for the quantitative evaluation of disentangled representations. ICLR, 2018.
Lasse Espeholt, Hubert Soyer. Remi Munos, Karen Simonyan, Volodymyr Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, lain Dunning, Shane Legg, Koray Kavukcuoglu. IMPALA: Scalable distributed deep-rl with importance weighted actor-learner architectures. arxiv, 2018.
Marta Garnelo, Kai Arulkumaran, and Murray Shanahan. Towards deep symbolic reinforcement learning. arXiv preprint arXiv: 1609.6600.5... 18, 2016.
Robert Gens and Pedro M. Domingos. Deep symmetry networks. NIPS, 2014,
David R. Hardoon, Sandor Szedmak, and Taylor Canonical correlation analysis; an overview with application to learning methods. Neural Computation, 16( 12):2639 2664, 2004.
Matteo Hessel, Joseph Modayil, Hado van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver. Rainhow: Combining improvements in deep reinforcement learning. arxiv, 2017.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. 3-VAE: Learning basic visual concepts with constrained variational framework. ICLR, 2017a.
Irina Higgins. Arka Pal, Andrei Rusu, Loic Matthey, Christopher Burgess, Alexander Pritzel, Matthew Botvinick. Charles Blundell, and Alexander Lerehner. DARLA: Improving zero-shot transfer in reinforcement learning. ICML, 2017b.
Trina Higgins, David Amos, David Pfau, Schastien Racaniere, Loic Matthey, Danilo Rezende, and Alexander Lerchner. Towards a definition of disentangled representations. arXiv, 2018a. 8a.
Irina Higgins, Nicolas Sonnerat, Loic Matthey, Arka Pal, Christopher Burgess, Matko Bosnjak, Murray Shanahan, Matthew Botvinick, Demis Hassabis, and Alexander Lerchner. SCAN: Learning hierarchical compositional visual concepts. ICLR, 2018b.
Jie Hu, Li Shen, and Gang Sun. Squeeze-and-excitatie networks. CVPR, 2018.
Frank Hutter, Holger Hoos, and Kevin Leyton-Brown. Brown. Sequential model-based optimization for general algorithm configuration. Learning and Intelligent Optimization, 2011.
Max Jaderberg, Valentin Dalibard, Simon Osindero, Wojciech M. Czamecki, Jeff Donahue, Ali Razavi, Oriol Vinyals, Tim Green, lain Dunning, Karen Simonyan, Chrisantha Fernando, and Koray Kavukcuoglu. Population based training of neural networks. arXiv, 2018.
Hyunjik Kim and Andriy Mnih. Disentangling by factorising. ICLR, 2018.
Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. 2014,
2014.
Nikolaus Kricgeskorte, Maricke Mur, and Peter Bandettini. Representational similarity analysis connecting the branches of systems neuroscience. Front Syst Neurosci., 4(2), 2008
Abhishek Kumar, Prasanna Sattigeri, and Avinash Balakrishnan. Variational inference of disentangled latent concepts from unlabeled observations. arxiv, 2017.
10

---

Published as a conference paper at ICLR 2020
Brenden M. Lake, Tomer D. Ullman, Joshua B. Tenenbaum, and Samuel J. Gershman. Building machines that learn and think like people. Behavioral and Brain Sciences, pp. 1-101, 2016.
Guillaume Lample, Myle Out, Alexis Conneau, Ludovic Denoyer, and Marc' Aurelio Ranzato. Phrase-based neural unsupervised machine translation. arxiv, 2018.
Adrien Laversanne- Finot, Alexandre Péré, and Pierre-Yves Oudeyer. Curiosity driven exploration of learned disentangled goal spaces, arxiv, 2018.
Yixuan Li, Jason Yosinski. Jeff Clune. Hod Lipson, and John Hoperoft. Convergent learning: different neural networks learn the same representations? ICLR, 2016.
Francesco Locatello, Stefan Bauer, Mario Lucic, Sylvain Gelly, Bernhard Schölkopf. and Olivier Bachem. Challenging common assumptions in the unsupervised learning of disentangled representations. ICML, 97: 4114-4124, 4124,2018 2018.
Francesco Locatello, Gabriele Abbati, Tom Rainforth, Stefan Bauer, Bernhard Schölkopf, and Olivier Bachem. On the fairesss of disentangled representations. arxiv, 2019.
Gary Marcus. Deep learning: A critical appraisal. arxiv, 2018.
Emile Mathieu, Tom Rainforth, N. Siddharth, and Yee Whye Teh. Disentangling disentanglement in variational autoencoders. 2019.
Loic Matthey, Trina Higgins, Demis Hassabis, and Alexander Lerchner. dsprites: Disentanglement testing sprites dataset, 2017. URL wttpsttp///// http:///ggtttttooccoccc///m/////
com
-eteitteesdddttttee datt-ddttsseee set//
Risto Miikkulainen, Jason Liang, Elliot Meyerson, Aditya Rawal, Dan Fink, Olivier Francon, Bala Raju, Hormoz Shahrzad, Arshak Navruzyan, Nigel Duffy, and Babak Hodjat. Evolving Deep Neural Networks. arxiv, 2017.
Ari S. Morcos, Maithra Raghu, and Samy Bengio. Insights on representational similarity in neural networks with canonical correlation. NIPS, 2018.
Ashvin Nair, Vitchyr Pong, Murtaza Dalal, Shikhar Bahl, Steven Lin, and Sergey Levine. Visual reinforcement learning with imagined goals. arxiv, 2018
Aaron van den Oord, Sander Dicleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. Wavenet: generative model for raw audio. arXiv 2016.
preprint arXiv:
Maithra Raghu, Justin Gilmer, Jason Yosinski, and Jascha Dicksttinn Sveca: Singular vector canonical correlation analysis for deep learning dynamics and interpretability. NIPS, 2017.
Scott Reed, Kihyuk Sohn, Yuting Zhang, and Honglak Lee, Learning to disentangle factors of variation with manifold interaction. /CML, 2014.
Danilo Rezende and Fabio Viola. Generalized elbo with constrained optimizationn, geco. Workshop on Bayesian Deep Learning, NeurlPS, 2018.
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. ICML, 32(2):1278 1286, 2014.
Karl Ridgeway and Michael C Mozer. Learning deep disentangled embeddings with the f-statistic loss. 2018.
Michal Rolinek, Dominik Zietlow, and Georg Martius. Variational autoencoders pursue pea directions (by accident). CVPR, 2019.
Jürgen Schmidhuber. Learning factorial codes by predictability minimization. Neural Computation, 4(6): 863-869, 1992.
David Silver. Thomas Hubert, Julian Schrittwieser, loannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre. Dharshan Kumaran, Thore Graepel, Timothy Lillicrap, Karen Simonyan, and Demis Hassabis. A general reinforcement learning algorithm that masters chess, shogi, and go through self self-play. Science, 362 (64 (6419))1110-1) 144, 2018. doi: 10.1 26/sciennce.aa66404.
J. Snock, H. Larochelle, and R. P. Adams. Practical Bayesian Optimization of Machine Learning Algorithms. arXiv, 2012.
Stefano Soatto. Steps toward theory of visual information. Technical Report UCLA-CSDIO00RR 2010.
11

---

Published as a conference paper at ICLR 2020
Xander Steenbrugge, Sam Leroux, Tim Verbelen, and Bart Dhoedt. Improving generalization for abstract reasoning tasks using disentangled feature representations. arxiv, 2018.
Raphael Suter, Dorde Miladinovic, Stefan Bauer, and Bernhard Scholkopf. Interventional robustness of deep latent variable models. arxiv, 2018.
C. Thornton, F. Hutter, H. H. Hoos, and K. Leyton-Brown, Auto-WEKAl Combined Selection and Hyperparameter Optimization of Classification Algorithms. arXiv, 2012.
Sjoerd Steenkiste, Francesco Locatello, Jürgen Schmidhuber, and Olivier Bachem. Are disentangled representations helpful for abstract visual reasoning? arxiv, 2019.
Liwei Wang. Lunjia Hu, Jiayuan Gu, Yue Wu, Zhiqiang Hu, Kun He, and John Hoperoft. Towards understanding learning representations: To what extent different neural networks learn the same representation. NeurIPS, 2018.
Nicholas Watters, Loic Matthey, Matko Bosnjak, Christopher Burgess, Alexander Lerchner. Cobra: Data- efficient model-basee rl through unsupervised object discovery and curiosity driven exploration. arxiv, 2019.
12

---

Published as a conference paper at ICLR 2020
A SUPPLEMENTARY MATERIAL
A.1 USEFUL. PROPERTIES OF DISENTANGLEED REPRESENTATIONS
Disentangled representations are particularly useful because they re-represent the information contained in the data in way that enables semantically meaningful compusitionaltyy For example, having discovered that the data generated using two factors, colour and shape, such a model would be able to support meaningful reasoning about fictitious objects, like pink elephants, despite having never seen one during training (iiggin et al. [01 7b 2018b). This opens up opportunities for counterfactual reasoning, more robust and interpretable inference and model podel--base based planning (Higgins et al. [2018a] SSter ct al. 2018). Furthermore, such representation would support more data efficient learning for subsequent tasks, like classification objective for diffcrentiating clephants from cats. This could be achieved by ignoring the nuisance variables irrelevant for the task, e-g. the colour variations, by simply masking out the disentangled subspaces that learnt to represent such nuisances, while only paying attention to the task-relevant subspaces, e.g. the units that learnt to represent shape (Cohen & Welling [016 Gens & Domingos, 2014 /Saatoo 2010). Hence, the semantically meaningful compositional nature of disentangled representations perhaps the most sought after aspect of disentangling, due to its strong implications for generalisation, data efficiency and interpretability (Schmidhuber) 1992 Bengio et al. 2013 Higgins et al. 2
A.2 ASPECTS OF DISENTANGLEMENT MEASURED BY DIFFERENT METRICS
Methods for evaluating and comparing representations have been proposed the past. The most similar approaches to ours are the DCI Disentanglement score from Eastwood & Williams (0118) and the axis alignment comparison of representations in trained classifiers proposed in Li et al. (2016). The former is not directly applicable for unsupervised disentangled model selection, since requires access the ground truth attribute labels. Even when adapted compare two latent representations, our preliminary experiments suggested that the entropy based aggregation proposed in Fastwood & Williams (2018) is inferior to our aggregation Eq. 2 when used in the UDR sctup. The approach by et al. (2016) shares the similarity matrix cal calculation step with us, however they never go beyond that quantitatively. opting for qualitative evaluations of model representations instead. Hence, their approach not directly applicable to quantitative unsupervised disentangled model ranking.
Other related approaches worth mentioning are the Canonical Correlation Analysis (CCA) and its modifications (Hardoon ot al. 2004] Raghu et al.] 2017 Morcos et al. 20118). These approaches, however, tend to be invariant to invertible afline Tramattinn and therefore to the axis alignment of individual neurons, which makes them unsuitable for evaluating disentangling quality. Finally, Representation Similarity Matrix (RSM) (Kriegeskorte al. 2008) commonly used method in Neuroscience for comparing the representations of different systems to the same set of stimuli. This technique, however, is not applicable for measuring disentangling, because it ignores dimension -iies wise response properties.
When talking about disentangled representations, three properties are generally considered: modularity. compactesss and explicitees Pl (Ridgeway & Mozer 2018). Modularity measures whether each latent dimension encodes only one data generative factor, compactness measures whether each data generative factor is encoded by a single latent dimension, and explicimess measures whether all the information about the data generative factors can be decoded from the latent representation. We believe that modularity is the key aspect of disentangling. since it measures whether the representation is compositional, which gives disentangled representations the majority of their beneficial properties (see Sec. in Supplementary Materials for more details). Compactness, on the other hand, may not always be desirable. For example, according to a recent definition of disentangled representations (Higgins et al. 2018a), is thcoretically impossible to represent 3D rotation a single dimension (see also Ridgeway & Mozer (018)). Finally, while explicimess is clearly desirable for preserving information about the data that may be useful for subsequent tasks, in practice models often fail to discover and represent both the observed data distribution and the model
the full set of the data generative factors due to restrictions capacity (Mathieu et al. 2019). Hence, we suggest noting the explicitness of representation, but not necessarily punishing its disentang iement ranking if it is not fully explicit. Instead, we suggest that the practitioncr should have the choice to select the most disentangled model given particular number of discovered generative factors. will use the term "disentanglement" to refer to the compositional property
Hence, in the rest of the paper of representation related to the modularity measure. providds a summary of how the different metrics considered in the paper compare in terms of modularity, compactness and explicitness.
A.3 DATASET DETAILS
dSprites A commonly used unit test for evaluating disentangling is the dSprites datasct (Matthey ct al. 20017] This dataset consists of images of a single binary sprite pasted on a blank background and can be fully described by five generative factors: shape (3 values), position (32 values), position (32 values), size (6 values) and
Similar properties have also been referred to as disentanglement, completeness and informativeness respec- tively in the independent yet concurrent paper by Fastwood & (
13

---

Published as a conference paper at ICLR 2020
Table 3: Disentangled model selection metrics comparison. M --oduarrit C compactness, E
explicitness (Ridgeway & Mozer 2018)
METRIC
E
BAE
V
FACTORVAE
V
RVN
MIG
DCI DISENTANGLEEMET
UDR
rotation (40 values). All the generative factors are sampled from a uniform distribution. Rotation is sampled from the full 360 degree range. The generative process for this dataset fully deterministic, resulting in 737,280 total images produced from the Cartesian product of the generative factors.
Shapes dataset BBurgess & Kim 2018). This dataset consists of images of a single 3D object a room and is fully specified by six generative factors: floor colour (10 values), wall colour (10 values), object colour (10 values), size (8 values), shape (4 values) and rotation (15 values). All the generative factors are sampled from uniform distribution. Colours sampled from the circular hue space. Rotation sampled from the (-30, 30] degree angle range.
3D Shapes more complex dataset evaluating disentangling is
This dataset was adapted from Reed el. (014)) The full data generative process this dataset unknown. The labelled factors include 199 car models and rotations sampled from the full 360 degree of plane rotation range. An example of an unlabelled generative factor is the colour of the car this varies across the dataset.
3D Cars
A.4 UNSUPERVISED DISENTANGLED REEOLEEDDEEEEEEEEEEE REPRESENTATION LEARNING MODELS
As mentioned in Sec. current state of the art approaches unsupervised disentangled representation learning based the VAE (Kingma Welling 2014 Rezende et al. (014) objective presented in Eq. [ These approaches decompose the objective in Eq. into various terms change their relative weighting exploit the trade-off between the capacity of the latent information bottleneck with independent sources of noise, and the quality of the resulting reconstruction in order to learn disentangled representation. The first such modification introduced by Higgins al. 2017 in their & VAE framework:
Ep(x)
B (=|=) [log =)] 8 K L(go(2/ar) p(=))]
(4)
In order to achieve disentangling in 3AVAE, the KL term in Eq, is typically up- weighted by setting 3 > 1. This implicitly reduces the latent bottleneck capacity and, through the interaction with the reconstruction term, encoded by different independent
encourages the generative factors eı with different reconstruction profiles noisy channels z; latent bottleneck. Building on the 3-VAE ideas, CCI- VAE (Burgess et al.) (007)) suggested slowly increasing the bottleneck capacity during training, thus improving the final disentanglement and reconstruction quality:
Ex( [x (m)) [log p6 [2) x)] = -ry / (Lss(())))) -())()
(5)
Later approaches (Kim & Mnih 2018 Chen et 2018 Kumar et
2017 showed that the KL term Eq,
can be further decomposed according to: Ex(w) (Ls
==(=)) KL [l
Henec, penalising the full KL term as in Eqs. 4 5 not optimal, since it unnecessarily penalises the mutual infor- mation between the latents and the data. To remove this undesirable side effect, different authors suggested instead adding more targeted penalised terms to VAE objective function. These include different implementations of the total correlation penalty (Factor VAE by Kim & Mnih (2018) and TC-VAE by Chen et al. (2018)):
Lv - (z)
II (9(2)))
and different implementations of the penalty that pushes the marginal posterior towards factorised prior (DIP-VAE by/ Kumar et al. (2017))
LV (L(go>)) ((())
(8)
A.4.1 MODEL IMPLEMENTATION DETAILS
We re-used the trained checkpoints from f Locatello et al. (2018) hence recommend the readers to check the original paper for model implementation details. Hriefly, the following architecture and optimiser were used.
14

---

Published as a conference paper at ICLR 2020
Table 4: Encoder and Decoder Implementation details shared for all models
Encoder
Decoder
Input: 64 x 64 number of channels
Input: R 10
4 x conv, 32 ReLU, stride 2 ReLUU stride
FC, 256 ReLU
x conv,
FC, x x 64 ReLU
X conv, 64 ReLUU, stride 2
FC, 4 x 4 upconv, 64 ReL.U, stride 2
4 X 4 conv, 64 ReLU, stride 2
FC, 4 x upconv. 32 ReLU, stride 2
FC 256, F22 x 10
x upconv. 32 ReLU, stride
x upconv, number of channels, stride
Table 5: Hyperparameters used for each model architecture
Model
Parameters
Values
3-VAE
2,4, 6, 8, 16] [5, 10, 50, 75, 100] 100000
CCI-VAE
Cmax iteration threshold
Y
1000
FactorVAE
Y
[10,20, 30, 40, 50, 100] [1, 10, 20, 50] 10A.d
DIP-VAE-1
Add
Xd
DIP-VAF-II
Aod
[1,2,5, 20] Aod
Xd
TC-VAE
3
[1, 2,4, 6, 10]
(a) Common hyperparameters across all
(b) FactorVAE discrimina- tor architecture
(c) FactorVAE discriminator pa-
models
rameters
Parameter
Values
Discriminator
Parameter
Values
Batch Size
64
FC, 1000 leaky ReLU
Batch size
64
Latent space dimension
10
FC, 1000 leaky ReLU
Optimizer
Adam
Optimizer
Adam
FC, 1000 leaky ReI.U
Adam: betal
0.5
Adam: betal
0.9
FC, 1000 leaky ReLU
Adam: beta2
0.9
Adam: beta2
0.999
FC, 1000 leaky ReLU
Adam: epsilon
1e-8
Adam: epsilon
le-8
FC, 1000 leaky ReLU
Adam: learning rate
0.0001
Adam: learning rate
0.0001
FC, 2
Decoder type
Bernoulli
Table 6: Miscellaneous model details
For consistency, all the models were trained using the architecture, optimiser, and hyperparameters. All methods use deep neural network to encode and decode the latent embedding and parameters of the latent factors are predicted using a Gaussian encoder whose architecture is specified in Table All of the models predict latent vector with 10 factors. Each model was also trained with 6 different levels of regularisation strength specified in Table The ranges of the hyperparameters used for the various levels of regularisation were specified show diversity of different performance on different datasets without relying pre-existing intuition on good hyperparameters, however ranges were hased on hyperparameters that were used previously in literature. For each of the model classes outlined above, we tried 6 hyperparameter values with 50 seeds each.
3-VAE The 8-VAE Higgins et al. 2017a model is similar to the vanilla VAE model but with additional hyperparameter 8 modify the strength of the KT. regulariser.
15

---

Published as a conference paper at ICLR 2020
Ep(sn] (a) [[] K (=(g)
p(x))]
where a 3 value of corresponds to the vanilla VAE model. Increasing 3 enforees stronger prior on the latent distribution and encourages the representation to be independent.
CCI-VAE
The CCI- VAE model (Burgess al. 2017 is variant of the B- V-VAE where the KL divergence
encouraged match a controlled value C' which inereased gradually throughout training. This yields us the objective function for VAE.
Bp(=) (x [log pe (x [LL(g(<)))) (l
(10)
2018 specifically penalises the dependencies hetween the latent dimensions such that the "Total Correlation term targeted yielding modified version of the V.F.obeccii
FactorVAE FactorVAF (Kim & Mnih
Ep(z) Bag(e (log KL(go [(p())))
- (=2)))
(11)
The "Total Correlation" term is intractable in this case so for Factor samples are used from both g( and ((z) as well as the density-ratio trick to compute an estimate of the "Total Correlation" term. FactorVAF uses additional discriminator network approximate the density ratio the KI. divergence. The implementation details for the discriminator network and its hyperparameters be found in Table ((b]l 5(c)
TC- VAE
The TC-VAE model (Chen al. 2018) which independently from FactorVAF has similar ohjective regulariser which contains a "Total Correlation" term. the case of TC- the "Total Correlation" term is estimated using a biased Monte-Cate estimate.
(2) Select P models for pairwise comparisons with each of M models
(3)
(4) Aggregate UDR, scores for each of M models
Create MxP similarity matrices, for each calculate UDR, score
Train M-HxS models
Modal on
Hyper
0.43
UDR 0.47
Seedc
seed
seed
UDR,
Hyper 2
0.87
spedd
soar S
Hyper H
0.32
UDR,
Figure 4: Schematic illustration of the UDR method. See details in text.
DIP-VAE The VAP model also adds regularisation the aggregated posterior but instead additional loss term is added encourage it to match factorised prior. Since the divergence intractable, other measures of divergence are used instead. Coup(a) (x)) can be used, yielding the DIP- VAE-I objective
Ep(x) Egg(s w) [log po (x)l2) KL(go(z(a)) lI
-od
(12)
Ad (Coup (x)]
16

---

Published as a conference paper at ICLR 2020
Cougy [z] is used instead yielding the DIP-VAE-II
Py(t) Pag(= (o K (T [1(()))
-od
[oou [=]]
(13)
Aa (Coovas> [=]]
A.5 UDR IMPLEMENTATION DETAILS
Similarity matrix
To compute the similarity matrix Ray we follow the approach of et al. (2016) and Morcos et al. (2018). For given dataset X = {0)1 ",,,,,, ZN and a neuron a E (1,..., I.j of model i (denoted as 2t. we define Zi, to be the vector of mean inferred posteriors ( (z)) across the full dataset:
Xi.o = ( (211) (,,,),,(((xx) E IR. Note that this different from the often considered notion of a "latent representation vector". Here zi.a is a response of single latent dimension over the entire dataset, not an entire latent response for single input. We then calculate the similarity hetween each two of such vectors 21,a and Zj.h using either Lasso regression Spearman's 's correlation.
Lasso regression (UDRL) We trained L lasso regressors to predict each of the latent responses 2,,a from Each row in Ray (a) then filled in using the weights of the trained Lasso regressor for Zi.a. The lasso regressors were trained using the default Scikit-lcarn multi-task lasso objective min 2nzompies
2, using the dataset latent encodings Zi (",,))) (...(,,.N (Z,.N N)), 2i,
[XW - Ylliro + A||W where Fro is the Frohenius
norm: ||Al|Fro = at and the 4اب loss is computed ||A||21 = E. = a À chosen using cross validation and the lasso is trained until convergence until either 1000 itcrations have been rum or our updates are below a tolerance of 0.0001. Lasso regressors were trained on a dataset of 10000 latents from each model and training was performed using coordinate descent over the entire dataset. Ram is then computed by extracting the weights in the trained lasso regressor and computing their absolute value (Eastwood & Williams 2018). important that the representations are normalised per-laten such that the relative importances computed per latent are scaled to reflect their contribution to the output. Normalising our latents also ensures that the weights that are computed roughly lie in the interval [-1,]] 1,1].
Spearman's 's based similarity matrix (UDRs) calculate each entry in the similarity matrix according to R:, (a.b) = Corr(zi.a..3.) where Corr stands for Spearman's 's correlation. We use Spearman's correlation to measure the similarity between 2i,a and 2,,6, because do not want necessarily assume linear relationship between the two latent encodings, since the geometry of the representational space not crucial for measuring whether a representation is disentangled (see Sec.2). [). but we do hope to find monotonic dependence between them. Spearman correlation coefficients were computed by extracting 1000 samples from each model and computing the Spearman correlation over all the samples per-latent basis.
All-to-all calculations To make all-to-all comparisons, we picked 10 random seeds per hyperparameter setting and limited all the calculations those models. Hence we made the maximum of (60 choose 2) pairwise model comparisons when calculating UDR-A2A.
Informative latent thresholding Uninformative latents typically have KL 0.01 while informative latents have KL 0.01, so KL= 0.01 threshold in Eq. Eq.Bis somewhat arbitrarily chosen to pick out the informative latents z.
Sample reduction experiments We randomly sampled without replacement 20 different sets models for pairwise comparison from the original set of 50 models with the same hyperparameter setting for UDR or 60 models with different seeds and hyperparameters for UDR-A2A.
A.6 SUPERVISED METRIC IPPLEMENTATON DETAILS
Original 8- V-E metric. First proposed in Higgins et al. (2017a), this metric suggests sampling two batches of observations where in both batches the same single data generative factor is fixed to particular value, while the other factors are sampled randomly from the underlying distribution. These the corresponding latent representations qo (2)
batches are encoded into (z|æ) ax) and the pairwise di differences between the corresponding mean latent values from the two batches are taken. Disentanglement is measured as the ability of a linear classifier to predict the index of the data generative factor that was fixed when generating
We compute the 3-VAE score by first randomly picking single factor of variation and fixing the value of that factor to randomly sampled value. We then generate hatches of 64 where all the other factors are sampled
17

---

Published as a conference paper at ICLR 2020
randomly and take the mean of the differences between the latent mean responses in the two batches to generate training point. This process is repeated 10000 times generate a training by using fixed factor variation as the label. We then train logistic regression on the data using Scikit-learn and report the evaluation accuracy on test set of 5000 as the disentanglement scorc.
FactorVAB metric. Kim & Mnih (2018) proposed modification the 3-VAF metric which made the classifier non-parametric (majority vote based the index of the latent dimension with the least variance after the pairwise difference step). This made the Factor
metric more robust, since the classifier optimised. Furthermore. the Factor VAE metric is more accurate than the 3- VAE one, since the B- VAE metric often over-estimates the level of disentanglement by reporting 100% disentanglement even when only K - factors were disentangled.
not need to be
The Factor VAE score is computed similarly the 3- 3-VAE metric but with few modifications. First draw a set 10000 random samples from the dataset we estimate the variance the mean latent responses in the model. Latents with a variance of less than 0.05 are discarded. Then batches of 64 samples are generated by random set of generative factors with single fixed generative factor. The variances of all the latent responses over the 64 samples computed and divided the latent variance computed in the first step. The variances are averaged generate a single training point using the fixed factor variation as label. 10000 such training points are generated as the training set. majority vote classifier is trained to pick out fixed generative factor and the evaluation accuracy is computed on test set of 5000 and reported as the disentanglement score.
Spearman
Lasso
Trained model pool
V
B
Z
Entangled
Disentangled
VAE M
Z
A
C
Figure 5: A: Schematic illustration of the pairwise model comparison. Two trained models i and j sampled for pairwise comparison. Both models learnt a perfectly disentangled representation, learning represent (positions x/y) and three (positions x/y, and size) generative factors respectively. Similarity matrix Ray: white high similarity between latent dimensions, black - low. B: Similarity matrix Rıj for the same pair of models, calculated using either Spearman correlation Lasso regression. The latter is often cleaner. C: Examples of Lasso similarity matrices of an entangled vs a disentangled model.
Mutual Information Gap (MIG). The MIG metric proposed in Chen et al. (2018) proposes estimating the mutual information (MI) between each data generative factor and each latent dimension. For cach factor, they consider two latent dimensions with the highest MI scores. is assumed that in a disentangled representation only one latent dimension will have high MI with single data generative factor, and hence the difference between calculated the average normalised difference
these MI scores will be large. Hence, the MIG between such pairs MI scores per each data generative factor, Chen al. (018)) suggest that MIG score is more general and unbiased than the B- VAE and Factor VAE metrics.
We compute the Mutual Information Gap by taking the discretising the mean representation of 10000 samples into 20 hins. The disentanglement score is then derived by computing, per generative factor, the difference between the top two latents with the greatest mutual information with the generative factor and taking the mean.
18

---

Published as a conference paper at ICLR 2020
Table 7: Rank correlations between each of the scores produced by the four versions of UDR and four supervised metrics. The scores are averaged over three model classes, two datasets and four supervised metries. See Supplementary Material for details.
UDR
LASSO
SPEARMAN
SUPERVISED
HYPER 0.51+0.06
0.53
0.67
ALI-TOOALL 0.60+0.11
0.59+0.10
Hue (I(2)())
max J+
(14)
where K is the number of generative factors, from which th is single generative factor 2, is the mean representation and jLk) = argmax, (2);20) is the latent representation with the greatest mutual information with the generative factor. H., is the computed entropy of the generative factor.
jndj'
This is the disentanglement part of the three-part metric proposed by Fastwond & Williams 2018). The DCI disentanglement metric is somewhat similar our unsupervised metric, whereby the authors train random forest classifier predict the ground truth factors from the corresponding latent encodings (z](a)) They then use the resulting M x N matrix of feature importance weights to calculate the difference between the entropy of the probability that a latent dimension is important for predicting particular ground truth factor weighted by the relative importance of cach dimension.
DCI Disentanglement.
The DCI disentanglement metric is an implementation of the disentanglement metric as described in Eastwood & Williams (2018) using gradient boosted tree. was computed by first extracting the relative importance of each latent mean representation as predictor for each generative factor by training a gradient boosted tree using the de- fault Scikit-leam model on 10000 training and 1000 test points and extracting the importance weights. The weights are summarised into an importance matrix R.g with the number of rows equal to the number of generative factors and columns equal to the number of latents. The disentanglement score for each column computed as D;= (1- k= is the probability of the latent factor i in being important for predicting factor k. The weighted mean of the scores for the column is computed
Hx (P)) ) where Hx (P)=-E* K -1
k== k== Pullogk Pa denotes the entropy. Pa = R / K-1
using the relative predictive importance of each column the weight D -, D, where pt = E,, Rag Ens Rag-
A.7 ADDITIONAL RESULTS
We evaluated four UDR versions. which differed in terms of whether Spearman- and Lasso-based similarity matrices Ra were used (subscripts S and L respectively). and whether the models for pairwise similarity comparison are picked from the pool of different seeds trained with the same hy hyperaarameters or from the pool of all models (the latter indicated by the A2A suffix). The A2A correlations in Tbl. 7arc on average slightly higher, however these scores are more computationally expensive compute duc the higher number of total pairwise similarity calculations. For that reason, scores presented in the table are calculated using only 20% all the trained models. Hence, the results presented in the main text of the paper are computed using the UDR, score, which allowed us to evaluate all 5400 models and performed slightly better than the UDRs score. Figs, provide more details the performance of the different UDR versions.
To qualitatively validate that the UDR method is ranking models well, we look into more detail into the 8 VAE model ranking when evaluated with the DCI disentanglemen metric on dSprites dataset. This scenario resulted the worst disagreement between UDR and the supervised metric as shown Fig. [ We consider the UDR, version of our method, since it appears to give the hest trade off hetween overall correlations with the supervised metrics and hyperparameter selection accuracy. Fig. D demonstrates that the poor correlation between UDR; and DCI Disentanglement is due the supervised metric. Models ranked highly by UDR, but poorly by DCI Disentanglement appear to be qualitatively disentangled through visual inspection latent traversals. Conversely, models scored highly by DCI Disentanglement but poorly by UDR, appear entangled.
A.8 UDR CORRELATION WITH FINAL TASK PERFORMANCE
To illustrate the usefulness UDR to select disentangled models, we ran two experiments. We computed the UDR correlation with faireess scores and with data efficiency on model-based RI. task.
Fairness scores. Fig. (left) demonstrates that UDR correlates well with the classification faimess scores introduced by Locatelllo et al. 2019). We adopted similar setup described in Locatello et al. (0199) to compute fair- ness, using a gradient booting classifier over 10000 labelled examples. The faimess score was computed by taking
19

---

Published as a conference paper at ICLR 2020
Lasso
Spearman
1.44
6:73
3:31
3:11
0:01
6:12
8:37
0:01
0:00
Hyper
642
6:10
AMM
3.0
841
AMG
6:18
6:22
633
6:14
640
6:18
944
643
6:43
6:74
644
JAM Mare
M4
Cel Menearr glawnt
JAM Arr
6.13
6:37
5.5
10
6:2
4:13
5.0
041
All-to-all all
6.13
6:18
3.5
615
6:34
3.41
0.73
6:10
Figure 6: Rank correlation between different versions of UDR with different supervised metrics across two datasets and three model classes. We see that the UDR2 approaches slightly outperform the UDRs ones.
the mean of the fairness scores across all targets and all sensitive variables where the fairness scores computed by measuring the total variation after intervening the sensitive variable. The fairness scores were compared against the Lasso regression version UDR where models were paired only within the same hyperparameters.
Model-based RL data efficiency. We reproduced the results from the COBRA agent (Watters et al. 2019). to observe if UDR would correlate with the final tasks performance when using VAEs as state representations. More precisely, will look at the training data efficiency, reported as the number of steps needed to achieve 90% perfor- mance on the Clustering tasks (see Watters et al. (2019) for details), while using differently disentangled models.
The agent is provided with a pre-trained MONet (Burgess et al. 2019,, , an exploration policy and a transition model and has to learn a good reward predictor for the task in a dense reward setting. It uses Model Predictive Control in order to plan and solve the task, where sprites have to be clustered by color (e.g. two blue sprites and two red sprites). In COBRA, the authors use MONet with disentangled representation by using high 3 1.
When pre- pre-training MONet, we used 8 E f0.01,0.1,1} in order to introduce entanglement in the representations without compromising reconstruction accuracy and pre-trained 10 seeds for each value of J.. We use 5 random initialisations of the reward predictor for each possible MONet model, and train them to perform the clustering task as explained in Watters et al. (2019). We report the number of steps to reach 90% success, averaged across the initialisations. The UDR score computed by feeding images with single sprite to obtain an associated unique representation and proceeding as described in the main text.
As can seen in Figure TT (right), we find that the UDR scores correlate with this final data efficiency (linear regression shown, Spearman correlation p= 0.56). This indicates that one could leverage the UDR score as a metric to select representations for further tasks. In this analysis we used the version of UDR that uses Spearman correlations and within- within- hyperparameter model comparisons.
A.9 EVALUATING UDR ON MORF COMPLEX DATASETS
We evaluated whether UDR is useful for model selection on more complex datascts. In particular, we chose CelebA and ImageNet. While disentangling VAEs have been shown to perform well on CelebA in the past (c.g. (Higgins et al. (2018b)) ImageNet is notoriously too complex for even vanilla VAEs model. However, we still wanted to verify whether the coarse representations of VAEs on ImageNet could be disentangled, and if so, whether UDR would be useful for model selection. To this end, we ran a hyperparameter sweep for the 3-VAE and ranked its representations using UDR. Fig. 12 shows that UDR scores are clearly different for the different values of the 3 hyperparameter. It is also clear that the models were able learn about CelebA and produce reasonable reconstructions. but on ImageNet even the vanilla VAES struggled to represent anything but the coarsest information. Figs. 13 14
plot latent traversals for three randomly chosen models with high (>0.6) and low (<0.3) UDR scores. The latents are sorted by their informativeness, approximated by their batch-averaged per dimension KI. with the prior as per F It clear that for both datasets those models that are ranked high by the UDR have both more interpretable and more similar representations than those models that ranked low.
20

---

Published as a conference paper at ICLR 2020
dSprites
B-VAE
DIP-VAE
TC-VAE
1.0
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
Hosto oiin sttlyy intoe
0.1
Hyper
UDR
UDR
UDR AAAA
UDR UDR-A2A
UDR
UDR
UDR A2AA
UDR UDR-AZA AZA
UDR
UDR
UDR -2a
UDR AAA
2
3
Shapes 3D
4
B-VAE
DIP-AEE
TC-VAE
5
1.0
0.g
0.8
0.7
0.6
0.5
0.4
0.3
0.7
0.1
0.0
UDR,
UDR,
UDR A2A
UDR A2A
UDR
UDR,
UDR A2A4
UDR A24
UDR_
UDR,
UDR A224
UDR ABA
Figure 7: The range of scores for each hyperparameter setting for the dSprites and 3D Shapes datasets for various models and metrics. We see that the different versions of the UDR method broadly agree with each other.
A. QUALITATIVE EVALUATION MODEL REPRESENTATION RANKED BY UDR SCORES
section we attempt to qualitatively verify our assumption that "for particular dataset and VAF-based un- supervised disentangled representation learning model class, disentangled representations are all alike, while every entangled representation is entangled in its own way, to rephrase Tolstoy". The theoretical justification of the pro- posed UDR hinges the work by Rolinek al. (2019) However, that work only empirically evaluated their analy- sis the 3-VAE model class. Even though we have reasons believe that their thcoretical results would also hold for the other disentangling VAEs evaluated in this paper, in this section we empirically evaluate whether this is truc.
First, we check if all model classes operate in the so called "polarised regime", which was highlighted by Rolinek et (2019) as being important for pushing VA towards disentanglement, It is known that even vanilla VAΣS [Kingma Welling 2014 Rezende et al. 2014 enter the "polarised regime". which is often cited
of their shortcomings (c.g. see Rezende & Viola (018)). of disentangling VAEs considered in this paper augment the original ELBO objective with extra terms. None of these extra terms penalise entering the "polarised regime", apart from that of DIP- VAE-I. We tested empirically whether different model classes entered the "polarised regime" during our hyperparameter sweeps. We did this by counting the number of latents that were "switched off" in each of the 5400 models considered in our paper by using Eq. 3 We found that all models apart from DIP-VAE-I entered the polarised regime during the hyperparameter sweep. having on average 2.95/10 latents "switched off" (with a standard deviation of 1.97).
Second, we check if the models scored highly by the UDR do indeed have similar representations, and models that are scored low have dissimilar representations. We do this qualitatively by plotting latent traversals for three randomly chosen models within each of the six disentangling VAE model classes considered in this paper.
21

---

Published as a conference paper at ICLR 2020
Lasso
Spearman
0.70
0.5
060
0.5
Hyper
0.50
0.45
0.40
40
comparisons
comparisons
0.70
0.65
All-to-all
060
0.5
5
10
15
20
25
30
35
40
45
50
comparisons
# of comparisons
Figure 8: Rank correlations of the different versions of the UDR score with the 3- BAV metrie on the dSprites dataset for B- VAE hyperparameter search the number of pairwise comparisons per model were changed. Higher number of comparisons leads to more accurate and more stable rankings, however these are still decent even with 5 pairwise comparisons per model.
We groups these plots by UDR (UDR<003). Fig. 15 shows latent traversals
into three bands: high (UDR>0.4), medium (0.3<UDR<0.4) and low
model classes that were able achieve the high range of UDR scores (note that some model classes were not able to achieve high UDR values with the hyperparameter settings evaluated in this paper). We present the latent traversals for all ten latents model without sorting them in any particular way. We also colour code the latents by their semantic meaning (if the meaning is apparent from latent traversal). Fig [5]shows that the representations learnt by highly ranked models appear very similar (up to subsetting. sign inverse and permutation). Note that models also include many latents that are "switched aff". Fig. [6]shows latent traversals for two model classes that did not achieve high UDR scores. We see that these models now have fewer semantically meaningful latent dimensions, and fewer latents that uninformative or "switched off". Finally, Fig. [7 shows latent traversals for all model classes that had models which scored low on UDR. We see that many of these models not any uninformative latents, and their representations are hard to interpret. Furthermore, is hard to find similarity between the representations learnt by the different models. Together, Figs. 15 17 eempiricaly verify that our assumption holds for the model classes considered in this paper. However, classes developed in the future first verify that the assumptions the UDR hold for those models. We suggest training the new models on a number of toy but well studied datasets (like dSprites or Shapes3D) and checking the ranks produced by UDR correlate with those produced by the supervised metrics. Furthermore, we suggest qualitative evaluation of the traversals plots for the high and low scoring models.
recommend that any practitioner using UDR new disentangling model
22

---

Published as a conference paper at ICLR 2020
by hyperparameter value
by informative latents #
-3
3
20
20
Figure 9: Example latent traversals of some of the best and worst ranked B- 3-VAE models using the UDR, (ordinate) and DCI Disentanglement (abscissa) metrics, coloured either by hyperparameter value (top) or final informative latent number (bottom). Uninformative units are greyed out. The models ranked highly by UDR, do appear to be well disentangled, despite being ranked poorly by DCI Disentanglement 2, 2, 4). On the other hand, models ranked well by DCI Disentanglement but poorly by UDR, look quite entangled (5, 6). Finally, models ranked poorly by both metrics do appear entangled (3).
23

---

Published as a conference paper at ICLR 2020
UDR 0.579
d
UDR 0.806
d
UDR 0.589
d 6
-3
3
2
10
2
Z:
2
Z
UDR 0.165
9
UDR 0.204
d 10
UDR 0.148
d 10
-3
3
3
10
10
Figure 10: Example latent traversals of some of the best and worst ranked B-VAE models using the UDR, scores. Uninformative latents are greyed out.
Fairness
7000
TC-VAE
-0.80
-0.78
Steps to 90% RL perfomance
5000
*VVAE
-0.75
-0.89
3000
DIP-VAE
-0.66
-0.90
1000
0.10
0.15
0.20
0.25
0.30
0.35
0.60
dSprites
3DShapes
UDR
Figure 11: Left: Spearman correlation between UDR scores and classification fairness scores intro- duced by Locatello et al. (2019) across sixty models trained per each one of the three different model classes (rows) and over two datasets (columns). Right: Spearman correlation between UDR scores and data efficiency for learning a clustering task by the COBRA agent introduced by Watters et al. (0019) Lower step number is better.
24

---

Published as a conference paper at ICLR 2020
celeba
imagenet
Ho
UDR
UDR
0.5
0.5
0
1
2.5
10
25
50
2.5
5
10
25
50
G value
value
Input
Reconstruction
Input
Reconstruction
Figure 12: Distribution of UDRs scores (P=50) for 300 3- 3-V models trained with 6 settings of the hyperparameter and 50 seeds on CelebA and ImageNet datasets. The reconstructions shown are for the vanilla VAE (B=1). ImageNet a complex dataset that VAFs struggle to model well.
Traversal
UDR 0.6
Latent
Traversal
UDR 0.3
Latent
Y
Y
Figure 13: Latent traversals for the four most informative latents ordered by their KL from the prior for three different VAE models that ranked high or low according to UDR. Those models that were ranked high have learnt representations that both interpretable and very similar across models. Those models that were ranked low have learnt representations that are harder to interpret and they do not appear similar to each other across models.
25

---

Published as a conference paper at ICLR 2020
Traversal
UDR
UDR 0.3
Figure 14: Latent traversals for the six most informative latents ordered by their KL from the prior for three different 3-VAE models that ranked high or low according to UDR. Despite the fact that none of the 3-VAE VAE models were able learn to reconstruct this dataset well, those models that were ranked high by the UDR still managed to learn representations that are more interpretable and more similar across models. This unlike the representations of those models that were ranked low by the UDR.
26

---

Published as a conference paper at ICLR 2020
UDR > 0.4
B-VAE
Latent
Traversal
TC-VAE
Latent
Traversal
CCI-VAE
Latent
Traversal
Position
Position
Size
Rotation
Entangled
Uninformative
Figure 15: Latent traversals for all ten latent dimensions presented in particular ordering for three different models per model class. These models were ranked highly by the UDR. can be seen that they learnt interpretable and similar representations up permutation, sign inverse and subsetting. We included all model classes that achieved UDR scores in the range specified (UDR > 0.4).
27

---

Published as a conference paper at ICLR 2020
0.3 UDR < 0.4
Factor VAE
Latent
Traversal
DIP-VAE-II
Latent
Traversal
Position
Position
Size
Rotation
Entangled
Uninformative
Figure 16: Latent traversals for all ten latent dimensions presented in particular ordering for three different models per model class. These models received medium UDR scores. It can be seen that they learnt less interpretable and less similar representations than the models shown in Fig. [5] None of the models these model classes scored higher than the range specified (0.3 < UDR < <.4)..
28

---

Published as a conference paper at ICLR 2020
UDR < 0.3
B-VAE
DIP-VAE-I
Latonn
Labont
Travenal
TC-VAE
DIP-VAE-II
Latent
Latent
Triversal
Factor VAE
Position
Size
Traversal
Rotation
Entangled
Uninforrative
Figure 17: Latent traversals for all ten latent dimensions presented in particular ordering for three different models per model class. These models received low UDR scores. It can be seen that their representations are hard to interpret and they look quite different from each other. We included all model classes that achieved UDR scores in the range specified (UDR 0.3).
29


Reviews:
[{"review_id": "rkehdCaaKH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a metric for unsupervised model (and hyperparameter) selection for VAE-based models. The essential basis for the metric is to rank the models based on how much disentanglement they provide. This method relies on a key observation from this paper [A] viz., disentangled representations by any VAE-based model are likely to be similar (upto permutation and sign).\n\nI am inclined to accept the paper for the following reasons:\n1. The proposed approach is clear and easy enough to understand and well motivated\n2. The paper has clearly outlined the assumptions and limitations of their work\n3. The reported result show that models ranked by disentanglement correlate well with the supervised metrics for the various VAE models.\n4. This metric is unsupervised and thus can utilize far more data than the supervised metric methods and can be useful even when the dataset has no labels.\n5. The supplementary material also shows that the metric correlates well with the task performance.\n\n[A] Variational Autoencoders Pursue PCA Directions (by Accident), CVPR 2019\n\n---\n\nUpdate:\n\nThanks for the thoughtful rebuttal by the authors to all the reviewers' feedback.\n\nBased on the several discussions by the other reviewers and the discussion that happened, I am inclined to lower my scores to a weak accept. ", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "SJgQGjb39r", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper addresses the problem of unsupervised model selection for disentangled representation learning. Based on the understanding of “why VAEs disentangle” [Burgess et al. 2017, Locatello et al. 2018, Mathieu et al. 2019, Rolinek et al. 2019], the authors adopt the assumption that disentangled representations are all alike (up to permutation and sign inverse) while entangled representations are different, and propose UDR method and its variants. Experimental results clearly show that UDR is a good approach for hyperparameter/model selection.\nOverall, I think a reliable metric for model selection/evaluation is needed for the VAE-based disentangled representation learning. According to comprehensive experimental studies performed in this paper, UDR seems to be a potentially good choice.\n\nHowever, I am not sure if very good disentangled representations must benefit (general) subsequent tasks, though the authors provide experimental evidence on fairness classification and data efficiency tasks. Actually, the data generation process in the real-world may consist of different generative factors that are not independent of each other. Though good disentangled representation provides good interpretability, it needs not to be better than entangled representation for concrete tasks. Specifically, for concrete supervised classification tasks, VAE with beta smaller than 1 (not towards disentanglement) might be the best (Alexander A. Alemi et al. 2017, Deep VIB).\n\nAnother concern is about the choice of some key “hyperparameters”.\nFor the KL divergence threshold in equation 3, you set it to be 0.01. It looks like the choice would control how much the UDR favors a “sparse representation map”. The larger the value, the few “informative dimensions” would be considered.\nIn supplementary material, you say that “uninformative latents typically have KL<<0.01 while informative latents have KL >>0.01”. Is this judgment based on “qualitative feeling”? For me, as you are contributing a ``quantitative measurement”, it is interesting and important to see how this threshold would generally affect UDR’s behavior in one (or more) datasets you have tried. \nAnother hyperparameter I cared is P (number of models for pairwise comparison). In the paper, you validate the effect of P in the range [5,45]. How would P smaller than 5 affect UDR? According to Table 1, if I was using UDR, I’d rather using P>=20 (or at least 10) rather than 5.\nAlso, it seems to me P would grow up due to the size of factors that generate the data. Thus, I also have a little concern about the computation cost of the proposed metric (as also mentioned by the authors).\n\nOthers concerns:\n-- As a heavy experimental paper, most experimental results are in supplementary material, while the authors spent a lot of time in the main text explaining the conclusions found in other papers.\n-- To validate the fundamental assumption of UDR, the authors might consider to quantitatively validate that, disentangled representations learned by those approaches you used in the paper are almost the same (up to permutation and sign inverse). ", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "Byx8ouwAFr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a criterion called Unsupervised Disentanglement Ranking (UDR) score. The score is computed based on the assumption that good disentangled representations are alike, while the representations can be entangled in multiple possible ways.  The UDR score can be used for unsupervised hyperparameter tuning and model selection for variational disentangled method.\n\nThe problem this paper focuses on is essential because we usually apply unsupervised disentangled methods to analyze the data when the labels are unavailable. However, existing metrics for hyperparameter tuning and model selection requires ground-truth labels. This paper allows measuring model performance without supervision, making the hyperparameter tuning and model selection possible in practice.\n\nIt looks like some parts of this paper need rewriting. In the abstract, it is not mentioned at all what is the proposed approach. Most paragraphs in the introduction section review the related work and background but do not introduce what assumption and strategy the proposed method adopted.\n\nIt looks like the proposed UDR is theoretically supported by Rolinek et al. (2019). However, the proof given by Rolinek et al. (2019) is for $\\beta$-VAE, where the regularization can be turned into the constraint on KL divergence. I do not think the \"polarised regime\" holds for other disentangled model, for example, TCVAE, where a biased estimation of total correlation is introduced in the objective function. Therefore, I am not convinced that I should trust the results of the UDR, which combines multiple disentangled models.\n\nThe computational process of UDR is heuristic and somewhat arbitrary. There is no theoretical guarantee that UDR should be a useful disentanglement metric.  Although the UDR is supported by some experiments, I am not convinced that it is trustworthy for more complex real-world datasets.\n\nEquation (3) looks problematic. Note that it is possible to train a Bidirectional Generative Adversarial Network (BiGAN) that can generate complex images based on a uniform distribution (Donahue et al., 2016). The encoder of the BiGAN can be considered as the inverse of the generator, which maps images back to the uniform distribution. This suggests that under the encoder-decoder framework, it is possible that latent variables can be informative even the posterior distribution matches the prior distribution. Although VAEs are trained using a different strategy, I do not see why the posterior needs to diverge from the prior distribution for informative latent representations. The encoder might simply be the inverse of the decoder under a certain scenario.\n\nIn summary, this paper focuses on solving an important problem. However, the proposed method is not well supported by theorems as it seems. The paper also appears to contain minor technical issues. Therefore, I am inclined to reject this paper.\n\nReferences\nDonahue, Jeff, Philipp Krähenbühl, and Trevor Darrell. \"Adversarial feature learning.\" arXiv preprint arXiv:1605.09782 (2016).", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors address the important and understudied problem of tuning of unsupervised models, in particular variational models for learning disentangled representations.  They propose an unsupervised measure for model selection that correlates well with performance on multiple tasks.  After significant fruitful discussion with the reviewers and resulting revisions, many reviewer concerns have been addressed.  There are some remaining concerns that there may still be a gap in the theoretical basis for the application of the proposed measure to some models, that for different downstream tasks the best model selection criteria may vary, and that the method might be too cumbersome and not quite reliable enough for practitioners to use it broadly.  All of that being said, the reviewers (and I) agree that the approach is sufficiently interesting, and the empirical results sufficiently convincing, to make the paper a good contribution and hopefully motivation for additional methods addressing this problem.
Decision: Accept (Poster)