Disentangling Factors of Variations Using Few Labels
Authors: Francesco Locatello
Michael Tschannen
Stefan Bauer
Gunnar Rätsch
Bernhard Schölkopf
Olivier Bachem
Venue:  (2020)

Abstract: Learning disentangled representations is considered a cornerstone problem in representation learning. Recently, Locatello et al. (2019) demonstrated that unsupervised disentanglement learning without inductive biases is theoretically impossible and that existing inductive biases and unsupervised methods do not allow to consistently learn disentangled representations. However, in many practical settings, one might have access to a limited amount of supervision, for example through manual labeling of (some) factors of variation in a few training examples. In this paper, we investigate the impact of such supervision on state-of-the-art disentanglement methods and perform a large scale study, training over 52000 models under well-defined and reproducible experimental conditions.  We observe that a small number of labeled examples (0.01--0.5% of the data set), with potentially imprecise and incomplete labels, is sufficient to perform model selection on state-of-the-art unsupervised models. Further, we investigate the benefit of incorporating supervision into the training process. Overall, we empirically validate that with little and imprecise supervision it is possible to reliably learn disentangled representations.

Published as a conference paper at ICLR 2020
DISENTANGLING FACTORS OF VARIATION USING FEW LABELS
Francesco Locatello Michael Tschannen Stefan Bauer", Gunnar Rätsch', Bernhard Schölkopf Olivier Bachem²
Department of Computer Science, ETH Zurich
2 Max Planck Institute for Intelligent Systems, Tübingen
8 Google Research, Brain Team
Fr ancesco. locatello@ing eth2.ch bachem@google.com com
ABSTRACT
Learning disentangled representations is considered comeestone problem in repre- sentation learning. Recently, Locatello al. (2019) demonstrated that unsupervised disentanglement learning without inductive biases is theoretically impossible and that existing inductive biases and unsupervised methods do not allow to consistently learn disentangled representations. However, in many practical settings, one might have access to limited amount supervision, for example through manual labeling of (some) factors of variation in a few training examples, In this paper, we investigate the impact of such supervision on state-of-the-aa disentanglement methods and perform large scale study, training well-defined and reproducible experimental conditions. We observe that a small number of labeled examples (0.01-0.5% of the data set), with potentially imprecise and incomplete labels, is sufficient to perform model selection on state-of-the-an unsupervised models. Further, we investigate the benefit of incorporating supervi- sion into the training process. Overall, we empirically validate that with little and imprecise supervision it possible to reliably learn disentangled representations.
52 000 models under
INTRODUCTION
In machine learning, is commonly assumed that high-dimensional observations x (such as images) are the manifestation of a low-dimensional latent variable z of ground-truth factors of variation (Ben) gio al. [0133] Kulkarni et al., 20155 Chen al. [0166] Tschannen al., [208)). More specifically, one often assumes that there is a distribution p(z) over these latent variables and that observations in this ground-truth model are generated by sampling z from p(z) first. Then, the observations sampled from conditional distribution p(x(z). The goal of disentanglement learning is to find a representation of the data r(x) which captures all the ground-truth factors of variation in z independently. The hope is that such representations will be interpretable, maximally compact, allow for counterfactual reasoning and be useful for a large variety of downstream tasks (Bengio et al. 2013 Peters et al.] 2017 LeCun et al.) 2015 Bengio al. [2007] Schmidhuber 1992| Lake et al.
2017 Goodfellow et al. 2009 Lenc & Vedaldi 2015 Tschannen et al. 2018 Higgins et al 2018 Suter et al. 2019; Adel et al. 2018 van Steenkiste et al. [2019] Locatello al., 2019a Gondal et
2019). As all these applications rely on the assumption that disentangled representations can be
reliably learned in practice, we hope learn them with as little supervision as possible (Bengio
et al. 2013 Schölkopf et al. 2012 Peters et al. 2017 Pearl 2009 Spirtes et al. 2000).
Current state-of-the-art unsupervised disentanglement approaches enrich the Variational Autoencoder (VAE) (Kingma & Welling 2014) objective with different unsupervised regularizers that aim to encourage disentangled representations (Higgins et al., 2017a Burgess et al. 20188) Kim & Mnih
2018 Chen al. 2018 Kumar et al.) 2018 Rubenstein et al. 2018 Mathieu et al.) 2018 Rolinek et al. 2019). The disentanglement of the representation of such methods exhibit a large variance and while models turn out to be well disentangled it appears hard to identify them without supervi- sion [Locatello et al., 2019b). This consistent with the theoretical result of Locatello et al. (2019b) that the unsupervised learning of disentangled representations is impossible without inductive biases.

---

Published as a conference paper at ICLR 2020
Figure 1: Latent traversals (each column corresponds different latent variable being varied) on S for the 3-TCVAE model with best validation MIG (top) and for the semi-supervised 3-TCVAE model with best validation loss (bottom), both using only 1000 labeled examples for validation and/or supervision. Both models appear to be visually well disentangled.
While human inspection can be used to select good model runs and hyperparameters (c.g. Higgins al. (201bb, Appendix 5.1)), argue that such supervision should be made explicit. We hence consider the setting where one has access to annotations (which we call labels in the following) of the latent variables z a very limited number of observations x, for example through human annotation. Even though this setting is not universally applicable (e.g. when the observations are not human interpretable) and a completely unsupervised approach would be elegant, collecting small number of human annotations is simple and cheap via crowd-sourcing platforms such as Amazon Mechanical Turk, and is common practice in the development of real-world machine learning systems. As a consequence, the considered setup allows to explicitly encode prior knowledge and biases into the learned representation via annotation, rather than relying solely on implicit biases such as the choice of network architecture with possibly sard-to-contro effects.
Other forms of inductive biases such as relying on temporal information (video data) (Denton & Birodkar 2017 Yingzhen & Mandi 2018), allowing for interaction with the environment (TTomaa
et al. (017). )) or incorporating grouping information (Kulkarni et al.) 2015) [Bouchacourt et al. (2018) discussed in the literature candidates circumvent the impossibility result by Locatello et al. (2019b). However, there currently seems to be no quantitative evidence that such approaches will lead to improved disentanglement on the metrics and data sets considered in this paper. Furthermore, these approaches come with additional challenges: Incorporating time can significantly increase computational costs (processing video data) and interaction often results slow training (e.g. robot arm interacting with the real world). By contrast, collecting few labels is cheap and fast.
We first investigate whether disentanglement scores are sample efficient and robust to imprecise labels. Second, we explore whether it is more beneficial to incorporate the limited amount of labels available into training and thoroughly test the benefits and trade-offs of this approach compared supervised validation. For this purpose, perform a reproducible large scale experimental study] training over 52 000 models on four different data sets. We found that unsupervised training with supervised vali- dation enables reliable learning of disentangled representations, On the other hand, using some of the labeled data for training is beneficial both terms of disentanglement and downstream performance. Overall, we show that very small amount of supervision is enough reliably learn disentangled representations as illustrated in Figure I Our key contributions can be summarized as follows:
We observe that some of the existing disentanglement metrics (which require observations of z) can be used to tune the hyperparameters of unsupervised methods even when only very few labeled examples are available (Section 3) Therefore, training variety of models and introducing supervision to select the good runs is viable solution to overcome the impossibility result Locatell al. (2019b).
"Reeroducing these experiment requires approximately 8.57 GPU years (NVIDIA P100).
2

---

Published as a conference paper at ICLR 2020
We find that adding simple supervised loss, using as little 100 labeled examples, outperforms unsupervised training with supervised model validation both in terms of disentanglement scores and downstream performance (Section 4.2).
We discover that both unsupervised training with supervised validation and semi-supervised training are surprisingly robust to label noise (Sections (3. and 4.3) and tolerate coarse and partial annotations, the latter being particularly important if not all factors of variation are known.
Based on our findings we provide guidelines helpful for practitioners to leverage disentangled representations in practical scenarios.
2 BACKGROUND AND RELATED WORK
Problem definition: Consider a generative model with latent variable z with factorized density
p(z) II: p(z)), where d > 1. and observations x obtained as samples from p(x z). Intuitively, the goal of disentanglement learning is to find representation r(x) separating the factors of variation into independent components so that a change in dimension of z corresponds to change in a dimension of r(x) (Bengio al. 2013). Refinements of this definition include disentangling independent groups in the topological sense (Higgins et al.) 2018) and learning disentangled causal models (Suter et al. 2019). These definitions are reflected in various disentanglement metrics that aim at measuring some structural property of the statistical dependencies between and r(x).
The BetaVAE (Higgins et al., 2017a) and Factor- VAE (Kim & Mnih 2018) scores measures disentanglement by performing an intervention on the factors of variation and predicting which factor was intervened on. The Mutual Information Gap (MIG) (Chen et al. 2018), Modularity (Ridgeway & Mozer (018)) DCI Disentanglement (Eastwood & Williams, 2018) and SAPscore (Kumar et al., 2018) first compute a matrix relating factors of varia- tion and codes (for example via pairwise mutual information, feature importance and predictability. Then, they average a normalized difference between the top two entries of the matrix either row or column wise. For more details, see Appendix C of (Locatello et al. 2019b).
Evaluating disentangled representations:
Learning disentangled representations: Since all these metrics require access to labels z they cannot be used for unsupervised training. Many state-of-the-art unsupervised disentanglement methods therefore extend VAES (Kingma & Welling 2014) with regularizer (g (z) x)) that enforces structure the latent space of the VAE induced by the encoding distribution qs x)) with
the hope that this leads to disentangled representations. These approaches (Higgins et al. 2217aa
(Burgess al., [2188] Kim & Mnih, 2018 Chen et al. 2018 Kumar et al. 2018) can cast under the following optimization template:
max 9.9
Ex [ (x)x) [log po (x/z)) (x (zx()))))) + RR (=((g()
.EI (0.))
The 3-VAE (Higgins et al. 2017a) and AnnealedVAE (Burgess et al. 2018 reduce the capacity of the VAE bottleneck under the assumption that encoding the factors of variation is the most efficient way to achieve good reconstruction (Peters al., 2017). The Factor-VAE. (Kim & Mnih 2018) and 3-TCVAE both penalize the total correlation of the aggregated posterior q(z) (i.c. the encoder distribution after marginalizing the training data). The DIP-VAE variants (Kumar et al. 2018) match the moments of the aggregated posterior and a factorized distribution. We refer to Appendix B of 2019b) and Section 3 of (Tschannen et al., 2018) for a more detailed description.
(Locatello al.
Supervision and inductive biases: While there is prior work on semi-supervised disentanglement [0144] Cheung al., 2014 Mathieu et al.) 2016 Narayanaswamy al., 2017 Kingma et al. [0144] Klys et 2018), these methods aim to disentangle only some observed factors of variation from the other latent variables which themselves remain entangled. These approaches are briefly discussed in Section 4 Exploiting relational information or knowledge of the effect of the factors of variation have both been qualitatively studied to learn disentangled representations (Hinton et al. M001i/Coher & Welling 2014 Karaletsos al. 2015 Goroshin al. [013 W Whitney et al. 2016 Fraccaro et al. 2017 & Birodkar| 2201 7 Hsu et al. 2017 Yingzhen & Mandt [018)] 1 Lo
learning (Reed al.
catello et al. 2018 Kulkarni et al. 2015 Ruiz et al. [019 Bouchacourt et 2018 Adel al. 2018)..
Other related work. Due to the lack of a commonly accepted formal definition, the term "disen- tangled representations" has been used in very different lines of work. There is for example rich
3

---

Published as a conference paper at ICLR 2020
literature in disentangling pose from content in 3D objects and content from motion in videos Yang et al. 2015 Yingzhen & Mandt, 2018 Hsiih et al. 2018 Fortuin et al. 2019 Deng et al. 2017
Goroshin et al. 2015: Hyvarinen & Morioka 2016). This can be achieved with different degrees of
supervision, ranging from fully unsupervised to semi-supervised. Another line of work aims at disen- tangling class labels from other latent variables by assuming the existence of causal model where the latent variable z has an arbitrary factorization with the class variable y. In this setting, y is partially observed (Reed al. [0044] Chheng et al. 2014, Mathieu et al. 2016 Narayanaswamy et al. 2017 Kingma et al.] 2014]] Klys et al. (2018). Without further assumptions on the structure of the graphical model, this equivalent to partially observed factors of variation with latent confounders. Except for very special cases, the recovery of the structure of the generative model is known to be impossible with purely observational data (Peters et al. 2017 D Amour 2019) Suter et al. 2019). Here, we intend to disentangle factors of variation in the sense of (Bengio et al., 2013] /Suter et al. 2019; Higgins et al.) 2018). We aim at separating the effects of all factors of variation, which translates to learning a rep- resentation with independent components. This problem has already been studied extensively in the non-linear ICA literature (Comon 1994, Bach & Jordan 2002, Jutten & Karhunen] 2003) Hyvarinen
& Morioka 2016 Hyvärinen & Pajunen 1999: Hyvarinen et al. [019]] Gresele 2019).
3 UNSUPERVISED TRAINING WITH SUPERVISED MODEL SELECTION
In this section, we investigate whether commonly used disentanglement metries can be used to identify good models if a very small number of labeled observations is available. While existing metrics are often evaluated using as much as 10 000 labeled examples, it might be feasible in many practical settings to annotate 100 to 1000 data points and use them to obtain disentangled representation. At the same time, it is unclear whether such an approach would work as existing disentanglement metrics have been found to be noisy (even with more samples) (Kim & Mnih| 2018). Finally, we emphasize that the impossibility result of Locatello et al. (2019b) does not apply in this setting as we do observe samples from z.
3.1 EXPERIMENTAL SETUP AND APPROACH
Data sets. We consider four commonly used disentanglement data sets where one has explicit access to the ground- trouud generative model and the factors of variation: dSprites (Higgins et al. 7a) Cars3D (Reed et al. 2015), SmallNORB (LeCun et al. (004)) and Shapes3D (Kim & Mnih |00 8). Following (Locatello et al. 2019b), we consider the statistical setting where one directly samples from the generative model, effectively side- stepping the issuc of empirical risk minimization and overfitting. For each data set, we assume to have either 100 or 1000 labeled examples available and large amount of unlabeled observations. We note that 100 labels correspond to labeling 0.01 % of dSprites, 0.5% of Cars3D, 0.4% of SmalINORB and 0.02% of Shapes3D.
Perfect vs. imprecise labels. In addition to using the perfect labels of the ground-trute generative model, we also consider the setting where the labels are imprecise. Specifically, we consider the cases were labels are binned to take most five different values, are noixy (each observation of factor of variation has 10% chance of being random) or partial (only two randomly drawn factors of variations are labeled). This meant to simulate the trade-offs in the process of practitioner quickly labeling a small number of images.
Model selection metrics. We use MIG (Chen et al. 2018). DCI Disentanglement (Eastwood & Williams 2018) and SAP score (Kumar et al. 2018) for model selection as they can be used on purely observational data. In contrast, the Beta (Higgins al. 2017a) and RAE (Kim & Mnih] 2018) scores cannot be used for model selection on observational data because they require access to the true generative model and the ability to perform interventions. At the same time, prior work has found all these disentanglement metrics to be substantially correlated (Locatello al. 2019b).
Experimental protocol. In total, we consider 32 different experimental settings where an experi- mental setting corresponds to a data set of labeled examples (100/1000), and a labeling setting (perfectbimeedhoiiy/pataaa))) For each con- sidered setting, generate five different sets of labeled examples using five different random seeds. For each of these labeled sets, we train cohorts of 3 3-VAEs (Higgins et al.] (0177a). 3-TCVAEs (Chen) et al. 2018), Factor- VAFs (Kim& & Mnih 2018), and
a specific number
(Kumar al. 2018) where each
4

---

Published as a conference paper at ICLR 2020
Dataset dippites
Dacaset dSprites
SAP (M5380)
20
(952000)
MG (MS3000
92
(MS1000)
70
DC (MS160)
79
68
08
54
-1
59
DO (15100005
75
93
99
-4
67
(2)
d
(4)
(81
(00
16)
(5)
(a) 100 perfect labels.
(b) 1000 perfect labels.
deprits
Dataset
ebjorttes
Natasgt diprtes
SAP 016100)
31
SAP 1963000
SAP (M6100)
-10
MIG (M5100)
77
78
90
-14
57
M4G (M5300))
70
62
185
59
-10
68
MIG (M5100))
60
35
53
55
-7
14
DO (15200)
DC (MB300)
-13
(MS200))
33
(5)
(1)
6
FF
(1)
di
(F)
850
(1)
F1
69
(c) 100 binned labels.
(d) 100 noisy labels.
(e) 100 partial labels.
Figure 2: Rank correlation of validation metrics and test metrics on dSprites. Validation metrics are computed on different types of labels. Legend: (A)=BetaVAB Score, (B)=PactorVAE. Score, (C)=MIG, (D)=DCI Disentanglement, (E)=Modularity, (F)=SAP.
model cohort consists of 36 different models with 6 different hyperparameters for each model and 6 random seeds. For a detailed description of hyperparameters, architecture, and model training we refer to Appendix [ For each of these 23 040 models, we then compute all the model selection metrics on the set of labeled examples and use these scores to select the best models in each of the cohorts. We prepend the prefix U/S for unsupervised training with supervised model selection to the method name. Finally, we evaluate robust estimates of the BetaVAE score, the FactorVAE score, MIG, Modularity, DCI disentanglement and SAP score for each model based on an additional test set of 10 000 sample<" in the same way as in Locatello et al. (201bb).
3.2 KEY FINDINGS
We highlight our key findings with plots picked to be representative of our main results. In Appen- dices we provide complete sets of plots for different methods, data sets and metrics.
In Figure (a), we show the rank correlation between the validation metrics computed on 100 samples and the test metrics on dSprites. We observe that MIG and DCI Disentanglement generally correlate well with the test metrics (with the only exception of Modularity) while the correlation for the SAP score is substantially lower. This is not surprising given that the SAP score requires to train multi- class support vector machine for each dimension of r(x) predicting each dimension of z. For example, on Cars3D the factor determining the object type can take 183 distinct values which can make it hard to train a classifier using only 100 training samples. In Figure 2 (b). we observe that the rank correla- tion improves considerably for the SAP score if we have 1000 labeled examples available and slightly for MIG and DCI Disentanglement. In Figure | (top) we show latent traversals for the U/S model achieving maximum validation MIG on 1000 examples on Shapes3D. Figure 2 (c) shows the rank correlation between the model selection metrics with binned values and the test metrics with exact labels. We observe that the binned labeling does not seem detrimental the performance of the model selection with few labels. We interpret these results as follows: For the purpose of disentanglement, fine- mme-grained labeling not critical as the different factors of variation can already be disentangled us- ing coarse feedback. Interestingly, the rank correlation of the SAP score and the test metrics improves significantly (in particular for 100 labels). This is to be expected, as now we only have five classes for each factor of variation so the classification problem becomes easier and the estimate of the SAP score more reliable. In Figure (d) we observe that noisy labels are only slightly impacting the performance. In Figure (e). we can see that observing only two factors of variation still leads to a high correlation with the test scores, although the correlation is lower than for other forms of label corruption.
Conclusions. From this experiment, we conclude that it is possible to identify good runs and hyperparameter settings on the considered data sets using the MIG and the DCI Disentangleme based on 100 labeled examples. The SAP score may also be used, depending on how difficult the underlying classification problem is. Surprisingly, these metrics are reliable even if we do not collect the labels exactly. We conclude that labeling a small number of examples for supervised validation appears to be a reasonable solution to learn disentangled representations in practice. Not observing
For the Beta VAF score and the Factor score, this includes specific realizations based on interventions on the latent space.
5

---

Published as a conference paper at ICLR 2020
all factors of variation does not have dramatic impact. Whenever it is possible, it seems better to label more factors of variation in a coarser way rather than fewer factors more accurately.
4 INCORPORATING LABEL INFORMATION DURING TRAINING
Using labels for model selection evenn only a small amount-rainn the natural question whether these labels should rather be used for training good model directly. In particular, such approach also allows the structure of the ground-truth factors of variation to be used, for example ordinal information. In this section, we investigate a simple approach to incorporate the information of very few labels into existing unsupervised disentanglement methods and compare that approach to the alternative of unsupervised training with supervised model selection (as described in Section
The key idea is that the limited labeling information should be used to ensure a latent space of the VAE with desirable structure w.r.t. the ground-truth factors of variation (as there is not enough labeled samples to learn good representation solely from the labels). We hence incorporate supervision by equipping Equation [ with a constraint R. (z)x), z) S K, where Re(gs (z)x)))xx, x), z) is function computed on the (few) available observation-label pairs and >> 0 is threshold. We can Baaah-Kuhn-Tuuse
now include this constraint into the loss as a regularizer under the
conditions:
max 6.0
ELBO 0) + B Ru, (2)x)))))) + Bx Rs Rg (z)x);
where Ysup O. We rely on the binary cross-entropy loss to match the factors their targets, i.e., Rg (1)((z))) x), z) = = - 2; log(oa + (1 z,) log(1 - (r(())))) where the targets 2; are normalized to [0, 1]. a()) the logistic function and r(x) corresponds to the mean (vector) of qo (z x). When z has more dimensions than the number of factors of variation, only the first d dimensions are regularized (where d is the number of factors of variation). While the z; do not model probabilities of binary random variable but factors of variation with potentially more than two discrete states, we have found the binary cross-entropy loss to work empirically well out-of-the-box. We also experimented with a simple L2 loss (r(x)) z12 for R., but obtained significantly worse results than for the binary cross-entropy. Similar observations were made in the context of VAFs where the binary cross- entropy as reconstruction loss is widely used and outperforms the Lg loss even when pixels have con- tinuous values in [0, 1] (see, e.g. the code accompanying Chen al. (2018): Locatello et al. (201bb)) Many other candidates for supervised regularizers could be explored in future work. However, given the already extensive experiments in this study, this is beyond the scope of the present paper.
Differences to prior work on semi-supervised disentanglement. Existing semi-supervised ap- proaches tackle the different problem of disentangling some factors of variation that are (partially) observed from the others that remain entangled (Reed et al. 2014 Cheung et al.) 2014: Mathieu et al. 2016 Narayanaswamy et al. 2017: 7 Kingma et al. 2014). In contrast, we assume to observe all ground-truth generative factors but only for a very limited number of observations. Disentangling only some of the factors of variation from the others is an interesting extension of this study. How- ever, is not clear how to adapt existing disentanglement scores to this different setup as they are designed to measure the disentanglemert the experiments in this section is to compare the two different approaches to incorporate supervision into slate-of-the-at unsupervised disentanglement methods.
of all the factors of variation. We remark that the goal of
4.1 EXPERIMENTAL SETUP
True vs. imprecise labels. As in Section 3 we compare the effectiveness of the ground-truth labels with binned, noisy and partial labels on the performance of our semi-supervised approach. To understand the relevance of ordinal information induced by labels, we further explore the effect of randomly permuting the labels in Appendix A
Experimental protocol. To include supervision during training we split the labeled examples in 90%/100% train/validation split. We consider 40 different experimental settings each corresponding to a data set (100/1000), and a labeling setting inductive biases of R, and report the analysis with randomly permuted labels in Appendix [A] For each considered setting, we generate the same five different sets of labeled examples we used for the U/S models. For each of the labeled sets, we train cohorts of 3-VAFS, 3-TCVAFs, Factor- VAEs, and
a specific number of labeled examples wwtpee///iiine/ooii//aarrta/aata
permuted). We discuss the

---

Published as a conference paper at ICLR 2020
Mode PPsseTcve 2r P.
Vf Racarinn CONA poop
Model
6.30
mosfacsrnii Supers
69 VAE
37/FPacto-MA
Donaset - Card30
6.20
Chaptarphemett
0 01553
Sun Lobels
0583
120
0.30
1200
20 20433
044
2.0
Reguintocion strength
Modal
Figure 3: (left) Median across the draws of the labeled data set of the DCI Disentanglement test score on SmalINORB after validation with 100 and 1000 labeled examples. U/S were validated with the MIG. (center) Increasing the supervised regularization strength makes the matrix of pairwise mutual
information [(z, r(x)) closer to diagonal (avgMI = [l r(x)) - diag( / (xx)))]l? (right) Probability of each method being the best random downstream task. Legend: 0=5²/S-B: TCVAE, 1=5² /=52//---VAI 2= /S-DI-------LL,, 3=5² E-FactorVVAD 4=U//S-B-TCVa 5=U/S- 3-VAE, G-U//SDIP-VAAE- 7=U/S- For the U/S methods we sample the validation
metric uniformly.
DIP- VAE-Is with the additional supervised regularizer R. (4s (z) x), z). Each model cohort consists of 36 difffernnt models with 6 different hyperparameters for each of the two regularizers and one random seed. Details on the hyperparameter values can be found in Appendix B For each of these 28 800 models, we compute the value of R, on the validation examples and use these scores to select the best method in each of the cohorts. For these models we use the prefix 5² for semi-supervised training with supervised model selection and compute the same test disentanglement metries in Section 3
Fully supervised baseline. We further consider a fully supervised baseline where the encoder is trained solely based the supervised loss (without any decoder, KI divergence and reconstruction loss) with perfectly labeled training examples (again with a 90%/10% vainn/valdatton split). The supervised loss does not have any tunable hyperparameter, and for each labeled data set, we run cohorts of six models with different random seeds. For each of these 240 models, we compute the value of R. on the validation examples and use these scores to select the best method in the cohort.
4.2 SHOULD LABELS BE USED FOR TRAINING?
First, we investigate the benefit of including the label information during training by comparing semi-supervised training with supervised validation in Figure 3 (left). Each dot in the plot corresponds to the median of the DCI Disentanglement score across the draws of the labeled subset on SmaIINORB (using 100 vs 1000 examples for validation). For the U/S models we use MIG for validation (MIG has a higher rank correlation with most of the testing metrics than other validation metrics, see Figure 2) From this plot one can see that the fully supervised bascline performs worse than the ones that make use of unsupervised data. As expected, having more labels can improve the median downstream performance for the S²/S approaches (depending the data set and the test metric) but does not improve the U/S approaches (recall that we observed in Figure 2 (a) that the validation metrics already perform well with 100 samples).
To test whether incorporating the label information during training is better than using it for validation only, we report in Figure 4 (a) how often each approach outperforms all the others on random disentanglement metric and data set. We observe that semi-supervised training often outperforms supervised validation. In particular, S2/S-B-TC-VAL seems improve the most, outperforming the 8² S-eto-F VAE which was the best method for 100 labeled examples. Using 100 labeled examples, the S² s²/S S approach already wins in 70.5% of the trials. In Appendix D we observe similar trends even when we use the testing metrics for validation (based on the full testing set) in the U/S models. The /2/S approach seem to overall improve training and to transfer well across the different disentanglement metrics. In Figure [ (bottom)
show the latent traversals for the best S²/S 3-TCVAE using 1000 labeled examples. We observe that it achieves excellent disentanglement and that the unnecessary dimensions of the latent space are unused, as desired.
In their Figure 27,
Locatello et al. (009b) showed that increasing regularization in unsupervised methods does not imply that the matrix holding the mutual information between all pairs of entries of r(x) becomes closer to diagonal (which can be seen as a proxy for improved disentanglement). For the semi-supervised approach, in contrast, we observe in Figure (center) that this is actually the case.
7

---

Published as a conference paper at ICLR 2020
023-
9:5
031
444-
230
100
1200
3.5
2.00
390
Vedel
Model
(a) Perfect labels.
(b) Binned labels.
(c) Noisy labels.
(d) Partial labels.
Figure 4: Probability of each method being the best random test metric and random data set after validation with different types of labels. Legend: 0=S*/S-B-TCVAL 1=5² /S-B-VAE, /S-DIP S--P--------LLLL 3=5² / (S-FactorVAE, 4=U/S-B-TCVAAE 5=U//SB- 6=U//-DIPP VAE-I, T=U//SFactorVAE Overall, seem more beneficial to incorporate supervision during training rather than using only for validation. Having more labels available increases the gap.
Finally, we study the effect of supii-sper training on the (natural) downstream task of predicting the ground-truth factors of variation from the latent representation. We four different training set sizes for this downstream task: 10, 100, 1000 and 10 000 samples. We train the same cross-validated logistic regression and gradient boosting classifier as used in Locatello et al.) (2019b). We observe in Figure 3 (right) that /²/S S methods often outperform U/$ in downstream performance. From Figure 20 the Appendix, one can see that the fully unsupervised method has often significantly worse performance.
Conclusions: Even though our semi- cumi-supereiee training does not directly optimize the disentangle- ment scores, it seem beneficial compared to unsupervised training with supervised selection. The more labels are available the larger the benefit. Finding extremely sample efficient disentanglement metrics is however an important research direction for practical applications of disentanglement.
4.3
HOw ROBUST IS SUPERVISED TRAINING TO IMPRECISE LABELS?
Metric = MIG
In this section, we explore the performance and the robustness of the / methods compared to the U/S methods. In Figure we observe that imprecise labels do not significantly worsen the per- formance of both the supervised validation and the semi semi-supervised training. Sometimes the regularization induced by simplifying the labels appears improve generalization, arguably due a reduction in overfitting (see Figure 21 in the Appendix). We observe that the model selection metrics slightly more robust than the semi- supervised loss. This effect is more evident especially when only 100 labeled examples are available, see Figure 2 in the Appendix. How- ever, shown in Figure 4 (b-d), the semi-supervised approaches still outperform supervised model selection in 64.8% and 67.5% of the trials with 100 binned and noisy labels respectively. The only exception appears to be with partial labels, where the two approaches are essentially equivalent (50.0%) with 100 labeled examples and the semi-supervised improves (62.6%) only with 1000 labeled examples.
0.40
0.35
GESIEO = 1950200
0.25
velie
0.15
0.10
0.00
1
4 Model
6
Figure 5: Distribution of mod- els trained with different types of labels with 1000 samples, U/S validated with MIG. Leg- end: O=U/S perfect, 1=5² / perfect, 2-1/S binned, 3= binned, 4=U/S
These results show that the S2/S methods are also robust to imprecise labels. While the U/S methods appear to be more robust, 52 methods are still outperforming them.
Conclusion:
noisy, 5=5² /S noisy, 6=U/S
partial, 7=5² partial.
5 CONCLUSION
In this paper, we investigated whether very small number of labels can be sufficient to reliably learn disentangled representations. We found that existing disentanglement metrics can in fact be used to perform model selection models trained in completely unsupervised fashion even when the number of labels is very small and the labels are noisy. In addition, we showed that one obtain even better results if one incorporates the labels into the learning process using simple supervised regularizer. In particular, both unsupervised model selection and cemi-supervised training surprisingly robust to imprecise labels (inherent with human annotation) and partial
8

---

Published as a conference paper at ICLR 2020
labeling of factors of variation (in case not all factors can be labeled), meaning that these approaches are readily applicable in real-world machine learning systems. The findings of this paper provide practical guidelines for practitioners develop such systems and, as we hope, will help advancing disentanglement research towards more practical data sets and tasks.
Francesco Locatello is supported by the Max Planck ETH Center for Learning Systems, by ETH core grant (to Gunnar Rätsch), and by Google Ph.D. Fellowship. This work was partially done while Francesco Locatello was Google Research, Brain Team, Zurich.
Acknowledgments:
REFERENCES
Tameem Adel, Zoubin Ghahramani, and Adrian Weller. Discovering interpretable representations for both deep generative and discriminative models. In International Conference Machine Learning, 2018.
Francis Bach and Michael Jordan. Kernll independent component analysis. Journal of Machine Learning Research, 3(7):1-48, 2002.
Yoshua Bengio, Yann LeCun, et al. Scaling learning algorithms towards Al. Large-scale Kernel Machines, (4(5):1-41,200) 2007.
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE Transactions Pattern Analysis and Machine Intelligence, 35(8):1798- 1828, 2013.
Diane Bouchacourt, Ryota Tomioka, and Sebastian Nowozin. Multi-level variational autoencoder: Learning disentangled representations from grouped observations. AAAI Conference on Artificial Intelligence, 2018.
Christopher P Burgess, Irina Higgins, Arka Pal, Loic Matthey, Nick Watters, Guillaume Des- jardins, and Alexander Lerchner. Understanding disentangling in beta- VAE. arXiv preprint arXiv:1804.03599 2018
Tian Qi Chen, Xuechen Li, Roger Grosse, and David Duvenaud. Isolating sources of disentanglement in variational autoencoders. In Advances in Neural Information Processing Systems, 2018.
Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. Infogan: Interpretable representation learning by information maximizing generative adversarial nets. In Advances in Neural Information Processing Systems, 2016.
Brian Cheung, Jesse A Livezey, Arjun K Bansal, and Bruno A Olshausen. Discovering hidden factors of variation in deep networks. arXiv preprint 1rXiv:1412.6588 2014.
Taco Cohen and Max Welling. Learning the irreducible representations commutative lie groups. In International Conference on Machine Learning, 2014.
Pierre Comon. Independent component analysis, new concept? Signal Processing, 36(3):287-314, 1994.
Alexander D' Amour. On multi-cause approaches causal inference with unobserved counfounding: Two cautionary failure cases and a promising alternative. In International Conference on Artificial Intelligence and Statistics, 2019.
Zhiwei Deng, Rajitha Navarathna, Peter Carr, Stephan Mandt, Yisong Yue, Iain Matthews, and Greg Mori. Factorized variational autoencoders for modeling audience reactions to movies. In IEEE Conference on Computer Vision and Pattern Recognition, 2017.
Emily L Denton and Vighnesh Birodkar. Unsupervised learning of disentangled representations from video. In Advances Neural Information Processing Systems, 2017.
Cian Eastwood and Christopher KI Williams. A framework for the quantitative evaluation of disentangled representations. In International Conference Learning Representations, 2018.

---

Published as a conference paper at ICLR 2020
Vincent Fortuin, Matthias Hüser, Francesco Locatello, Heiko Strathmann, and Gunnar Rätsch. Deep self-organizztion: Interpretable discrete representation learning on time series. In Internationaa Conference Learning Representations, 2019.
Marco Fraccaro, Simon Kamronn, Ulrich Paquet, and Ole Winther. disentangled recognition and nonlinear dynamics model for unsupervised learning. In Advances in Neural Information Processing Systems, 2017.
Muhammad Waleed Gondal, Manuel Wüthrich, Djordje Miladinovié, Francesco Locatello, Martin Breidt, Valentin Volchkov, Joel Akpo, Olivier Bachem, Bernhard Schölkopf, and Stefan Bauer. On the transfer of inductive bias from simulation the real world: new disentanglement dataset. In (To appear) in Advances in Neural Information Processing Systems, 2019.
Ian Goodfellow, Honglak Lee, Quoc V Le, Andrew Saxe, and Andrew Y Ng. Measuring invariances in deep networks. In Advances in Neural Information Processing Systems, 2009.
Ross Goroshin, Michael F Mathieu, and Yann LeCun. Learning linearize under uncertainty. In Advances in Neural Information Processing Systems, 2015.
Luigi Gresele, Paul K. Rubenstein, Arash Mchrjou, Francesco Locatello, and Bernhard Schölkopf.
The incomplete rosetta stone problem: Identifiability results for multi-view nonlinear ica. In Conference on Uncertainty Artificial Intelligence (UAI), 2019.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-VAE: Learning basic visual concepts with constrained variational framework. In International Conference on Learning Representations, 2017a.
Irina Higgins, Arka Pal, Andrei Rusu, Loic Matthey, Christopher Burgess, Alexander Pritzel, Matthew Botvinick, Charles Blundell, and Alexander Lerchner. Darla: Improving zero-shot transfer in reinforcement learning. In International Conference Machine Learning, 2017b.
Irina Higgins David Amos, David Pfau, Sebastien Racaniere, Loic Matthey, Danilo Rezende, and Alexander Lerchner. Towards definition of disentangled representations. arXiv preprint arXiv: 2018.
Geoffrey E Hinton, Alex Krizhevsky, and Sida D Wang. Transforming auto-encoders. In International Conference on Artificial Neural Networks, 2011.
Jun-Ting Hsieh, Bingbin Liu, De-An Huang, Li F Fei-Fei, and Juan Carlos Niebles. Learning to decompose and disentangle representations for video prediction. In Advances in Neural Information Processing Systems, 2018.
Wei-Ning Hsu, Yu Zhang, and James Glass. Unsupervised learning of disentangled and interpretable representations from sequential data. In Advances in Neural Information Processing Systems, 2017.
Aapo Hyvarinen and Hiroshi Morioka. Unsupervised feature extraction by time-contrastive learning and nonlinear ica. In Advances in Neural Information Processing Systems, 2016.
Aapo Hyvärinen and Petteri Pajunen. Nonlinear independent component analysis: Existence and uniqueness results. Neural Networks, 1999.
Aapo Hyvarinen, Hiroaki Sasaki, and Richard E Turner Nonlinear ica using auxiliary variables and generalized contrastive learning. International Conference on Artificial Intelligence and Statistics, 2019.
Christian Jutten and Juha Karhunen. Advances in nonlinear blind source separation. International Symposium Independent Component Analysis and Blind Signal Separation, pp. 245-256, 2003.
Theofanis Karaletsos, Serge Belongie, and Gunnar Rätsch. Bayesian representation learning with oracle constraints. arXiv preprint 1506.000000500
2015.
10

---

Published as a conference paper at ICLR 2020
Hyunjik Kim and Andriy Mnih. Disentangling by factorising. In International Conference Machine Learning, 2018.
Diederik P Kingma and Max Welling, Auto-encoding variational Bayes. In International Conference on Learning Representations, 2014.
Diederik P Kingma, Shakir Mohamed, Danilo Jimenez Rezende, and Max Welling. Semi-supervised learning with deep generative models. Advances Neural Information Processing Systems, 2014.
Jack Klys, Jake Snell, and Richard Zemel. Learning latent subspaces in variational autoencoders. In Advances in Neural Information Processing Systems. 2018.
Tejas D Kulkamni, William F Whitney, Pushmeet Kohli, and Josh Tenenbaum. Deep convolutional inverse graphics network. In Advances in Neural Information Processing Systems, 2015.
Abhishek Kumar, Prasanna Sattigeri, and Avinash Balakrishnan. Variational inference of disen- tangled latent concepts from unlabeled observations. In International Conference on Learning Representations, 2018.
Brenden M Lake, Tomer D Ullman, Joshua Tenenbaum, and Samuel Gershman. Building machines that learn and think like people. Behavioral and Brain Sciences, 40, 2017.
Yann LeCun, Fu Jie Huang, and Leon Bottou. Learning methods for generic object recognition with invariance to pose and lighting. In IEEE Conference on Computer Vision and Pattern Recognition, 2004.
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. Nature, 521(733)):436, 2015.
Karel Lene and Andrea Vedaldi. Understanding image representations by measuring their equivariance and equivalence. In IEEE Conference on Computer Vision and Pattern Recognition, 2015.
Francesco Locatello, Damien Vincent, Ilya Tolstikhin, Gunnar Rätsch, Sylvain Gelly, and Bernhard Schölkopf. Competitive training of mixtures of independent deep generative models. In Workshop at the 6th International Conference Learning Representations (ICLR), 2018.
Francesco Locatello, Gabriele Abbati, Tom Rainforth, Stefan Bauer, Bernhard Schölkopf, and Olivier Bachem. On the fairnsss of disentangled representations. In (To appear) in Advances in Neural Information Processing Systems, 2019a
Francesco Locatello, Stefan Bauer, Mario Lucic, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. Challenging common assumptions in the unsupervised learning of disentangled represen- tations. In International Conference on Machine Learning, 2019b.
Emile Mathieu, Tom Rainforth, N. Siddharth, and Yee Whye Teh. Disentangling disentanglement in variational auto-encoders. arXiv preprint arXiv: 01/1182222233... 2018.
Michael F Mathieu, Junbo J Zhao, Aditya Ramesh, Pablo Sprechmann, and Yann LeCun. Disentan- gling factors of variation in deep representation using adversarial training. In Advances Neural Information Processing Systems, 2016.
Siddharth Narayanaswamy, Brooks Paige, Oil-Willem Van Meent, Alban Desmaison, Noah Goodman, Pushmeet Kohli, Frank Wood, and Philip Torr. Learning disentangled representations with semi-supervised deep generative models. In Advances in Neural Information Processing Systems, 2017.
Judea Pearl. Causality. Cambridge University Press, 2009.
Jonas Peters, Dominik Janzing, and Bernhard Schölkopf. Elements of Causal Inference Foundations and Learning Algorithms. Adaptive Computation and Machine Learning Series. MIT Press, 2017.
Scott Reed, Kihyuk Sohn, Yuting Zhang, and Honglak Lee. Learning to disentangle factors of variation with manifold interaction. In International Conference Machine Learning, 2014.

---

Published as a conference paper at ICLR 2020
Scott Reed, Yi Zhang, Yuting Zhang, and Honglak Lee. Deep visual mmaogy-makinng In Advances in Neural Information Processing Systems, 2015.
Karl Ridgeway and Michael C Mozer.
Learning deep disentangled embeddings with the f-statistic loss. In Advances Neural Information Processing Systems, 2018.
Michal Rolinek, Dominik Zietlow, and Georg Martius. Variational autoencoders recover pea directions (by accident). In Proceedings IEEE Conf. on Computer Vision and Pattern Recognition, 2019.
P. K. Rubenstein, B. Schölkopf, and I. Tolstikhin. Learning disentangled representations with wasser- stein auto-encoders. In Workshop at the 6th International Conference on Learning Representations (ICLR), 2018.
Adrià Ruiz, Oriol Martinez, Xavier Binefa, and Jakob Verbeek. Learning disentangled representations with reference-based variational autoencoders, arXiv preprint urXiv:1901.08534 2019.
Jürgen Schmidhuber. Learning factorial codes by predictability minimization. Neural Computation, 4(6):863-879, 1992.
Bernhard Schölkopf, Dominík Janzing, Jonas Peters, Eleni Sgouritsa, Kun Zhang, and Joris Mooij. On causal and anticausal learning. In International Conference Machine Learning, 2012.
Spirtes, C. Glymour, and R. Scheines. Causation, prediction, and search. MIT Press, 2000.
Raphael Suter, Djordje Miladinović, Stefan Bauer, and Bernhard Schölkopf. Interventional robustness of deep latent variable models. In (To appear) International Conference on Machine Learning. 2019.
Valentin Thomas, Emmanuel Bengio, William Fedus, Jules Pondard, Philippe Beaudoin, Hugo Larochelle, Joelle Pineau, Doina Precup, and Yoshua Bengio. Disentangling the independently con- trollable factors of variation by interacting with the world. Learning Disentangled Representations Workshop at NeurlPS, 2017.
Michael Tschannen, Olivier Bachem, and Mario Lucie. Recent advances in autoencoder-based representation learning, arXiv preprint arXiv:1812.0500 1812.0506,, 2018.
Sjoerd van Steenkiste, Francesco Locatello, Jürgen Schmidhuber, and Olivier Bachem. Are disentan- gled representations helpful for abstract visual reasoning? In (To appear) in Advances Neural Information Processing Systems, 2019.
William F Whitney, Michael Chang, Tejas Kulkarni, and Joshua B Tenenbaum. Understanding visual concepts with continuation learning. arXiv preprint arXiv: arXiv:/602.06822 2016.
Jimei Yang, Scott E Reed, Ming-Hsuan Yang, and Honglak Lee. Weakly-supervised disentan- gling with recurrent transformations for 3D view synthesis. In Advances in Neural Information Processing Systems, 2015.
Li Yingzhen and Stephan Mandt. Disentangled sequential autoencoder. In International Conference on Machine Learning, 2018.
12

---

Published as a conference paper at ICLR 2020
Method
Type
SAP 100
SAP 1000
MIG 100
MIG 1000
DCI 100
DCI 1000
/ permuted U/S perfect
54.8%
55.6%
33.9%
48.0%
34.9%
48.0%
3-VAE
45.2%
44.4%
66.1%
52.0%
65.1%
52.0%
/2 permuted S/S perfect
48.0%
44.4%
39.5%
44.4%
41.1%
44.4%
FactorVAE
52.0%
55.6%
60.5%
55.6%
58.9%
55.6%
/ permuted U/S perfect
64.8%
55.5%
34.4%
42.3%
36.8%
43.5%
8-TCVAE
35.2%
44.5%
65.6%
57.7%
63.2%
56.5%
/ permuted 56.8%
61.9%
30.5%
46.8%
36.4%
46.5%
DIP-VAE-I
U/S perfect
43.2%
38.1%
69.5%
53.2%
63.6%
53.5%
Table Removing the ordering information significantly worsen the performances of the 52/5 on each method. The standard deviation is between 3% and 5% and can be computed as /p(1 p)/
A ORDERING AS AN INDUCTIVE BIAS
We emphasize that the considered supervised regularizer R. uses an inductive bias in the sense that assumes the ordering of the factors of variation matter. This inductive bias valid for many ground truth factors of variation both in the considered data sets and the real world (such as spatial positions, sizes, angles even color). We argue that such inductive biases should generally be exploited whenever they are available, which is the case if we have few manually annotated labels. To better understand the role of the ordinal information, we test its importance by removing from the labels (via random permutation of the label order) before applying our semi-supervised approach. We find that this significantly degrades the disentanglement performance, i.e., ordinal information is indeed an important inductive bias. Note that permuting the label order should not harm the performance on test metries as they are invariant to permutations.
In this section, verify that that the supervised regularizer considered relies on the inductive bias given by the ordinal information present in the labels. Note that all the continuous factors of variation binned in the considered data sets. We analyze how much the performance of the semi-supervised approach degrades when the ordering information is removed. To this end, we permute the order of the values of the factors of variation. Note that after removing the ordering information the supervised loss will still be at its minimum if r(x) matches z. However, the ordering information is now useless and potentially detrimental as it does not reflect the natural ordering of the generative factors. We also remark that none of the disentanglement metrics make use of the ordinal information, so the performance degradation cannot be explained by fitting the wrong labels. In Figure 6 we observe that the 52 52/S approaches heavily rely on the ordering information and removing significantly harms the performances of the test disentanglement metries regardless of the fact that they are blind to ordering. This result is confirmed in Table 1 where we compute how often each $2 5²/S method with permuted labels outperforms the corresponding U/S on random disentanglement metric and data set with perfect labels. We observe that in this case U/S S is superior most of the times but the gap reduces with more labels.
Conclusions:
Imposing suitable inductive bias (ordinal structure) the ground- truth generative model the form of supervised regularizer is useful for disentanglement if the assumptions about the bias are correct. If the assumptions are incorrect, there is no benefit anymore over unsupervised training with supervised model selection (which is invariant to the ordinal structure).
B ARCHITECTURES AND DETAILED EXPERIMENTAL DESIGN
The architecture shared across every method is the default one in the aisentaagllemeent. which we describe here for completeness in Table 2 along with the other fixed hyperparameters in Table 4a and the discriminator for total correlation estimation in Factor Table 4bl with hyperparameters in Table 4c The hyperparameters that were swept for the different methods can be found in Table 3 All the hyperparameters for which we report single values were not varied and are selected based on the literature.
13

---

Published as a conference paper at ICLR 2020
syeic
Ld
1.00
00.0%
E.0
6.8
0.3
8:3
64
0.90
0.8
0.00
014
3:53
0.20
012
6.55
0.0
0.85
1.0
1.3
033
0.40
0.3
605
034
Ditaes Smrul RCCD
3 6:15
sac-
022
0.41
67c
03-
0.70
0.4
0.10
0.m9
3,005
65-
0.00
0.00
0.6
3900
032
0.80
0.25
3.-
4.-
0.55
0.99
0.50
66
0.05
01
0.80
1900
Vade
Figure 6: Violin plot showing the effect of removing the inductive bias given by the ordering of the labels semi-supervised methods. Models are abbreviated 0=U//S with perfect labels, 1=52/ / with perfect labels, 2=52/ /S training with permuted labels.
Table Encoder and Decoder architecture for the main experiment.
Encoder
Decoder
Input: 64 64x number of channels
Input: 10
4 x 4 conv, 32 ReLU, stride 2
FC, 256 ReLU
4 conv, 32 ReLU, stride
FC, 4 x 64 ReLU
2 x 2 conv, 64 ReLU, stride 2
4 x upconv, 64 ReLU, stride 2
x 2 conv, 64 ReLU, stride
4 x upconv, ReLU, stride 2
FC 256, FC 2 x 10
4 x upconv, 32 ReLU, stride 2
4 x upconv, number of channels, stride
14

---

Published as a conference paper at ICLR 2020
Table 3: Hyperparameters explored for the different disentanglement methods.
Model
Parameter
Values
B-VAE
B
[1
4,
6,
8,
16
S2/S B-VAE
B
1.
2,
4,
6.
8,
16
Ysup
1.
2,
4,
6,
8,
FactorVAE
7
[0,
20,
30,
40, 50, 100]
5²/S Factor VAE
7
10, 20,
30,
40,
50, 100
ysup
10, 20,
30,
40,
50, 100
DIP-VAE-I
Rod
1. 2, 5, 10,
20, 50]
Xd
10Aod
8 DIP-VAE-I
Aod
[1, 2, 5, 10, 20, 50]
Ad
10yod
Ysup
[1, 2, 5,
10,
20,
50]
3-TCVAE S /2/S 3-TCVAE
B
1.
2,
4,
6,
8,
10
B
6,
8,
10
Ysup
1.
2,
4,
6,
8, 10
Table 4: Other fixed hyperparameters.
(a) Hyperparameters common to
(b) Architecture for the dis- criminator in Factor VAE.
(c) Parameters for the discrimina- tor in FactorVAE.
all considered methods.
Parameter
Values
Discriminator
Parameter
Values
Batch size
64
FC, 1000 Ieaky ReLU
Batch size Optimizer
64
Latent space dimension
FC, 1000 leaky ReLU
Adam
Optimizer
Adam
FC, 1000
leaky ReLU
Adam: betal
0.5
Adam: betal
0.9
FC, 1000
leaky ReLU
Adam: beta2
0.9
Adam: beta2
0.999
FC, 1000 leaky ReLU
Adam: epsilon
1e-8
Adam: epsilon
le-8
FC, 1000 leaky ReLU
Adam: learning rate
0.0001
Adam: learning rate
0.0001
Decoder type
Bernoulii 300000
FC, 2
Training steps
C DETAILED PLOTS FOR SECTION 3
In Figure we compute the rank correlation between the validation metrics computed on 100 samples and the test metrics each data set. In Figure 8 observe that the correlation improves if we consider 1000 labeled examples. Figures 10 to 15 show the rank correlation between the validation metrics with binned/noisy/partial observations and the test metrics with exact labels for both 100 and 1000 examples. These plots are the extended version of Figure showing the results on all data sets for both sample sizes.
In Figure g we plot for each unsupervised model its validation MIG with 100 samples against the DCI test score on dSprites. We can see that indeed there strong linear relationship.
D DETAILED PLOTS FOR SECTION 4
D. DOES SUPERVISION HELP TRAINING?
In Figure 18 we plot the median each score across the draws of the labeled subset achieved by the best models each data set (using 100 vs 1000 examples). For the U/S models validation (MIG has a higher rank correlation with most of the testing metric than other validation metrics, Figure 2. This plot extends Figure (left) to all data set and test score.
MIG for
15

---

Published as a conference paper at ICLR 2020
Dajaset
(Sprites
Datapet y SrçalINoRE
Dataset Çars3D
Datqset
Shapes3D
(M5300)
(MS100)
(0)
(MS1100)
(H5300)
(MS10D)
MIO
MIO (MS1100)
DCI (NS100)
68
88
58
DCI (MS100)
17 39
DIC (MS100)
18
13
DCI (MS100)
56
64
93
BelavV Score
FactorVi Score
NI
Disntannlener
Medularity
SKP
BetoVAEE Score
MB
DCI Disemannlemeer
medularty
MB
DC Dbentannleeeer
Modularity
SAP
BeravAE Scora
TactoriiE Score
SMP
BetavAE
Eactorizz
Figure 7: Rank correlation of validation metrics computed with 100 examples and metrics on each data set.
Dataset
4Spritas
Datapet
SmallyORg
Dapasoe
CarsaD
Datasat
Shaper30
(M51000)-
65
50
69
69
(MS10CO)-
-40
.50
23
36
33
SAP
16
17
(ME1000)
49
13
(M51000)
84
76
94
70
(MS1000)
18
97
66
IMS1000)
64
98
58
(MS1000)
65
100
82
(MS1000)
Besivi Score
18G
DC Diventanammnnt
SAP
BenwAr Soorn
MG
Diventangaamen
Modiilaiit
FactorviE Score
Disentangement
ioddllaiity
BetevAt Score
FactorVAE Score
MB
DCI bicantanglmmnn
SAP
FactorVAE
Figure 8: Rank correlation of validation metrics computed with 1000 examples and test metrics each data set.
3-VAE
FactorvAE
S-TCAAE
DIP-VAE-I
MIG (M550000
SSP 0105
0.3
(MS500)
0.8
Score
Dicomtngtemmnn
Modulantt
Figure 9: Scatter plot of validation and test metrics on dSprites before model selection. The validation metrics computed with 100 examples.
16

---

Published as a conference paper at ICLR 2020
Dataset dSprttee
Pataper
SmaliNOR
Dataser
(arsa
Datpsot
Spapee3D
SAP (NS100)-
30
31
31
0
SAP (MS100)
:18
3
11
SAP (M5100)
-10
3
SAP (MS100)
15
19
23
12
MIS (MS100)
77
94
80
14
MIG (MS100)
80
17
54
MIG (MS100)
18
32
17
15
NIG (MS100)
51
96
65
33
OCI (MA3100)
78
92
64
-13
DO (MS1 00)
2
60
74
-10
41
DCI (MS100)
24
30
54
61
53
DCI (MSSL00)
55
81
98
49
56
FoctorVAE Score
MMG
DC Saannnnggmmii
SAP
BetAVA Soore
FoctorVAs Score
IMG
bc Dissattinglemn
Mobillrtt
SAP-
BetoVA Scor
Dd Dizentingeems
Modilaaty
DC Intentanglemeent
Modilerit
Belavnt
FoctorVAE
Figure 10: Rank correlation validation metrics and test metrics. Validation metrics are computed with 100 examples with labels binned to five categories.
Dataset
@Sprites,
pataset + SmallyORi
Dapaset
parçaD
pataset: Spapes30
SAP (MS1000)
75
71
85
SAP (MS1000)
52
39
41
40
SAP (MS1000)
12
14
SAP (MS1000)
58
03
26
MIS (N51000)
53
81
95
91
MIG (MS1900)
MIG (MS1000)
43
59
40
NIG (MS1000)
52
07
39
80
23
DCI (MS1000)-
83
82
91
98
5
DOI (M51000)
88
-10
DCI (MS1000)
40
74
DCI (M51000)
50
8
81
100
49
FactoruAE Scone
MB
DO Divertangieeent
Modulartty
SAP
Wetavie score
MG
DO Disertangiement
SAP
BebaVAE Score
MG
Dicentanglenenn Modularit
SAP
Factervie Score
MG
DC Divertangie weeeteeeeeeee.....
Medulaiity
BelaViE
FoctorViE
Factorvil
BetaviE
Figure 11: Rank correlation validation metrics and test metrics. Validation metrics are computed with 1000 examples with labels binned to five categories.
Dataset
dSprites
Datape:
SmallNORg
Dataset
CarsaD
Datpset
Shapes
SAP (M5100)-
SAP (MS100)
SAP (M5100)
6
-1
SAP (MS100)
10
MIS (MS100)-
69
110
MIG (MS180))
56
MIG (MS1001
12
35
15
MIG (MS100)
64
26
DCI (M81100)
66
83
63
13
61
DO (MS1 00)
57
74
40
DCI (MS1000)
25
28
11
DCI (MS1100)
80
97
FactonAX score
MG
SAP
FacoovA Score
MG
DC Dicentinnle eennntnngeeeent
Upa.uarrty
SAP
BateVAS Score
MO
DC Dicentanglemssn
updularity
SAP
DC pntenttnglement
Modulerit
DO Diventangleenn
BetAVAL
Focton/AE
FactorVAE BeteVAL
Figure 12: Rank correlation validation metrics and test metrics. Validation metrics are computed
with 100 examples where each labeled factor has 10% chance of being random.
Datasat
@Sprites
Patasot
SmallNORB
Dapasot
FarsaD
Pataset
Shapps3D
SAP IM51000)
75
71
85
22
SAP (MS1000)
-39
52
39
41
40
SAP (MS1000)
12
14
SAP (MS1000)
58
33
26
MIC (M51000)
81
95
MIG (MS1000)
MIG (M51000)
46
NIG (MS1000)
52
39
25
DCI [m811000
83
82
94
98
6
DO (MS) (00)
-10
DCI (MS1000)
A0
90
DCI (M611000)
50
81
100
49
FacturvaF scare
DO Doertanggeeeem
Modulanty
SAP
DelaviE Score
MIG
DO Disertanglemem
Modularity
SAP
BebivE Score
MGG
DC Doertanglement
Mosialeriy
50P
Faztorvil score
MG
DO Diven anglement
Modulaiit
BebiViE
Factorval
BetaviE
Figure 13: Rank correlation of validation metrics and test metrics. Validation metrics are computed with 1000 examples where each labeled factor has 10% chance of being random.
17

---

Published as a conference paper at ICLR 2020
Dapaset eSprites
Datape: SmalnNORg
Dakaset çarspp
Datpeet
Spape:3D
SAP (NS1000)-
(MS/100)
(MS5000)
(MS100))
MIC (MS10n))-
(MS1100)
(MS3000)
NIG: (MS100))
DCI (MS100))
26
DCI (MS100)
DCI (MS100)
8
14
DCI (MS100))
84
FactorvaE Score
MI
DC Disentnngeman
SAP
BeLAVA Score
Factorvns Score
NIG
DC Disentanni pocnttaniommnnt
Modulenty
SAP
Retavar Score
DO Disentangg emmenneeeentt.....
Modulaity
DO Disentangient meentnnggeeeett...
Modulaiity
50P
Factorvas
DetaMiE
FactorvAE
Figure 14: Rank correlation of validation metrics and test metrics. Validation metrics are computed
with 100 examples with only two factors labeled.
Dataset
4Sprites,
pataset
SmallyoRs
Dapaset
parsa
pataset
Spapes30
(MS1000)
(MS1000)
155
(MS1000)
(MS1000))
14
MIS (NS1000)
(MS1800)
(MS1000))
(MS10C0)
DCI (MS1000)
DO (MS1000)
2
DCI (MS1000)
51
DCI (MS1000)
85
8
FactorvAF Scare
NIG
DCI Diventangeen emeen.
Modulanty
SAP
Beavvn Score
NIG
DC Deentrngemen emmnt
SAP
BecaviEE Sooe
NI
DCI Disentanneemen eemmen.
Masilliit
SAP
BesaiiE Score
FoctorvtE sciee
NIG
DCI Disentann tement
Modulaitt
BecaVAE
FactorvAs
Figure 15: Rank correlation of validation metries and test metrics. Validation metrics are computed with 1000 examples with only two factors labeled.
In Table we compute how often each 5²/5 method outperforms the corresponding U/S on random disentanglement metric and data set. We observe that S S2/S often outperforms U/S especially when more labels are available.
In Figure To can be observed that with 1000 samples the semi-supervised method is often better than the corresponding U/S even using the test MIG computed with 000 samples for validation. We conclude that the semi-supervised loss improves the training and transfer better to different metrics than the MIG. In Figure [7] we observe similar trends if we use the test DCI Disentanglement with 10 000 samples for validation of the U/S methods.
in Figure 19 we observe that increasing the supervised regularization makes that the matrix holding the mutual information between all pairs of entries of z and rxx) closer to diagonal. This plots extend Figure 3 to all data sets.
In Figure 20 we compare the median downstream performance after validation with 100 1000 samples. Finally, we observe in Table 6 that cemi-supervieed methods often outperforms U/S in downstream performance, especially when more labels are available.
D.2 WHIAT HAPPENS IF WE COLLECT IMPRECISE LABELS?
In Figures 21 and 22
observe that imprecisions do not significantly worsen the performance of both the supervised validation and the semi- superviuer training. These plots extend Figure 5 to both sample sizes, all test scores and data sets.
In Tables 9 we show how often each /S/S method outperforms the corresponding U/S on a random disentanglement metric and data set with the different types of imprecise labels (binned/noisy/partiaa))
18

---

Published as a conference paper at ICLR 2020
6.5
646
6085
0.9
0.5
646
040
601s
Dateec Softa
0.91
0.0
0.36
Oetant Smot 2013
620
0.4
3.4.
3.0
0:30
0.30
375
0.91
199
1.3
630
0.4
0:3
Detaodd Can30
3.9
6.66
1.0
3.0
3.11
533
0.30
050
030
3.87
645
339
600
030 600
Figure 16: Test scores of the U/S methods using the test MIG as validation and the S /S models with 1000 labeled examples. Legend: O=U/S-B-VAE -=Ü//S--TTCVA. 2=U//S-FactorVAE 3=U/S- DIP- VAE-I, 4=5²/S /-S//S--VAE, S --S//S--VVE T=S"/S-FactorVal
Method
Type
SAP 100
SAP 1000
MIG 100
MIG 1000
DCI 100
DCI 1000
$2/S
72.6%
79.2%
53.9%
74.2%
53.9%
69.2%
8-VAE
U/S
27.4%
20.8%
46.1%
25.8%
46.1%
30.8%
52/5
71.5%
79.4%
64.5%
75.2%
68.5%
77.6%
FactorVAE
U/S
28.5%
20.6%
35.5%
24.8%
31.5%
22.4%
$2/8
79.5%
80.6%
58.5%
75.0%
62.9%
74.4%
3-TCVAE
U/S
20.5%
19.4%
41.5%
25.0%
37.1%
25.6%
52/S
81.6%
83.5% 16.5%
64.9%
74.8%
67.7%
70.5%
DIP-VAE-I
U/S
18.4%
35.1%
25.2%
32.3%
29.5%
Table 5: Percentage how often 5²/S improves upon U/S on for each approach separately. The standard deviation is between 3% and 5% and can be computed as /p(1 -0pp)) pp)/120.
19

---

Published as a conference paper at ICLR 2020
Seta,
848-4
006
3080
0.90
0.00
0.90
Dataec Softo
0.80
0.40
0.75
3.0
645
0.0
0.55
Oetant Soot WO13
024-
0.40
4.90
538
0.00
1.00
0.00
1.00
37
0.5
Delane Cana0
0:3
0.96
66
8.45
0.00
0.30
1.30
1.00
0.900
0.00
35
010
045
3350
0.00
31
0.30
0,0
Figure 17: Test scores of the U/S methods using the test DCI as validation and the S2/S models with 1000 labeled examples. Legend: Q=U/S-B-VAE. I=U//S-B-TCVAE 2=U//S-FactorVAE, 3=U/S- DIP- VAE-I, 4=5² /S-3/V-.V /=S//SDIP-/ASAAAAAAAAA 6=5* 7=5²/ /S-Facttrr/A
Method
Type
SAP 100
SAP 1000
MIG 100
MIG 1000
DCI 100
DCI 1000
$2/S
70.0%
75.6%
53.8%
75.0%
43.8%
71.9%
8-VAE
U/S
30.0%
24.4%
46.2%
25.0%
56.2%
28.1%
52/5
61.2%
71.9%
62.5%
78.1%
63.8%
71.2%
FactorVAE
U/S
38.8%
28.1%
37.5%
21.9%
36.2%
28.8%
$2/8
70.6%
72.7%
51.9%
71.9%
55,6%
67.5%
3-TCVAE
U/S
29.4%
27.3%
48.1%
28.1%
44.4%
32.5%
52/S
71.9%
80.6%
50.6%
75.6%
50.6%
65.0%
DIP-VAE-I
U/S
28.1%
19.4%
49.4%
24.4%
49.4%
35.0%
Table Percentage of how often 52/S improves upon U/S on the downstream performance. The standard deviation is between 3% and 4% and can be computed as /p(1 - p)/160.
20

---

Published as a conference paper at ICLR 2020
Model
6918 A VAE
87 FactorvRE
I- VAE
Venr BETANSE Store
HeTK Farperid Styre
Merk
Mertic Mockraryy
MetrY SAP
6
0.09
035
0.30
0.25
64
0.5
10
115
0.33
0.0
0.25
0.33
0.68
0.00
9.85
3.06
0.5
BLOM BERD
100
02
01
65
0.30
3.9
03
1.5
12
60
0.16
0.32
832
145
47
0.68
6.70
532
0.5
638
0.5
0.7
COLID cempp
0.10
0.2
3.93
0.50
3.54
3/60
3.5
0.5
0.91
145
00
32
0.52
475
4.5
310
0.5
644
3.92
69-
3.0
1.00
0.3
centing 2wOrd
0.8
0.2
2.5
634
335
0.00
1.0
100
3.8
35
10
33
355
3.55
0.50
3.55
3.50
6.55
1
0.6
Figure 18: Median across the draws of the labeled data of the scores each data set after validation with 100 and 1000 labeled examples. U/S were validated with the MIG.
#'/SEartrVAE
/DD---AA--
0.025
Datacet dSpritos
Datzsot 5ma INORB
0.000
0attset C202D
0.25
Datazet Shapo530
0.34
0.045
0.050
0.32
0.235
3.08
0.040
0.30
3.325
0.000
2.04
0.025
0.26
0.2
08
0.2
0.8
02
0.6
0.8
0.0
02
0.8
Rogulanizttion strength supervised
Regu Rlguubrizzttion strength supervied
Regu Riguirrization strength supervised
Reguiriizatio strength supervised
Figure 19: Increasing the supervised regularization strength makes the matrix of pairwise mutual
information (z,, rx))) closer to diagonal.
Method
Type
SAP 100
SAP 1000
MIG 100
MIG 1000
DCI 100
DCI 1000
52/S U/S
66.9%
76.3%
50.4%
75.2%
44.5%
74.6%
3-VAE
33.1%
23.7%
49.6%
24.8%
55.5%
25.4%
52/S
72.4%
67.9%
60.5%
63.2%
56.8%
62.4%
FactorVAB
U/S
27.6%
32.1%
39.5%
36.8%
43.2%
37.6%
52/5
79.2%
77.9%
58.5%
74.0%
61.2%
72.7%
8-TCVAE
U/S
20.8%
22.1%
41.5%
26.0%
38.8%
27.3%
$2/S
67.7%
75.8%
57.4%
71.8%
53.4%
69.6%
DIP-VAE-I
U/S
32.3%
24.2%
42.6%
28.2%
46.6%
30.4%
Table 7: Percentage of how often S2/S improves upon U/S for each method random disentan- glement score and data set with binned labels. The standard deviation is between 3% and 5% and can be computed as / /((1 p)/120.
21

---

Published as a conference paper at ICLR 2020
Mo3d
UIN TOWNE
Superoed
E30
3130
A1G
332
44
020
M
1.4
032
0.2
0.5
04
0.5 a a.s
024
030
215
030 6A5
955 02) 875
631
0.5
02
04
012
415 620 322
023 635 0.5
03
30
50
035 829
325
039
2.0 6.0
CB6 085 2
AM4 0.64
Figure 20: Comparison of the median downstream performance after validation with 100 vs. 1000 examples on each data set. The downstream tasks are: cross-validated Logistic Regression (LR) and Gradient Boosting classifier (GBT) both trained with 10, 100, 1000 and 10 000 examples.
032
083
0.2
0.04
DeaI:: CAROO
0.00
1.00
0.5
08C
Model
Model
Model
Mode
Modell
Figure 21: Distribution models trained with perfect and imprecise labels with 100 samples, U/S validated with MIG. Legend: 0=U/S perfect, 1=52/5 /S perfect, 2-U/S binned, 3=5² / binned, 4=U/S noisy, 5=S² /S noisy, 6-U/S partial, 7=S² / partial.
22

---

Published as a conference paper at ICLR 2020
011
039
2.30
033
0.70
1.5
235
6.0
1.99
055
634-
Outaunt Seuehoo
645
623
0.35
6:0
1.5
6.9
0.00
109
144
000
125
10009
200055
233
0.6
0.0
M0
Model
Model
Model
Figure 22: Distribution of models trained with perfect and imprecise labels with 1000 samples, U/S validated with MIG. Legend: 0=U/S perfect, 1=52 / perfect, 2-1// binned, 3=5² /8 binned, 4=U/S noisy, 5=S²/5 /S noisy, 6=U/S partial, 7=S²/5 partial.
Method
Type
SAP 100
SAP 1000
MIG 100
MIG 1000
DCI 100
DCI 1000
$2/S
71.0%
80.2%
49.6%
70.5%
53.5%
72.1%
3-VAE
U/S
29.0%
19.8%
50.4%
29.5%
46.5%
27.9%
52/8
70.9%
75.4%
63.2%
67.5%
64.8%
64.6%
FactorVAE
U/S
29.1%
24.6%
36.8%
32.5%
35.2%
35.4%
$2/5
81.3%
82.7%
63.3%
72.9%
52.3%
73.5%
8-TCVAE
U/S
18.7%
17.3%
36.7%
27.1%
47.7%
26.5%
52/S
79.5%
77.2%
53.0%
63.7%
51.9%
62.2%
DIP-VAE-I
U/S
20.5%
22.8%
47.0%
36.3%
48.1%
37.8%
Table 8: Percentage of how often /S improves upon U/S for each method on a random disentan- glement score and data set with noisy labels. The standard deviation between 3% and 5% and can be computed as /p(1 - p2/120.
23

---

Published as a conference paper at ICLR 2020
Method
Type
SAP 100
SAP 1000
MIG 100
MIG 1000
DCI 100
DCI 1000
52/S
62.7%
65.6%
34.1%
47.2%
45.6%
56.0%
3-VAE
U/S
37.3%
34.4%
65.9%
52.8%
54.4%
44.0%
$2/S
59.2%
68.8%
46.4%
63.7%
50.4%
63.2%
FactorVAB
U/S
40.8%
31.2%
53.6%
36.3%
49.6%
36.8%
$2/5
69.6%
75.8%
37.0%
68.8%
49.2%
66.9%
8-TCVAE
U/S
30.4%
24.2%
63.0%
31.2%
50.8%
33.1%
82/8
59.2%
72.8%
48.4%
61.6%
41.4%
55.5%
DIP-VAE-I
U/S
40.8%
27.2%
51.6%
38.4%
58.6%
44.5%
Table Percentage of how often S / improves upon U/S for each method on a random disentan- glement score and data set with partial labels (only two factors observed). The standard deviation is between 3% and 5% and can be computed as /p(1 p)/120.
24


Reviews:
[{"review_id": "ryxyH98XYr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "After rebuttal edit:\nNo clarifications were made, so I keep my score as is.\n\n------------------------------------------------------\nClaims: Explicitly requiring a small number of labels allows for successful learning of disentangled features by either using them as a validation set for hyper parameter tuning, or using them as a supervised loss. \n\nDecision: Reject. This paper needs a substantial rewrite to make clear what specific contributions are from the multitude of experiments run in this study. As is, the two contributions stated in the introduction are both obvious and not particularly significant -- that having some labels of the type of disentanglement desired helps when used as a validation set and as a small number of labels for learning a disentangled representation space. There are no obviously stated conclusions about which types of labels are better than others (4.2). Section 3.2 seems to have some interesting findings that small scale supervision can help significantly and fine-grained labeling is not necessarily needed, but I don't understand why that finding is presented there when Fig. 4 seems to perform a similar experiment on types of labels with no conclusion based on its results. Conclusion sentence of 4.3 is hard to decipher, but I assume is just saying S^2/S beats U/S even when S^2/S is subject to noisy labels. Overall, I find it very difficult to absorb the huge amount of results and find the analysis not well presented.\n\n", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "r1x0CupCYH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper presents evidence that even a tiny bit of supervision over the factors of variation in a dataset presented in the form of semi-supervised training labels or for unsupervised model selection, can result in models that learn disentangled representations. The authors perform a thorough sweep over multiple datasets, different models classes and ways to provide labeled information. Overall, this work is a well executed and rigorous empirical study on the state of disentangled representation learning. I think experimental protocol and models trained models will prove extremely useful for future work and advocate for accepting this paper.\n\nComments\n\n1) Would it be possible to use the few labeled factors of variation in a meta-learning setup rather than as a regularizer?\n\n2) The paper provides high level conclusions about the impact of having supervised model selection or semi-supervised learning in models in general, but doesn’t offer much discussion into their behavior under specific settings (i.e.) it seems to be hard to pick a winner amongst presented model. Some are better with 100 labeled examples but don’t scale as well as others when an order of magnitude more labeled data is available. It is certainly hard to discuss all the thousands of experimental observations, but the paper can benefit from some more fine-grained analysis.\n\nMinor\nFigure 4 is hard to comprehend without a model to index mapping similar to Figure 3", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "SJeN77VTYH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper considers the challenge of learning disentangled representations---i.e. learning representations of data points x, r(x), that capture the factors of variation in an underlying latent variable z that controls the generating process of x---and studies two approaches for integrated a small number of data points manually labeled with z (or noisier variants thereof): one using these to perform model selection, and another incorporating them into unsupervised representation learning via an additional supervised loss term.  This investigation is motivated by recent results concluding that inductive biases are needed otherwise learning disentangled representations in an unsupervised fashion is impossible.  The paper poses its overall goal as making this injection of inductive biases explicit via a small number (~100 even) of (potentially noisy) labels, and reports on exhaustive experiments on four datasets.\n\nI think that this paper merits acceptance because (a) the motivation of taking a necessity in practice (somehow selecting models / injecting inductive biases) and making it more explicit in the approach is a good one, and because the thorough empirical survey (and simple, but novel, contribution of a new semi-supervised representation learning objective) are likely valuable contributions to this community.\n\nOne negative comment overall would be that the results are not that surprising: that is, the fact that using labels either (a) to do model validation or (b) in a semi-supervised fashion would help is not too surprising.  However, I believe in the context of (a) making more explicit a practical (and theoretically) necessary step in the pipeline of learning representations, and (b) contributing a comprehensive empirical study, this is a worthwhile contribution.\n\nMinor notes:\n- Fig. 1 isn't the most intuitive- ideally would be better explained for the headlining figure\n- 8.57 P100 GPU years is ~= $75k based on a cursory glance at cloud instance pricing at monthly rates... this is a lot to reproduce these experiments...", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper addresses the problem of learning disentangled representations and shows that the introduction of a few labels corresponding to the desired factors of variation can be used to increase the separation of the learned representation. 

There were mixed scores for this work. Two reviewers recommended weak acceptance while one reviewer recommended rejection. All reviewers and authors agreed that the main conclusion that the labeled factors of variation can be used to improved disentanglement is perhaps expected. However, reviewers 2 and 3 argue that this work presents extensive experimental evidence to support this claim which will be of value to the community. The main concerns of R1 center around a lack of clear analysis and synthesis of the large number of experiments. Though there is a page limit we encourage the authors to revise their manuscript with a specific focus on clarity and take-away messages from their results. 

After careful consideration of all reviewer comments and author rebuttals the AC recommends acceptance of this work. The potential contribution of the extensive experimental evidence warrants presentation at ICLR. However, again, we encourage the authors to consider ways to mitigate the concerns of R1 in their final manuscript. 

Decision: Accept (Poster)