Disentangling Factors of Variations Using Few Labels

1 INTRODUCTION . In machine learning , it is commonly assumed that high-dimensional observations x ( such as images ) are the manifestation of a low-dimensional latent variable z of ground-truth factors of variation ( Bengio et al. , 2013 ; Kulkarni et al. , 2015 ; Chen et al. , 2016 ; Tschannen et al. , 2018 ) . More specifically , one often assumes that there is a distribution p ( z ) over these latent variables and that observations in this ground-truth model are generated by sampling z from p ( z ) first . Then , the observations x are sampled from a conditional distribution p ( x|z ) . The goal of disentanglement learning is to find a representation of the data r ( x ) which captures all the ground-truth factors of variation in z independently . The hope is that such representations will be interpretable , maximally compact , allow for counterfactual reasoning and be useful for a large variety of downstream tasks ( Bengio et al. , 2013 ; Peters et al. , 2017 ; LeCun et al. , 2015 ; Bengio et al. , 2007 ; Schmidhuber , 1992 ; Lake et al. , 2017 ; Goodfellow et al. , 2009 ; Lenc & Vedaldi , 2015 ; Tschannen et al. , 2018 ; Higgins et al. , 2018 ; Suter et al. , 2019 ; Adel et al. , 2018 ; van Steenkiste et al. , 2019 ; Locatello et al. , 2019a ; Gondal et al. , 2019 ) . As all these applications rely on the assumption that disentangled representations can be reliably learned in practice , we hope to learn them with as little supervision as possible ( Bengio et al. , 2013 ; Schölkopf et al. , 2012 ; Peters et al. , 2017 ; Pearl , 2009 ; Spirtes et al. , 2000 ) . Current state-of-the-art unsupervised disentanglement approaches enrich the Variational Autoencoder ( VAE ) ( Kingma & Welling , 2014 ) objective with different unsupervised regularizers that aim to encourage disentangled representations ( Higgins et al. , 2017a ; Burgess et al. , 2018 ; Kim & Mnih , 2018 ; Chen et al. , 2018 ; Kumar et al. , 2018 ; Rubenstein et al. , 2018 ; Mathieu et al. , 2018 ; Rolinek et al. , 2019 ) . The disentanglement of the representation of such methods exhibit a large variance and while some models turn out to be well disentangled it appears hard to identify them without supervision ( Locatello et al. , 2019b ) . This is consistent with the theoretical result of Locatello et al . ( 2019b ) that the unsupervised learning of disentangled representations is impossible without inductive biases . While human inspection can be used to select good model runs and hyperparameters ( e.g . Higgins et al . ( 2017b , Appendix 5.1 ) ) , we argue that such supervision should be made explicit . We hence consider the setting where one has access to annotations ( which we call labels in the following ) of the latent variables z for a very limited number of observations x , for example through human annotation . Even though this setting is not universally applicable ( e.g . when the observations are not human interpretable ) and a completely unsupervised approach would be elegant , collecting a small number of human annotations is simple and cheap via crowd-sourcing platforms such as Amazon Mechanical Turk , and is common practice in the development of real-world machine learning systems . As a consequence , the considered setup allows us to explicitly encode prior knowledge and biases into the learned representation via annotation , rather than relying solely on implicit biases such as the choice of network architecture with possibly hard-to-control effects . Other forms of inductive biases such as relying on temporal information ( video data ) ( Denton & Birodkar , 2017 ; Yingzhen & Mandt , 2018 ) , allowing for interaction with the environment ( Thomas et al. , 2017 ) , or incorporating grouping information ( Kulkarni et al. , 2015 ; Bouchacourt et al. , 2018 ) were discussed in the literature as candidates to circumvent the impossibility result by Locatello et al . ( 2019b ) . However , there currently seems to be no quantitative evidence that such approaches will lead to improved disentanglement on the metrics and data sets considered in this paper . Furthermore , these approaches come with additional challenges : Incorporating time can significantly increase computational costs ( processing video data ) and interaction often results in slow training ( e.g . a robot arm interacting with the real world ) . By contrast , collecting a few labels is cheap and fast . We first investigate whether disentanglement scores are sample efficient and robust to imprecise labels . Second , we explore whether it is more beneficial to incorporate the limited amount of labels available into training and thoroughly test the benefits and trade-offs of this approach compared to supervised validation . For this purpose , we perform a reproducible large scale experimental study1 , training over 52 000 models on four different data sets . We found that unsupervised training with supervised validation enables reliable learning of disentangled representations . On the other hand , using some of the labeled data for training is beneficial both in terms of disentanglement and downstream performance . Overall , we show that a very small amount of supervision is enough to reliably learn disentangled representations as illustrated in Figure 1 . Our key contributions can be summarized as follows : • We observe that some of the existing disentanglement metrics ( which require observations of z ) can be used to tune the hyperparameters of unsupervised methods even when only very few labeled examples are available ( Section 3 ) . Therefore , training a variety of models and introducing supervision to select the good runs is a viable solution to overcome the impossibility result of Locatello et al . ( 2019b ) . 1Reproducing these experiment requires approximately 8.57 GPU years ( NVIDIA P100 ) . • We find that adding a simple supervised loss , using as little as 100 labeled examples , outperforms unsupervised training with supervised model validation both in terms of disentanglement scores and downstream performance ( Section 4.2 ) . • We discover that both unsupervised training with supervised validation and semi-supervised training are surprisingly robust to label noise ( Sections 3.2 and 4.3 ) and tolerate coarse and partial annotations , the latter being particularly important if not all factors of variation are known . • Based on our findings we provide guidelines helpful for practitioners to leverage disentangled representations in practical scenarios . 2 BACKGROUND AND RELATED WORK . Problem definition : Consider a generative model with latent variable z with factorized density p ( z ) = ∏d i=1 p ( zi ) , where d > 1 , and observations x obtained as samples from p ( x|z ) . Intuitively , the goal of disentanglement learning is to find a representation r ( x ) separating the factors of variation into independent components so that a change in a dimension of z corresponds to a change in a dimension of r ( x ) ( Bengio et al. , 2013 ) . Refinements of this definition include disentangling independent groups in the topological sense ( Higgins et al. , 2018 ) and learning disentangled causal models ( Suter et al. , 2019 ) . These definitions are reflected in various disentanglement metrics that aim at measuring some structural property of the statistical dependencies between z and r ( x ) . Evaluating disentangled representations : The BetaVAE ( Higgins et al. , 2017a ) and FactorVAE ( Kim & Mnih , 2018 ) scores measures disentanglement by performing an intervention on the factors of variation and predicting which factor was intervened on . The Mutual Information Gap ( MIG ) ( Chen et al. , 2018 ) , Modularity ( Ridgeway & Mozer , 2018 ) , DCI Disentanglement ( Eastwood & Williams , 2018 ) and SAPscore ( Kumar et al. , 2018 ) first compute a matrix relating factors of variation and codes ( for example via pairwise mutual information , feature importance and predictability ) . Then , they average a normalized difference between the top two entries of the matrix either row or column wise . For more details , see Appendix C of ( Locatello et al. , 2019b ) . Learning disentangled representations : Since all these metrics require access to labels z they can not be used for unsupervised training . Many state-of-the-art unsupervised disentanglement methods therefore extend VAEs ( Kingma & Welling , 2014 ) with a regularizer Ru ( qφ ( z|x ) ) that enforces structure in the latent space of the VAE induced by the encoding distribution qφ ( z|x ) with the hope that this leads to disentangled representations . These approaches ( Higgins et al. , 2017a ; Burgess et al. , 2018 ; Kim & Mnih , 2018 ; Chen et al. , 2018 ; Kumar et al. , 2018 ) can be cast under the following optimization template : max φ , θ Ex [ Eqφ ( z|x ) [ log pθ ( x|z ) ] −DKL ( qφ ( z|x ) ‖p ( z ) ) ] ︸ ︷︷ ︸ = : ELBO ( φ , θ ) +βEx [ Ru ( qφ ( z|x ) ) ] . ( 1 ) The β-VAE ( Higgins et al. , 2017a ) and AnnealedVAE ( Burgess et al. , 2018 ) reduce the capacity of the VAE bottleneck under the assumption that encoding the factors of variation is the most efficient way to achieve a good reconstruction ( Peters et al. , 2017 ) . The Factor-VAE ( Kim & Mnih , 2018 ) and β-TCVAE both penalize the total correlation of the aggregated posterior q ( z ) ( i.e . the encoder distribution after marginalizing the training data ) . The DIP-VAE variants ( Kumar et al. , 2018 ) match the moments of the aggregated posterior and a factorized distribution . We refer to Appendix B of ( Locatello et al. , 2019b ) and Section 3 of ( Tschannen et al. , 2018 ) for a more detailed description . Supervision and inductive biases : While there is prior work on semi-supervised disentanglement learning ( Reed et al. , 2014 ; Cheung et al. , 2014 ; Mathieu et al. , 2016 ; Narayanaswamy et al. , 2017 ; Kingma et al. , 2014 ; Klys et al. , 2018 ) , these methods aim to disentangle only some observed factors of variation from the other latent variables which themselves remain entangled . These approaches are briefly discussed in Section 4 . Exploiting relational information or knowledge of the effect of the factors of variation have both been qualitatively studied to learn disentangled representations ( Hinton et al. , 2011 ; Cohen & Welling , 2014 ; Karaletsos et al. , 2015 ; Goroshin et al. , 2015 ; Whitney et al. , 2016 ; Fraccaro et al. , 2017 ; Denton & Birodkar , 2017 ; Hsu et al. , 2017 ; Yingzhen & Mandt , 2018 ; Locatello et al. , 2018 ; Kulkarni et al. , 2015 ; Ruiz et al. , 2019 ; Bouchacourt et al. , 2018 ; Adel et al. , 2018 ) . Other related work . Due to the lack of a commonly accepted formal definition , the term “ disentangled representations ” has been used in very different lines of work . There is for example a rich literature in disentangling pose from content in 3D objects and content from motion in videos ( Yang et al. , 2015 ; Yingzhen & Mandt , 2018 ; Hsieh et al. , 2018 ; Fortuin et al. , 2019 ; Deng et al. , 2017 ; Goroshin et al. , 2015 ; Hyvarinen & Morioka , 2016 ) . This can be achieved with different degrees of supervision , ranging from fully unsupervised to semi-supervised . Another line of work aims at disentangling class labels from other latent variables by assuming the existence of a causal model where the latent variable z has an arbitrary factorization with the class variable y . In this setting , y is partially observed ( Reed et al. , 2014 ; Cheung et al. , 2014 ; Mathieu et al. , 2016 ; Narayanaswamy et al. , 2017 ; Kingma et al. , 2014 ; Klys et al. , 2018 ) . Without further assumptions on the structure of the graphical model , this is equivalent to partially observed factors of variation with latent confounders . Except for very special cases , the recovery of the structure of the generative model is known to be impossible with purely observational data ( Peters et al. , 2017 ; D ’ Amour , 2019 ; Suter et al. , 2019 ) . Here , we intend to disentangle factors of variation in the sense of ( Bengio et al. , 2013 ; Suter et al. , 2019 ; Higgins et al. , 2018 ) . We aim at separating the effects of all factors of variation , which translates to learning a representation with independent components . This problem has already been studied extensively in the non-linear ICA literature ( Comon , 1994 ; Bach & Jordan , 2002 ; Jutten & Karhunen , 2003 ; Hyvarinen & Morioka , 2016 ; Hyvärinen & Pajunen , 1999 ; Hyvarinen et al. , 2019 ; Gresele et al. , 2019 ) .

Summary: This paper considers the challenge of learning disentangled representations---i.e. learning representations of data points x, r(x), that capture the factors of variation in an underlying latent variable z that controls the generating process of x---and studies two approaches for integrated a small number of data points manually labeled with z (or noisier variants thereof): one using these to perform model selection, and another incorporating them into unsupervised representation learning via an additional supervised loss term.  This investigation is motivated by recent results concluding that inductive biases are needed otherwise learning disentangled representations in an unsupervised fashion is impossible.  The paper poses its overall goal as making this injection of inductive biases explicit via a small number (~100 even) of (potentially noisy) labels, and reports on exhaustive experiments on four datasets.