On Disentangled Representations Learned From Correlated Data

Despite impressive progress in the last decade , it still remains an open challenge to build models that generalize well across multiple tasks and datasets . One path to achieve this is to learn meaningful and compact representations , in which different semantic aspects of data are structurally disentangled . The focus of disentanglement approaches has been on separating independent factors of variation despite the fact that real-world observations are often not structured into meaningful independent causal variables . In this work , we bridge the gap to real-world scenarios by analyzing the behavior of most prominent methods and disentanglement scores on correlated data in a large scale empirical study ( including 4260 models ) . We show that systematically induced correlations in the dataset are being learned and reflected in the latent representations , while widely used disentanglement scores fall short of capturing these latent correlations . Finally , we demonstrate how to disentangle these latent correlations using weak supervision , even if we constrain this supervision to be causally plausible . Our results thus support the argument to learn independent mechanisms rather than independent factors of variations . 1 INTRODUCTION Due to the induced structure , disentangled representations promise generalization to unseen scenarios ( Higgins et al. , 2017b ) , increased interpretability ( Adel et al. , 2018 ; Higgins et al. , 2018 ) and faster learning on downstream tasks ( van Steenkiste et al. , 2019 ; Locatello et al. , 2019a ) . While the advantages of disentangled representations have been well established , they generally assume the existence of natural factors that vary independently within the given dataset , which is rarely the case in real-world settings . As an example , consider a scene with a table and some chairs ( see Fig . 1 ) . The higher-level factors of this representation are in fact correlated and what we actually want to infer are independent ( causal ) mechanisms ( Peters et al. , 2017 ; Parascandolo et al. , 2018 ; Suter et al. , 2019 ; Goyal et al. , 2019 ) . A complex generative model can be thought of as the composition of independent mechanisms or “ causal ” modules , which generate highdimensional observations ( such as images or videos ) . In the causality community , this is often considered a prerequisite to achieve representations which are robust to interventions upon variables determined by such models ( Peters et al. , 2017 ) . One particular instantiation of this idea in the machine learning community is the notion of disentangled representations ( Bengio et al. , 2013 ) . The goal of disentanglement learning is to find a representation of the data which captures all the ground-truth factors of variation ( FoV ) independently . Despite the recent growth of the field , the performance of state-of-the-art disentanglement learners remains unknown for more realistic settings where FoV are correlated during training . Given the potential societal impact in the medical domain ( Chartsias et al. , 2018 ) or fair decision making ( Locatello et al. , 2019a ; Madras et al. , 2018 ; Creager et al. , 2019 ) , the evaluation of the usefulness of disentangled representations trained on correlated data is of high importance . To go beyond the highly idealized settings considered thus far , we conducted a large scale empirical study to systematically assess the effect of induced correlations between pairs of factors of variation in training data on the learned representations . To provide a qualitative and quantitative evaluation , we investigate multiple datasets with access to ground-truth labels . Moreover , we study the generalization abilities of the representations learned on correlated data as well as their performance in particular for the downstream task of fair decision making . Contributions . Our main contributions can be summarized as follows : • We present the first large-scale empirical study ( 4260 models ) 1 that examines how modern disentanglement learners perform when ground truth factors of the observational data are correlated . • We find that factorization-based inductive biases are insufficient to learn disentangled representations from observational data . Existing methods fail to disentangle correlated factors of variation , resulting in correlated latent space dimensions . Moreover , standard disentanglement metrics do not reveal these persisting correlations . • We investigate the usefulness of semi-supervised and weakly-supervised approaches to resolve latent entanglement . For the latter setting , we focus on multiple observational and interventional distributions . 2 BACKGROUND AND RELATED WORK . Disentanglement . Current state-of-the-art disentanglement approaches use the framework of variational auto-encoders ( VAEs ) ( Kingma & Welling , 2014 ; Rezende et al. , 2014 ) . The ( high-dimensional ) observations x are modelled as being generated from some latent features z with chosen prior p ( z ) according to the probabilistic model pθ ( x|z ) p ( z ) . The generative model pθ ( x|z ) as well as the proxy posterior qφ ( z|x ) can be parameterized by neural networks , which are optimized by maximizing the variational lower bound ( ELBO ) of log p ( x1 , . . . , xN ) . LV AE = N∑ i=1 Eqφ ( z|x ( i ) ) [ log pθ ( x ( i ) |z ) ] −DKL ( qφ ( z|x ( i ) ) ‖p ( z ) ) ( 1 ) The above objective does not enforce any structure on the latent space , except for similarity ( in KL-divergence ) to the prior p ( z ) ( typically chosen as an isotropic Gaussian ) . However , the structure and semantic meaning of latent representations can be relevant to study generation properties . Consequently , various proposals for structure-imposing regularization and commonly used evaluation metrics measuring different notions of disentanglement of the learned representations have been made ( Higgins et al. , 2017a ; Kim & Mnih , 2018 ; Burgess et al. , 2018 ; Kumar et al. , 2018 ; Chen et al. , 2018 ; Eastwood & Williams , 2018 ; Mathieu et al. , 2018 ) . Recently , it has been shown that unsupervised disentangling by optimising marginal likelihood in a generative model is impossible without further inductive bias ( Locatello et al. , 2019b ) . To address this theoretical limitation , methods have been proposed that do not require explicitly labelled data but only some weak labeling information ( Locatello et al. , 2020 ; Shu et al. , 2019 ) . Ideas related to disentangling the factors of variations date back to the non-linear ICA literature ( Bach & Jordan , 2002 ; Comon , 1994 ; Jutten & Karhunen , 2003 ; Hyvärinen & Pajunen , 1999 ; Hyvarinen et al. , 2019 ; Hyvarinen & Morioka , 2016 ; Gresele et al. , 2019 ) . Recent work combines non-linear ICA with disentanglement ( Khemakhem et al. , 2020 ; Sorrenson et al. , 2020 ; Klindt et al. , 2020 ) Correlations . A set of random variables Xi=1 , ... , n is not independent , if and only if their joint distribution does not factorize P ( X1 , X2 , . . . , Xn ) 6= n∏ i=1 P ( Xi ) . ( 2 ) In this case , we speak of dependence between the random variables , also commonly referred to as correlation.2 Correlation between two variables can either stem from a direct causal relationship ( one 1Each model has been trained for 300,000 iterations on Tesla V100 GPUs . Reproducing these experiments requires approximately 0.79 GPU years . 2We use the term correlation here in a broad sense of any statistical association , not just linear dependencies . variable affects the other ) , but can also be due to different unobserved circumstances ( confounders ) affecting both . Real-world datasets display many of these ( “ spurious ” and often a priori unknown ) correlations ( Geirhos et al. , 2020 ) . However , most work on learning disentangled representations assumes that there is an underlying set of independent ground truth variables that govern the generative process of observable data . These methods are hence predominantly evaluated on data that obey independence in the true factors of variation , which we then consider to be the correct factorization . In the real world , the observation generating process is likely not always as clearly “ disentangled ” and we do expect correlations in the collected datasets . It is thus an open question to what degree existing inductive biases from the encoder/decoder architecture , but more importantly the dataset biases , affect the learned representation . In our experiments , we introduce dataset correlations in a controlled manner to understand to what degree state-of-the-art approaches can cope with such correlations . We believe these correlations to reflect a major feature of more realistic environments . Other Related Work . Most popular datasets in the disentanglement literature exhibit perfect independence in their FoV . At some level this is sensible as it reflects the underlying assumption in the inductive bias being studied . However , this assumption is unlikely to hold in practice as shown by Li et al . ( 2019 ) , who propose methods based on a pairwise independence assumption instead . The literature so far has not thoroughly measured how popular inductive biases such as factorized priors behave when learning from correlated datasets , although several smaller experiments along these lines can be acknowledged . Chen et al . ( 2018 ) studied correlated 3DFaces ( Paysan et al. , 2009 ) by fixing all except three factors in which the authors conclude that the β-TC-VAE regularizer can help to disentangle imposed correlations . Brekelmans et al . ( 2019 ) show that Echo noise results in superior disentanglement compared to standard betaVAE in a small experiment on a downsampled dSprites variant where randomly selected factor pairs are excluded . However , the latent structure was not studied in detail ; our findings suggest that global disentanglement metrics are insufficient to diagnose issues when models learn from correlated data . Creager et al . ( 2019 ) based some of the evaluations of a proposed new autoencoder architecture in the fairness context on a biased dSprites variant and Yang et al . ( 2020 ) study a linear SCM in a VAE architecure on datasets with dependent variables . However , their studies focused on representation learners that require strong supervision via FoV labels at train time . 3 THE EFFECT OF CORRELATED DATA ON DISENTANGLEMENT LEARNERS . In this section , we want to present the key findings from our empirical study of unsupervised disentanglement learning on a particular variant of correlated data . We start by outlining the experimental design of our study in Section 3.1 . Based on this , we analyze the latent spaces in Section 3.2 and find that factorization-based inductive biases are insufficient to learn disentangled representations from observational data . Persisting pairwise correlations in the latent space are not sufficiently revealed by standard disentanglement metrics that might be particularly relevant and problematic for fairness applications . Finally , in Section 3.3 , we show extrapolation and generalization capabilities of the learned representations towards unseen factor combinations due to the induced correlations . 3.1 EXPERIMENTAL DESIGN . For our first experiments we introduce correlations between single pairs of factors of variation on the following three datasets : Shapes3D with object size and azimuth ( denoted “ A ” ) , dSprites with orientation and X-position ( “ B ” ) and finally the real-world observations dataset MPI-3D with first and second degree of freedom ( “ C ” ) . We focus on linear correlations with Gaussian noise between the two variables , which we denote by c1 , c2 . The ground truth factors for c1 , c2 take values zc1 ∈ { 0 , . . . , zmaxc1 } and zc2 ∈ { 0 , . . . , zmaxc2 } respectively . We then parameterize correlations by sampling the training dataset from the joint P ( zc1 , zc2 ) ∝ N ( zc2−αzc1 , σ ) where α = zmaxc2 /zmaxc1 . The strength of the correlations can be tuned by σ , for which we choose 0.2 , 0.4 , 0.7 in normalized units with respect to the range of values in zc1 , c2 . Lower σ indicates stronger correlation . See Fig . 5 for an example of P ( zc1 , zc2 ) for correlating azimuth and object size in Shapes3D with σ = 0.2 . Additionally , we study the uncorrelated limit ( σ =∞ ) , which amounts to the case typically studied in the literature . We train the same six VAE methods as discussed in Locatello et al . ( 2019b ) , including β-VAE , FactorVAE , AnnealedVAE , DIP-VAE-I , DIP-VAE-II and β-TC-VAE , each with six hyperparameter settings . Each method has been trained using five different random seeds . All remaining factors of variation are sampled uniformly at random . This first study sums up to a total of 2160 trained models , or 180 models per dataset and correlation strength3 . Appendix A describes additional implementation details .

Summary: This paper systematically presents a large-scale empirical study on the disentangled representation learning when the underlying factors are possibly entangled. From the results of purely unsupervised settings, the authors have discovered the shortcomings of the existing metrics of disentanglement as well as the poor learned representations (in terms of disentanglement). However, with the help of small amount of factor labels or other weak supervision signals, recent approaches could learn fairly perfect representation.