Disentangling One Factor at a Time

With the overabundance of data for machines to process in the current state of machine learning , data discovery , organization , and interpretation of the data becomes a critical need . Specifically of need are unsupervised methods that do not require laborious labeling by human observers . One promising approach to this enedeavour is Disentanglement , which aims at learning the underlying generative latent factors of the data . The factors should also be as human interpretable as possible for the purposes of data discovery . Unsupervised disentanglement is a particularly difficult open subset of the problem , which asks the network to learn on its own the generative factors without any link to the true labels . This problem area is currently dominated by two approaches : Variational Autoencoder and Generative Adversarial Network approaches . While GANs have good performance , they suffer from difficulty in training and mode collapse , and while VAEs are stable to train , they do not perform as well as GANs in terms of interpretability . In current state of the art versions of these approaches , the networks require the user to specify the number of factors that we expect to find in the data . This limitation prevents “ true ” disentanglement , in the sense that learning how many factors is actually one of the tasks we wish the network to solve . In this work we propose a novel network for unsupervised disentanglement that combines the stable training of the VAE with the interpretability offered by GANs without the training instabilities . We aim to disentangle interpretable latent factors “ one at a time ” , or OAT factor learning , making no prior assumptions about the number or distribution of factors , in a completely unsupervised manner . We demonstrate its quantitative and qualitative effectiveness by evaluating the latent representations learned on three benchmark datasets ; DSprites , 3DShapes and CelebA . 1 INTRODUCTION . Deep learning models , which are now widely adopted across multiple A.I . tasks ranging from vision to music generation to game playing ( Krizhevsky et al. , 2017 ; Oord et al. , 2016 ; Mnih et al. , 2015 ) , owe their success to their ability to learn representations from the data rather than requiring handcrafted features that older models required . However , this self-learning of abstract representations comes at the known cost of the resulting representations being cryptic and inscrutable to human observers . These learned representations might be dramatically affected by noise or spurious correlations between the data and the labels - the representations might encode ’ useless ’ information from the input data which is correlated with the corresponding label Geirhos et al . ( 2020 ) . This makes them more vulnerable to slight changes in the data distribution . A more comprehensive understanding of the data down to essential indivisible factors would allow us to learn insights , sort and label data , and facilitate downstream learning more efficiently . This also requires , critically , that these factors be somehow interpretable as well . This approach , dubbed Disentanglement . requires that we learn the data from its fundamental building block , so-called “ disentangling ” the true factors - the factors of variation - that generate the data . If one were to learn these factors , one would learn all possible causes of variation in a given dataset , and would in some sense gain complete understanding of the underlying machinery , so to speak . In this work , we propose a new method that attempts to learn the true disentangled factors one at a time , in a way that maintains interpretability for future use . 0Code available at https : //github.com/OATFactor/OATFactor 1.1 WHAT IS A DISENTANGLED REPRESENTATION ? . While it is easy to informally talk of factors of variation , actually pinning down concrete definitions of disentangled learning has proved a somewhat more difficult task . Though there is no commonly accepted formalized notion of disentanglement or validation metrics ( Higgins et al. , 2018 ) , recent works have characterized disentangled representations , based on natural intuition , as one which encodes each informative factor of variation of the data in separate latent dimenions ( Bengio , 2013 ) , such that a change in single factor of variation produces a change in only a subset of the learned latent representation . This is refered to as the separability ( Do & Tran , 2020 ) quality of the representations , also called disentanglement ( Eastwood & Williams , 2018 ) and modularity ( Ridgeway & Mozer , 2018 ) . Separability ensures that the downstream tasks which depend on a certain subset of factors are not affected by changes in other factors thus facilitating robust models Suter et al . ( 2019 ) . Some previous works ( Ridgeway & Mozer , 2018 ; Eastwood & Williams , 2018 ) have suggested that a single dimension of the learned disentangled representation should completely describe a factor . This constraint along with separability entails a bijective mapping between the true factors of variation and the learned representation . However , enforcing this constraint might not be conducive to learn complex factors in a single latent variable Esmaeili et al . ( 2018 ) . Hence in this work we focus primarily on separability . For an in depth discussion on interpretability and how we enforce it in the proposed work , we refer the reader to A.1 . 1.2 WHAT IS MISSING FROM CURRENT APPROACHES TO SOLVING DISENTANGLEMENT ? . Due to the monumentous task of learning data without supervision , current methods have attempted to tackle the subproblems of separability and interpretability , but few have successfully solved both at once . Learning these disentangled representations in a semi-supervised setting Kulkarni et al . ( 2015 ) ; Siddharth et al . ( 2017 ) ; Locatello et al . ( 2019 ) is a relatively easier task where additional annotated data is available that give a strong backpropgation gradient to the network to guide it to cleanly separate factors . However , if the main goal of disentanglement is to discover unknown factors in large data corpuses , then it is not the right direction to require labels ; rather the network should learn in an unsupervised fashion . For unstructured data , all current state of the art approaches based on unsupervised disentanglement rely on a deep generative neural network , built either on a Variational Autoencoder or Generative Adversarial Network structure ( See Related Work . ) Many popular VAE based methods are able to do well in separating factors out but make strict assumptions on the number of factors and their structure and those that do , do not explicitly attempt to also make the factors interpretable . Other methods based on GANs suffer from common issues from training of GANs , but also tend to only learn a small subset of factors ( See 2 ) . Here we propose a novel hybrid network that provides the stability and strong gradients of VAE training with the performance of GAN training , without mode collapse . Most current VAE-based state of the art methods make the assumption that there are a fixed number of independent factors for all the data points in the dataset . However in real datasets , in addition to the independent factors common to all points in the dataset , there might also be some correlated , nuisance or noisy factors pertinent to specifically only certain data points . Moreover , approaches rely on a heuristically chosen latent dimension d , sufficiently large to encompass all true factors . However , this suffers from the same pitfalls for many of the same reasons that algorithms like kmeans cluster do , namely that given a new , unseen dataset , we do not necessarily know how many independent factors there are . In fact , this is one of the main goals of disentanglement in the first place , to glean insights about the data . Our method instead assumes that there is a set of independent factors , and one of entangled nuisance and correlated factors , and separates them out . We then iteratively learn to disentangle each factor of variation one at a time , such that the network learns on its own different independently controllable factors thus removing a current hand-tuned roadblock on the way to true unsupervised disentanglement . Second , in the same training loop , we ensure that the disentangled latent representation follows intepretable latent code manipulation ( Section . 1.1 ) , which says that a change in a single latent should make a distinct and noticeable change in the output ( Eastwood & Williams , 2018 ) . Together , we demonstrate our proposed model is able to learn both critical qualities of disentanglement , in a completely unsupervised manner . 1.3 CONTRIBUTIONS . Our main contributions are as follows : • We introduce a new completely unsupervised generative neural model , One at a Time ( OAT ) factor learning that combines the stability of VAE training with the accuracy of GAN learning , which contains both a set of independent separable factors and a set of entangled factors , that produces separable and interpretable latent factor codes • Our proposed model is the first unsupervised method that is capable of learning an arbitrary number of latent factors via incremental unsupervised interventions in the latent space • We test and evaluate our algorithm on three datasets and across multiple metrics . Our empirical results strongly suggest that our proposed method is effective in finding interpretable factors and competitive with the most recent disentanglement strategies 2 RELATED WORK . Various authors have attempted to learn unsupervised disentangled representations using generative models in recent years . State-of-the-art approaches for unsupervised disentanglement learning can be broadly classified into two categories based on the type of generative model used ; one via Variational Autoencoders ( VAE ) ( Kingma & Welling , 2014 ; Rezende et al. , 2014 ) , and another via Generative Adversarial Networks ( GAN ) ( Goodfellow et al. , 2014 ) . 2.1 VIA VARIATIONAL AUTOENCODERS . Variational Autoencoders are a deep generative neural network model which learns an approximate posterior distribution of the latent representations from the data , while trying to maximize the data log-likelihood . They can be thought of as an autoencoder with an additional loss term that drives the reconstructions to be closer together in latent space . VAEs assume that the data x is generated from a set of latent features z with a prior p ( z ) according to the model pθ ( x|z ) p ( z ) . The top-down generator pθ ( x|z ) and a bottom up inference network qφ ( z|x ) are modeled as multilayer neural networks and trained jointly to maximize the marginal log-likelihood of the empirical distribution of the training data . However , since the marginal log-likelihood is intractable , VAEs optimize a a tractable lower bound L on the data log-likelihood pθ ( x ) called the Evidence Lower Bound ( ELBO ) : L = Eq ( x ) [ Eqφ ( z|xi ) [ log pθ ( xi|z ) ] − KL ( qφ ( z|xi ) ||p ( z ) ) ] ( 1 ) where q ( x ) = 1N ∑N i=1 δ ( xi ) is the distribution of the training data . The first term measures the reconstruction error and the second term measures the distance between the approximate posterior distribution qφ ( z|x ) and the assumed prior distribution p ( z ) . Many state-of-the-art unsupervised disentanglement methods extend the above objective function to impose additional constraints on the structure of the latent space to match the independent prior assumption . β-VAE ( Higgins et al. , 2017 ) and AnnealedVAE ( Burgess et al. , 2018 ) heavily penalize the KL divergence term thus forcing the learned posterior distribution qφ ( z|x ) to be independent like the prior . Factor-VAE ( Kim & Mnih , 2019 ) and β-TCVAE ( Chen et al. , 2019 ) penalize the total correlation of the aggregated posterior qφ ( z ) . TC = KL ( q ( z ) || ∏K i=1 q ( zi ) ) where the aggregated posterior is calculated as qφ ( z ) = Ep ( x ) [ q ( z|xi ) ] = 1N ∑N i=1 qφ ( z|xi ) using adversarial and statistical techniques respectively . DIPVAE ( Kumar et al. , 2018 ) forces the covariance matrix of the aggregated posterior q ( z ) to be close to the indentity matrix by method of moment matching . Other works improved the performance by making a specific design for discrete factors ( Dupont , 2018 ; Jeong & Song , 2019 ) ; and use optimization techniques based on annealing to encode information effectively in the discrete and continuous factors .

Summary: The authors propose a new method to learn a disentangled representation. The proposed method is Variational Autoencoder (VAE) based. The latent variables consist of nuisance factors and disentangled factors that form the $k$ generative factors per dataset. To help learn the disentangled factors of variation the authors propose to use an additional discriminator and interventions on the disentangled factors. The authors demonstrate the performance of the proposed model on the dSprites and CelebA dataset.