Disentangling One Factor at a Time

With the overabundance of data for machines to process in the current state of machine learning , data discovery , organization , and interpretation of the data becomes a critical need . Specifically of need are unsupervised methods that do not require laborious labeling by human observers . One promising approach to this enedeavour is Disentanglement , which aims at learning the underlying generative latent factors of the data . The factors should also be as human interpretable as possible for the purposes of data discovery . Unsupervised disentanglement is a particularly difficult open subset of the problem , which asks the network to learn on its own the generative factors without any link to the true labels . This problem area is currently dominated by two approaches : Variational Autoencoder and Generative Adversarial Network approaches . While GANs have good performance , they suffer from difficulty in training and mode collapse , and while VAEs are stable to train , they do not perform as well as GANs in terms of interpretability . In current state of the art versions of these approaches , the networks require the user to specify the number of factors that we expect to find in the data . This limitation prevents “ true ” disentanglement , in the sense that learning how many factors is actually one of the tasks we wish the network to solve . In this work we propose a novel network for unsupervised disentanglement that combines the stable training of the VAE with the interpretability offered by GANs without the training instabilities . We aim to disentangle interpretable latent factors “ one at a time ” , or OAT factor learning , making no prior assumptions about the number or distribution of factors , in a completely unsupervised manner . We demonstrate its quantitative and qualitative effectiveness by evaluating the latent representations learned on three benchmark datasets ; DSprites , 3DShapes and CelebA . 1 INTRODUCTION . Deep learning models , which are now widely adopted across multiple A.I . tasks ranging from vision to music generation to game playing ( Krizhevsky et al. , 2017 ; Oord et al. , 2016 ; Mnih et al. , 2015 ) , owe their success to their ability to learn representations from the data rather than requiring handcrafted features that older models required . However , this self-learning of abstract representations comes at the known cost of the resulting representations being cryptic and inscrutable to human observers . These learned representations might be dramatically affected by noise or spurious correlations between the data and the labels - the representations might encode ’ useless ’ information from the input data which is correlated with the corresponding label Geirhos et al . ( 2020 ) . This makes them more vulnerable to slight changes in the data distribution . A more comprehensive understanding of the data down to essential indivisible factors would allow us to learn insights , sort and label data , and facilitate downstream learning more efficiently . This also requires , critically , that these factors be somehow interpretable as well . This approach , dubbed Disentanglement . requires that we learn the data from its fundamental building block , so-called “ disentangling ” the true factors - the factors of variation - that generate the data . If one were to learn these factors , one would learn all possible causes of variation in a given dataset , and would in some sense gain complete understanding of the underlying machinery , so to speak . In this work , we propose a new method that attempts to learn the true disentangled factors one at a time , in a way that maintains interpretability for future use . 0Code available at https : //github.com/OATFactor/OATFactor 1.1 WHAT IS A DISENTANGLED REPRESENTATION ? . While it is easy to informally talk of factors of variation , actually pinning down concrete definitions of disentangled learning has proved a somewhat more difficult task . Though there is no commonly accepted formalized notion of disentanglement or validation metrics ( Higgins et al. , 2018 ) , recent works have characterized disentangled representations , based on natural intuition , as one which encodes each informative factor of variation of the data in separate latent dimenions ( Bengio , 2013 ) , such that a change in single factor of variation produces a change in only a subset of the learned latent representation . This is refered to as the separability ( Do & Tran , 2020 ) quality of the representations , also called disentanglement ( Eastwood & Williams , 2018 ) and modularity ( Ridgeway & Mozer , 2018 ) . Separability ensures that the downstream tasks which depend on a certain subset of factors are not affected by changes in other factors thus facilitating robust models Suter et al . ( 2019 ) . Some previous works ( Ridgeway & Mozer , 2018 ; Eastwood & Williams , 2018 ) have suggested that a single dimension of the learned disentangled representation should completely describe a factor . This constraint along with separability entails a bijective mapping between the true factors of variation and the learned representation . However , enforcing this constraint might not be conducive to learn complex factors in a single latent variable Esmaeili et al . ( 2018 ) . Hence in this work we focus primarily on separability . For an in depth discussion on interpretability and how we enforce it in the proposed work , we refer the reader to A.1 . 1.2 WHAT IS MISSING FROM CURRENT APPROACHES TO SOLVING DISENTANGLEMENT ? . Due to the monumentous task of learning data without supervision , current methods have attempted to tackle the subproblems of separability and interpretability , but few have successfully solved both at once . Learning these disentangled representations in a semi-supervised setting Kulkarni et al . ( 2015 ) ; Siddharth et al . ( 2017 ) ; Locatello et al . ( 2019 ) is a relatively easier task where additional annotated data is available that give a strong backpropgation gradient to the network to guide it to cleanly separate factors . However , if the main goal of disentanglement is to discover unknown factors in large data corpuses , then it is not the right direction to require labels ; rather the network should learn in an unsupervised fashion . For unstructured data , all current state of the art approaches based on unsupervised disentanglement rely on a deep generative neural network , built either on a Variational Autoencoder or Generative Adversarial Network structure ( See Related Work . ) Many popular VAE based methods are able to do well in separating factors out but make strict assumptions on the number of factors and their structure and those that do , do not explicitly attempt to also make the factors interpretable . Other methods based on GANs suffer from common issues from training of GANs , but also tend to only learn a small subset of factors ( See 2 ) . Here we propose a novel hybrid network that provides the stability and strong gradients of VAE training with the performance of GAN training , without mode collapse . Most current VAE-based state of the art methods make the assumption that there are a fixed number of independent factors for all the data points in the dataset . However in real datasets , in addition to the independent factors common to all points in the dataset , there might also be some correlated , nuisance or noisy factors pertinent to specifically only certain data points . Moreover , approaches rely on a heuristically chosen latent dimension d , sufficiently large to encompass all true factors . However , this suffers from the same pitfalls for many of the same reasons that algorithms like kmeans cluster do , namely that given a new , unseen dataset , we do not necessarily know how many independent factors there are . In fact , this is one of the main goals of disentanglement in the first place , to glean insights about the data . Our method instead assumes that there is a set of independent factors , and one of entangled nuisance and correlated factors , and separates them out . We then iteratively learn to disentangle each factor of variation one at a time , such that the network learns on its own different independently controllable factors thus removing a current hand-tuned roadblock on the way to true unsupervised disentanglement . Second , in the same training loop , we ensure that the disentangled latent representation follows intepretable latent code manipulation ( Section . 1.1 ) , which says that a change in a single latent should make a distinct and noticeable change in the output ( Eastwood & Williams , 2018 ) . Together , we demonstrate our proposed model is able to learn both critical qualities of disentanglement , in a completely unsupervised manner . 1.3 CONTRIBUTIONS . Our main contributions are as follows : • We introduce a new completely unsupervised generative neural model , One at a Time ( OAT ) factor learning that combines the stability of VAE training with the accuracy of GAN learning , which contains both a set of independent separable factors and a set of entangled factors , that produces separable and interpretable latent factor codes • Our proposed model is the first unsupervised method that is capable of learning an arbitrary number of latent factors via incremental unsupervised interventions in the latent space • We test and evaluate our algorithm on three datasets and across multiple metrics . Our empirical results strongly suggest that our proposed method is effective in finding interpretable factors and competitive with the most recent disentanglement strategies 2 RELATED WORK . Various authors have attempted to learn unsupervised disentangled representations using generative models in recent years . State-of-the-art approaches for unsupervised disentanglement learning can be broadly classified into two categories based on the type of generative model used ; one via Variational Autoencoders ( VAE ) ( Kingma & Welling , 2014 ; Rezende et al. , 2014 ) , and another via Generative Adversarial Networks ( GAN ) ( Goodfellow et al. , 2014 ) . 2.1 VIA VARIATIONAL AUTOENCODERS . Variational Autoencoders are a deep generative neural network model which learns an approximate posterior distribution of the latent representations from the data , while trying to maximize the data log-likelihood . They can be thought of as an autoencoder with an additional loss term that drives the reconstructions to be closer together in latent space . VAEs assume that the data x is generated from a set of latent features z with a prior p ( z ) according to the model pθ ( x|z ) p ( z ) . The top-down generator pθ ( x|z ) and a bottom up inference network qφ ( z|x ) are modeled as multilayer neural networks and trained jointly to maximize the marginal log-likelihood of the empirical distribution of the training data . However , since the marginal log-likelihood is intractable , VAEs optimize a a tractable lower bound L on the data log-likelihood pθ ( x ) called the Evidence Lower Bound ( ELBO ) : L = Eq ( x ) [ Eqφ ( z|xi ) [ log pθ ( xi|z ) ] − KL ( qφ ( z|xi ) ||p ( z ) ) ] ( 1 ) where q ( x ) = 1N ∑N i=1 δ ( xi ) is the distribution of the training data . The first term measures the reconstruction error and the second term measures the distance between the approximate posterior distribution qφ ( z|x ) and the assumed prior distribution p ( z ) . Many state-of-the-art unsupervised disentanglement methods extend the above objective function to impose additional constraints on the structure of the latent space to match the independent prior assumption . β-VAE ( Higgins et al. , 2017 ) and AnnealedVAE ( Burgess et al. , 2018 ) heavily penalize the KL divergence term thus forcing the learned posterior distribution qφ ( z|x ) to be independent like the prior . Factor-VAE ( Kim & Mnih , 2019 ) and β-TCVAE ( Chen et al. , 2019 ) penalize the total correlation of the aggregated posterior qφ ( z ) . TC = KL ( q ( z ) || ∏K i=1 q ( zi ) ) where the aggregated posterior is calculated as qφ ( z ) = Ep ( x ) [ q ( z|xi ) ] = 1N ∑N i=1 qφ ( z|xi ) using adversarial and statistical techniques respectively . DIPVAE ( Kumar et al. , 2018 ) forces the covariance matrix of the aggregated posterior q ( z ) to be close to the indentity matrix by method of moment matching . Other works improved the performance by making a specific design for discrete factors ( Dupont , 2018 ; Jeong & Song , 2019 ) ; and use optimization techniques based on annealing to encode information effectively in the discrete and continuous factors .

Summary: This study proposes a disentangled representation learning method called "one at a time", or OAT factor learning which is a VAE-GAN network to generate high resolution samples and to learn variational factors in an unsupervised manner, without knowing the number of ground-truth factors a priori. The authors formulate the latent space as a union of two disjoint sets, $z_1$ and $z_2$, where $z_1$ corresponds to the variational (disentangled) factors, and $z_2$ denotes either noise or sample-specific factors. To train the proposed network, the authors use a two-step training approach in which they first train the VAE network with only $z_2$ latent factors. Then, they start to learn $z_{1_k}$, one dimension at the time. To enforce $z_1$ to encode the disentangled factors, they use an intervention-based approach, in which the value of one dimension in $z_1$ is changed, while the remaining $z_1$ factors are kept unchanged. The authors reported their experimental results for two datasets, i.e., dSprites and CelebA.