Disentangling One Factor at a Time
Authors: Vaishnavi S Patil
Matthew S Evanusa
Joseph JaJa
Venue: ICLR 2022 Submitted (2022)
Keywords: unsupervised representation learning
disentanglement
Variational Autoencoders
Generative Adversarial Networks

Abstract: With the overabundance of data for machines to process in the current state of machine learning, data discovery, organization, and interpretation of the data becomes a critical need.   Specifically of need are unsupervised methods that do not require laborious labeling by human observers.  One promising approach to this enedeavour is \textit{Disentanglement}, which aims at learning the underlying generative latent factors of the data.  The factors should also be as human interpretable as possible for the purposes of data discovery. \textit{Unsupervised disentanglement} is a particularly difficult open subset of the problem, which asks the network to learn on its own the generative factors without any link to the true labels. This problem area is currently dominated by two approaches: Variational Autoencoder and Generative Adversarial Network approaches.  While GANs have good performance, they suffer from difficulty in training and mode collapse, and while VAEs are stable to train, they do not perform as well as GANs in terms of interpretability. In current state of the art versions of these approaches, the networks require the user to specify the number of factors that we expect to find in the data.  This limitation prevents "true" disentanglement, in the sense that learning how many factors is actually one of the tasks we wish the network to solve.  In this work we propose a novel network for unsupervised disentanglement that combines the stable training of the VAE with the interpretability offered by GANs without the training instabilities.   We aim to disentangle interpretable latent factors "one at a time", or OAT factor learning, making no prior assumptions about the number or distribution of factors, in a completely unsupervised manner.  We demonstrate its quantitative and qualitative effectiveness by evaluating the latent representations learned on two benchmark datasets, DSprites and CelebA. 

Under review as a conference paper at ICLR 2022
DISENTANGLING ONE FACTOR AT A TIME
Anonymous authors Paper under double-blind review
ABSTRACT
With the overabundance of data for machines to process in the current state of machine learning, data discovery, organization, and interpretation of the data be- comes critical need. Specifically of need are unsupervised methods that do not require laborious labeling by human observers. One promising approach to this enedeavour is Disentanglement, which aims at learning the underlying genera- tive latent factors of the data. The factors should also be as human interpretable possible for the purposes of data discovery. Unsupervised disentanglement is a particularly difficult open subset of the problem, which asks the network to learn on its own the generative factors without any link the true labels. This problem area is currently dominated by two approaches: Variational Autoencoder and Generative Adversarial Network approaches. While GANs have good perfor- mance, they suffer from difficulty in training and mode collapse, and while VAEs are stable to train, they do not perform as well as GANs terms of interpretabil- ity. In current state of the art versions of these approaches, the networks require the user to specify the number of factors that we expect to find in the data. This limitation prevents "true" disentanglement, in the sense that learning how many factors is actually one of the tasks we wish the network to solve. In this work we propose novel network for unsupervised disentanglement that combines the stable training of the VAE with the interpretability offered by GANs without the training instabilities. We aim disentangle interpretable latent factors "one at a time", or OAT factor learning, making no prior assumptions about the number or distribution of factors, in a completely unsupervised manner. We demonstrate its quantitative and qualitative effectiveness by evaluating the latent representations learned on three benchmark datasets; DSprites, 3DShapes and CelebA.
INTRODUCTION
Deep learning models, which are now widely adopted across multiple A.I. tasks ranging from vision to music generation to game playing (Krizhevsky et al.) 2017] Oord al. 2016 Mnih et al. 2015), owe their success to their ability to learn representations from the data rather than requiring hand- crafted features that older models required. However, this self-learning of abstract representations comes at the known cost of the resulting representations being cryptic and inscrutable to human observers. These learned representations might be dramatically affected by noise or spurious corre- lations between the data and the labels the representations might encode 'useless' information from the input data which is correlated with the corresponding label Geirhos et al. (020). This makes them more vulnerable to slight changes in the data distribution. A more comprehensive understand- ing of the data down to essential indivisible factors would allow us learn insights, sort and label data, and facilitate downstream learning more efficiently. This also requires, critically, that these factors somehow interpretable as well. This approach, dubbed Disentanglement requires that we learn the data from its fundamental building block, so-called "disentangling" the true factors the factors of variation - that generate the data. If one were to Iearn these factors, one would Iearn all possible causes of variation in a given dataset, and would in some sense gain complete understand- ing of the underlying machinery, so to speak. In this work, we propose a new method that attempts to leam the true disentangled factors one time, in way that maintains interpretability for future use.
"Code available https//gghha..cmmmoooAaaaaaaAAaa

---

Under review as a conference paper at ICLR 2022
1.1 WHAT IS A DISENTANGLED REPRESENTATION?
While it is easy informally talk of factors of variation, actually pinning down concrete definitions of disentangled learning has proved a somewhat more difficult task. Though there is no commonly accepted formalized notion of disentanglement or validation metries (Higgins et al. 2018). recent works have characterized disentangled representations, based on natural intuition, as one which encodes each informative factor of variation of the data in separate latent dimenions (Bengio 2013), such that a change in single factor of variation produces a change in only subset of the learned latent representation. This is refered to as the separability (Do & Tran (2020) quality of the representations, also called disentanglement (Fastwood & Williams| 2018) and modularity (Ridgeway & Mozer, 2018). Separability ensures that the downstream tasks which depend on a certain subset of factors not affected by changes in other factors thus facilitating robust models Suter et al. (2019).
Some previous works (Ridgeway & Mozer 2018 Eastwood Williams 2018) have suggested that a single dimension of the learned disentangled representation should completely describe factor. This constraint along with separability entails a bijective mapping between the true factors of variation and the learned representation. However, enforcing this constraint might not be conducive to learn complex factors in a single latent variable [Emaaili et al. (001).. Hence in this work we focus primarily separability. For in depth discussion on interpretability and how enforce it in the proposed work, we refer the reader to A.
1 1.2 WHAT IS MISSING FROM CURRENT APPROACHES TO SOLVING DISENTANGLEMENT?
Due to the monumentous task of learning data without supervision, current methods have attempted to tackle the subproblems of separability and interpretability, but few have successfully solved both at once. Learning these disentangled representations in semi-supervised setting Kulkarni et al. (2013): Siddharth et al. (2017); Locatello et al. (2019) is a relatively easier task where additional annotated data is available that give a strong backpropgation gradient the network to guide it to cleanly separate factors. However, if the main goal of disentanglement is to discover unknown factors in large data corpuses, then it is not the right direction to require labels; rather the network should learn in an unsupervised fashion. For unstructured data, all current state of the art approaches based on unsupervised disentanglement rely on deep generative neural network, built either on a Variational Autoencoder or Generative Adversarial Network structure (See Related Work.) Many popular VAE based methods are able to do well in separating factors out but make strict assumptions on the number of factors and their structure and those that do, do not explicitly attempt to also make the factors interpretable. Other methods based on GANs suffer from common issues from training of GANs, but also tend to only learn small subset of factors (See Here we propose novel hybrid network that provides the stability and strong gradients of VAE training with the performance of GAN training, without mode collapse.
Most current VAE-based state of the art methods make the assumption that there are a fixed number of independent factors for all the data points in the dataset. However in real datasets, in addition to the independent factors common to all points in the dataset, there might also be some correlated, nuisance or noisy factors pertinent to specifically only certain data points. Moreover, approaches rely on a heuristically chosen latent dimension d, sufficiently large to encompass all true factors. However, this suffers from the same pitfalls for many of the same reasons that algorithms like k- means cluster do, namely that given a new, unseen dataset, we do not necessarily know how many independent factors there are. In fact, this is one of the main goals of disentanglement in the first place. to glean insights about the data. Our method instead assumes that there is set of independent factors, and one of entangled nuisance and correlated factors, and separates them out. We then iteratively learn to disentangle each factor of variation one at a time, such that the network learns on its own different independently controllable factors thus removing a current hand-tuned roadblock on the way to true unsupervised disentanglement. Second, in the same training loop, we ensure that the disentangled latent representation follows intepretable latent code manipulation (Section. 1.I, which says that change in single latent should make distinct and noticeable change in the output (Eastwood & Williams 2018). Together, we demonstrate our proposed model is able to learn both critical qualities of disentanglement, in completely unsupervised manner.
2

---

Under review as a conference paper at ICLR 2022
1.3 CONTRIBUTIONS
Our main contributions are as follows:
We introduce a new completely unsupervised generative neural model, One a Time (OAT) factor learning that combines the stability of VAE training with the accuracy of GAN learning, which contains both a set of independent separable factors and a set of entangled factors, that produces separable and interpretable latent factor codes
Our proposed model is the first unsupervised method that is capable of learning an arbitrary number of latent factors via incremental unsupervised interventions in the latent space
We test and evaluate our algorithm on three datasets and across multiple metrics. Our em- pirical results strongly suggest that our proposed method is effective in finding interpretable factors and competitive with the most recent disentanglement strategies
2 RELATED WORK
Various authors have attempted to learn unsupervised disentangled representations using genera- tive models in recent years. State-of-the-art approaches for unsupervised disentanglement learning be broadly classified into categories based the type of generative model used; one via Variational Autoencoders (VAE) (Kingma & Welling 2014 Rezende et al. (014), and another via Generative Adversarial Networks (GAN) (Goodfellow al. 2014).
2.1
VIA VARIATIONAL AUTOENCODERS
Variational Autoencoders are deep generative neural network model which learns an approximate posterior distribution of the latent representations from the data, while trying to maximize the data log likelihhodd. They can be thought of as an autoencoder with an additional loss term that drives the reconstructions to be closer together in latent space. VAEs assume that the data is generated from set latent features with a prior p(z) according to the model pp (ar (z))))(z) The top-down generator po(x]z) and bottom up inference network 4a(2 (2(zz) are modeled as multilayer neural networks and trained jointly to maximize the marginal log-likelihood of the empirical distribution the training data. However, since the marginal log- likelioodood is intractable, VAEs optimize tractable lower bound on the data log-likes likelihokh pa (z) called the Evidence Lower Bound (ELBO):
=
Eg(a)) [Bgo(elo)) po (oo - (L((((()=:)ll(()))
(1)
where ((x)= = N E 5(x;) is the distribution of the training data. The first term measures the reconstruction error and the second term measures the distance between the approximate posterior distribution go (z(xx) and the assumed prior distribution p(z). Many state-of-the-art unsupervised disentanglemmet methods extend the above objective function impose additional constraints on the structure of the latent space to match the independent prior assumption. 3-VAE (Higgins et al., 2017) and AnnealedVAE (Burgess al. 2018) heavily penalize the KL divergence term thus forcing the learned posterior distribution go (
be independent like the prior. Factor VAE (Kim & Mnih 2019 and 3-TCVAE (Chen et al. 2019) penalize the total correlation of the aggregated posterior (z)... TC = (L(g(>)]]
(2x)) where the aggregated posterior is calculated as (x(2) Ep(a) [g(2|=.)] + EN (0(2/z,) using adversarial and statistical techniques respectively. DIP- VAE (Kumar et al., 2018) forces the covariance matrix of the aggregated posterior q(2) to be close
to the indentity matrix by method of moment matching. Other works improved the performance by making specific design for discrete factors (Dupont) 2018 Jeong & Song 2019); and use
optimization techniques based on annealing to encode information effectively in the discrete and continuous factors.
2.2
VIA GENERATIVE ADVERSARIAL NETWORKS
Models based GANs, explicitly condition the generator network with set of independent latent variables c (by concatenation with random noise 2), and train the generator to generate data which has high mutual information with The most prominent work from the GAN family is InfoGAN
3

---

Under review as a conference paper at ICLR 2022
(Chen et al. 2016) which learns disentangled, semantically meaningful representations by maximiz- ing a lower bound on the intractable mutual information between the conditioning latent variables c and the generated samples G(z,c).
min max - ((
(2)
G D
where the adversarial loss is given by;
G(D,G) = Ez- (log(,)(((())))) + Bes-pts),--p(eelllg D(G(2,
Assuming a perfect discriminator, the generator tries to minimize the Jensen-Shannon Divergence between the true data distributing and the generated distribution. By changing the value of the conditioning variables, the generator is forced to make distinct and noticeable changes in the data, such that the value of the conditioning variables can be recovered from the generated data alone. InfoGAN-CR (Lin et al. 2020) add a contrastive regularizer to the InfoGAN model, which is trained to predict the changes in the latent space given only the pairs of images generated from the respective latent codes. (Zhu et al. 2020) augment their objective with a similar self supervised learning task to predict the dimension of the latent variable which is different from the pair of images. Some other works based GANs are (Jeon al. 2019 Liu et al. 2020 (Liu et al. 2020) add orthogonal regularization to encourage independent representations.
Alternatively, approaches based the InfoGAN framework find interpretable factors of variation through the Information Maximization principle (InfoMAX). However, in many cases these ap- proaches suffer from mode collapse phenomenon that causes complete failure in training. Crit- ically, GANs lack the ability to provide an explicit modeling of the latent space, as VAEs do (by explicitly learning the parameters of an encoder distribution), as the entire GAN model relies on implicit sampling. We aim to address this issue by our hybrid approach combining VAEs with GAN components.
Suter et al. (2019) introduced the concept of interventions to study the robustness of the learned representations under the Independent Mechanisms (IM) (Schoelkopf et al. 2012) assumption. In this paper, however, we use the method of interventions while training the model to find disentangle common factors from their entangled set. (Lee al., 2020) use VAE to disentangle and then pass into a GAN to generate high-fidelity images; our approach differs in that the GAN component is tightly integrated into our training loop, we perform interventions, and we split the latents into two spaces (See Sec. 3). Hu et al. (2018) condition the latent space of entangled representations with disentangled code, which they learn in a supervised way using specific attribute discriminators. As far as the authors are aware, are the first split the latent dimension into entangled and disentan- gled in a completely unsupervised way, as well as the first to combine this with interventions, and with incremental learning.
OUR METHOD
3.1
GENERATIVE DISENTANGLEMENT
Most previous works on disentanglement take generative view of data where they assume that an unknown generative model has produced the data. The data is assumed be composed from an a-priori set of factors of variation G& = 1.... ,K"), which contains the different numan-defined atomic features that assume different values for specific instances. Here, K' is assumed to be the "true" number of independent factors of variation in the data. The data is then assumed to be gener- ated by a two-step process. First, the values of the different factors of variation, g. are sampled from factorized distribution p(G)= = II P(GG), where y can be any probability distribution. A gen- erator function p(a (G) then maps the specific values, g, sampled from pl (G), a high-dimensional datapoint Thus, p( (z())) describes causal mechanism (Suter et al. 2019) invariant changes in the distributions p(g)). In this generative view the aim of disentangles led representation learning then becomes to uncover these "true" factors of variation from the data alone and re-encode each factor as an independent latent representation.
We posit, however, that this generative view of disentanglement is too limiting and restrictive when it comes to real-world data, as described in section [.I Even though the factors are independent concepts, knowing what observation of obtained renders the different latent causes dependent
4

---

Under review as a conference paper at ICLR 2022
certain factor realizations tend co-oceur more than others characteristic of the dataset. In the proposed work we do not assume any particular factorization of the distribution of the factors of variation and instead separate out the factors through iterative interventions from their entangled counterpart as discussed in further sections. These interventions ensure that the dimensions of the latent representation independently controllable and have causal effect each other. More- over, independent interventions also ensure that the confounding variables for the different factors are integrated out. In addition the generative factors of variation of the dataset, we also model second set of entangled, correlated nuisance factors pertinent to that particular data point. We train our network systematically discern the meaningful latent factors shared across the dataset from the nuisance ones, both of which are important to maximize the log-likelihood of the data (See 3.3 for details).
3.2 DISJOINT ENTANGLED AND DISENTANGLED LATENT SETS
In practise training VAE, guided by the reconstruction loss to maximize the log-likelihood of the data does not lead to disentangled representations, i.c., do not adequately separate out all of the factors within one latent space z. While constraints can be imposed on the latent space to enforce independence between the different dimensions of the learned representations, they can lead to blurry reconstructions (Kim & Mnih 2019) where some factors might be ignored altogether. Because of these practical limitations, we propose a novel method of splitting the latent space z into two disjoint sets: the disentangled set 21 = {2] 2f. ........................
........ 2 where of E R and the entangled set Z2 E Rd_ Here, K is the number of factors that the network learns, and is not predetermined but upper- bounded. This factorizes the generative model as: pe(x, 2) (2(221) II; p(21).
During training, the factors common to the dataset are disentangled into Zı. while the nuisance factors that are either correlated with the other factors in the dataset, or are noise factors that are specific individual samples, but are not representative of the dataset which has a higher encoding capacity. This ensures that the learning of the disentangled latent set not solely driven by the reconstruction cost. Our proposed model is the first VAE-based model to incorporate this notion of two sets of latent variables, which were used in somewhat similar manner in GANs (Chen et al. 2016 Lin et al. 2020).
whole are encoded in z2
3.3 PROPOSED ARCHITECTURE
Our proposed model consists of encoder and decoder in standard VAE, modeled as deep convolutional neural networks, with an additional discriminator network attached (see Fig. [ The encoder and decoder network parameterize the posterior distribution of the latent representation (z(2)z) and the generative model distribution po (z/22) respectively. In addition to the standard VAE, attach a discriminator network D., the output of the VAE decoder. This ensures that the distri- bution of images generated by the intervention procedure (See Sec (.4.2p is close to the distribution of the images in the training set and thus changes in the latent space translate only to interpretable changes. This discriminator distinguishes between the true images in the dataset and generated images by the decoder by using the "real or fake" paradigm of GAN training.
Per the discussion in Section 3.2 the latent representation layer is divided into two sets, a disentan- gled set 2 and a correlated entangled set 22. Thus the entire latent space denoted as = (21, 22). We use the boldface z to represent a vector-valued random variable. The encoder network encodes the data into the two sets of latent variables go, (z) (x) and goz (Z2 (z)]z)) with Φi and 62 sharing weights for set number of initial layers. The two sets are then passed through the decoder po (2) to yield the reconstructed images f.
3.4 TRAINING PROCEDURE
Our training procedure consists of pre- pre-training phase and main learning procedure: a pre-training phase where we train a VAE to maximize the data likelihododd and the main training phase, where the OAT factor learning is performed.
5

---

Under review as a conference paper at ICLR 2022
21
Zk
4x(2)
X
Encoder
Decoder
Disc RVF
(g(z)
X
22
The complete OAT architecture. First, input image, æ, is passed into the VAE en- coder, a deep convolutional neural network (CNN), and encoded via two multi-layered components qa, (21 (x) and qo (22(z:), into two distinct latent spaces, "factorized" and correlated space, 22, which is then decoded by a deep transpose convolutional neural network, produce a reconstructed image â. The insight of OAT training is that may be possible to decorrelate all of the data for various reasons, so we first group the correlated latents into one space, 22, and then "peel off"" each independent factor one at time. Next, an intervention is made on one latent variable in the new disentangled space, 2 creating new latent z². which is passed through the decoder to produce a new image &t_ This factor-reconstruuted a-1 is then passed back through the encoder to ensure the encoder learns how to encode that particular factor change into the same intervention-alle altered disentangled latent z The fector-reconstrutee values & are then passed into discriminator D., along with real images to ensure that the factor-altered reconstructions remain realistic.
Figure 1:
disentangled space, 21,
3.4.1 VAE PRE-TRAINING
We first train the encoder and the decoder to maximize the likelihood of the training data under the generative model Pa (a: 22) as detailed in equation [ In this step, only the Ze latent set is learned; which encodes all the informative factors variation albeit in an entangled way. Since we do not enforce any extra independence constraints the latent space, the learning of the latent space is driven by the reconstruction loss. The posterior distribution Tøs (Zg (z) is regularized to be similar to the zero mean, unit variance, isotropic Gaussian prior p( (2). Thus, the objective function aim to minimize is evidence lower-bound (ELBO) of the data log-likelihood as in a regular VAE.
LELBO
Begg.s (B (Z2) (p 22)] KI. (2g (2) (z))22)(((2
where p(z2) (0, (a)
The first term in the above objective function minimizes the reconstruction error of the data points from the latent representations alone. This ensures that the latent representations encode the impor- tant information the data all the different factors of variation in the dataset.
3.4.2 OAT FACTOR DISENTANGLING
The main contribution of our work occurs the this stage, where perform OAT factor disentan- glement. We outline the process as a two-step process:
Step 1: Passing Through The Disentangled Latents
Once the pre-training is completed, the reconstruction loss saturates and all the informative factors encoded in 22, however they highly entangled. In step perform the same VAE training pass as in the pre- pre-training phase, but of dimensions learned until that point. This will eventually allow the model to encode information in
pass the data also through 2 where & is the number
Zi from the data in disentangled way instead of their entangled representations in 22. For brevity

---

Under review as a conference paper at ICLR 2022
denote the set (z1 l:ke Z2) as Thus objective function for this step is as follows:
K
C1 = Bggn L (2/22/2 Dac
KL (p1xx)
(5)
where p(21)= = I p(2)) W 1). Here Y/t is a weighting factor that acts as a mask, which indicates turning off latent elements of 21 that are not being currently trained as part of the OAT iterative procedure. During the pre-training o = 1: K) are set to zero. At the beginning of the OAT training procedure, only the first element of 21, 21. is learned, with ?
and for the latents with N 2 k} 0 the gradients are not calculated and the weights are not updated. This constitutes a core of the "one at time" component, and is critical for the network's functionality: by focusing on one latent factor at a time, we can iteratively learn and discover each latent factor at different points during the training. As the training proceeds the value of more y are flipped to which allows more latents in Z1 to be seen by the network.
The value of 3 is increased linearly during training to encourage the model to encode information in 21 instead of z2- This along with the intervention described in the further sections ensure that the posterior distribution of the disentangled set does not collapse to the prior.
Step 2: Interventions and Discrii-isrr In order to ensure that independent, in- terpretable factor of variation is encoded in zi. we perform interventions on each of the learned dimensions of 21. Thus it is important for any partieular dimension i in 21, that changing the value of another dimension 2 # i) should not change the value of of when the corresponding gener- ated data from the intervention is re-encoded. Interventions: For the intervention procedure, we start with a learned representation z {Zz, žz} such that 21 ~ qo, (Zz (z) and Z2 qø (z /z)), encoded from an some datapoint x. We then uniformly select dimension & E [K] from the learned dimensions of 2 to change it's value during intervention. We sample a new value for the dimension
k from the prior distribution p(=f) say to create a new representation zt (fc, 2 }. 22). Thus the two representations are same in all dimensions except k. The new representation then passed through the decoder to generate a data point ht. This generated data is then re- re-passed back through the encoder to obtain the representations 2 (1è, " 2222 sampled from the encoder distribu- tions qs, (z (=*) and qdu (zzlä") respectively. Both the encoder and the decoder are then trained to reconstruct xt according to 6 We refer to this procedure of altering a single factor, combined with data generation and re-passing through the encoder, an intervention, per Suter et al. (2019).
While we want to reconstruct the intervened latent variable, we also want to ensure that the changes, the intervened latent variable makes in the corresponding generated data, are interpretable. This is ensured by constraining the distribution of the generated data to lie strictly in the true data man- ifold. Thus particular generated datum differs from the training datum in subset of factors of variation, ideally in only one factor corresponding to the intervened latent variable. To address this, train the discriminator Da to differentiate between the training data and the generated data. The decoder weights are updated to generate data which would fool the discriminator. Thus the complete objective function can be written as follows;
C2 Epp (p (=)(((=)))))))))))))()) [1s [xs(2* [=*))*)] (KL [1)))))))
The first term in the above objective function minimizes the reconstruction error of the re-encoded, intervened latent representation from the generated data. Because the intervened representations are reconstructed from the generated data alone, the decoder is forced to make distinct changes in the generated images corresponding to the different dimensions of 2 This ensures that the disen- tangled set 2 is independently controllable and interpretable, desirable property of disentangled representations.
Since it is difficult to compute analytically the high-dimensional true data distribution from the sam- ples alone, the KL divergence in the second term is replaced with the (JS (JS-divergence). In Goodfellow et al. (2014) given optimal discriminator network D described in Sec. 3.3 training the decoder to fool the discriminator essentially minimizes the JS-divergence between the generated distribution and the training images.
Divergence
During the training, step and step 2 are performed one after the other in an alternative fashion, similar other two-step learning algorithms such as wake-sleep (Hinton et al. 2006). This process
7

---

Under review as a conference paper at ICLR 2022
Model
Factor Vae
MIG
DCI
BetaVAE
VAE
0.63#.06
0.10
0.30
3-VAE
0.63*.10
0.21
0.41
FactorVAE (y=40)
0.82+.01
0.43+.01
0.74
0.84+.01
3-TCVAE
0.62+.07
0,45
0.29
InfoGAN
0.82#.01
0.22+.01
0.60#.02
0.87#.01
InfoGAN-CR
0.88+.01
0.37+.01
0.71
0.95#.01
OAT (3,=10, 22=10)
0,82
0,36 + I3
0.78# .01
0.80
OAT (2) = z2=10)
0.84 = .12
0,44 + .03
0.74 .05
0.82 .10
OAT (z 22=10)
0.78 + .08
0.33 + .11
0.77# .01
0.76 .08
Table 1: Comparisons of the popular disentanglement metries on the dSprites dataset. A perfect disentanglement corresponds to 1.0 scores. The scores are averages over 10 runs with different random seeds.
has the effect of minimizing the symmetric KL divergence between the joint generative and inference distributions pe (2. 2) and (x,
During training. the first latent variable of the set 2 is trained and thus 71 's value is changed from 0 to This latent is trained using interventions until it's KL-divergence saturates and it can encode more information. Then, start training the next latent variable from the value of 72 is switched from 0 to 1. The stopping criteria for this process depends on the dataset. If the dataset has fixed number of factors which independent like in the synthetic datasets dSprites and 3DShapes, the process of adding more latents is stopped when the KL-divergence of the entangled set, 22, goes zero and it encodes no more information. That is when we can say that all the factors of variation have been found. For real datasets like CelebA, we can find continue finding factors until the desired number of factors have been found.
i.e. of and the
EMPIRICAL EVALUATION
For quantitative evaluation, we run experiments on two synthetic datasets generated from indepen- dent ground truth factors of variation; dSprites Matthey et al. (2017) and 3DShapes [Burgess & Kim (2018). For qualitative evaluation we use real dataset with unknown factors of variation, the CelebA dataset Liu et al. (2015).
We evaluate the learned representations with one metric from each the three kinds of metrics as described in (aidi et al. 2021). We use the FactorVAR (Kim & Mnih 2019), the Beta BAFHHgggisg (ct al.) 2017). Mutual Information Gap (MIG) (Chen et al. 2019) and the Disentanglement- pwaplleteeees-nfooonatvveee
(DCI) (Eastwood & Williams/ 2018) metric to quantitatively eval- uate disentanglement on dSprites (T) and 3DShapes (2). We also qualitatively evaluate the models trained on CelebA 4 dSprites 4 3D shapes (ref: A.4). More details regarding the implementation and the metrics and the latent traversals can be found in the appendix ??
We also perform ablation studies the dimension of 21, K, and Z2, d, on the dSprites dataset. We find that increasing d beyond a certain range (5-10), does not affect the metrics but make the reconstructions sharper shown in A.4 The value of K affects how the factors that encoded in the different dimensions of z and thus affects the metrics.
4.1 DISCUSSION
From Table we see that OAT beats the previous baselines on the DCI metric while performing comparably the other metrics. We noticed that in the DCI metric, OAT consistently got high scores for informativeness and disentanglement, but lower scores for completeness. The complete- ness score is high if each factor is completely captured by only a single latent variable. However, as pointed in Eastwood & Williams (2018) enforcing completeness might be counter-productive to learning disentangled representation for complicated factors like rotation.
With the latent traversals we see that the factors are not confined to single latent dimensions but instead encoded several dimensions. For example, complicated factors like rotation and
8

---

Under review as a conference paper at ICLR 2022
Figure 2: (Left): Traversals for the CelebA dataset show that 01,(=;=10,22=24) is able to disen- tangle multiple unique factors smoothly on real-world data, with unknown true factors. (Right): Results showing the OAT (z1=5,2-=10) architecture disentangling various factors for a synthetic dataset. From top to bottom: oval heart, oval to square, rotation, and size. The rest of the val- ues for different factors are encoded in different dimensions as completeness is not enforced during disentanglement
Model
Factor Vac
MIG
DCI
FactorVAE (y=10)
0.33 = .06
0.14 .02
0.75 ,01
Factor 40)
0.42 .01
0.32 # .03
0.73 2 02
3-TCVAE
0 .01
0.38 .02
0.76 - 001
InfoGAN-CR
0.68 .01
0.26 .03
0.53 + 00
OAT (2) 10=
0.64 .18
0.39 + 0.04
0.77 .01
Table 2: Comparisons of the disentanglement metries on the 3DShapes dataset averaged over 10 runs with different random seeds.
shape are encoded over two latent dimensions. This possible explanation for the lower MIG scores the MIG score essentially computes compactness along with informativeness while giving less importance to modularity. As pointed in (Zaidi et al. 2021) MIG has a high score even if one latent dimension encodes information about multiple factors as long one factor is only encoded by one dimension. This also explains why OAT(2==5, 10= has a higher MIG score than the other OAT models. However simpler factors like size are consistently encoded in a single latent variable.
From the ablation studies we see that when |21| is large, some factors are encoded in more dimen- sions of 21, whereas when |2)] is small, more than one factor forced into single dimension of 21. Thus, this model performs best when |=] is larger than the true number of factors of variation if known.
We also notice that for some factors like y-axis, x-axis there is an abrupt change in the traversals. We believe this is an artifact of sampling interventions from the prior, and would change if a different intervention method is used. We use zero mean unit variance Gaussian distribution for sampling. In future works we plan to experiment with sampling from different distribution like learned priors and approximate posterior distribution.
CONCLUSION
this work we present a novel generative neural network framework for unsupervised disentan- glement, One at Time (OAT) Factor Learning. We demonstrate that with the use of unsupervised interventions, the network is able to learn smooth traversals across each latent dimension, and that the latent dimensions learned informative, interpretable and separate without the of labels. With the addition of the two separate latent spaces, OAT is able to learn an arbitrary number of factors, whereas before the latent dimension had be pre-determined. Due to its design, it is able

---

Under review as a conference paper at ICLR 2022
to find balance between the training stability of VAEs with the generative quality of GANs, while catering to real datasets with arbitrary number of factors. We hope that this work paves the path towards learning disentangled representations from real world datasets.
REFERENCES
Yoshua Bengio. Deep learning of representations: Looking forward, 2013.
Chris Burgess and Hyunjik Kim. 3d shapes dataset. http://gihibbboomeeeemmm////aass 2018.
Christopher Burgess, Irina Higgins, Arka Pal, Loic Matthey, Nick Watters, Guillaume Desjardins, and Alexander Lerchner. Understanding disentangling in f-vae, 2018.
Tong Che, Yanran Li, Athul Paul Jacob, Yoshua Bengio, and Wenjie Li. Mode regularized generative adversarial networks, 2017.
Ricky T. Q. Chen, Xuechen Li, Roger Grosse, and David Duvenaud. Isolating sources of disentan- glement in variational autoencoders, 2019.
Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. Info- gan: Interpretable representation learning by information maximizing generative adversarial nets, 2016.
Kien and Truyen Tran. Theory and evaluation metrics for learning disentangled representations, 2020.
Emilien Dupont. Learning disentangled joint continuous and discrete representations, 2018.
Cian Eastwood and Christopher K. I. Williams. framework for the quantitative evaluation of disentangled representations. In International Conference on Learning Representations, 2018. URL htttps:///peeneeriii wttp://iwww...eeeooimmmeeeemeeee
Babak Esmaeili, Hao Wu, Sarthak Jain, Alican Bozkurt, N. Siddharth, Brooks Paige, Dana H. Brooks, Jennifer Dy, and Jan-Willem van de Meent. Structured disentangled representations, 2018.
Robert Geirhos, Jörn-Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A. Wichmann. Shortcut learning in deep neural networks. Na- ture Machine Intelligence, 2(11):665-673, Nov 2020. ISSN 2522-5839, doi: 10.1038/
Utthttp:// wtw. www.dai........e.ooeo..eooeee..e 10.
Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley. Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks, 2014.
I. Higgins, Loie Matthey, A. Pal, C. Burgess, Xavier Glorot, M. Botvinick, S. Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with constrained variational framework. In ICLR, 2017.
Irina Higgins, David Amos, David Pfau, Sebastien Racaniere, Loic Matthey, Danilo Rezende, and Alexander Lerehner. Towards definition of disentangled representations, 2018.
Geoffrey E. Hinton, Simon Osindero, and Yee Whye Teh. fast learning algorithm for deep belief nets. Neural Computation, 18:1527-1554, 2006.
Zhiting Hu, Zichao Yang, Xiaodan Liang, Ruslan Salakhutdinov, and Eric P. Xing. Toward con-
trolled generation text, 2018.
Insu Jeon, Wonkwang Lec, and Gunhee Kim. IB-GAN: Disentangled representation learning
with information bottleneck GAN, 2019. URL
Fy15V2A5KX
Yeonwoo Jeong and Hyun Oh Song. Learning discrete and continuous factors of data via alternating disentanglement, 2019.
10

---

Under review as a conference paper at ICLR 2022
Hyunjik Kim and Andriy Mnih. Disentangling by factorising, 2019.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2017.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes, 2014.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convo- lutional neural networks. Communications of the ACM, 60(6):84-90, 2017.
Tejas D. Kulkarni, Will Whitney, Pushmeet Kohli, and Joshua B. Tenenbaum. Deep convolutional inverse graphies network, 2015.
Abhishek Kumar, Prasanna Sattigeri, and Avinash Balakrishnan. Variational inference of disen- tangled latent concepts from unlabeled observations. In International Conference on Learning Representations, 2018. URL httpa/////pprreveeeeeeeeee//eeee
Wonkwang Lee, Donggyun Kim, Seunghoon Hong, and Honglak Lee. High-fidelity synthesis with disentangled representation, 2020.
Zinan Lin, Kiran Koshy Thekumparampil, Giulia Fanti, and Sewoong Oh. Infogan-cr and model- centrality: Self-supervised model training and selection for disentangling gans, 2020.
Bingchen Liu, Yizhe Zhu, Zuohui Fu, Gerard de Melo, and Ahmed Elgammal. Oogan: Disentan- gling gan with one-hot sampling and orthogonal regularization, 2020.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV), December 2015.
Francesco Locatello, Stefan Bauer, Mario Lucic, Gunnar Raetsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. Challenging common assumptions in the unsupervised learning of disentangled representations. In Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 4114-4124. PMLR, 09-15 Jun 2019.
Loic Matthey, Irina Higgins, Demis Hassabis, and Alexander Lerchner. dsprites: Disentanglement testing sprites dataset. http://hhhb...oo////ee/mn///ssss
2017.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Belle- mare, Alex Graves, Martin Riedmiller, Andreas Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning, nature. 5 (177540):529-5333 2015.
Aaron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. Wavenet: A generative model for raw audio. arXiv preprint
2016.
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models, 2014.
Karl Ridgeway and Michael C. Mozer. Learning deep disentangled embeddings with the f-statistic loss, 2018
Bernhard Schoelkopf, Dominik Janzing, Jonas Peters, Eleni Sgouritsa, Kun Zhang, and Joris Mooij. On causal and anticausal learning, 2012.
N. Siddharth, Brooks Paige, Jan-Willem van de Meent, Alban Desmaison, Noah D. Goodman, Push- meet Kohli, Frank Wood, and Philip H. S. Torr. Learning disentangled representations with semi- supervised deep generative models, 2017.
Raphael Suter, ore Miladinović, Bernhard Schölkopf, and Stefan Bauer. Robustly disentangled causal mechanisms: Validating deep representations for interventional robustness, 2019.
Julian Zaidi, Jonathan Boilard, Ghyslain Gagnon, and Marc-André Carbonneau. Measuring disen- tanglement: A review of metrics, 2021.
Xinqi Zhu, Chang Xu, and Dacheng Tao. Learning disentangled representations with latent variation predictability, 2020.
11

---

Under review as a conference paper at ICLR 2022
A APPENDIX
A.1 INTERPRETABILITY OF DISENTANGLED REPRESENTATIONS
To be of use for downstream tasks, successful transfer learning and domain adaptation (Bengio) 2013) or to glean insights from observers be them machines or humans, the representations must somehow be interpretable. Interpretability has a simple intuitive understanding: that each factor represents a human-defined concept regarding the data, which humans could easily understand and identify. In practice, however, this is only intuitive because these are concepts pre-trained human brains, and are not as clearly separable for machines without any imported biases; it is easy for the network to fuse multiple of what appear to humans as "essential factors" into one latent code while still keeping the factors independent and orthogonal one another other; As example, the network simply can learn a rotation of the "human" representation, which would appear from an observer combine factors together. To address this, define interpretability here new form not as uniquely human interpretable persay but as interpretable latent representation manipulation: each individual atomic code should make a unique and noticeable change in the output. Critically we are not defining noticable as being for humans only, as emphasized by the rotational example above, but by any observer.
A.2 PROOFS
We assume a generative model pa 2) po (=]2))((() where p(z) is the prior assumed on the latent space and po (z]=z) is modelled by the generator. The corresponding inference model is (o =) (2(2(z))) (x)) where ((x) governs the empirical data distribution of observed data while the inference model is qo (2(x).
The ELBO in [ can further be written as the KL divergence between the inference model and the generative model as follows:
L Extx) (ogg (L(
(1,,) [log (x((2)) (xtr:))(( [[(2))
BB (=lx) log (=(x))
(m.s) [og po
po (x|²)]
p(2)
(m.,))
[log po (o (x|2) log qo (=(1r)) + +oggp(2)]
(m.)) blog po (2, 2) log (2(x)
(m.,)) po 2) log (=(1)) log ((x) - log {(x)]
(t.,))
logpo(<; 2) - log (, 2)] + E (x (2x)
(L (xl 2)) H(((z))
S (1, (1
By maximizing the lower bound L we minimize the KL divergence, which is equivalent to the maximum likelihood objective as follows:
Minimizing the first term gives us the maximum likelihood objective. Given the asymmetric nature of the KL divergence this ensures that generative model po(x:) has non zero probability for every x ~ q(x). However, this does not ensure that the model gives low probability to images not in q(x). Thus seek to also minimize the reverse KL divergence.
The ELBO in [ can further be written as the KL divergence between the generative model and the inference model as follows:
12

---

Under review as a conference paper at ICLR 2022
2 = (yss) Po log (2|2=)) - -L
E pu [log (x(22) - E (Le (=pp(<
E pu 9o (x|22] - Bp(a) (1rlz) log
g(xr)
po (x)z)) - log (o log g (x))
pp [log log pa (a [z)]
(oo(e,))0oogg0))) log (z. 21) log p())
(1) (p(z)))
By minimizing the first term ensure that the learned aggregated posterior gx(2) is similar to the factorized prior distribution. This is important for disentangling the factors, as we want the learned representation to be independent like the prior.
Thus we see that by maximizing both L1 + L2 we minimize the symmetric KL. divergence.
A.3 EXPERIMENTAL DETAILS
For our experiments we use the standard VAE architecture proposed in Kim Mnih (2019) for the encoder and the decoder model. We use [0, normalised data as targets for the mean of Bernoulli distribution log po (z)]z). We use the negative cross entropy as the reconstruction loss and optimize the model using the Adam optimiser (Kingma & Ba 2017) with learning rate of le 04 for the dSprites dataset and 5e 05 for the 3D shapes dataset and the CelebA dataset.
For the discriminator use the architecture proposed in Lin al. (2020) and use the Adam op- timizer with a learning rate of le 04 and le 05 for dSprites and 3Dshapes respectively with 31 = 0.9 and 32 0.999.
We train the models on the dSpirtes dataset for 30 epochs and on the 3DShapes and the CelebA dataset. for 50 epochs with batch size of 128. To calculate the metries we use 10 different random seeds and calculate the average of all the runs for the scores.
A.4 FURTHER ABLATION RESULTS
We experimented with different dimensions of Z2 and noticed that a larger Zzl produces crisp re- constructions for datasets with fixed number of factors of variation can be seen in(A.4 and A.4 However, the metrics do not change much as the dimension of 22 is increased beyond 10.
For real datasets too notice that larger 22 produces reconstructions with more details and more factors. This facilitates the discovery of more factors in the disentangled set during the OAT proce- dure be seen in ??
A.S EVALUATION METRICS
For implementation details and hyperparameter settings of the metrics, we directly follow the set- tings in Locatello et al. (2019). Our VAE architecture is the one Kim & Mnih (2019) use in their experiments and the discriminator architecture is based on Lin et al. (002)).
A.5.1 BETAVAE, FACTORVAE
We evaluate the learned representations with one metric from each of the three kinds of metrics as described in (Zaidi et al. 2021). The intervention- metrics compare representations by more ground-truth factors kept constant. These metrics
creating subsets of data in which do not make any assumptions on the factor-code relations which is their main advantage. We use the Factor- VAE metric from the intervention-based metrics kind. In this metrie, in a batch factor
13

---

Under review as a conference paper at ICLR 2022
Figure 3: Top: The original input, Bottom: Reconstructions after passing through trained full OAT (x)=10,z==10) model (as described in Experimental Setup.) Note that the model produces crisp, non-blurry, reconstructions.
Figure 4: Training images (left) and the corresponding reconstructions (right) for 21=10 and z2=20.
G2 is chosen randomly. Then, a fixed number of pairs from the data are selected where the value of the factor G; is the same. The intuition is that representation dimensions associated with the fixed factor should have the same value, which means a smaller difference than the other representation dimensions. Finally, linear classifier is trained on the data set to predict which factor was fixed. The accuracy of the classifier is the score.
A.5.2 DCI
Predictor-basee metrics use regressors or classifiers to predict factors from the representations. These metrics train models to predict factor realizations from the representations. Then the use- fulness of each code dimension predicting given factor is analyzed. These methods are naturally suited to measure explicitness. We use the DCI-Lasso and (Fastwood & Williams 2018 metrics to measure explicitness and modularity.
A.5.3 MIG
Information- based metrics compute disentanglement by estimating the mutual information (MI) between the factors and the representations. These methods require fewer than intervention-based and predictor-based metrics. Moreover, they do not make assumptions
14

---

Under review as a conference paper at ICLR 2022
Figure 5: More Traversals for the CelebA dataset, for the same (no cherry picking between runs.) OAT is able to disentangle multiple unique factors smoothly on real-world data, with unknown true factors.
Figure 6: Traversals for the 3D shapes dataset with |2j |= and |22|=10 Factors from top to bottom:
shape, floor color, orientation, size, wall color, floor color.
15

---

Under review as a conference paper at ICLR 2022
the nature of the factor-representttion relations. We use the MIG (Cen et al. 2016) measure all the three facets of disentanglement.
A.5.4 LATENT TRAVERSALS
For the CelebA dataset the ground truth values or the factors of variations are not known apriori. Therefore we use latent traversals as a way to measure disentanglement qualitatively (Che et al. 2017). Disentanglement evaluated qualitatively by traversing the latent space, by fixing all the dimensions of the representations except one and varying the values of that one dimension. For the varying dimension discrete values are sampled over its distribution and the resulting generated samples visualized. A model has better disentanglement if for different values of the varying dimension, the resulting generated samples have distinct and noticeable change for an interpretable factor of variation. We also qualitatively evaluate the models trained on dSprites and 3D shapes (ref: A.4)
A.6 DOWNSTREAM CELEBA TASKS
To show that our method learns more informative factors than current state the art approaches, we perform number of downstream tasks on the learned disentangled representations. The CelebA dataset has 40 binary attributes for each image in the dataset. We extract the latent representations from the best disentangling models, namely, 8-VAE, Factor- VAE and the 3-TCVAE and train 40 SVMs to predict each the binary attributes. We use 1000 (1k) and 10000 (10k) examples as a training set for these SVMs and evaluate it's performance 2k examples in the test set. The results of the table are calculated by taking the average of the accuracies of the 40 SVMs.
Model
kemel: 10k
kernel:linear, 1k
kernel.rbf, 10k
kernel.rbf, 1k
3-VAB
0.64
0.62
0.69
0.66
FactorVAE (y=40)
0.72
0.71
0.75
0.74
3-TCVAE
0.74
0.74
0.79
0.77
OAT (2:=10, =0)
0.79
0.78
0.81
0.80
Table 3: Average accuracy over 40 SVMs trained for each model representation, using different number training examples.
Thus we see that the representations learned by OAT are more informative than the state of the art VAE-based approaches. We believe that OAT will perform better in the downstream tasks where certain factors might not be present in all the images but only in a subset of them.
16


Reviews:
[{"review_id": "XnlXYJO94sf", "reviewer": "Reviewer_Uqhc", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper addresses an important problem. However, it has several issues, including inaccurate claims, insufficient experiments, and many typos. See below for more details.\n\n*** Inaccurate claims ***\n\n* Page 2: 'Current state of the art methods make the assumption that there are a ﬁxed number of independent factors for all the data points in the dataset. However in real datasets, in addition to the independent factors common to all points in the dataset, there might also be some correlated, nuisance or noisy factors pertinent to speciﬁcally only certain data points'. Note that many GAN-based approaches (e.g., InfoGAN) only regularize a small part of latents; they can indeed model 'correlated, nuisance or noisy factors' (using the non-regularized latents) besides the 'independent factors' (using the regularized latents). Please fix this statement. (I see that you do admit it later at the end of Section 3.2.)\n\n* The paper mentions in many places that the idea of progressively learning one factor at a time is new. However, a similar idea has already been proposed in 'Robust Disentanglement of a Few Factors at a Time' at NeurIPS 2020.\n\n*** Insufficient experiments and results ***\n\n* From table 1 it seems like the proposed approach performs much worse than the state-of-the-arts like FactorVAE and InfoGAN-CR. \n\n* If the main selling point of the paper is the ability to disentangle real-world datasets with **unknown number of factors**, then the paper should conduct experiments to demonstrate that (i.e., on the settings when the approaches that set a pre-defined number of factors fail, and show how your approach can improve that)\n\n*** Typos ***\n\n* Page 2: 'related Work'\n\n* Page 3: '/mse'\n\n* Page 5: missing a period after 'factors are integrated out'\n\n* Figure 1 caption: 'so we ﬁrst group the correlated latents into one space, z_1'. Here z_1 should be z_2?\n\n* Page 7: 'z_2.For brevity' -> 'z_2. For brevity'\n\n* Page 7: '\\gamma_ii = 1:K' -> '\\gamma_i i = 1:K'\n\n* Page 7: '\\gamma_1=1 and \\gamma_i1=2: k=0'\n\n* Page 7: sometimes you use k to denote the number of learned disentangled latents (e.g., the line before Eq. 5), and sometimes you use K instead (e.g., Eq. 5 and 'We then uniformly select a dimension k ∈ [K]').\n\n* Page 8: 'Kim & Mnih (2019)use' -> 'Kim & Mnih (2019) use'\n\n*** Other suggestions ***\n\n* Page 2: You could probably add refs to the corresponding sections for 'related work' and 'preliminaries'\n\n* Eq. 6: I understand from the text that you are matching the intervened latents, but this equation does not express it precisely.\n\n* Figure 2, 3: You could label the factors besides the images to make them easier to read.\n", "summary_of_the_paper": "The paper proposes a new approach for training disentangled generative models. On top of VAE, it separates the latents into two disjoint groups: disentangled and entangled ones, and progressively increases the size of the disentangled group one at a time. To encourage disentanglement, it changes one random dimension of disentangled latent and pushes it to decode and then encode into the same one, and has a GAN loss to make sure that the changed generated distribution matches the ground-truth distribution. The paper shows the performance of the proposed approach on dSprites and CelebA datasets.", "summary_of_the_review": "In summary, (1) the main selling idea of 'disentangling one factor at a time' is not new, (2) the results are worse than the state-of-the-arts, and are not sufficient to show the benefit of 'disentangling one factor at a time'. Therefore, I have to give a negative score.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "THenzoxfc9b", "reviewer": "Reviewer_E94u", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Strengths\n- Interesting approach to combine the strengths of VAE for disentangled representation with the power of a discriminator.\n\n### Weaknesses\n- Not really convincing experimental results in Table 1\n- Lack of clarity and details regarding the training procedure (see questions)\n- Lack of ablations with respect to the number of learned disentangled factors $k$, which is a central part of the proposed pipeline\n\n### Questions\n- The authors of the ID-GAN paper (Lee et al: High-fidelity synthesis with disentangled representation, 2020) write about the trade-off between generative quality and disentangled representation. Did the authors also experience that? Or was the generative quality not part of their evaluation?\n- What about model selection? The authors do not write much about how they selected their models. The training procedure involves multiple steps and to me it is not straightforward how to select the model in an unsupervised manner for every training step out of a range of multiple possible hyper-parameters. It would be interesting to also see an ablation study regarding the most important hyper-parameters.\n- Table 1 is a bit confusing. What is the scheme of making some entries bold? It does not seem the highest value per column. In general, I would have appreciated a more detailed description of the results presented in Table 1. Also, I cannot find a reference to table 1 in the text.\n- There is no ablation on the number of learned factors of variations $k$. To me, it is not obvious that the right number of disentangled factors can be found. And for table 1 it is not even disclosed what the respective k is, nor compared to the latent space size of previous works.\n- What are the motivation for another researcher to use your method? It does not seem to be able to outperform previous methods, nor does it seem to have a simple training procedure.\n- In section 3.4.2., step 1 the authors write that the $\\gamma_i$ are set to zero and, hence the corresponding latent factors are not trained. Although it is true, the regularisation for these factors is not part of the objective, in my opinion,  information flows through this part of the network nevertheless and the weights will be updated through their contribution to the reconstruction loss. I would appreciate the authors' thoughts on that.\n- Also in section 3.4.2, step 1, the authors describe their one-at-a-time training approach by switch the $\\gamma_i$ from 0 to 1. I cannot find any information on how this is scheduled during training nor how it is stopped. I understood that the number of disentangled factors $k$ is not a hyper-parameter (only upper-bounded) as it is learned during training. I would appreciate a bit more details on that.\n- Is it a goal of the proposed method that no information correlated to the disentangled factors $z_1$ is encoded in $z_2$ (low mutual information between the two representation)? If so, how is this achieved? Did the authors explore that direction?\n", "summary_of_the_paper": "The authors propose a new method to learn a disentangled representation. The proposed method is Variational Autoencoder (VAE) based. The latent variables consist of nuisance factors and disentangled factors that form the $k$ generative factors per dataset. To help learn the disentangled factors of variation the authors propose to use an additional discriminator and interventions on the disentangled factors.\nThe authors demonstrate the performance of the proposed model on the dSprites and CelebA dataset.", "summary_of_the_review": "Although this work presents an interesting approach to learning disentangled representations, in my opinion it is not ready to be published at ICLR at this stage. It lacks convincing arguments why a researcher in this area should choose the proposed method. In my opinion what is needed to improve the paper is more clarity and details regarding the training procedure, e.g. model selection criteria for the different training stages, ablation studies on the learned number of disentangled factors $k$ and a better description of their central and only results table.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "KHDJTNk_fXO", "reviewer": "Reviewer_EBrM", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: \nI think this work lacks notable originality and novelty. The VAE-GAN networks (proposed by Larsen et al., 2016) have been studied in many papers. The proposed formulation of latent space that splits latent factors to two disjoint sets is also not novel. The only novel part is assuming an upper bound for variational factor, where it is not clear how the model can find the true number of factors, while $|z_1|$ is gradually increasing up to $K$ over iterations of training.\n\nQuality: \nThe motivation of the work is good; learning a disentanglement representation is quite a challenging problem. However, the paper could not appropriately address this challenge and the authors’ contribution is quite limited. The paper does not offer any theoretical analysis or profound experimental studies. The reported results for dSprites and CelebA do not showcase the superiority of the proposed method, neither in terms of existing disentanglement scores nor quality of the reconstructed images. \n\n\nLimitations: \n- It is not clear how the suggested formulation for the latent space is enhancing the disentitlement.  I cannot see why we need to encode $z_2$ while it is expected to encode noise or sample-specific variations, which is not desirable in the latent representation learning. \n\n- While the authors claim that the model does not need to know $|z_1|$, it is not clear how the model learns the true $|z_1|$. If I am not wrong the model iteratively unmasks the dimensions of $z_1$, without any stopping criterion. \n\n- The experimental study is limited to two datasets, without sufficient ablation studies on each dimension of $z_1$ and $z_2$. It is critical to explore the impact of $z_2$ vs. $z_1$ and quantify the correlation between these factors. \n\n- Additionally, I think the recent study by Träuble et al. ICML2021, and its discussion on the correlated latent space is quite relevant to this work. The authors might need to refine the discussion on the casual factors and disentailment studies, based on the findings of this paper.\n\nMinors comment:\n- Page 7: “The value of $\\beta$ is increased linearly during training to encourage the model to encode information in $z_1$ instead of $z_2$.” $\\beta$ or $\\gamma$? I got confused. \n- There are some typos in the text, e.g., “/mse”, “a a” at page 3.\n- The table and Figures are not cited in the manuscript. \n", "summary_of_the_paper": "This study proposes a disentangled representation learning method called \"one at a time\", or OAT factor learning which is a VAE-GAN network to generate high resolution samples and to learn variational factors in an unsupervised manner, without knowing the number of ground-truth factors a priori. The authors formulate the latent space as a union of two disjoint sets, $z_1$ and $z_2$, where $z_1$ corresponds to the variational (disentangled) factors, and $z_2$ denotes either noise or sample-specific factors. To train the proposed network, the authors use a two-step training approach in which they first train the VAE network with only $z_2$ latent factors. Then, they start to learn $z_{1_k}$, one dimension at the time. To enforce $z_1$ to encode the disentangled factors, they use an intervention-based approach, in which the value of one dimension in $z_1$ is changed, while the remaining $z_1$ factors are kept unchanged. The authors reported their experimental results for two datasets, i.e., dSprites and CelebA.", "summary_of_the_review": "Learning a disentangled and interpretable representations is an important topic in the machine learning. The proposed method tried to address this challenge. However, I think the contribution is not notable and the paper lacks theoretical and empirical justifications to support its claim. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "8D0VS-N95fD", "reviewer": "Reviewer_roR5", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is well motivated and the OAT technique is kind of novel and impressive. However, paper clarity can be improved, which has impeded reader's understanding and rating of the paper.\n\nContribution wise, this paper is not the first ones to use an \"entangled+disentangled\" latent space (eg paper named \"Toward Controlled Generation of Text\" in 2017), but the OAT learning approach is interesting. \n\nThe presentation can be improved, given so many typos, improper subscripts, equation-naming (Eq 6 or Eq 3.4.2), etc. Moreover, the central equation (Eq.6) is hard to follow, say how q(x) and its KL divergence is evaluated is unknown.\n\nIn Eq 5, whether or not the p(z_1) in last KL term is also factorizable. \n\nSince the intervention is key idea, it's a miss to see it not shown in Figure 1 plot.\n\nThe paper needs more explanation on how to avoid posterior collapse and avoid the collapse of disentangled latent variables.    \n\nThe table column in Table 1 is hard to follow. From Table 1, it seems OAT model has not outperformed other baselines, right?", "summary_of_the_paper": "This paper presents an unsupervised learning model for disentangled latent representation learning. Specifically, the model works on a combined latent space including both entangled variable and separable variable. The most impressive part is the one-at-a-time (OAT) factor learning approach that iteratively uncovers disentangled dimension and learned from reconstructed samples. The OAT approach generally sounds. ", "summary_of_the_review": "The paper is well motivated and the idea of OAT technique generally sounds. However, the paper reads like a manuscript in a rush: we see many typos, unclear subscripts. Moreover, we see unclear equations, such as Eq. 6, which is hard to follow, but the most important equation for the paper. Weird Table 1 columns and not strong experimental metrics. Overall, the paper needs a polish to improve its clarity and readiness to publish.  ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a method for unsupervised learning of disentangled representations by first training a VAE with a tangled set of latents, and then sequentially learning disentangled latent variables one at a time from the entangled initial VAE latent space. On several toy disentanglement benchmarks, the method is shown to perform competitively with previous VAE and GAN approaches. 

There were several concerns from reviewers around the clarity and description of the proposed one-factor-at a time (OAT) training procedure. While the updated draft addressed several typos and some clarity issues, multiple reviewers continued to find the method description problematic. There were additional concerns around the viability of the method on real-world datasets where the number of factors are not known, and as the authors stated the proposed method can also result in one factor of variation encoded into mulitple latent variables, which hurts on many of the disentanglement metrics.  The addition of CelebA downstream task evaluation begins to address this concern of real-world data, but more rigorous experiments (including more description of how models were selected) and discussion of the limtiations of the proposed method are needed. There is also no theoretical motivation as to why the proposed intervention-based factor learning algorithm should recover the ground truth factors.

Given the concerns over experimental results, clarity, and lack of theoretical motivation, I suggest rejecting this paper in the current form.
Decision: Reject