Clearing the Path for Truly Semantic Representation Learning

1 INTRODUCTION . The task of unsupervised learning of interpretable data representations has a long history . From classical approaches using linear algebra e.g . via Principle Component Analysis ( PCA ) ( Pearson , 1901 ) or statistical methods such as Independent Component Analysis ( ICA ) ( Comon , 1994 ) all the way to more recent approaches that rely on deep learning architectures . The cornerstone architecture is the Variational Autoencoder ( Kingma & Welling , 2014 ) ( VAE ) which clearly demonstrated both high semantic quality as well as good performance in terms of disentanglement . Until today , derivates of VAEs ( Higgins et al. , 2017 ; Kim & Mnih , 2018a ; Chen et al. , 2018 ; Kumar et al. , 2017 ) excel over other architectures in terms of disentanglement metrics . The extent of VAE ’ s success even prompted recent deeper analyses of its inner workings ( Rolinek et al. , 2019 ; Burgess et al. , 2018 ; Chen et al. , 2018 ; Mathieu et al. , 2018 ) . If we treat the overloaded term disentanglement to the highest of its aspirations , as the ability to recover the true generating factors of data , fundamental problems arise . As explained byLocatello et al . ( 2019 ) , already the concept of generative factors is compromised from a statistical perspective : two ( or in fact infinitely many ) sets of generative factors can generate statistically indistinguishable datasets . Yet , the scores on the disentanglement benchmarks are high and continue to rise . This apparent contradiction needs a resolution . In this work , we claim that all leading disentanglement architectures can be fooled by the same trick . Namely , by introducing a small change of the correlation structure , which , however , perfectly preserves the set of generative factors . To that end , we provide an alternative version of the standard disentanglement datasets in which each image undergoes a modification barely visible to a human eye . We report drastic drops of disentanglement performance on the altered datasets . On a technical level , we build on the findings by Rolinek et al . ( 2019 ) who argued that VAEs recover the nonlinear principle components of the datasets ; in other words nonlinearly computed scalars that are the sources of variance in the sense of classical PCA . The small modifications of the datasets we propose , aim to change the leading principle component by adding modest variance to an alternative Datasets will be released here : https : //sites.google.com/view/sem-rep-learning candidate . The “ to-be ” leading principle component is specific to each dataset but it is determined in a consistent fashion . With the greedy algorithm for discovering the linear PCA components in mind , we can see that any change in the leading principle component is reflected also in the others . As a result , the overall alignment changes and the generating factors get entangled leading to low disentanglement scores . We demonstrate that , even though the viewpoint of ( Rolinek et al. , 2019 ) only has theoretical support for the ( β ) -VAE , it empirically transfers to other architectures . Overall , we want to encourage evaluating new disentanglement approaches on the proposed datasets , in which the generative factors are intact but the correlation structure is less favorable for the principle component discovery ingrained in VAE-style architectures . We hope that providing a more sound experimental setup will clear the path for a new set of disentanglement approaches . 2 RELATED WORK . The related work can be categorized in three research questions : i ) defining disentanglement and metrics capturing the quality of latent representations ; ii ) architecture development for unsupervised learning of disentangled representations ; and iii ) understanding the inner workings of existing architectures , as for example of β-VAEs . This paper is building upon results from all three lines of work . Defining disentanglement . Defining the term disentangled representation is an open question ( Higgins et al. , 2018 ) . The presence of learned representations in machine learning downstream tasks , such as object recognition , natural language processing and others , created the need to “ disentangle the factors of variation ” ( Bengio et al. , 2013 ) very early on . This vague interpretation of disentanglement is inspired by the existence of a low dimensional manifold that captures the variance of higher dimensional data . As such , finding a factorized , statistically independent representation became a core ingredient of disentangled representation learning and dates back to classical ICA models ( Comon , 1994 ; Bell & Sejnowski , 1995 ) . For some tasks , the desired feature of a disentangled representation is that it is semantically meaningful . Prominent examples can be found in computer vision ( Shu et al. , 2017 ; Liao et al. , 2020 ) and in research addressing interpretability of machine learning models ( Adel et al. , 2018 ; Kim , 2019 ) . Based on group theory and symmetry transformations , ( Higgins et al. , 2018 ) provides the “ first principled definition of a disentangled representation ” . Closely related to this concept is also the field of causality in machine learning ( Schölkopf , 2019 ; Suter et al. , 2019 ) , more specifically the search for causal generative models ( Besserve et al. , 2018 ; 2020 ) . Architecture development . The leading architectures for disentangled representation learning are based on VAEs ( Kingma & Welling , 2014 ) . Despite originally developed as a generative modeling architecture , its variants have proven to excel at representation learning tasks . First of all the β-VAE ( Higgins et al. , 2017 ) , which exposes the trade-off between reconstruction and regularization via the additional hyperparameter , performs remarkably well . Other architectures have been proposed that additionally encourage statistical independence in the latent space , e.g . FactorVAE ( Kim & Mnih , 2018b ) and β-TC-VAE ( Chen et al. , 2018 ) . The DIP-VAE ( Kumar et al. , 2017 ) suggests using moment-matching to close the distribution gap introduced in the original VAE paper . As the architectures developed , so did the metrics used for measuring the disentanglement quality of representations ( Chen et al. , 2018 ; Kim & Mnih , 2018b ; Higgins et al. , 2017 ; Kumar et al. , 2017 ) . Understanding inner workings . With the rising success and development of VAE based architectures , the question of understanding their inner working principles became dominant in the community . One line of work searches for an answer to the question why these models disentangle ( Burgess et al. , 2018 ) . Another , closely related line of work shows up and tightens the connection between the vanilla ( β- ) VAE objective and ( probabilistic ) PCA ( Tipping & Bishop , 1999 ) ( Rolinek et al. , 2019 ; Lucas et al. , 2019 ) . Building on such findings , novel approaches for model selection were proposed ( Duan et al. , 2020 ) , emphasizing the value of thoroughly understanding the working principles of these methods . On a less technical side , ( Locatello et al. , 2019 ) conducted a broad set of experiments , questioning the relevance of models given the variance over restarts and the choice of hyperparameter . They also formalized the necessity of inductive bias as a strict requirement for unsupervised learning of disentangled representations . Our experiments are built on their code-base . 3 BACKGROUND . 3.1 QUANTIFYING DISENTANGLEMENT . Among the different viewpoints on disentanglement , we follow recent literature and focus on the connection between the discovered data representation and a set of generative factors . Multiple metrics have been proposed to quantify this connection . Most of them are based on the understanding that , ideally , each generative factor is encoded in precisely one latent variable . This was captured concisely by Chen et al . ( 2018 ) who proposed the Mutual Information Gap ( MIG ) , the mean ( over the Nw generative factors ) normalized difference of the two highest mutual information values between a latent coordinate and the single generating factor : 1 Nw Nw∑ i=0 1 H ( wi ) ( max k I ( wi ; zk ) −max k 6=k′ I ( wi ; zk ) ) , ( 1 ) where k′ = argmaxκ I ( wi , zκ ) . More details about MIG , its implementations , and an extension to discrete variables can be found in ( Chen et al. , 2018 ; Rolinek et al. , 2019 ) . While multiple other metrics were proposed such as SAPScore ( Kumar et al. , 2017 ) , FactorVAEScore ( Kim & Mnih , 2018a ) and DCI Score ( Eastwood & Williams , 2018 ) ( see the supplementary material of Klindt et al . ( 2020 ) ) , in this work , we focus primarily on MIG . 3.2 VARIATIONAL AUTOENCODERS AND THE MYSTERY OF A SPECIFIC ALIGNMENT . Variational autoencoders hide many intricacies and attempting to compress their exposition would not do them justice . For this reason , we limit ourselves to what is crucial for understanding this work : the objective functions . For well-presented full descriptions of VAEs , we refer the readers to ( Doersch , 2016 ) . As it is common in generative models , VAEs aim to maximize the log-likelihood objective N∑ i=1 log p ( x ( i ) ) , ( 2 ) in which { x ( i ) } Ni=1 = X is a dataset consisting of N i.i.d . samples x ( i ) of a multivariate random variable X that follows the true data distribution . The quantity p ( x ( i ) ) captures the probability density of generating the training data point x ( i ) under the current parameters of the model . This objective is , however , intractable in its general form . For this reason , Kingma & Welling ( 2014 ) follow the standard technique of variational inference and introduce a tractable Evidence Lower Bound ( ELBO ) : Eq ( z|x ( i ) ) log p ( x ( i ) | z ) +DKL ( q ( z | x ( i ) ) ‖ p ( z ) ) . ( 3 ) Here , z are the latent variables used to generate samples from X via a parameterized stochastic decoder q ( x ( i ) | z ) . The fundamental question of “ How do these objectives promote disentanglement ? ” was first asked by Burgess et al . ( 2018 ) . This is indeed far from obvious ; in disentanglement the aim to encode a fixed generative factor in precisely one latent variable . From a geometric viewpoint this requires the latent representation to be axis-aligned ( one axis corresponding to one generative factor ) . This question becomes yet more intriguing after noticing ( and formally proving ) that both objective functions ( 2 ) and ( 3 ) are invariant under rotations ( Burgess et al. , 2018 ; Rolinek et al. , 2019 ) . In other words , any rotation of a fixed latent representation results in the same value of the objective function and yet β-VAEs consistently produce representations that are axis-aligned and in effect are isolating the generative factor into individual latent variables . 3.3 RESOLUTION VIA NON-LINEAR CONNECTIONS TO PCA . A mechanistic answer to the question raised in the previous subsection was given by Rolinek et al . ( 2019 ) . The formal argument showed that under specific conditions which are typical for β-VAEs ( called polarized regime ) , the model locally performs PCA in the sense of aligning the “ sources of variance ” with the local axes . The resulting global alignment often coincides with finding nonlinear principal components of the dataset ( see Fig . 1 ) . This behavior stems from the convenient but uninformed choice of a diagonal posterior , which breaks the symmetry of ( 2 ) and ( 3 ) . This connection with PCA was also reported by Stuehmer et al . ( 2020 ) , alternatively formalized by Lucas et al . ( 2019 ) and converted into performance improvements in an unsupervised setting by Duan et al . ( 2020 ) . Since our dataset perturbation method is based on this interpretation of the VAE , we offer further context . In more technical terms , Rolinek et al . ( 2019 ) simplify the closed-form KL-term of the objective ( 3 ) under the assumption of polarized regime into the form LKL ( x ( i ) ) ≈ 1 2 ∑ j∈Va µ2j ( x ( i ) ) − log ( σ2j ( x ( i ) ) ) , ( 4 ) where Va is the set of active latent variables , µ2j ( x ( i ) ) is the mean embedding of input x ( i ) and σ2j ( x ( i ) ) the corresponding term in the diagonal posterior which also plays the role of noise applied to the latent variable j . This form of the objective , when studied independently , lends itself to an alternative viewpoint . When the only “ moving part ” in the model is the alignment of the latent representation , i.e . application of a rotation matrix , the induced optimization problem has the following interpretation1 : Distribute a fixed amount of noise among the latent variables such that the L2 reconstruction error increases the least . This extracted problem can be solved in closed form and the solution is based on isolating sources of variance . Intuitively , important scalar factors whose preservation is crucial for reconstruction , need to be captured with high precision ( low noise ) . For that , it is economic to isolate them from other factors . This high-level connection with PCA is then further formalized by Rolinek et al . ( 2019 ) . One less obvious observation is that the “ isolation ” of different sources of variance relies on the accuracy of the linearization of the decoder at around a fixed µ ( x ( i ) ) . Since in many datasets the local and global correlation structure is nearly identical , β-VAE recover sound global principle components . If , however , the local structure obeys a different “ natural ” alignment , the Variational autoencoder prefers it to the global one as captured in a synthetic experiment displayed in Fig . 1 . The sensitivity of β-VAE ’ s global representation to small local changes is precisely our point of attack . 1the precise formal statement can be found in ( Rolinek et al. , 2019 , Equations ( 22 ) and ( 23 ) )

Summary: The authors first describe the shortcomings of existing work for disentanged representation learning and the general problem of unidentifiability. Here however, the presentation could be more precise. For example, they state that the log-likelihood objective (2) and the evidence lower bound (3) would be "invariant under rotations", and give (Burgess et al., 2018; Rolinek et al., 2019) as a reference. However, the log likelihood or the evidence lower bound themselves are not rotationally invariant per-se. Instead these objectives are invariant under rotations for the standard VAE with a rotationally invariant prior p(z) (e.g. the standard normal distribution).