On Disentangled Representations Extracted from Pretrained GANs

1 INTRODUCTION . Unsupervised learning of disentangled representations is currently one of the most important challenges in machine learning . Identifying and separating the factors of variation for the data at hand provides a deeper understanding of its internal structure and can bring new insights into the data generation process . Furthermore , disentangled representations are shown to benefit certain downstream tasks , e.g. , fairness ( Locatello et al. , 2019a ) and abstract reasoning ( van Steenkiste et al. , 2019 ) . Since the seminal papers on disentanglement learning , such as InfoGAN ( Chen et al. , 2016 ) and β-VAE ( Higgins et al. , 2017 ) , a large number of models were proposed , and this problem continues to attract much research attention ( Alemi et al. , 2016 ; Chen et al. , 2018 ; Burgess et al. , 2017 ; Kim & Mnih , 2018 ; Kumar et al. , 2018 ; Rubenstein et al. , 2018 ; Esmaeili et al. , 2019 ; Mathieu et al. , 2019 ; Rolinek et al. , 2019 ; Nie et al. , 2020 ; Lin et al. , 2020 ) . The existing models achieve disentanglement in their latent spaces via specific regularization terms in their training objectives . Typically , these terms determine the trade-off between disentanglement and generation quality . For example , for β-VAE ( Higgins et al. , 2017 ) , one introduces the KLdivergence regularization term that constrains the VAE bottleneck ’ s capacity . This term is weighted by the β multiplier that enforces better disentanglement for β > 1 while resulting in worse reconstruction quality . Similarly , InfoGAN utilized a regularization term approximating the mutual information between the generated image and factor codes . As has been shown in the large scale study Locatello et al . ( 2019b ) , hyperparameter values can critically affect the obtained disentanglement . In the unsupervised setting , the values of ground truth latent factors utilized by disentanglement metrics are unknown , and thus selection of correct hyperparameters becomes a nontrivial task . In this paper , we investigate if disentangled representations can be extracted from the pretrained nondisentangled GAN models , which currently provide the state-of-the-art generation quality ( Karras et al. , 2020 ) . These GANs are trained without disentanglement terms in their objectives ; therefore , we do not need to tune the hyperparameters mentioned above . Our study is partially inspired by a very recent line of works on controllable generation ( Voynov & Babenko , 2020 ; Shen & Zhou , 2020 ; 1https : //bit.ly/3ipb6dW Härkönen et al. , 2020 ; Peebles et al. , 2020 ) , which explore the latent spaces of pretrained GANs and identify the latent directions useful for image editing . The mentioned methods operate without external supervision , therefore , are valid to use in the unsupervised disentanglement . As shown by the comparison on the common benchmarks , the proposed post hoc disentanglement is competitive to the current state-of-the-art in terms of existing metrics , becoming an important alternative to the established “ end-to-end ” disentanglement . Overall , our contributions are the following : • We investigate an alternative paradigm to construct disentangled representations by extracting them from non-disentangled models . In this setting , one does not need to tune hyperparameters for disentanglement regularizers . • We bridge the fields of unsupervised controllable generation and disentanglement learning by using the developments of the former to benefit the latter . As a separate technical contribution , we propose a new simple technique , which outperforms the existing prior methods of controllable generation . • We extensively evaluate all the methods on several popular benchmarks employing commonly used metrics . In most of the operating points , the proposed post hoc disentanglement successfully reaches competitive performance . 2 RELATED WORK . 2.1 DISENTANGLED REPRESENTATIONS . Learning disentangled representation is a long-standing goal in representation learning ( Bengio et al. , 2013 ) useful for a variety of downstream tasks ( LeCun et al. , 2004 ; Higgins et al. , 2018 ; Tschannen et al. , 2018 ; Locatello et al. , 2019a ; van Steenkiste et al. , 2019 ) . While there is no strict definition of disentangled representation , we follow the one considered in ( Bengio et al. , 2013 ) : disentangled representation is a representation where a change in one dimension corresponds to the change only in one factor of variation while leaving other factors invariant . Natural data is assumed to be generated from independent factors of variations , and well-learned disentangled representations should separate these explanatory sources . The most popular approaches so far were based on variational autoencoders ( VAEs ) . Usually , to make representations “ more disentangled ” , VAEs objectives are enriched with specific regularizers ( Alemi et al. , 2016 ; Higgins et al. , 2017 ; Chen et al. , 2018 ; Burgess et al. , 2017 ; Kim & Mnih , 2018 ; Kumar et al. , 2018 ; Rubenstein et al. , 2018 ; Esmaeili et al. , 2019 ; Mathieu et al. , 2019 ; Rolinek et al. , 2019 ) . The general idea behind these approaches is to enforce an aggregated posterior to be factorized , thus providing disentanglement . Another line of research on disentangled representations is based on the InfoGAN model ( Chen et al. , 2016 ) . InfoGAN is an unsupervised model , which adds an extra regularizer to GAN loss to maximize the mutual information between the small subset of latent variables ( factor codes ) and observations . In practice , the mutual information loss is approximated using an encoder network via Variational Information Maximization . InfoGAN-CR ( Lin et al. , 2020 ) is a modification of InfoGAN that employs the so-called contrastive regularizer ( CR ) , which forces the elements of the latent code set to be visually perceptible and distinguishable between each other . A very recently proposed InfoStyleGAN model ( Nie et al. , 2020 ) incorporates similar ideas into the state-of-the-art StyleGAN architecture , allowing for producing both disentangled representations and achieving excellent visual quality of samples . In contrast to these approaches , we use no regularizers or additional loss functions and simply study state-of-the-art GANs trained in a conventional manner . 2.2 CONTROLLABLE GENERATION . Based on rich empirical evidence , it is believed that the latent space of GANs can encode meaningful semantic transformations , such as orientation , appearance , or presence of objects in scenes , of generated images via vector arithmetic ( Radford et al. , 2016 ; Zhu et al. , 2016 ; Bau et al. , 2019 ; Chen et al. , 2016 ) . This means that for an image produced by some latent code , such a transformation can be obtained by simply shifting this latent code in a certain carefully constructed direction , independent from the chosen latent code . E.g. , in the case of human faces , we may have separate directions for such factors as hair color , age , gender . The main applications of this property have been in the field of controllable generation , i.e. , building software to allow a user to manipulate an image to achieve a certain goal while keeping the result photorealistic . Powerful generative models are an appealing tool for this task since the generated images lie on the image manifold by construction . To manipulate a real image , the standard approach is to invert the generator , i.e. , to compute a latent code that approximately corresponds to the desired image , and then apply previously discovered directions ( Shen et al. , 2020 ) . The discovery of directions that allow for interesting image manipulations is a nontrivial task , which , however , can be performed in an unsupervised manner surprisingly efficiently ( Voynov & Babenko , 2020 ; Shen & Zhou , 2020 ; Härkönen et al. , 2020 ; Peebles et al. , 2020 ) . In the heart of these methods lies the idea that the deformations produced by these directions should be as distinguishable as much as possible , which is achieved via maximizing a certain generator–based loss function or by training a separate regressor network attempting to differentiate between them . We thoroughly discuss these approaches further in the text . An important common feature of these methods is that they do not depend on sensitive hyperparameters or even do not have them at all , which makes them appealing for usage in unsupervised settings . Contrary to previous applications of such interpretable directions , we attempt to show they allow us to solve a more fundamental task of building disentangled representations , useful in a variety of downstream tasks . 3 TWO-STAGE DISENTANGLEMENT USING PRETRAINED GANS . In this section , we discuss how disentangled representations of data can be learned with a two-step procedure . Briefly , it can be described as follows . First , we search for a set of k orthogonal interpretable directions in the latent space of the pretrained GAN in an unsupervised manner . This step is performed via one of the methods of controllable generation described below . These directions can be considered as the first k vectors of a new basis in the latent space . By ( virtually ) completing it to a full orthogonal set of vectors , we can obtain ( presumably , disentangled ) representations of synthetic points by a simple change of bases and truncating all but the first k coordinates ; this can be computed by single matrix multiplication . To obtain such representations for real data , we can now train an encoder on a synthetic dataset where targets are constructed using the procedure above . We stick to orthogonal directions for several reasons . Experimentally , it has been shown that this constraint does not significantly affect the quality of discovered directions and is imposed by construction in several further discussed methods . Additionally , it makes the formulas less cumbersome . Reminder on style–based architectures . Traditionally , the generator network in GANs transforms a latent code z ∈ N ( 0 , 1 ) to an image x ∈ RC×H×W . Contrary to this , style–based generators introduce a so-called style network , usually realized as a trainable MLP , which “ preprocesses ” random latent codes to the so-called “ style vectors ” ( elements of the style space ) . The obtained style vectors are , in turn , fed into the convolutional layers of the generator . It has been shown ( Karras et al. , 2019 ) that direct manipulations over the style vectors , rather than latent codes themselves , leads to visually plausible image interpolations and image mixing results . Intuitively , the style network performs a “ rectification ” of the latent space so that the new latent codes are more interpretable and disentangled . In this paper , we work with the style–based generators and perform image editing in the style space , denoted byW ; its elements are denoted as w. Let us now discuss the steps mentioned above in more detail . Recall , that we are interested in finding directions n such that G ( w′ ) with w′ = w + αn performs a certain interpretable deformation of the image G ( w ) . We now thoroughly describe the existing approaches to obtaining them in an unsupervised manner as well as various hyperparameters one needs to specify for each method . 3.1 DISCOVERING INTERPRETABLE DIRECTIONS . We consider several recently proposed methods : ClosedForm ( Shen & Zhou , 2020 ) , GANspace ( Härkönen et al. , 2020 ) , LatentDiscovery ( Voynov & Babenko , 2020 ) . Inspired by these methods , we also propose another family of methods termed DeepSpectral . ClosedForm ( CF ) . The authors of the ClosedForm method propose to move along the singular vectors of the first layer of generator . More specifically , for the transformation in the first layer given by the matrix A ( e.g. , this may be the weight in a fully-connected or a transposed convolutional layer ) , the direction n is found as n∗ = argmax { n∈RD : nTn=1 } ‖An‖22 . ( 1 ) All local maximas of Equation ( 1 ) form the set of singular vectors of the matrix A , and the authors propose to choose k singular vectors , associated with the corresponding k highest singular values . For the style–based GANs , the matrix A is obtained by concatenating the style mapping layers of each convolutional block . Hyperparameters : this method is hyperparameter–free and requires only a pretrained GAN model . GANspace ( GS ) . This method searches for important , meaningful directions by performing PCA in the style space of StyleGAN . Specifically , these directions are found in the following manner : first , randomly sampled noise vectors z1 : N ∈ N ( 0 , 1 ) are converted into style vectors w1 : N . The interpretable directions then correspond to principal axes of the set w1 : N ; in practice , we consider top k directions according to the corresponding singular values . Hyperparameters : for this approach , we only need to provide the number of sampled points which can be taken fixed and large , as well as a random seed for sampling . LatentDiscovery ( LD ) . LatentDiscovery is an unsupervised model-agnostic procedure for identifying interpretable directions in the GAN latent space . Informally speaking , this method searches for a set of directions that can be easily distinguished from one another . The resulting directions are meant to represent independent factors of generated images and include humaninterpretable representations . The trainable components are the following : a matrix N ∈ RD×k and a reconstructor network R , which evaluates the pair ( G ( w ) , G ( w+N ( ek ) ) . The reconstructor model aims to recover the shift in the latent space corresponding to the given image transformation . These two components are optimized jointly by minimizing the following objective function : N∗ , R∗ = argmin N , R Ew , k , L ( N , R ) = argmin N , R Ew , k , [ Lcl ( k , k̂ ) + λLr ( , ̂ ) ] . ( 2 ) Here , Lcl ( · , · ) is a reconstructor classification loss ( cross-entropy function ) , Lr ( · , · ) – regression term ( mean absolute error ) , which forces shifts along found directions to be more continuous . This method utilizes a number of hyperparameters ; however , as was shown in Voynov & Babenko ( 2020 ) , it is quite stable , and the default values provide good quality across various models and datasets . Hyperparameters : the hyperparameters include the number of latent directions k and the multiplier of the reconstructor term λ ; additionally , we can select different architectures for the regressor , as well as different training hyperparameters and random seed for initialization . DeepSpectral ( DS ) . We propose a novel approach to finding interpretable directions in the GAN latent space . Our motivation is as follows . While CF and GS both produce decent results , they effectively ignore all the layers in the generator but the first few ones . We hypothesize that by studying outputs of deeper intermediate layers of the generator , one can obtain a richer set of directions unavailable for these methods . Concretely , we propose the following simple approach . Let G ( i ) ( w ) denote the output of the i-th hidden layer of the generator . In order to obtain k directions in the latent space , we compute k singular vectors of JG ( i ) ( w ) with the highest singular values ( at some fixed point w ) . Here , JG ( i ) denotes the Jacobian matrix of a mapping . In a way , our approach generalizes CF since a linear map and its Jacobian coincide ( when bias is zero ) . By using automatic differentiation and an iterative approach to computing Singular Value Decomposition ( thus , we do not form the full Jacobian matrix ) , such directions can be found basically instantly ( Khrulkov & Oseledets , 2018 ) . The only hyperparameters in this approach are the choice of the layer and the choice of the base point w to compute the Jacobian . We experimentally verify the benefits of DS by considering various intermediate layers in Section 4 . Hyperparameters : we need to specify the layer and the base point w. Another recently proposed method ( Peebles et al. , 2020 ) searches for interpretable directions by utilizing the so-called Hessian penallty , penalizing the norm of the off-diagonal elements of the Hessian matrix of a network . However , in our implementation , we were not able to obtain convincing results ; we plan to analyze in the future with the authors ’ implementation when released .

Summary: This paper proposes a new approach to learn disentangled representations from Generative models that were trained without a loss that explicitly enforces disengagement. The motivation behind this work is to address the computational time cost regarding hyperparameter tuning that is typically required for VAEs trained with a disentangled loss. The proposed approach can be summarized in 3 steps: (1) Training a GAN (in an unsupervised way) (2) Find  ‘k’ new basis vectors in ‘W’ space (3) Train a new encoder on the synthetic dataset {(z,G(z))} with loss that encourages the encodings to correspond to the subspace spanned by the basis vectors. Furthermore, the authors propose a new approach for identifying meaningful basis vectors which the authors claim that it is the generalized version of ClosedFrom. For evaluation, the authors then perform a set of experiments involving disentanglement, abstract reasoning, fairness on a variety of datasets such as 3Dshapes, MPI, and Cars3D.