On Disentangled Representations Extracted from Pretrained GANs

1 INTRODUCTION . Unsupervised learning of disentangled representations is currently one of the most important challenges in machine learning . Identifying and separating the factors of variation for the data at hand provides a deeper understanding of its internal structure and can bring new insights into the data generation process . Furthermore , disentangled representations are shown to benefit certain downstream tasks , e.g. , fairness ( Locatello et al. , 2019a ) and abstract reasoning ( van Steenkiste et al. , 2019 ) . Since the seminal papers on disentanglement learning , such as InfoGAN ( Chen et al. , 2016 ) and β-VAE ( Higgins et al. , 2017 ) , a large number of models were proposed , and this problem continues to attract much research attention ( Alemi et al. , 2016 ; Chen et al. , 2018 ; Burgess et al. , 2017 ; Kim & Mnih , 2018 ; Kumar et al. , 2018 ; Rubenstein et al. , 2018 ; Esmaeili et al. , 2019 ; Mathieu et al. , 2019 ; Rolinek et al. , 2019 ; Nie et al. , 2020 ; Lin et al. , 2020 ) . The existing models achieve disentanglement in their latent spaces via specific regularization terms in their training objectives . Typically , these terms determine the trade-off between disentanglement and generation quality . For example , for β-VAE ( Higgins et al. , 2017 ) , one introduces the KLdivergence regularization term that constrains the VAE bottleneck ’ s capacity . This term is weighted by the β multiplier that enforces better disentanglement for β > 1 while resulting in worse reconstruction quality . Similarly , InfoGAN utilized a regularization term approximating the mutual information between the generated image and factor codes . As has been shown in the large scale study Locatello et al . ( 2019b ) , hyperparameter values can critically affect the obtained disentanglement . In the unsupervised setting , the values of ground truth latent factors utilized by disentanglement metrics are unknown , and thus selection of correct hyperparameters becomes a nontrivial task . In this paper , we investigate if disentangled representations can be extracted from the pretrained nondisentangled GAN models , which currently provide the state-of-the-art generation quality ( Karras et al. , 2020 ) . These GANs are trained without disentanglement terms in their objectives ; therefore , we do not need to tune the hyperparameters mentioned above . Our study is partially inspired by a very recent line of works on controllable generation ( Voynov & Babenko , 2020 ; Shen & Zhou , 2020 ; 1https : //bit.ly/3ipb6dW Härkönen et al. , 2020 ; Peebles et al. , 2020 ) , which explore the latent spaces of pretrained GANs and identify the latent directions useful for image editing . The mentioned methods operate without external supervision , therefore , are valid to use in the unsupervised disentanglement . As shown by the comparison on the common benchmarks , the proposed post hoc disentanglement is competitive to the current state-of-the-art in terms of existing metrics , becoming an important alternative to the established “ end-to-end ” disentanglement . Overall , our contributions are the following : • We investigate an alternative paradigm to construct disentangled representations by extracting them from non-disentangled models . In this setting , one does not need to tune hyperparameters for disentanglement regularizers . • We bridge the fields of unsupervised controllable generation and disentanglement learning by using the developments of the former to benefit the latter . As a separate technical contribution , we propose a new simple technique , which outperforms the existing prior methods of controllable generation . • We extensively evaluate all the methods on several popular benchmarks employing commonly used metrics . In most of the operating points , the proposed post hoc disentanglement successfully reaches competitive performance . 2 RELATED WORK . 2.1 DISENTANGLED REPRESENTATIONS . Learning disentangled representation is a long-standing goal in representation learning ( Bengio et al. , 2013 ) useful for a variety of downstream tasks ( LeCun et al. , 2004 ; Higgins et al. , 2018 ; Tschannen et al. , 2018 ; Locatello et al. , 2019a ; van Steenkiste et al. , 2019 ) . While there is no strict definition of disentangled representation , we follow the one considered in ( Bengio et al. , 2013 ) : disentangled representation is a representation where a change in one dimension corresponds to the change only in one factor of variation while leaving other factors invariant . Natural data is assumed to be generated from independent factors of variations , and well-learned disentangled representations should separate these explanatory sources . The most popular approaches so far were based on variational autoencoders ( VAEs ) . Usually , to make representations “ more disentangled ” , VAEs objectives are enriched with specific regularizers ( Alemi et al. , 2016 ; Higgins et al. , 2017 ; Chen et al. , 2018 ; Burgess et al. , 2017 ; Kim & Mnih , 2018 ; Kumar et al. , 2018 ; Rubenstein et al. , 2018 ; Esmaeili et al. , 2019 ; Mathieu et al. , 2019 ; Rolinek et al. , 2019 ) . The general idea behind these approaches is to enforce an aggregated posterior to be factorized , thus providing disentanglement . Another line of research on disentangled representations is based on the InfoGAN model ( Chen et al. , 2016 ) . InfoGAN is an unsupervised model , which adds an extra regularizer to GAN loss to maximize the mutual information between the small subset of latent variables ( factor codes ) and observations . In practice , the mutual information loss is approximated using an encoder network via Variational Information Maximization . InfoGAN-CR ( Lin et al. , 2020 ) is a modification of InfoGAN that employs the so-called contrastive regularizer ( CR ) , which forces the elements of the latent code set to be visually perceptible and distinguishable between each other . A very recently proposed InfoStyleGAN model ( Nie et al. , 2020 ) incorporates similar ideas into the state-of-the-art StyleGAN architecture , allowing for producing both disentangled representations and achieving excellent visual quality of samples . In contrast to these approaches , we use no regularizers or additional loss functions and simply study state-of-the-art GANs trained in a conventional manner . 2.2 CONTROLLABLE GENERATION . Based on rich empirical evidence , it is believed that the latent space of GANs can encode meaningful semantic transformations , such as orientation , appearance , or presence of objects in scenes , of generated images via vector arithmetic ( Radford et al. , 2016 ; Zhu et al. , 2016 ; Bau et al. , 2019 ; Chen et al. , 2016 ) . This means that for an image produced by some latent code , such a transformation can be obtained by simply shifting this latent code in a certain carefully constructed direction , independent from the chosen latent code . E.g. , in the case of human faces , we may have separate directions for such factors as hair color , age , gender . The main applications of this property have been in the field of controllable generation , i.e. , building software to allow a user to manipulate an image to achieve a certain goal while keeping the result photorealistic . Powerful generative models are an appealing tool for this task since the generated images lie on the image manifold by construction . To manipulate a real image , the standard approach is to invert the generator , i.e. , to compute a latent code that approximately corresponds to the desired image , and then apply previously discovered directions ( Shen et al. , 2020 ) . The discovery of directions that allow for interesting image manipulations is a nontrivial task , which , however , can be performed in an unsupervised manner surprisingly efficiently ( Voynov & Babenko , 2020 ; Shen & Zhou , 2020 ; Härkönen et al. , 2020 ; Peebles et al. , 2020 ) . In the heart of these methods lies the idea that the deformations produced by these directions should be as distinguishable as much as possible , which is achieved via maximizing a certain generator–based loss function or by training a separate regressor network attempting to differentiate between them . We thoroughly discuss these approaches further in the text . An important common feature of these methods is that they do not depend on sensitive hyperparameters or even do not have them at all , which makes them appealing for usage in unsupervised settings . Contrary to previous applications of such interpretable directions , we attempt to show they allow us to solve a more fundamental task of building disentangled representations , useful in a variety of downstream tasks . 3 TWO-STAGE DISENTANGLEMENT USING PRETRAINED GANS . In this section , we discuss how disentangled representations of data can be learned with a two-step procedure . Briefly , it can be described as follows . First , we search for a set of k orthogonal interpretable directions in the latent space of the pretrained GAN in an unsupervised manner . This step is performed via one of the methods of controllable generation described below . These directions can be considered as the first k vectors of a new basis in the latent space . By ( virtually ) completing it to a full orthogonal set of vectors , we can obtain ( presumably , disentangled ) representations of synthetic points by a simple change of bases and truncating all but the first k coordinates ; this can be computed by single matrix multiplication . To obtain such representations for real data , we can now train an encoder on a synthetic dataset where targets are constructed using the procedure above . We stick to orthogonal directions for several reasons . Experimentally , it has been shown that this constraint does not significantly affect the quality of discovered directions and is imposed by construction in several further discussed methods . Additionally , it makes the formulas less cumbersome . Reminder on style–based architectures . Traditionally , the generator network in GANs transforms a latent code z ∈ N ( 0 , 1 ) to an image x ∈ RC×H×W . Contrary to this , style–based generators introduce a so-called style network , usually realized as a trainable MLP , which “ preprocesses ” random latent codes to the so-called “ style vectors ” ( elements of the style space ) . The obtained style vectors are , in turn , fed into the convolutional layers of the generator . It has been shown ( Karras et al. , 2019 ) that direct manipulations over the style vectors , rather than latent codes themselves , leads to visually plausible image interpolations and image mixing results . Intuitively , the style network performs a “ rectification ” of the latent space so that the new latent codes are more interpretable and disentangled . In this paper , we work with the style–based generators and perform image editing in the style space , denoted byW ; its elements are denoted as w. Let us now discuss the steps mentioned above in more detail . Recall , that we are interested in finding directions n such that G ( w′ ) with w′ = w + αn performs a certain interpretable deformation of the image G ( w ) . We now thoroughly describe the existing approaches to obtaining them in an unsupervised manner as well as various hyperparameters one needs to specify for each method . 3.1 DISCOVERING INTERPRETABLE DIRECTIONS . We consider several recently proposed methods : ClosedForm ( Shen & Zhou , 2020 ) , GANspace ( Härkönen et al. , 2020 ) , LatentDiscovery ( Voynov & Babenko , 2020 ) . Inspired by these methods , we also propose another family of methods termed DeepSpectral . ClosedForm ( CF ) . The authors of the ClosedForm method propose to move along the singular vectors of the first layer of generator . More specifically , for the transformation in the first layer given by the matrix A ( e.g. , this may be the weight in a fully-connected or a transposed convolutional layer ) , the direction n is found as n∗ = argmax { n∈RD : nTn=1 } ‖An‖22 . ( 1 ) All local maximas of Equation ( 1 ) form the set of singular vectors of the matrix A , and the authors propose to choose k singular vectors , associated with the corresponding k highest singular values . For the style–based GANs , the matrix A is obtained by concatenating the style mapping layers of each convolutional block . Hyperparameters : this method is hyperparameter–free and requires only a pretrained GAN model . GANspace ( GS ) . This method searches for important , meaningful directions by performing PCA in the style space of StyleGAN . Specifically , these directions are found in the following manner : first , randomly sampled noise vectors z1 : N ∈ N ( 0 , 1 ) are converted into style vectors w1 : N . The interpretable directions then correspond to principal axes of the set w1 : N ; in practice , we consider top k directions according to the corresponding singular values . Hyperparameters : for this approach , we only need to provide the number of sampled points which can be taken fixed and large , as well as a random seed for sampling . LatentDiscovery ( LD ) . LatentDiscovery is an unsupervised model-agnostic procedure for identifying interpretable directions in the GAN latent space . Informally speaking , this method searches for a set of directions that can be easily distinguished from one another . The resulting directions are meant to represent independent factors of generated images and include humaninterpretable representations . The trainable components are the following : a matrix N ∈ RD×k and a reconstructor network R , which evaluates the pair ( G ( w ) , G ( w+N ( ek ) ) . The reconstructor model aims to recover the shift in the latent space corresponding to the given image transformation . These two components are optimized jointly by minimizing the following objective function : N∗ , R∗ = argmin N , R Ew , k , L ( N , R ) = argmin N , R Ew , k , [ Lcl ( k , k̂ ) + λLr ( , ̂ ) ] . ( 2 ) Here , Lcl ( · , · ) is a reconstructor classification loss ( cross-entropy function ) , Lr ( · , · ) – regression term ( mean absolute error ) , which forces shifts along found directions to be more continuous . This method utilizes a number of hyperparameters ; however , as was shown in Voynov & Babenko ( 2020 ) , it is quite stable , and the default values provide good quality across various models and datasets . Hyperparameters : the hyperparameters include the number of latent directions k and the multiplier of the reconstructor term λ ; additionally , we can select different architectures for the regressor , as well as different training hyperparameters and random seed for initialization . DeepSpectral ( DS ) . We propose a novel approach to finding interpretable directions in the GAN latent space . Our motivation is as follows . While CF and GS both produce decent results , they effectively ignore all the layers in the generator but the first few ones . We hypothesize that by studying outputs of deeper intermediate layers of the generator , one can obtain a richer set of directions unavailable for these methods . Concretely , we propose the following simple approach . Let G ( i ) ( w ) denote the output of the i-th hidden layer of the generator . In order to obtain k directions in the latent space , we compute k singular vectors of JG ( i ) ( w ) with the highest singular values ( at some fixed point w ) . Here , JG ( i ) denotes the Jacobian matrix of a mapping . In a way , our approach generalizes CF since a linear map and its Jacobian coincide ( when bias is zero ) . By using automatic differentiation and an iterative approach to computing Singular Value Decomposition ( thus , we do not form the full Jacobian matrix ) , such directions can be found basically instantly ( Khrulkov & Oseledets , 2018 ) . The only hyperparameters in this approach are the choice of the layer and the choice of the base point w to compute the Jacobian . We experimentally verify the benefits of DS by considering various intermediate layers in Section 4 . Hyperparameters : we need to specify the layer and the base point w. Another recently proposed method ( Peebles et al. , 2020 ) searches for interpretable directions by utilizing the so-called Hessian penallty , penalizing the norm of the off-diagonal elements of the Hessian matrix of a network . However , in our implementation , we were not able to obtain convincing results ; we plan to analyze in the future with the authors ’ implementation when released .

Summary: This paper proposes a non-conventional approach to learning disentangled representations through finding interpretable directions in the latent space of a pre-trained Style-GAN (Karras et al., 2018). To find these directions, several existing techniques from the recent literature on controllable image generation are considered. To apply this technique to existing (real) images, an encoder network is trained on the generated images to learn a mapping  from images to disentangled representations.