Do Not Escape From the Manifold: Discovering the Local Coordinates on the Latent Space of GANs

1 INTRODUCTION . Generative Adversarial Networks ( GANs , Goodfellow et al . ( 2014 ) ) , such as ProGAN ( Karras et al. , 2018 ) , BigGAN ( Brock et al. , 2018 ) , and StyleGANs ( Karras et al. , 2019 ; 2020b ; a ) , have shown tremendous performance in generating high-resolution photo-realistic images that are often indistinguishable from natural images . However , despite several recent efforts ( Goetschalckx et al. , 2019 ; Jahanian et al. , 2019 ; Plumerault et al. , 2020 ; Shen et al. , 2020 ) to investigate the disentanglement properties ( Bengio et al. , 2013 ) of the latent space in GANs , it is still challenging to find meaningful traversal directions in the latent space corresponding to the semantic variation of an image . The previous approaches to find the semantic-factorizing directions are categorized into local and global methods . The local methods ( e.g . Ramesh et al . ( 2018 ) , Latent Mapper in StyleCLIP ( Patashnik et al. , 2021 ) , and attribute-conditioned normalizing flow in StyleFlow ( Abdal et al. , 2021 ) ) suggest a sample-wise traversal direction . By contrast , the global methods , such as GANSpace ( Härkönen et al. , 2020 ) and SeFa ( Shen & Zhou , 2021 ) , propose a global direction for the particular semantics ( e.g . glasses , age , and gender ) that works on the entire latent space . Throughout this paper , we refer to these global methods as the global basis . These global methods showed promising results . However , these methods are successful on the limited area , and the image quality is sensitive to the perturbation intensity . In fact , if a latent space does not satisfy the global disentanglement property itself , all global methods are bound to show a limited performance on it . Nevertheless , to the best of our knowledge , the global disentanglement property of a latent space has not been investigated except for the empirical observation of generated samples . In this regard , we need a local method that describes the local disentanglement property and an evaluation scheme for the global disentanglement property from the collected local information . ∗Equal contribution In this paper , we suggest that the semantic property of the latent space in GANs ( i.e . disentanglement of semantics and image collapse ) is closely related to its geometry , because of the sample-wise optimization nature of GANs . In this respect , we propose an unsupervised method to find a traversal direction based on the local structure of the intermediate latent spaceW , called Local Basis ( Fig 1a ) . We approximateW with its submanifold representing its local principal variation , discovered in terms of the tangent space TwW . Local Basis is defined as an ordered basis of TwW corresponding to the approximating submanifold . Moreover , we show that Local Basis is obtained from the simple closed-form algorithm , that is the singular vectors of the Jacobian matrix of the subnetwork . The geometric interpretation of Local Basis provides an evaluation scheme for the global disentanglement property through the global warpage of the latent manifold . Our contributions are as follows : 1 . We propose Local Basis , a set of traversal directions that can reliably traverse without escaping from the latent space to prevent image collapse . The latent traversal along Local Basis corresponds to the local coordinate mesh of local-geometry-describing submanifold . 2 . We show that Local Basis leads to stable variation and better semantic factorization than global approaches . This result verifies our hypothesis on the close relationship between the semantic and geometric properties of the latent space in GANs . 3 . We propose Iterative Curve-Traversal method , which is a way to trace the latent space in the curved trajectory . The trajectory of the images with this method shows a more stable variation compared to the linear traversal . 4 . We introduce the metrics on the Grassmannian manifold to analyze the global geometry of the latent space through Local Basis . Quantitative analysis demonstrates that theW-space of StyleGAN2 is still globally warped . This result provides an explanation for the limited success of the global basis and proves the importance of local approaches . 2 RELATED WORK . Style-based Generators . In recent years , GANs equipped with style-based generators ( Karras et al. , 2019 ; 2020b ) have shown state-of-the-art performance in high-fidelity image synthesis . The style-based generator consists of two parts : a mapping network and a synthesis network . The mapping network encodes the isotropic Gaussian noise z ∈ Z to an intermediate latent vector w ∈ W . The synthesis network takes w and generates an image while controlling the style of the image through w. Here , W-space is well known for providing a better disentanglement property compared to Z ( Karras et al. , 2019 ) . However , there is still a lack of understanding about the effect of latent perturbation in a specific direction on the output image . Latent Traversal for Image Manipulation . The impressive success of GANs in producing highquality images has led to various attempts to understand their latent space . Early approaches ( Radford et al. , 2016 ; Upchurch et al. , 2017 ) show that vector arithmetic on the latent space for the semantics holds , and StyleGAN ( Karras et al. , 2019 ) shows that mixing two latent codes can achieve style transfer . Some studies have investigated the supervised learning of latent directions while assuming access to the semantic attributes of images ( Goetschalckx et al. , 2019 ; Jahanian et al. , 2019 ; Shen et al. , 2020 ; Yang et al. , 2021 ; Abdal et al. , 2021 ) . In contrast to these supervised methods , some recent studies have suggested novel approaches that do not use the prior knowledge of training dataset , such as the labels of human facial attributes . In Voynov & Babenko ( 2020 ) , an unsupervised optimization method is proposed to jointly learn a candidate matrix and a corresponding reconstructor , which identifies the semantic direction in the matrix . GANSpace ( Härkönen et al. , 2020 ) finds a global basis forW in StyleGAN using a PCA , enabling a fast image manipulation . SeFa ( Shen & Zhou , 2021 ) focuses on the first weight parameter right after the latent code , suggesting that it contains essential knowledge of an image variation . SeFa proposes singular vectors of the first weight parameter as meaningful global latent directions . StyleCLIP ( Patashnik et al. , 2021 ) achieves a stateof-the-art performance in the text-driven image manipulation of StyleGAN . StyleCLIP introduces an additional training to minimize the CLIP loss ( Radford et al. , 2021 ) . Jacobian Decomposition . Some works use the Jacobian matrix to analyze the latent space of GAN ( Zhu et al. , 2021 ; Wang & Ponce , 2021 ; Chiu et al. , 2020 ; Ramesh et al. , 2018 ) . However , these methods focus on the Jacobian of the entire model , from the input noise z to the output image . Ramesh et al . ( 2018 ) suggested the right singular vectors of the Jacobian as local disentangled direc- tions in the Z space . Zhu et al . ( 2021 ) proposed a latent perturbation vector that can change only a particular area of the image . The perturbation vector is discovered by taking the principal vector of the Jacobian to the target area and projecting it into the null space of the Jacobian to the complementary region . On the other hand , our Local Basis utilizes the Jacobian matrix of the partial network , from the input noise z to the intermediate latent code w , and investigates the black-box intermediate latent space from it . The top-k Local Basis corresponds to the best-local-geometry-describing submanifolds . This intuition leads to exploiting Local Basis to assess the global geometry of the intermediate latent space . 3 TRAVERSING A CURVED LATENT SPACE . In this section , we introduce a method for finding a local-geometry-aware traversal direction in the intermediate latent spaceW . The traversal direction is refered to as the Local Basis at w ∈ W . In addition , we evaluate the proposed Local Basis by observing how the generated image changes as we traverse the intermediate latent variable . Throughout this paper , we assess Local Basis of theWspace in StyleGAN2 ( Karras et al. , 2020b ) . However , our methodology is not limited to StyleGAN2 . See appendix for the results on StyleGAN ( Karras et al. , 2019 ) and BigGAN ( Brock et al. , 2018 ) . 3.1 FINDING A LOCAL BASIS . Given a pretrained GAN model M : Z → X , from the input noise space Z to the image space X , we choose the intermediate layer W̃ to discover Local Basis . We refer to the former part of the GAN model as the mapping network f : Z → W̃ . The image of the mapping network is denoted as W = f ( Z ) ⊂ W̃ . The latter part , a non-linear mapping from W̃ to the image space X , is denoted by G : W̃ → X . Local Basis at w ∈ W is defined as the basis of the tangent space TwW . This basis can be interpreted as a local-geometry-aware linear traversal direction starting from w. To define the tangent space of the intermediate latent space W properly , we assume that W is a differentiable manifold . Note that the support of the isotropic Gaussian prior Z = RdZ and the ambient space W̃ = RdW̃ are already differentiable manifolds . The tangent space at w , denoted by TwW , is a vector space consisting of tangent vectors of curves passing through point w. Explicitly , TwW = { γ̇ ( 0 ) | γ : ( − , ) →W , γ ( 0 ) = w , for > 0 } . ( 1 ) Then , the differentiable mapping network f gives a linear map dfz between the two tangent spaces TzZ and TwW̃ where w = f ( z ) . dfz : TzZ −→ TwW ↪−→ TwW̃ , γ̇ ( 0 ) 7−→ ˙ ( f ◦ γ ) ( 0 ) ( 2 ) We utilize the linear map dfz , called the differential of f at z , to find the basis of TwW . Based on the manifold hypothesis in representation learning , we posit that the latent space of the image space X in W̃ is a lower-dimensional manifold embedded in W . In this approach , we estimate the latent manifold as a lower-dimensional approximation ofW describing its principal variations . The approximation manifold can be obtained by solving the low-rank approximation problem of dfz . The manifold hypothesis is supported by the empirical distribution of singular values σzi . The analysis is provided in Fig 9 in the appendix . The low-rank approximation problem has an analytic solution defined by Singular Value Decomposition ( SVD ) . Because the matrix representation of dfz is a Jacobian matrix ( ∇zf ) ( z ) ∈ RdW̃×dZ , Local Basis is obtained as the following : For the i-th right singular vector uzi ∈ RdZ , i-th left singular vector vwi ∈ RdW̃ , and i-th singular value σzi ∈ R of ( ∇zf ) ( z ) with σz1 ≥ · · · ≥ σzn , dfz ( u z i ) = σ z i · vwi for ∀i , ( 3 ) Local Basis ( w = f ( z ) ) = { vwi } 1≤i≤n . ( 4 ) Then , the k-dimensional approximation ofW around w is described as the following because Z = RdZ ( if σzk > 0 ) . Note that Wkw is a submanifold1 of W corresponding to the k components of Local Basis , i.e . TwWkw = span { vwi : 1 ≤ i ≤ k } . Wkw = { f ( z + ∑ i ti · uzi ) | ti ∈ ( − i , i ) , for 1 ≤ i ≤ k } ( 5 ) Locally affine mapping network In this paragraph , we focus on the locally affine mapping network f , which is one of the most widely adopted GAN structures , such as MLP or CNN layers with ReLU or leaky-ReLU activation functions . This type of mapping network has several well-suited properties for Local Basis . f ( z ) = ∑ p∈Ω 1z∈p ( Apz + bp ) ( 6 ) where Ω denotes a partition ofZ , and Ap and bp are the parameters of the local affine map . With this type of mapping network f , it is clear that the intermediate latent spaceW satisfies a differentiable manifold property at least locally on the interior of each p ∈ Ω . The region where the property may not hold , the intersection of several closure of p ’ s in Ω , has measure zero in Z . Moreover , the Jacobian matrix ( ∇zf ) ( z ) becomes a locally constant matrix . Then , the approximating manifold Wkw ( Eq 5 ) satisfies the submanifold condition , and is consistent locally for each p , avoiding being defined for each w. In addition , the linear traversal of the latent variable w along vwi can be described as the curve on W ( Eq 7 ) . Most importantly , these curves on W ( Eq 7 ) , starting from w in the direction of Local Basis , corresponds to the local coordinate mesh ofWkw . Traversal ( w = f ( z ) , vwi ) : ( − , ) −→ Z f−−→W , t 7→ ( z + t σzi · uzi ) 7→ ( w + t · vwi ) ( 7 ) Equivalence to Local PCA To provide additional intuition about Local Basis , we prove the following proposition . The proposition shows that Local Basis is equivalent to applying a PCA on the samples onW around w. Proposition 1 ( Equivalence to Local PCA ) . Consider the Local PCA problem around the base latent variable wb = f ( zb ) onW , i.e . PCA of the latent variable samples w′ around wb . w′ = T1f ( zb + c · ) with ∼ N ( 0 , I ) and for some small c > 0 . ( 8 ) where T1f ( z ) = wb + ( ∇zbf ) ( z − zb ) is the linear approximation of f around zb . Then , the principal components discovered in the Local PCA problem are equivalent to Local Basis at wb .

Summary: The authors proposed a geometric method for image editing with pretrained StyleGAN2. The idea is simple. Editing is performed by constraining the editing path on the manifold. The simplest way of making this is to use the tangent space. A natural realization of the tangent space is the principal subspace of the Jacobian. The authors harness the Jacobian regarding the mapping from $z$ to $w$. The extensive experimental results verify the effectiveness of the method.