A Geometric Analysis of Deep Generative Image Models and Its Applications

1 BACKGROUND . Generative adversarial networks ( GANs ) learn patterns that characterize complex datasets , and subsequently generate new samples representative of that set . In recent years , there has been tremendous success in training GANs to generate high-resolution and photorealistic images ( Karras et al. , 2017 ; Brock et al. , 2018 ; Donahue & Simonyan , 2019 ; Karras et al. , 2020 ) . Well-trained GANs show smooth transitions between image outputs when interpolating in their latent input space , which makes them useful in applications such as high-level image editing ( changing attributes of faces ) , object segmentation , and image generation for art and neuroscience ( Zhu et al. , 2016 ; Shen et al. , 2020 ; Pividori et al. , 2019 ; Ponce et al. , 2019 ) . However , there is no systematic approach for understanding the latent space of any given GAN or its relationship to the manifold of natural images . Because a generator provides a smooth map onto image space , one relevant conceptual model for GAN latent space is a Riemannian manifold . To define the structure of this manifold , we have to ask questions such as : are images homogeneously distributed on a sphere ? ( White , 2016 ) What is the structure of its tangent space — do all directions induce the same amount of variance in image transformation ? Here we develop a method to compute the metric of this manifold and investigate its geometry directly , and then use this knowledge to navigate the space and improve several applications . To define a Riemannian geometry , we need to have a smooth map and a notion of distance on it , defined by the metric tensor . For image applications , the relevant notion of distance is in image space rather than code space . Thus , we can pull back the distance function from the image space onto the latent space . Differentiating this distance function on latent space , we will get a differential geometric structure ( Riemannian metric ) on the image manifold . Further , by computing the Riemannian metric at different points ( i.e . around different latent codes ) , we can estimate the anisotropy and homogeneity of this manifold . The paper is organized as follows : first , we review the previous work using tools from Riemannian geometry to analyze generative models in section 2 . Using this geometric framework , we introduce an efficient way to compute the metric tensor H on the image manifold in section 3 , and empirically investigate the properties of H in various GANs in section 4 . We explain the properties of this metric in terms of network architecture and training in section 5 . We show that this understanding provides a unifying principle behind previous methods for interpretable axes discovery in the latent space . Finally , we demonstrate other applications that this geometric information could facilitate , e.g . gradient-free searching in the GAN image manifold in section 6 . 2 RELATED WORK . Geometry of Deep Generative Model Concepts in Riemannian geometry have been recently applied to illuminate the structure of latent space of generative models ( i.e . GANs and variational autoencoders , VAEs ) . Shao et al . ( 2018 ) designed algorithms to compute the geodesic path , parallel transport of vectors and geodesic shooting in the latent space ; they used finite difference together with a pretrained encoder to circumvent the Jacobian computation of the generator . While promising , this method did not provide information of the metric directly and could not be applied to GANs without encoders . Arvanitidis et al . ( 2017 ) focused on the geometry of VAEs , deriving a formula for the metric tensor in order to solve the geodesic in the latent space ; this worked well with shallow convolutional VAEs and low-resolution images ( 28 x 28 pixels ) . Chen et al . ( 2018 ) computed the geodesic through minimization , applying their method on shallow VAEs trained on MNIST images and a low-dimensional robotics dataset . In the above , the featured methods could only be applied to neural networks without ReLU activation . Here , our geometric analysis is architecture-agnostic and it ’ s applied to modern large-scale GANs ( e.g . BigGAN , StyleGAN2 ) . Further , we extend the pixel L2 distance assumed in previous works to any differentiable distance metric . 3 METHODS . Formulation A generative network , denoted by G , is a mapping from latent code z to image I , G : Rn → I = RH×W×3 , z 7→ I . Borrowing the language of Riemannian geometry , G ( z ) parameterizes a submanifold in the image space with z ∈ Rn . Note for applications in image domain , we care about distance in the image space . Thus , given a distance function in image space D : I × I → R+ , ( I1 , I2 ) 7→ L , we can define the distance between two codes as the distance between the images they generate , i.e . pullback the distance function to latent space through G. d : Rn × Rn → R+ , d ( z1 , z2 ) : = D ( G ( z1 ) , G ( z2 ) ) . The Hessian matrix ( second order partial derivative ) of the squared distance function d2 can be seen as the metric tensor of the image manifold ( Palais , 1957 ) . The intuition behind this is as follows : consider the squared distance to a fixed reference vector z0 as a function of z , fz0 ( z ) = d 2 ( z0 , z ) . Obviously , z = z0 is a local minimum of fz0 ( z ) , thus fz0 ( z ) can be locally approximated by a positive semi-definite quadratic form H ( z0 ) as in Eq.1 . This matrix induces an inner product and defines a vector norm , ‖v‖2H = vTH ( z0 ) v. This squared vector norm approximates the squared image distance , d2 ( z0 , z0 + δz ) ≈ ‖δz‖2H = δTzH ( z0 ) δz . Thus , this matrix encodes the local distance information on the image manifold up to second order approximation . This is the intuition behind Riemannian metric . In this article , the terms ” metric tensor ” and ” Hessian matrix ” are used interchangeably . We will call αH ( v ) = vTHv/vTv the approximate speed of image change along v as measured by metric H . d2 ( z0 , z ) ≈ δzT ∂2d2 ( z0 , z ) ∂z2 |z0δz , H ( z0 ) : = ∂2d2 ( z0 , z ) ∂z2 |z0 ( 1 ) Numerical Method As defined above , the metric tensor H can be computed by doubly differentiating the squared distance function d2 . Here we use a convolutional neural network ( CNN ) -based distance metric , LPIPS ( Zhang et al. , 2018 ) , as it has been demonstrated to approximate human perceptual similarity judgements . The direct method to compute Hessian is by building a computational graph towards the gradient g ( z ) = ∂zd2|z=z0 and then computing the gradient towards each element in g ( z ) . This method computes H column by column , therefore its time complexity is proportional to the latent-space dimension n and the backpropagation time through this graph . For situations when direct backpropagation is too slow ( e.g . FC6GAN , StyleGAN2 ) , we developed an approximation method to compute the major eigen-dimensions of the Hessian more efficiently . These top eigen-pairs are useful in applications like optimization and exploration ; moreover , they form the best low-rank approximation to the Hessian . As we will later discover , the spectra of these Hessians have a fast decay , thus far less than n eigenvectors are required to approximate them , cf . Sec 4 . As a matrix , the Hessian is a linear operator , which could be defined as long as one can compute the Hessian vector product ( HVP ) . Since the gradient to z commutes with inner product with v , HVP can be rewritten as the gradient to vTg , or the directional derivative to the gradient vT∂zg ( Eq.2 ) . The first form ∂z ( vTg ) is easy to compute in reverse-mode auto-differentiation , and the directional derivative is easy to compute in forward-mode auto-differentiation ( or finite differencing ) . Then , Lanczos iteration is applied to the HVP operator defined in these two ways to solve the largest eigen pairs , which can reconstruct an approximate Hessian matrix . The iterative algorithm using the two HVP definitions are termed Backward Iteration and Forward Iteration respectively . For details and efficiency comparison , see Appendix A.2 . HV P : v 7→ Hv = ∂z ( vTg ( z ) ) = vT∂zg ( z ) ≈ ( g ( z + v ) − g ( z − v ) ) /2‖ v‖ ( 2 ) Note a similar computational method has been employed to understand the optimization landscape of deep neural networks recently ( Ghorbani et al. , 2019 ) , although it has not been applied towards the geometry of latent space of GANs before . Connection to Jacobian This formulation and computation of the Riemannian metric is generic to any mapping into a metric space . Consider a mapping φ ( z ) : Rn → RM , which could be the feature map of a layer in the GAN , or a CNN processing the generated image . We can pull back the squared L2 distance and metric from RM , d2φ ( z1 , z2 ) = 1 2‖φ ( z1 ) −φ ( z2 ) ‖ 2 2 , and define a manifold . The metric tensor Hφ of this manifold can be derived as Hessian of d2φ . Note , there is a simple relationship between the Hessian of d2φ , Hφ and the Jacobian of φ , Jφ ( Eq . 3 ) . Through this we know the eigenvalues and eigenvectors of the Hessian matrix Hφ correspond to the squared singular values and right singular vectors of the Jacobian Jφ . This allows us to examine the geometry of any representation throughout the GAN , and analyze how the geometry in the image space builds up . Hφ ( z0 ) = ∂2 ∂z2 1 2 ‖φ ( z0 ) − φ ( z ) ‖22|z0 = Jφ ( z0 ) TJφ ( z0 ) ( 3 ) vTHφ ( z0 ) v = ‖Jφ ( z0 ) v‖2 , Jφ ( z0 ) = ∂zφ ( z ) |z0 ( 4 ) In this work , we use LPIPS , which defines image distance based on the squared L2 distance of the first few layers of a pretrained CNN . If we concatenate the activations and denote this representational map by ϕ ( I ) : I → RF , then the metric tensor of the image manifold can be derived from the Jacobian of the composite of the generator and the representation map ϕ , H ( z ) = JTϕ◦GJϕ◦G , Jϕ◦G = ∂zϕ ( G ( z ) ) . This connection is crucial for understanding how geometry depends on the network architecture . 4 EMPIRICAL OBSERVATIONS . Using the above method , we analyzed the geometry of the latent space of the following GANs : DCGAN ( Radford et al. , 2015 ) , DeePSiM/FC6GAN ( Dosovitskiy & Brox , 2016 ) , BigGAN ( Brock et al. , 2018 ) , BigBiGAN ( Donahue & Simonyan , 2019 ) , Progressive Growing of GANs ( PGGAN ) ( Karras et al. , 2017 ) , StyleGAN 1 and 2 ( Karras et al. , 2019 ; 2020 ) - model specifications reviewed in Sec . A.3 . These GANs are progressively deeper and more complex , and some employ a style-based architecture instead of the conventional DCGAN architecture ( e.g . StyleGAN1,2 ) . This diverse set of models allowed us to test the broad applicability of this new approach . In the following sections , ” top ” and ” bottom ” eigenvectors refer to the eigenvectors with large and small eigenvalues . Top Eigenvectors Capture Significant Image Changes . In differential geometry , a metric tensor H captures an infinitesimal notion of distance . To determine whether this quantity represents evident image changes , we randomly picked a latent code z0 , then computed the metric tensorH ( z0 ) and its eigendecomposition H ( z0 ) = ∑ i λiviv T i . Then we explored linearly in the latent space 1along the eigenvectorsG ( z0 +µivi ) . We found that images changed much faster when moving along top than along bottom eigenvectors , both per visual inspection and LPIPS ( Fig.1 ) . More intriguingly , eigenvectors at different ranks encoded qualitatively different types of changes ; for example , in BigGAN noise space , the top eigenvectors encoded head direction , proximity and size ; while lower eigenvectors encoded background changes , shading or much more subtle pixel-wise changes . Moreover , PGGAN and StyleGANs trained on the face dataset ( celebA , FFHQ ) have top eigenvectors that represent similar interpretable transforms of faces , such as head direction , sex or age ( Fig.10 ) . These observations raised the possibility that top eigenvectors also captured perceptually relevant changes : we tested this directly with positive results in Sec . 6 . A B C DBigGAN Class Space BigGAN Noise Space StyleGAN2 Cat StyleGAN2 Face Spectrum Structure of GANs To explore how eigenvalues were distributed , for each GAN , we randomly sampled 100-1000 z in the latent space , used backpropagation to compute H ( z ) and then performed the eigendecomposition . In Fig . 2 , we plotted the mean and 90 % confidence interval of the spectra and found that they spanned 5-10 orders of magnitude , with fast decays ; each spectrum was dominated by a few eigenvectors with large eigenvalues . In other words , only a small fraction of dimensions were responsible for major image changes ( Table 2 ) , while most dimensions introduced nuanced changes ( e.g . shading , background ) — thus GAN latent spaces were highly anisotropic . We found this anisotropy in every GAN we tested , which raises the question of why it has not been discussed more frequently . One possibility is that the statistical properties of high dimensionality create an illusion of isotropy . When traveling along a random direction v in latent space , the approximate rate of image change αH ( v ) = vTHv/vTv is a weighted average of all eigenvalues as in Eq . 9 . In Sec A.6 , we show analytically that the variance of α ( v ) across random directions will be 2/ ( n+2 ) times smaller than the variance among eigenvalues . For example , in BigGAN latent space ( 256 dimensions ) , the eigenvalues span over six orders of magnitude , while the α ( v ) for random 1For some spaces , we used spherical linear exploration ( i.e . SLERP ) , where we restrict the vector to a sphere of certain norm . We project vi onto tangent space of z0 and travel on the big circle from z0 along vi . directions has a standard deviation less than one order of magnitude ( Figs . 2 , 6 ) . Further , the center of this distribution was closer to the top of the spectrum , and thus provided a reasonable rate of change , while masking the existence of eigendimensions of extremely large and small eigenvalues . Global Metric Structure Because the metric H ( z ) describes local geometry , the next question is how it varies at different positions in the latent space . We computed the metric H ( z ) at randomly selected z and examined their similarity using a statistic adopted from Kornblith et al . ( 2019 ) . In this statistic , we applied the eigenvectors Ui = [ u1 , ... un ] from a metric tensor Hi at position zi to the metric tensor Hj at zj , as uTi Hjui . These values formed a vector Λij , representing the effects of metric Hj on eigenvectors of Hi . Then we computed the Pearson correlation coefficient between Λij and the target eigenvalues , Λj , as corr ( Λj , Λij ) . This correlation measured the similarity of the action of metric tensors on eigenframes around different positions . As the spectrum usually spanned several orders of magnitude , we computed the correlation on the log scale CHlogij , where the eigenvalues distribute more uniformly . Λij = diag ( U T i H ( zj ) Ui ) ( 5 ) CHij = corr ( Λij , Λj ) , C Hlog ij = corr ( log ( Λij ) , log ( Λj ) ) ( 6 ) Using this correlation statistic , we computed the consistency of the metric tensor across hundreds of positions within each GAN latent space . As shown in Fig . 3C , the average correlation between eigenvalues and vHv values of two points CHlogij was 0.934 in BigGAN . For DCGAN-type architecture , mean correlations on the log scale ranged from 0.92-0.99 ; for StyleGAN-1,2 , 0.64-0.73 in the Z space , and 0.87-0.89 in the W space ( Fig . 3D , Tab.4 ) . Overall , this shows that the local directions that induce image changes of different orders of magnitude are highly consistent at different points in the latent space . Because of this , the notion of a ” global ” Hessian makes sense , and we estimated it for each latent space by averaging the Hessian matrices at different locations . Implication of the Null Space As the spectra have a large portion of small eigenvalues and the metric tensors are correlated in space , the bottom eigenvectors should create a global subspace , in which latent traversal will result in small or even imperceptible changes in the image . This is supported by our perceptual study , as over half of the subjects can not see any change in image when latent vector move in bottom eigenspace . ( Sec . 6 ) . This perceptually ” null ” space has implications about exploration in the GAN space and interpretable axes discovery . As G ( z + v ) ≈ G ( z ) , if one axis u encodes an interpretable transformG ( z ) → G ( z+u ) , then shifting this vector by a vector in the null space v will still result in an interpretable axis G ( z ) → G ( z + v + u ) ≈ G ( z + u ) . Thus , each interpretable axis have a family of ” equivalent ” axes which encode similar transforms , differing from each other by a vector in ” null ” space . However , adding component v in the null space will decrease the rate of image change along that axis . In this sense , the vectors using a smallest step size to achieve that transform should be the ” purest ” axes of the family . Further , the cosine angle between two interpretable axes may not represent the similarity of the transforms they encode . A large angle can be found between two axes of the same family but at different image traversal speed . We compared the axes from previous works in A.9 and observed that projecting out a large part of their axes did not affect the semantics it encoded ( Fig . 8 ) .

Summary: This work intends to explore the geometry of the latent space and proposes to define the distance in latent space as the distance between the corresponding generated images and use the Hessian of that squared distance as metric tensor to define the manifold. Using Learned Perceptual Image Patch Similarity (LPIPS), they show that the Hessian can either be computed through backpropagation or if that is not efficient, it is sufficient to iteratively compute the eigenvectors corresponding to the largest eigenvalues. With the proposed method, the empirical observations showed 1) the impact of those eigenvectors through examples, 2) consistent geometric local changes over different positions in the latent space, and 3) the impact of top eigenvectors on particular layers. Further, the authors discuss three areas of possible application (gradient-based GAN inversion, gradient-free image search, interpretable axes discovery).