A Geometric Analysis of Deep Generative Image Models and Its Applications
Authors: Binxu Wang
Carlos R Ponce
Venue: ICLR 2021 Poster (2021)
Keywords: Deep generative model
Interpretability
GAN
Differential Geometry
Optimization
Model Inversion
Feature Visualization

Abstract: Generative adversarial networks (GANs) have emerged as a powerful unsupervised method to model the statistical patterns of real-world data sets, such as natural images. These networks are trained to map random inputs in their latent space to new samples representative of the learned data. However, the structure of the latent space is hard to intuit due to its high dimensionality and the non-linearity of the generator, which limits the usefulness of the models. Understanding the latent space requires a way to identify input codes for existing real-world images (inversion), and a way to identify directions with known image transformations (interpretability). Here, we use a geometric framework to address both issues simultaneously. We develop an architecture-agnostic method to compute the Riemannian metric of the image manifold created by GANs. The eigen-decomposition of the metric isolates axes that account for different levels of image variability. An empirical analysis of several pretrained GANs shows that image variation around each position is concentrated along surprisingly few major axes (the space is highly anisotropic) and the directions that create this large variation are similar at different positions in the space (the space is homogeneous). We show that many of the top eigenvectors correspond to interpretable transforms in the image space, with a substantial part of eigenspace corresponding to minor transforms which could be compressed out. This geometric understanding unifies key previous results related to GAN interpretability. We show that the use of this metric allows for more efficient optimization in the latent space (e.g. GAN inversion) and facilitates unsupervised discovery of interpretable axes. Our results illustrate that defining the geometry of the GAN image manifold can serve as a general framework for understanding GANs. 

Published as a conference paper at ICLR 2021
THE GEOMETRY OF DEEP GENERATIVE IMAGE MOD- ELS AND ITS APPLICATIONS
Binxu Wang
Carlos R. Ponce
Department of Neuroscience
Department of Neuroscience
Washington University in St Louis St Louis, MO, USA binxu. wwngwwustl. edu
Washington University in St Louis St Louis, MO, USA
crponce@wustl..em
ABSTRACT
Generative adversarial networks (GANs) have emerged a powerful unsuper- vised method to model the statistical patterns real-world data sets, such as natu- ral images. These networks are trained to map random inputs in their latent space to new samples representative latent space is hard to intuit due to its high dimensionality and the non non-linearity of the generator, limiting the usefulness of the models. Understanding the latent space requires way identify input codes for existing real-world images (inver- sion), and way to identify directions with known image transformations (inter- pretability). Here, we use a geometric framework to address both issues simul- taneously. We develop an architecture-agnuee aggnssicc method to compute the Rieman- nian metric of the image manifold created by GANs. The of the metric isolates axes that account for different levels of image variability. An empirical analysis of several pretrained GANs shows that image variation around each position is concentrated along surprisingly few major axes (the space is highly anisotropic) and the directions that create this large variation are similar at different positions in the space (the space is homogeneous). We show that many the top eigenvectors correspond to interpretable transforms in the image space, with substantial part of eigenspace corresponding to minor transforms which could be compressed out. This geometric understanding unifies key previous re- sults related to GAN interpretability. We show that the of this metric allows for more efficient optimization in the latent space (e.g. GAN inversion) and facilitates unsupervised discovery of interpretable axes. Our results illustrate that defining the geometry of the GAN image manifold can serve as a general framework for understanding GANs.
the learned data. However, the structure the
BACKGROUND
Generative adversarial networks (GANs) learn patterns that characterize complex datasets, and sub- sequently generate new samples representative of that set. In recent years, there has been tremendous success in training GANs to generate high-resolution and photorealistic images (Karras al. 2017 Brock et al. 2018 Donahue & Simonyan, 2019 Karras et al.) 2020). Well-trained GANs show smooth transitions between image outputs when interpolating in their latent input space, which makes them useful in applications such as high-level image editing (changing attributes of faces), object segmentation, and image generation for art and neuroscience (Zhu al., 2016)) Shen al.) 2020 Pividori et al. 2019 Ponce et al. 2019). However, there is no systematic approach for under- standing the latent space of any given GAN its relationship to the manifold natural images.
Because a generator provides smooth map onto image space, one relevant conceptual model for GAN latent space is Riemannian manifold. To define the structure of this manifold, we have to ask questions such as: images homogeneously distributed on sphere? (Whitel 2016) What the structure of its tangent space - do all directions induce the same amount of variance in image transformation? Here develop method to compute the metric of this manifold and investigate its geometry directly, and then use this knowledge to navigate the space and improve several applications.

---

Published as a conference paper at ICLR 2021
To define a Riemannian geometry, we need to have smooth map and notion of distance on it, de- fined by the metric tensor. For image applications, the relevant notion of distance is in image space rather than code space. Thus, we can pull back the distance function from the image space onto the latent space. Differentiating this distance function on latent space, we will get differential geomet- ric structure (Riemannian metric) on the image manifold. Further, by computing the Riemannian metric at different points (i.e. around different latent codes), we can estimate the anisotropy and homogeneity of this manifold.
The paper is organized as follows: first, we review the previous work using tools from Riemannian geometry to amalyze generative models in section 2 Using this geometric framework, we introduce an efficient way to compute the metric tensor H on the image manifold in section [] and empirically investigate the properties of H in various GANs in section 4 We explain the properties of this
metric in terms of network architecture and training in section 5 We show that this understanding provides a unifying principle behind previous methods for interpretable axes discovery in the latent space. Finally, we demonstrate other applications that this geometric information could facilitate, e.g. gradient- free searching in the GAN image manifold in section
RELATED WORK
Geometry of Deep Generative Model Concepts in Riemannian geometry have been recently ap- plied to illuminate the structure of latent space of generative models (i.e. GANs and variational autoencoders, VAEs). [Shao et al. (2018) designed algorithms to compute the geodesic path, parallel transport of vectors and geodesic shooting in the latent space; they used finite difference together with a pretrained encoder to circumvent the Jacobian computation of the generator. While promis- ing, this method did not provide information of the metric directly and could not be applied to GANs without encoders. AAvanitiiis et al.] (2017) focused for the metric tensor in order to solve the geodesic in the latent space; this worked well with shallow convolutional VAEs and low-resolution images (28 x 28 pixels). Chen et al. (2018) computed the geodesic through minimization, applying their method on shallow VAΣs trained on MNIST images and low-dimensional robotics dataset. In the above, the featured methods could only be applied to neural networks without ReLU activation. Here, our geometric analysis is prchitecture-agnssii it's applied to modern large-scale GANs (e.g. BigGAN, StyleGAN2). Further, we extend the pixel 1.2 distance assumed in previous works to any differentiable distance metric.
the geometry of VAEs, deriving formula
and
3 METHODS
Formulation A generative network, denoted by G. is a mapping from latent code to image R" + I = " x3 + I. Borrowing the language of Riemannian geometry, G(z) I, :.:
parameterizes submanifold in the image space with z RR.. Note for applications in image domain, we care about distance in the image space. Thus, given distance function in image space D : I x I - R4, (I1, + L, we can define the distance between two codes as the distance between the images they generate, i.e. pullback the distance function to latent space through G. d: R x R + R,,,d d(z1, 22) == ((G())), G(z2)).
The Hessian matrix (second order partial derivative) of the squared distance function d2 can be seen as the metric tensor of the image manifold (Palais 1957). The intuition behind this is as follows: consider the squared distance Obviously, z = Zg is a local minimum of fzo (z), thus fza (z) can be locally approximated by a positive semi-definite quadratic form H ((x)) in Eq [ This matrix induces an inner product and defines a vector norm, || l = (())). This squared vector norm approximates the squared image distance, d( 2p + 8z) x || = 87 H (20)02. Thus, this matrix encodes the local distance information the image manifold up to second order approximation. This is the intuition behind Riemannian metric. In this article, the terms "metric tensor" and "Hessian matrix" are used interchangeably. We will call ( = v Hv/v" T the approximate speed of image change along as measured by metric H.
fixed reference vector 2 as a function of z. f (z) d= z).
l( x) ~ 52 PP z)
(Pllzzn 2)
82, H(zo) =
(1)
dz2
dz²
20
2

---

Published as a conference paper at ICLR 2021
Numerical Method As defined above, the metric tensor H can be computed by doubly differen- tiating the squared distance function d², Here we use a convolutional neural network (CNN)-based distance metric, LPIPS (Zhang et al.] 2018), as it has been demonstrated to approximate human perceptual similarity judgements. The direct method to compute Hessian is by building a compu- tational graph towards the gradient g(z) 0_d === and then computing the gradient towards each element in (2)). This method computes H column by column, therefore its time complexity proportional the latent-space dimension and the backpropagation time through this graph.
For situations when direct backpropagation is too slow (e.g. FC6GAN, StyleGAN2), we developed an approximation method to compute the major eigen- digen-dimenssons of the Hessian more efficiently. These top eigen-pairs are useful in applications like optimiattion and exploration; moreover, they form the best low-rank approximation to the Hessian. As we will later discover, the spectra of these Hessians have fast decay, thus far less than n eigenvectors are required to approximate them, cf. Sec As a matrix, the Hessian is a linear operator, which could be defined as long as one can com- pute the Hessian vector product (HVP). Since the gradient to commutes with inner product with v. HVP can be rewritten as the gradient to v² g, or the directional derivative to the gradient v' 02g (Eq The first form 0² (v" g) is easy compute in reverse- monde nuto-differentiation and the di- rectional derivative is easy to compute in forward -ooee auto-diie tito-dfffrrentationn (or finite differencing). Then, Lanezos iteration is applied to the HVP operator defined in these two ways to solve the largest eigen pairs, which can reconstruct an approximate Hessian matrix. The iterative algorithm using the two HVP definitions are termed Backward Iteration and Forward Iteration respectively. For details and efficiency comparison, see Appendix A.2
(2) Note a similar computational method has been employed understand the optimization landscape of deep neural networks recently (Ghorbani et al. 2019), although it has not been applied towards the geometry of latent space of GANs before.
HVP: v + He = 00(0" g ()) v² 0zg(z) (g(z + ev) - g(= (((=-e)))/2) e2))\2||ebl
Connection to Jacobian This formulation and computation of the Riemannian metric is generic to any mapping into a metric space. Consider a mapping d(z) R" + RRM which could be the feature map of layer in the GAN, or CNN processing the generated image. We can pull back the squared L2 distance and metric from R.M de z2) = 11l((z)) - x(z) ((z))l [12) and define a manifold.
The metric tensor Ho of this manifold can be derived as Hessian of dz. Note, there is a simple relationship between the Hessian of dz. Ha and the Jacobian of ø, Ja (Eq. 3). Through this we know the eigenvalues and cigenvectors of the Hessian matrix H& correspond to the squared singular values and right singular vectors of the Jacobian Jo. This allows us to examine the geometry of any representation throughout the GAN, and analyze how the geometry in the image space builds up.
82 1 110(z)) - (=) = J (Z)
2 (z))) d2 2
(3)
o7 Hz (Z)0()) (z(z = 020(2))=
(4)
In this work, we use LPIPS, which defines image distance based on the squared L2 distance of the first few layers of a pretrained CNN. If we concatenate the activations and denote this representational map by p([) I - RF. then the metric tensor of the image manifold can be derived from the Jacobian of the composite of the generator and the representation map i H(z) poct JpoG (Gz)) This connection is crucial for understanding how ge- ometry depends on the network architecture
4 EMPIRICAL OBSERVATIONS
Using the above method, we analyzed the geometry of the latent space of the following GANs: DCGAN (Radford et al.] 2015). DeePSiM/FCGGAN (Dosovitskiy & Brox] 2016). BigGAN (Brock) ct al., 2018), BigB (Donahue & Simonyan 2019), Progressive Growing of GANs (PGGAN) Karras et al. 2017), StyleGAN I and 2 (Karras et al. 2019 2020 model specifications reviewed in Sec. A.3 These GANS are progressively deeper and more complex, and some employ a style-based based architecture instead of the conventional DCGAN architecture (e.g. StyleGAN1.2). This diverse set of models allowed us to test the broad applicability of this new approach. In the following sections, "top" and "bottom" eigenvectors refer the eigenvectors with large and small eigenvalues.
3

---

Published as a conference paper at ICLR 2021
Top Eigenvectors Capture Significant Image Changes. In differential geometry, a metric tensor H captures an infinitesimal notion of distance. To determine whether this quantity represents evident image changes, we randomly picked latent code &o. then computed the metric tensor H (zn) and its eigendecomposition H (2o) = Li \, Then we explored linearly in the latent space along the eigenvectors G( + (. We found that images changed much faster when moving along top than along bottom eigenvectors, both per visual inspection and LPIPS (Fig D. More intriguingly. eigen- vectors at different ranks encoded qualitatively different types of changes; for example, in BigGAN noise space. the top eigenvectors encoded head direction, proximity and size; while lower eigen- vectors encoded background changes, shading or much more subtle pixel-wise changes. Moreover, PGGAN and StyleGANs trained on the face dataset (celebA,FFHQ) have top eigenvectors that rep- resent similar interpretable transforms of faces, such as head direction, sex or age (Fig TOp. These observations raised the possibility that top eigenvectors also captured perceptually relevant changes: we tested this directly with positive results in Sec. 6
BigGAN Class Space
BigGAN Noise Space
StyleGAN2 Cat
StyleGAN Face
0.6
0.6
OMS -EE
Image Distacce
0.6
04
04
MALL
AIES
0.4
64.
02
02
02
300
0.0
0.0
0.0
-050
-0.25
0.00
025
050
-0.18
000
018
-25
00
GD space
in lafennt
Angle(rad] in latent space
latent spare
L2in Iatenttnt space
Figure 1: Images change at different rates along top vs bottom eigenvectors. Each panel (A-D) shows perturbations around a randomly chosen reference image (center column); each row shows perturbations introduced by moving along each of five eigenvectors; each contiguous column is separated by the same distance in latent space. Eigenvectors are shown in descending order of their eigenvalues. Line plots under each montage show the LPIPS image distance to the reference image as a function of positively and negatively perturbed distance along each eigenvector (x-axis). The rate of image change differed across eigenvectors; top vs bottom eigenvectors encoded changes such as object class, head direction, pose, color, shading or other subtle details (e.g. fur variations panel A, bottom). Eigenvector rank and associated eigenvalue labeled in the legend.
Spectrum Structure of GANS To explore how eigenvalues were distributed, for each GAN, we randomly sampled 100-1000- z in the latent space, used backpropagation to compute Hz)) and then performed the cigendecomposition. In Fig. [ we plotted the mean and 90% confidence interval of the spectra and found that they spanned 5-10 orders of magnitude, with fast decays; each spectrum was dominated by a few eigenvectors with large eigenvalues. In other words, only a small fraction of dimensions were responsible for major image changes (Table while most dimensions introduced nuanced changes (e.g. shading, background) - thus GAN latent spaces were highly anisotropic.
We found this anisotropy in every GAN we tested, which raises the question of why has not been discussed more frequently. One possibility is that the statistical properties of high dimensionality create an illusion of isotropy. When traveling along a random direction e in latent space, the approx- imate rate of image change (v) v" H p/u" v is weighted average of all cigenvalues in Eq. 9 In Sec A.6 we show analytically that the variance of (D) across random directions will be 2/(n+2) times smaller than the variance among eigenvalues. For example, in BigGAN latent space (256 dimensions), the eigenvalues span over six orders of magnitude, while the avv) for random
'oor some spaces, we used spherical linear exploration (i.e. SLERP), where we restrict the vector to sphere of certain norm. We project v, onto tangent space of zg and travel the big circle from zo along e;.
4

---

Published as a conference paper at ICLR 2021
0
Spectra Compared GANS
C
BROAY
Reequero Density
122
Speed Image Char
botropic for Rendomiiestors
BloBIGNN
BigGAN
DCGAN-fashion
O.B.
ad of random
vector
dignn
BigGAN
gelDeegleegmma
POG
SG-face Z 5Gface
0.4
DCGAN
6.
40
80
1200
0.0
-2
log10leigval) 4
-8
-2
-10
Sca-faoe512 W
Scrace
-12
StyleGAN Z. W
100
200 ranks
300
400
1000 2000 3000
0 40
80
120
Figure 2: Spectra of GANs, shown as a function of individual model (A) and types of architecture (B, C). DCGAN-type (green-blue), Z. and W space for StyleGAN (SG1, 2) (red, orange). Lines and shaded areas show the averaged spectra and the 5-95% percentile of each eigenvalue among samples (for quantification, Table 2. D. Histogram of approximate speed of image change ((v) for eigenvectors and random directions, visualizing the "illusion of isotropy".
directions has a standard deviation less than one order of magnitude (Figs. 2 6 Further, the center of this distribution was closer to the top of the spectrum, and thus provided a reasonable rate of change, while masking the existence of eigendimensions of extremely large and small eigenvalues.
Global Metric Structure Because the metric (z) describes local geometry. the next question is how it varies at different positions in the latent space. We computed the metric H (z) at randomly selected z and examined their similarity using statistic adopted from Kornblith al. (2019). In this statistic, we applied the eigenvectors U; = [ul,...un] from a metric tensor H1; at position z, to the metric tensor Hy Zj. ui fu These values formed vector Acj. representing the effects of metric H, on eigenvectors of H,. Then we computed the Pearson correlation coefficient between Aij and the target eigenvalues, Aj. as corr(A,.A1g) This correlation measured the similarity of the action of metric tensors on eigenframes around different positions. As the spectrum usually spanned several orders of magnitude, we computed the correlation on the log scale C.ff where the eigenvalues distribute more uniformly.
Aij diog(U, (I (2;)
(5)
- Ag), C.llo corr(log(Ass). (og())
(6)
Using this correlation statistic, we computed the consistency of the metric tensor across hundreds of positions within each GAN latent space. As shown in Fig. 3C, the average correlation between eigenvalues and vHv values of two points Offllgg was 0.934 in BigGAN, For DCGAN-type architec- ture. mean correlations on the log scale ranged from 0.92-0.99; for StyleGAN- 1,2, 0.64-0.73 in the space, and 0.87-0.89 in the W space (Fig. 3D, Tab[4 Overall, this shows that the local directions that induce image changes of different orders of magnitude are highly consistent at different points in the latent space. Because of this, the notion of "global" Hessian makes sense, and we estimated it for each latent space by averaging the Hessian matrices at different locations.
Implication of the Null Space As the spectra have large portion of small eigenvalues and the metric tensors are correlated in space, the bottom eigenvectors should create a global subspace, in which latent traversal will result in small even imperceptible changes in the image. This is supported by our perceptual study, as over half of the subjects cannot see any change in image when latent vector move in bottom eigenspace. (Sec. 6). This perceptually "null" space has implications about exploration in the GAN space and interpretable axes discovery. As G(z + v) R G(z). if one axis encodes an interpretable transform G(z) - G(z+uz then shifting this vector by a vector in the null space v will still result in an interpretable axis G(z) + G(z + v + u) x G(z + u). Thus, each interpretable axis have a family of "equivalent" axes which encode similar transforms, differing from each other by a vector in "null" space. However, adding component e in the null space will decrease the rate of image change along that axis. In this sense, the vectors using a smallest step size to achieve that transform should be the "purest" axes of the family. Further, the cosine angle between two interpretable axes may not represent the similarity of the transforms they encode. A large angle can be found between two axes of the same family but different image traversal speed. We compared the axes from previous works in A.9 and observed that projecting out a large part of their axes did not affect the semanties it encoded (Fig.
5

---

Published as a conference paper at ICLR 2021
A. Eigen Frame of Metric Tensor Har) at position.z,
Consistency In Latent Space
B.
Consistency of MagAN tonsor across positions
loglu log squared squaredd norm of oomnoo-eggennn elgenveccors at source,
30
4
375
2
810
-2
2
0
2
4
4
R
mesoured at target
30
pos 0566
2
shreeffect
Log Scale
2
4
20
2
Metric Comelation
BigEGGAN
BigGAN (256d)
log CDIT
DOGAN
dendity
0.8
SG2-Cat256 W
pos 638
N
10
SG Face
10
0.6
2
0
2
4
-2
0.8 0.9 10
02
0.4
0.6
08
1.0
po5
p5s
658
com(legg (HU). log(A))
linscale com
log(A): log eigenvalues @ttrget
Figure 3: Globalization of metric structure: A. Schematic of the geometric picture. In the latent space (green area), the metric eigen frames at each point (blue and violet) are mapped to transfor- mations in image space (blue area); the length and saturation of image-space vectors represent the eigenvalue (i.e. amplification factor) of G. We show that the top eigen space are relatively aligned at different positions. B. Distributions of log(Ai;) and log(A,), showing the action of metric are correlated 3 different points. C. Histogram of correlation Ciflog between all pairs among 1000 points in BigGAN space. D. Comparison of correlation values on linear and log scales for different GAN models. DCGAN-type (blue), Z. and W space for StyleGAN1,2 (red and orange).
MECHANISM
Above, we showed an intriguingly consistent geometric structure across multiple GANs. Next, we sought to understand how this structure emerged through network architecture and training.
To link the metric tensor to the generator architecture, is helpful to highlight the relationship between the metric tensor and Jacobian matrix H(z) = Je (Eq. 3). As the latent space gets warped and mapped onto image space, directions in latent spaces are scaled differently by the Jacobian of the map; specifically, directions that undergo the most amplification will become the top eigenvectors (Fig. 4A). As the Jacobian of the generator is a composition of Jacobians of each layer di,, the scaling effect on the image manifold is product of the scaling effects of each intermediate layer. We can analyze the scaling effect of different layers v/s by applying a set of vectors onto the metric tensors of these layers H.,- BigGAN, when we apply the eigenvectors of the first few layers onto the metric of other layers, the top eigenvectors are still strongly amplified by subsequent layers, thus forming the top eigendimensions of the manifold. Of note, this is not true for weight-shuffled control BigGAN: in that case, the top eigendimension of the first few layers was not particularly amplified on the image manifold, and vice versa (Fig. 4 B). This shows that the amplification effect of layers becomes more aligned through training, with the top eigenspace shared across layers. Further, as the amplification effects are not lined up across layers of weight shuffled networks, these networks should exhibit a more isotropic geometry on their image manifold. Indeed, we find their spectra to be flatter and the largest cigenvalue smaller (Fig.
A. day
fyyery:
fie (rik t-at)
Aapery.
B. 10
MgGAA
Weighe Shuflled
GerBood00
Po (retr))
GorBo300
GerBesd04
Can Gantiloice
Gor
Sattitontion
Gor CorBBo300
Disction Mertippus
Vedooranggh i
Gan
Irrege
0 Rank
50
100
150
200
Blod00)
Rark oworroooooooooooooo GorBBoodi
100
150
200
Figure 4: Anisotropy is induced and maintained throughout the GAN architecture. A. As latent space gets warped and mapped into image space, directions latent spaces are scaled differently by the Jacobian of the map. B. Amplification of eigenvectors of the metric tensor of the first conv layer (GenBlook0O) in all major layers in BigGAN, C. Same, but for weight-shuffled BigGAN

---

Published as a conference paper at ICLR 2021
6 APPLICATIONS
By defining the geometry of the latent space via the metric tensor, we gain an understanding of which directions in this space are more informative than others. This understanding leads to improvements in three applications: 1) finding numan-interpreed based optimizers. 3) accelerating gradient-fred free search.
axes in the latent space, 2) improving gradient-
Interpretable Axes Discovery When users wish manipulate generated images via their latent code, it is useful to reduce the number of variables needed to effectuate that manipulation. Our method provides a systematic way to compute the most informative axes (top eigenspace) in the latent space to use as variables, and the resulting eigenvalues can serve to compute appropriate step sizes along each corresponding axes. We visualized the image changes corresponding to the top eigenvectors in BigGAN, BigBiGAN, PGGAN, StyleGANI,2 [Fig We found many of these eigenvectors appeared to capture interpretable transformations like zooming, head direction and object position, consistent across reference image.
To test if this was apparent to people other than the authors, we conducted a study using Amazon's Mechanical Turk. We tested the perceptual properties of the axes identified by the metric tensor, including the top 10 eigenvectors, random vectors orthogonal to the top 15d eigenspace, and bottom 10 eigenvectors. Images were generated using four different GANs (PGGAN, BigGAN noise space, StyleGAN2- and -Face), and were presented to 185 participants. In each trial, five randomly sampled reference images were perturbed along a given axis, and participants were asked if they could a) perceive a change, b) indicate an estimate of its magnitude (0%-100%% c) describe a com- mon change in their own words and how many of the five images shared this change, c) indicate how similar were the 5 image changes (consistency, score of 1-9, 9 most similar) and finally, d) state how difficult was to describe this change (difficulty score, scale of 1-9, most difficult).
Only 48.5% of the subjects reported to see any change happen for bottom eigenvectors, while the fraction was 93.5% and 89.8% for top and orthogonal directions respectively. Further, when subjects observed some change, they reported that the image transformations induced by top eigenvectors were larger (70.3% 0.6%) than those of orthogonal directions (66.8% 0.9%, P 7.0 x 10-1, 2 sample T-test) and than those of bottom eigenvectors (61.5%~ + 1.6%. P = 2.1 x 10-19). This was true even though we picked a step size in the top eigenspace that was 5-10 times smaller than in the orthogonal and bottom eigenspaces. Further, subjects reported the top 10 eigenvectors had higher mean perceptual consistency score (6.72 + 0.06). n = 929 responses) than the orthogonal (6.42 + 0.09, P = 5.8 x 10-3 n 448 responses) and bottom eigenvectors (6.12 + 0.14, P 1.3 x 10 n = 242 responses). Participants reported that the top eigenvectors were easier to interpret (4.91 + 0.08) than the bottom eigenvectors (5.69+0..1 0.114 P 3.8 x 10-6, albeit comparably to the orthogonal eigenvectors 4.82 + 0.11, P = 0.5). Thus, overall we conclude that the Hessian eigenvectors not only capture informative axes of image transformations, but that these were also perceptually relevant, corresponding to similar semantic changes when applied to different reference vectors (Fig. axes interpretable not just in local sense, but in global sense.
Improving Gradient-Based GAN Inversion. For applications like GAN-assisted drawing and photo editing (Zhu et al., 2016 Shen et al. 2020), one crucial step is to find a latent code correspond- ing to a given natural image (termed GAN inversion). For this problem, one basic approach is to minimize the distance between a generated image and the target image z' = arg min, D(G(z), l). Although second-order information (Hessian) is valuable in optimization, they are seldom used as they are expensive to compute and update. However, since we find that the local Hessians are highly correlated across the latent space, we can pre-compute it once for each latent space and use the global average Hessian to boost first-order optimization. As an example, ADAM is first-order optimization algorithm that adapts the learning rate of each parameter separately according to the moments of gradients that parameter (Kingma & Ba 2014). It can be seen quasi-second order optimizer that approximates a diagonal Hessian matrix based on first-order information. How- ever, if the true Hessian is far from diagonal, i.e. the space is anisotropic and the valley is not aligned with the coordinates, then this approximation could work poorly.
To test whether the metric can help overcome this problem, used the eigenvectors of the global average Hessian to rotate the latent space; this orthogonal change of variables should make the Hessian more diagonal and thus accelerate ADAM. This method can be seen as precondition-
7

---

Published as a conference paper at ICLR 2021
C
BasinOMA BasintMAInverttigg BigGAN E
Image Inversion Score
StyleGAN2 PGGAN BigGAN
0.30
ImageNet
GAN Generated
Face
Target Image
LPIPS Image Disanne
LPIPS Imoge Distance
0.25
CdlebA
facing 3d face rotating space
FFHQ
age
slightly NONE
0.20
0.25
GAN
ImageNett
Little same NOrow background hairstyle
0.10
0.15
space direction NONE
0.05
Hess None
Hess None
Hess NoneH
N
H
N
F
CMA for Act. Maximization
BigGAN
FOSGAN
120
B
Hess
None
Target
Activatinnoonn
Frequency
0.2
bottom
d=0.441
d-0273
(Imagnet)
80
0.1
2
Hess
None
Target (BigGAN)
(500d) (9796)) Hess Orig
Difficulty score
d=0.094
d=0.143
Hess Orig
Figure 5: Applications of the metric tensor A. Perceptual properties of eigenvectors. Word cloud shows subjects' descriptions (N = 24) of the image transforms induced by the top- (red) vs. the bottom eigenvector (black) in StyleGAN2-
B. Distribution of difficulty scores associated with top- vs. bottom eigenveetrrs (red, black) across all four GANs for N=185 subjects. Lines show mean frequency + standard error (per bootstrap). C-E. Eigenbasis pre-conditioning improves GAN inversion. C. BasinCMA with eigenbasis pre-conditioning (Hess,H) outperformed a method using normal basis (None.N) in inverting ImageNet and BigGAN generated images: D. Examples of fitted ImageNet and BigGAN images with our Hessian BasinCMA and original method (LPIPS distance below). E. Results for PGGAN and StyleGAN2 inverting samples from CelebA and FFHQ F. Hessian-CMAES (Hess) outperforms CMAES (Orig) in maximizing CNN activation in BigGAN, and increases sampling efficiency FC6GAN latent space. Each line represents a different layer of optimized units in AlexNet. *** denotes paired t-test p < 1 x 10-6
ing step which could be inserted into any pipeline involving ADAM. We tested this modification on the state-of-the-art algorithm for inverting BigGAN, i.e. BasinCMA (Huh et al., 2020), which interleaves ADAM and CMAES steps. We used our Hessian eigenbasis in the ADAM steps, and found that we could consistently lower the fitted distance to the target when inverting ImageNet and BigGAN--enerrate images (Fig. 5). Similarly, eigenbasis preconditioning consistently improved inversion of PGGAN and StyleGAN2-Face for real image sampled from both FFHQ and CelebA using ADAM method. In short, the understanding of homogeneity and anisotropy of the latent space can improve gradient-based optimization.
Improving Gradient-Free Search in Image Space In some domains, it important to optimize objectives in the absence of a gradient, for example, in black-box attacks against image recogni- tion systems via adversarial images, when searching for sidivity-maxmmiiine stimuli for neurons in primate visual cortex, when optimizing perceptual evaluation in the user (Ponce et al.) 2019
Xiao & Kreiman 2020 Chiu et al. 2020). These applications usually involve a low-dimensional parameter space (such as GANs) and efficient gradient- free search algorithm, such covariance matrix adaptation evolution strategy (CMAES). CMAES explores the latent space using a Gaussian distribution and adapts the shape of the Gaussian (covariance matrix) according to the search history and local landscape. However, online learning of a covariance matrix in high-dimensional space is computationally costly, and inaccurate knowledge of it can be detrimental optimization. Here we applied the prior geometric knowledge of the space to build the covariance matrix instead of learning from scratch. For example, as illustrated by natural gradient descent (Amaril 1998), one simple heuristic for optimizing on the image manifold is to move in smaller steps along dimensions that change the image faster to avoid overshoot. We built in this heuristic to improve CMAES, termed CMAES-Hessian. With our method, the search can be limited to the most informative directions, which should increase sampling efficiency; further, our method tunes the exploration step size in a theeeeeplattii
way that is inversely proportional to the rate of image change. To test this approach, we applied our CMAES-Hessian algorithm to the problem of searching for activation maximizing stimuli for units in AeexNet (Nguyen et al. (016)) in the latent space of FC6GAN and BigGAN. We found that
8

---

Published as a conference paper at ICLR 2021
the dimension of the search space could be reduced from 4096 to 500 for FC6GAN without impair- ing maximal activation values. Further, we found that SMMAES-Hesiizg consistently led to higher activation values compared to the classic CMAES algorithm in BigGAN space (Fig. 5F).
DISCUSSION AND FUTURE DIRECTIONS
In this work, we developed an efficient and ceatitectuur-annottte way to compute the geometry of the manifold learnt by generative networks. This method discovers axes accounting for the largest variation in image transformation, which frequently represent semantically interpretable changes. Subsequently, this geometric method can facilitate image manipulation, increase explainability, and accelerate optimization on the manifold (with or without gradients).
There have been multiple efforts directed at identifying interpretable axes in latent space using un- supervised methods, including (Ramesh et al. 2018 Härkööen et al. 2020; Shen & Zhou 2020 Voynov & Babenko 2020; |Peebles et al.J 2020). Our description of the connection between the metric tensor of the image manifold and the Jacobian matrices of intermediate layers unifies these previous results. As we have showed, the top right singular vectors of the weights (i.e. Jacobian) of the first few layers (as used in [Shen & Zhou (0200), correspond to the top eigenvectors of the metric tensor of the image manifold, and these usually relate to interpretable transforms. Similarly, the top principal components (PCs) of intermediate layer activations Härkönen et al. (2020) roughly correspond the top left singular vectors of the Jacobian, thus also to the interpretable top eigenvec- tors of the metric on the image manifold. Likewise, Ramesh et al. (2018) also observed that the top right singular vectors of the Jacobian of G are locally disentangled. Regarding Voynov & Babenko (020) and Peebles et al. (0200), we empirically compared their interpretable axes and our eigenvec- tors, and found that in some of the GANs, the discovered axes have a significantly larger alignment with our top eigenspace and they are highly concentrated on individual top axes than expected from random mixing. We refer readers to Sec A.9 and Fig & for further comparison.
Although have answered how the anisotropy comes into being mechanistically, there remains the question of why it should exist at all. Anisotropy may result from gradient training: theoretical findings deep-linear networks for classification show that gradient descent aligns the weights of layers, resulting in a highly anisotropic Jacobian (Ji & Telgarsky] 2018).. Whether that analysis transfers the setting of generative networks remains to be investigated.
Alternatively, assuming that a well-trained GAN faithfully represents the data distribution, this anisotropy may reveal the intrinsie dimensionality of the data manifold. Statistical dependencies of variation in real-world images imply that the images reside in a statistical manifold of much lower dimension. Further, among transformations that happen on this manifold, there will be some dimensions that transform images a lot and some that do not. In that sense, our method may be equivalent to performing a type of nonlinear PCA of the image space through the generator map. In fact, we have found that GANs trained on similar datasets (e.g. PGGAN, StyleGAN1.2 1,2 trained on the human face dataset CelebA.FFHQ) have top eigenvectors that represent the same set of trans- forms (e.g. head direction, gender, age; Fig. 10).. This supports the "PCA" hypothesis, as these transformations may account for much of the pixelwise variability in face space; the GANs are able to learn to represent these transformations linear directions, which our method can then identify.
This further raises the intriguing possibility that if the dataset is actually distributed on a lower dimensional space, one could learn generators with smaller latent spaces; or alternatively, it may be easier to learn generators with large latent spaces and reduce them after intensive training. These questions worth exploring.
ACKNOWLEDGMENTS
appreciate the conceptual and technical inspirations from Timothy Holy (WUSTL). We are grateful for the constructive discussion with Zhengdao Chen (NYU), whose pointers to the relevant literature helped launch this work. We thank Hao Sun (CUHK) in providing experience for the submission and rebuttal process. We thank friends and colleagues Yunyi Shen (W--Madioon,, Lingwei Kong (ASU) and Yuxiu Shao (ENS) who read and commented on our early manuscript. This work was supported by Washington University in St. Louis, the David and Lucille Packard Foundation, the McDonnell Center for Systems Neuroscience (pre- doctoral fellowship to B. J.). and a seed grant from the Center for Brains, Minds and Machines (CBMM; funded by NSF STC award 1231216.....

---

Published as a conference paper at ICLR 2021
AUTHOR CONTRIBUTIONS
B.W. conceptualized the project, designed the algorithm, developed the code base, performed the numerical experiment and analyzed the data. B. W. and C.R.P. interpreted the results. B.W. and C.R.P. designed the human MTurk task and analyzed the data, B. W. and C.R.P. prepared revised the manuscript.
REFERENCES
Shun-Ichi Amari. Natural gradient works efficiently in learning. Neural computation, 10(2):251- 276, 1998.
Shun-ichi Amari. Information geometry and its applications, volume 194. Springer, 2016.
Georgios Arvanitidis, Lars Kai Hansen, and Søren Hauberg. Latent space oddity: on the curvature of deep generative models. arXiv preprint arXiv: 1710. 1379, 2017.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. arXiv preprint arXiv: 1096, 2018.
Nutan Chen, Alexej Klushyn, Richard Kurle, Xueyan Jiang, Justin Bayer, and Patrick Smagt. Met- rics for deep generative models. In International Conference on Artificial Intelligence and Statis- fics, pp. 1550. PMLR, 2018.
Chia-Hsing Chiu, Yuki Koyama, Yu-Chi Lai, Takeo Igarashi, and Yonghao Yue. Human-in-the- loop differential subspace search in high-dimensional latent space. ACM Trans. Graph., (4)4), July 2020. ISSN 0730-0301. doi: 10. 10.1145/3386669933924499 /ttp::tss////o.orr
1145/338656 3392409
Jeff Donahue and Karen Simonyan. Large scale adversarial representation learning. In Advances in Neural Information Processing Systems, pp. 10542-10552, 2019.
Alexey Dosovitskiy and Thomas Brox. Generating images with perceptual similarity metrics based on deep networks. In Advances in neural information processing systems, pp. 358-666, 2016.
Behrooz Ghorbani, Shankar Krishnan, and Ying Xiao. An investigation into neural net optimization via hessian eigenvalue density. In International Conference on Muchine Learning, pp. 2232-
2241,2019)
Erik Härkönen, Aaron Hertzmann, Jaakko Lehtinen, and Sylvain Paris. Ganspace: Discovering interpretable gan controls. arXiv preprint wxXiv:200.022466, 2020.
Minyoung Huh, Richard Zhang, Jun- Yan Zhu, Sylvain Paris, and Aaron Hertzmann. Trans- forming and projecting images into class-conditional generative networks. arXiv preprint arXiv:2005.01 703, 2020.
Ziwei Ji and Matus Telgarsky. Gradient descent aligns the layers of deep linear networks. arXiv preprint 1XX:11810.02032. 2018
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for im- proved quality, stability, and variation. arXiv preprint arXiv: 2017.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4401-4410. 2019.
Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyz- ing and improving the image quality stylegan. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8110-8119, 2020.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 1rXiv:14/2.69888 2014.
Simon Komblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. arXiv preprint arXiv: 1905.0900 2019.
10

---

Published as a conference paper at ICLR 2021
Richard B Lehoueq. Danny C Sorensen, and Chao Yang. ARPACK users' guide: solution of large- scale eigenvalue problems with implicitly restarted Arnoldi methods. SIAM, 1998.
Anh Nguyen, Alexey Dosovitskiy, Jason Yosinski, Thomas Brox, and Jeff Clune. Synthesizing the preferred inputs for neurons in neural networks via deep generator networks. In Advances in neural information processing systems, pp. 3387-3395, 2016.
Richard S Palais. On the differentiability of isometries. Proceedings of the American Mathematical Society, 8(4):805-807, 1957.
William Peebles, John Peebles, Jun- Yan Zhu, Alexei Efros, and Antonio Torralba. The hessian penalty: A weak prior for unsupervised disentanglement arXiv preprint arXiv:2008. 10599, 2020.
Marcos Pividori, Guillermo L Grinblat, and Lucas C Uzal. Exploiting gan internal capacity for high-quality reconstruction of natural images. arXiv preprint arXiv: 1at.:19111..000 2019.
Carlos R Ponce, Will Xiao, Peter F Schade, Till S Hartmann, Gabriel Kreiman, and Margaret Livingstone. Evolving images for visual neurons using a deep generative network reveals coding principles and neuronal preferences. Cell, 177(4):999-1009 1009,, 2019.
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv:1511.06434, 2015.
Aditya Ramesh, Youngduck Choi, and Yann LeCun. spectral regularizer for unsupervised disen- tanglement. arXiv preprint
2018.
Hang Shao, Abhishek Kumar, and P Thomas Fletcher. The riemannian geometry of deep generative models. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition Workshops, pp. 315-323, 2018.
Yujun Shen and Bolei Zhou. Closed-form factorization of latent semantics in gans, arXiv preprint arXiv:2007.06600 2020.
Yujun Shen, Jinjin Gu, Xiaoou Tang, and Bolei Zhou. Interpreting the latent space of gans for semantic face editing. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 9243-9252, 2020.
A Van der Sluis and HA Van der Vorst. The convergence behavior of ritz values in the presence of close eigenvalues. Linear Algebra and its Applications, 88:651-694, 1987.
Andrey Voynov and Artem Babenko. Unsupervised discovery of interpretable directions in the gan latent space. arXiv preprint arXiv:2002.03754, 2020.
Tom White. Sampling generative networks. arXiv preprint 1609:::609.04400 2016.
Will Xiao and Gabriel Kreiman. Xdream: Finding preferred stimuli for visual neurons using gen- erative networks and gradient-free optimization. PLoS computational biology, 160)))71077773,
2020.
Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. Lsun: Construction of a large-scale image dataset using deep learning with humans in the loop. arXiv preprint arXiv: 15t..03366.066. 2015.
Xinru Yuan, Wen Huang, Fierre-Antoine Absil, and Kyle A Gallivan. Averaging symmetric positive- definite matrices. In Handbook of Variational Methods for Nonlinear Geometric Data, pp. 555- 575. Springer, 2020.
Richard Zhang, Phillip Isola, Alexei Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 586-595, 2018.
Jun- Yan Zhu, Philipp Krähenbühl, Eli Shechtman, and Alexei A Efros. Generative visual manipu- lation the natural image manifold. In European conference computer vision, pp. 597-613 Springer, 2016.
11

---

Published as a conference paper at ICLR 2021
A APPENDIX
A.1 CONNECTION TO INFORMATION GEOMETRY
It is useful to compare our work to the "information geometry" "(Amari 2016) on the space of distri- butions. In that formulation, KI. divergence is a pseudo-metre function on the space of distributions, and its Hessian matrix towards parameters of distribution is the Fisher information matrix. In infor- mation geometry, this Fisher information matrix could be considered as the metric on the manifold of distributions; this metric information can be further used to assist optimization distributions, termed natural gradient descent (Amari) 1998). In our formulation, the squared image difference function D is analogous to this KL-divergence; the image G(z) as parameterized by latent code z is analogous to the distribution py parameterized by 8. The metric tensor we com- puted is comparable the Fisher information matrix in their setting. Thus our way of using metric information to assist optimization on manifold is analogous to natural gradient descent.
the manifold of
A.2 METHODS FOR COMPUTING THE HESSIAN
One direct way to compute the Hessian of a given scalar function (e.g. squared distance df in case), is to compute 9z. (z) 02d x== create computational graph from code to the gradient 9zo (z), and back propagate from gradient vector Izo (z) element by element. In this way the computational time is linear time of single backward pass times the latent space dimension n.
Given large latent space deep network (e.g. 4096 dimensions in FC6GAN, or 512 in Style- GAN2), this method can be very slow. An efficient way is to use the Hessian vector product (HVP) operator and iterative eigenvalue algorithms like power iteration Lanesoz iterations solve the cigenvectors corresponding to eigenvalues of largest amplitudes. These largest eigen pairs create the best low rank approximation the real Hessian matrix. Note that, find the smallest amplitude eigen pairs, inverse Hessian vector product operator is required, which is much more expensive to compute. However, as the eigenspace with the smallest eigenvalues represent directions that do not change images much, the exact eigenvector does not matter. We can just define an arbitrary basis in the "null" space orthogonal to the eigenspace with large amplitude eigenvalues.
There are two ways to construct HVP operator: one way uses the 2nd order computational graph from z to the gradient g(z) to compute HVP by back-prop. i.e. HV Phackward; the other way uses finite difference on the first-order gradient to compute HVP i.e. HV Pforward. As it does not require backpropagation, a single operation of HV HV Pforward is faster than HV Pbockward but is less accurate and takes more iterations to converge. We use the ARPACK (Lehoucq et al. 1998) implementation of the Lanczos algorithm as our iterative eigenvalue solver.
HV Pbockwardd' U - O2 (x g(z)) (xFFg(z))
g(2 + Ev) - g(z -ew)
HV P Forward
2c||w/
We termed the direct method Full BackProp (BP), the iterative method using HV Poackward and HV forward Backward Iteration and Forward Iteration respectively. We computed the Hessian the same 20 using these three methods in different GANs and compared their temporal cost empirically in Table A.2
Note, our method can be employed to compute the singular values and right singular vectors of the Jacobian from latent space towards any intermediate layer representation. To obtain the left singular vector, iee the change in representation or image space caused by the direction, we need to push forward the right singular vectors through the Jacobian, which is feasible through forward-mode autodiff or finite difference.
A.3 SPECIFICATION OF GAN LATENT SPACE
The pretrained GANs used in the paper are from the following sources:
12

---

Published as a conference paper at ICLR 2021
Table 1: Computational Cost for Three Methods: Computation time measured on a GTX 1060 GPU. The iterative method has a variable runtime which depends on the number of eigenpairs required. In this table, we all use one half of the full dimension as the eigenpaired required, which the largest we can require using ARPACK implementation of Lanczos. Thus these numbers should be seen as an upper limit of time for Backward and Forward Iteration method. A shallower or narrower network will result in faster computation time. For StyleGAN2 which has configurable depth and width, we use the config-f. Dimension
Full BackProp
Backward Iter
Forward Iter Time
Time
Time
DCGAN
120
12.5
13.4
6.9
FC6 GAN
4096
282.4
101.2
90.2
BigGAN
256
69.4
70.6
67
BigBiGAN
120
15.3
15.2
13.3
PGGAN
512
95.4
95.7
61.2
StyleGAN
512
112.8
110.5
64.5
StyleGAN2*
512
221
217
149
DCGAN model was obtained from torch hub
bttps://pytorch.. org.hbh// Facebookresearch wwthh/pytttohghhheeeeeeee zoo _dd_an/c/nn] It's trained on 64 by 64 pixel fashion dataset. has a 120d latent space, using Gaussian as latent space distribution.
Progressive Growing GAN (PGGAN) was obtained from torch hub ptctps://pytoroos org/ hub/Faccbookreseazzeee otw.raooooreeeeeep..oooorrreraaa 200 200 Batll/pgaaa///// and we use the 256 pixel version. It's trained on celebrity faces dataset (CelebA). It has a 512d latent space, using Gaussian as latent space distribution.
DeePSim, FC6GAN model was re-written and translated into Pytorch, with weights ob- tained from official page https:://mmt in inforratiii uni
de/people/ desovits code html of Dosovitskiy & Brox (2016). The architecture is designed to mirror that AlexNet, and the FC6GAN model is trained to invert AlexNet's mapping from image to FC6 layer. Thus it has 4096d latent space. This model is highly expressive in fitting arbitrary pattern, but not particularly photorealistic.
BigGAN model was obtained through Hugging Face's translation of DeepMind's Tensorflow imple- mentation htt : / thub. ctt.ccm//////oe wwwuuggigggaann ngfaae/ www.ggggggggaccettgoaeeeeee httpa///yytoooo.peeeraieeeeeeeee gGAN, we version. It's trained ImageNet dataset in a class conditional way. It has 128d latent space called noise space, and a 188d embedding space for the 1000 classes called class space. The 2 vectors are concatenated and sent into the network. The distribution used to sample in noise space is truncated normal. Here we analyze the metric tensor computed in the concatenated 256d space (BigGAN) in the 128d noise space or class space separately (BigGAN-noiss class).
biggan-deep-
BigBiGAN model was obtained via translation of DeepMind's Tensorflow implementation https://tfhie wee.::eevvveeeeeeeeee http////pwm.a/ippiiiaaaaaaaaaaae
we use bigbigan-resnette version. It's trained on ImageNet dataset in unconditioned fashion. It has 120d latent space, using Gaussian as latent distribution. Note, the latent vector is split into six 20d trunks and sent into different parts of the model, which explains why the spectrum of BigBiGAN has the staircase form (in Fig. 2).
StyleGAN model was obtained via a translation of NVIDIA's Tensorflow implementation https: 7/github. com/ rosinal wawiinntttinno wataainnttytsyylleebbbaat baw.ylll--bedde-----aeaaa-
We used the 256 pixel out- put. It has a 512d latent space called Z space, where the latent distribution is Gaussian distribution. This Z distribution gets warped into another 512d latent space called W space, by a multi-layer perceptron. The latent vector W is sent into style-based generative network, in which the latent vector just modulates the feature maps in the conv layers, instead of serving as a spatial input as in DCGAN, FC6GAN, PGGAN.
StyleGAN2 models are obtained via a translation of NVIDIA's Tensorflow implementation https: / / thub. com/ ros i styaallttyyyytyyyennppp
This is improved version of StyleGAN: it also has a network mapping the 512d Z to the W space. and the style-based gen- erative network. The various pre-trained models are fetched from httpt:tt://// https:/pyythonawessnt
13

---

Published as a conference paper at ICLR 2021
Table Quantification of Spectra anisotropy (Models marked with are audio wave form gener- ating GANs using different distance metric function.)
dimen
dim.99
dim.999
dim.9999
dim.99999
FC6
4096
297
502
661
848
OCGAN-fashion
120
17
35
65
97
BigGAN
256
10
53
149
224
BigGAN _noise
128
29
88
120
127
BigGAN Lelass _class
128
38
98
123
BigBiGAN
120
21
41
62
73
PGGAN-face
512
57
167
325
450
StyleGAN--fce Z
512
27
52
StyleGAN2- 12.Z
512
7
17
41
78
MaleGAAN2-face2551 Z
512
28
63
103
StyleGAN2-caa Z
512
8
14
31
62
StyleGAN-face..
512
124
355
480
507
StyleGAAN2-face5112 12 W
512
153
345
471
506
StyleGAN2-face255
W
512
157
350
473
506
StyleGAN2-catt, W
512
23
57
126
269
WaveGAN MSEi
100
17
38
74
94
WaveGAN.STFTT
100
9
19
42
More specifically StyleGANN- and 512 both trained on FFHQ dataset, while Face256
generate lower resolution images and use narrower conv layers. StyleGANN-Cat is trained on I.SUN cat dataset (Yu al. 2013) at. 512 resolution.
WaveGAN model is
obtained from the repository https://githii com/ Its architecture resembles that DCGAN,
mostafaelarase beeaby/wavegar. saavegan-pytorchh but applied to the one dimensional wave form generation problem. We customly trained it on the wave forms of piano performance clips. has 100d latent space, using Gaussian as latent space distribution.
A.4 QUANTIFICATION OF POWER DISTRIBUTION IN SPECTRA
We quantified the anisotropy of the space, i.c. the low rankness of the metric tensor in Table To do this, we computed the number eigenvalues needed to account for the 0.99, 0.999, 0.9999, 0.99999 fraction of the sum of all eigenvalues. This can be thought of as the minimal number of dimensions needed achieve a low rank approximation of the Jacobian with 0.01, 0.001, 0.0001, 0.00001 residue in terms of the Frobenius norm.
There are few interesting patterns we noticed in this table. For BigGAN, we noted that the class subspace is more low-ranked than the noise subspace, i.e. fewer directions could account for most of the changes across object classes than within classes. For StyleGAN and 2, we analyzed the geometry of Z space and W space separately, and found that in all the models the metric in W space is significantly more isotropic i.e. less low rank than Z space. Thus, in this regard, the mapping warped the spherical distribution in Z space an elongated one in W space, but the mapping from W space to image is still more isotropic.
A.5 GEOMETRIC STRUCTURE ROBUST TO THE IMAGE DISTANCE METRIC
Our work used the LPIPS distance metric to compute the Riemannian metric tensor. To determine how much of the results depended on this choice of metric, we computed the metric tensor at the same hidden vector using different image distance functions, specifically a) structural similarity in- dex measure (SSIM) and b) Mean Squared Error (MSE) in pixel space, which do not depend on CNN. We computed the Hessian at 100 random sampled vectors in BigGAN, Progressive Growing GAN (Face), StyleGAN2 (Face 256), using MSE, SSIM and LPIPS, and then compared their Hes- sian spectra and cigenvectors. We found that the entry-wise correlation across the Hessian matrices (d2 elements) ranged from [0.94-0.99]. The correlation of eigenvalue spectra ranged from (0.987-
14

---

Published as a conference paper at ICLR 2021
Table 3: Comparison of Hessian Computed with Different Sample Dissimilarity Metric We experimented with BigGAN, PGGAN and StyleGAN2 (FFHQ 256 resolution), we compared the Hessian computed by LPIPS and that by MSE or SSIM 100 latent vectors. The statistics we showed are: element-wise Hessian correlation (H corr). eigen spectra correlation (eigval corr), the Hessian consistency measure Cf and Collog, The linear regression between the log spectra of LPIPS the and that of the alternative (SSIM or MSE) yields the slope (slope) and intercept (inter- cept). The mean and standard deviation (in parenthesis) of the the 100 statisties are shown. In the last row, WaveGANf is audio generating GAN. We measured the similarity of the Hessian us- ing MSE of wave forms (MSE) and MSE of spectrogram (STFT) as dissimilarity metric. Hessian computed using these two measures less similar to each other.
H corr
eigval corr Cillinn
Co
slope
intercept
BigGAN
MSE
0.973(0.033)
0,995(0.008)
0.999(0.001) 0.996(0.014)
1.01(0.03)
1.47(0.22)
SSIM
0.988(0.012)
0.997(0.003)
0.999(0.002)
0.999(0.001)
1.06(0.02)
-0.40(0.07)
PGGAN
MSE
0.938(0.047) 0.987(0.016)
0.987(0.038)
0.999(0.000)
1.02(0.02)
1.75(0.19)
SSIM
0.970(0.020) 0.993(0.009)
0.997(0.007)
0.999(0.000)
1.06(0.01)
-0.23(0.11)
StyleGAN2
MSE
0.945(0.035)
0.989(0011))
0,990(0.020)
0.987(0.011)
1.13(0.16)
1.32(0.34)
SSIM
0,945(0.047)
0.987(0.015)
0.988(0.027)
0.991(0.008)
1.08(0.18)
-0.17(0.26)
WaveGANf
STFT
0.529(0.229)
0.892(0.088)
0.938(0.062) 0.661(0.313)
1.09(0.05)
4.67(0.30)
0.995J. Measuring Hessian similarity using the statistics we developed Holo and CHilinn resulted in correlations concentrated at 0.99. Thus, we found that the Hessian matrices and their spectra were highly correlated across image distance metrics, and that the Hessian matrices had similar effect on the eigenvectors of each other.
One major difference across Hessians from different image distance metric was evident in the scale of the eigenvalues. We regressed the log Hessian spectra induced by SSIM or MSE onto the log Hessian spectrum induced by LPIPS, and found the intercepts of the regression were usually not ze- ros (Tab. 5). This result showed different image distance metrics exhibit different "unit" or intrinsic scale, although they all factored out the same structure in the GAN.
This result is contextualized by Section As equation showed, the Riemannian metric or Hessian of the generator manifold is the inner product matrix of the Jacobian of the representational map. The effect of image distance metric on the Riemannian metric to add a few more terms on top of the chain of Jacobians. The Jacobian terms from the layers of generator seem to have larger effect than the final terms coming from the image distance metric.
Note that this does not mean that the choice of sample space distance function is irrelevant. Going beyond image generation, when applying our method to an audio generating GAN, the WaveGAN, we found that the choice of distance function in the space of sound waves will substantially affect the Hessian obtained. We used the MSE of wave forms and MSE of spectrograms (denoted by STFT) to compute metric tensor of that sound wave manifold. We found the element-wise Hessian correlation between these is around 0.53, while the other Hessian similarity metric are also lower than the counterparts for BigGAN, PGGAN and StyleGAN2 (Tab. 5.. We think the MSE of spectrograms is more perceptually relevant distance metric of sound waves than MSE of wave forms, and this difference reflected in the geometry they induced i.e. anisotropy and homogeneity (Tab. 2 Thus, when and how the sample space distance metric will affect the geometry of generative model still requires more development to be answered.
A.6 RANDOM MIXING OF SPECTRA
Here we give a simple derivation of why a highly ill-conditioned Hessian matrix may appear normal, under the probe of random vectors. Given symmetric matrix H. and its eigen decomposition UAUT, we computed its effect on an isotropic random vector u, a(v) = v" Ho/e" u v. and v ~ (0, f). This random variable represents the effect of the symmetric matrix on random directions.
15

---

Published as a conference paper at ICLR 2021
Note that change of variable using the orthogonal matrix w U will not change the distribution w ~ I). Through this the random variable a could be rewritten as
x² 11
a(v)
X
~ 100,
(9)
Aow??
Σ
wis Wi ~ NO 1) i= 1, 2,2...
(10)
E Xmu Ead cds
(11)
Σ, u?
4 == w//
" 2
(12)
As each element in w is distributed as i..d unit normal, w is distributed as i..d. chi-square distribution of parameter 1. ~ ((0)~T(() x))x2())~TX) (()))) Thus the normalized weights c; wf/ > wf
conform to Dirichlet distribution Dirichlet( 3.....)) Through moment formula of Dirichlet distribution, it is straightforward compute the mean and variance of a(w) = c² X
E[a] =x F|c]
n A
1
Σ
(13)
Varla] =x² Coo(clA
2
Σ A
2
1132
(14)
n(n + 2)
n²(n + 2)
Var[N] -
Σ
x
1 n Σ
x=)
n n
Varla]
(15)
2
As we can see, the variance of the effect on random directions scales 1/n relative to the variance of original cigenvalue distribution. This is why the distribution is much tighter than the whole eigenvalue distribution.
This phenomenon may explain why the perceptual path length regularization (PPL) used in Karras et al. (2020) doesn't really result in a flat spectrum. In our notation, the PPL regularization min- imizes Eg || (v) - b1l2, which is to minimize the variance of the distribution of a(v) with v sampled from normal distribution and z sampled from latent distribution. The global minimizer for this regularization is indeed a mapping G with flat spectrum, i.e. an isometry up to some scaling. However, we can through our derivation and Fig that even for highly anisotropic spectrum, this variance will not be very large. Thus we should expect a limited effect of this regularization.
A.7 METHOD TO QUANTIFY METRIC SIMILARITY
We developed our own statistic to quantify the similarity of metric tensor between different points. Here we discuss the benefits and caveats of it.
Angles between eigenvectors per se are not used, since eigenvectors with close-by eigenvalues are likely to rotate into each other when computing eigendecomposition (Van der Sluis & Van der Vorst, 1987). However, the statistics should be invariant to this eigenvector mixing, and also take the eigenvalues into account. In our statistics, we applied the eigenvectors U1= [[,,,;...] of one matrix H1 to the other H2, i.c. examined the length of these vectors as measured by the other matrix as the metrie tensor, an,, (un,i uf Hzu, Recall that a (u1,)) uf, uf,, Hiu aH (u2.:) = ui,, Hzug, = A2 the eigenvectors fall the eigenspace of the same eigenvalue in H2, then a(u),)) will equal the eigenvalue, and thus our statistic is invariant to rotation within the cigenspace. If the eigenvectors are totally uncorrelated, the resulting a(u1,) will distribute like that of random vectors as in 6 As we compute the correlation between the eigenvalue 22. aH2 (uz,;) and the aH, ( summarize the similarity of action of H2 on eigenvectors Un and U2.
A1, and
However, this method assumes an anisotropy of spectra in both metric tensors. For example, if both tensors are identity matrices H, = H2 = I, then this correlation will yield NaN, as there is no variation in the spectra to be correlated. Similarly, if the metric tensor has more isotropic spectrum,
16

---

Published as a conference paper at ICLR 2021
Theory B Emppr16e-1136eee
Apparent and Real Spectral Anisotropy Comparison
Frequencyeessii
10-
DCGAN
3-
FC&GAN
Mean (std) of ampllfication for theory and empirical simulation note in title
2-
eigen value (Amplification of eigenvector)
Amplification of random vector
0.0
-12
-2
Theory 1112)/8411 Empir 1empi1111200 1.128 Se1)
Thesry 39,0/23300) Empir 3.e0022e0)
Emm 2/:2/22222
2.0
BigGAN
1.5
BigBiGAN
15-
PGGAN
08-
Proquency
10
10-
0.4
0.0
0
-4
0.0
-2
31e 1266e Empir 1(2.ee1
Theory 1.e00755 Empir 11s0(7.6e-1
Theory 13:e-(400ee1) Empii 34.00e))
StyleGAN-Face var
StyleGAN-Face Wepace
StyleGAN-Face 2space
Frequency Dersily
08
2-
0.4
-1
0
-8
-6
-4
-2
0
2
Theory 3.4e-1229e-11
34-11/99-1-)
Theory 9.6e 1/4.9e Empir 9.0 115.1e
Theory 4.6 1/40e Empir
4.6e 1(40c
9yyeGAN2-Facc512,
2.0
StyleGAN2-Face5122555 Wepace
1.0
StyleGAN2-acce5222555 Zspace
Frequency Derely
15
0.8
06
10.
05
logibtelgual)
00
-3
-2
loga0keigva))
2
0.0
-6
loglOfsgial)
-2
Figure 6: Spectral Histogram compared to Apparent Anistropy in Different GAN models (FC6GAN, DCGAN, BigGAN, BigBiGAN, PGGAN, StyleGAN, StyleGAN2) The apparent speed of image change (<v)) has much smaller variability than the variability in the whole spectra. Eq. 13 can predict the mean and std the apparent variability.
then it will generally have a smaller correlation with others. In that sense, spectral anisotropy also plays a role in our statisties for metric similarity or homogeneity of the manifold. In all the GAN spaces we examined, there is a strong anisotropy in the metric spectra, thus this correlation works fine. But there is caveat for comparing this correlation between two GANs when there is also difference in the anisotropy in their spectra, as smaller anisotropy can also results in a smaller metric similarity.
Finally, we are aware that there are different ways to average symmetric positive definite matrices (SPSD), induced by different measures of distance in the space SPSD Yuan et al., 2020). Here we picked the simplest one to estimate the global Hessian in the latent space: averaging the metric tensors element by element.
A.8 GEOMETRIC STRUCTURE OF WRIGHT-SHUFFLED GANS
Here we show the geometric analysis for the shuffled controls for all our GANs. Specifically, we shuffled the elements of the weight tensor from each layer keep the overall weight distribution
17

---

Published as a conference paper at ICLR 2021
Table 4: Quantification of Manifold Homogeneity by metric conisistency C on log scale and linear scale. Same data generating Fig. D. Models marked with i audio wave form generating GANs.
Log scale
Linear Scale
mean
std
mean
std
FC6GAN
0.984
0.002
0.600
0.119
DCGAN
0.920
0.028
0.756
0.192
BigGAN
0.934
0.024
0.658
0.186
BigBiGAN
0.986
0.007
0.645
0.180
PGGAN
0.928
0.014
0.861
0.123
StyleGAN-Face Z
0.638
0.069
0.376
0.160
StyleGAN2-Facee Face5 12
0.616
0.052
0.769
0.181
ByleGAAN-Facce2..
0.732
0.037
0.802
0.130
StyleGAN2-Ca1256 Z
0.700
0.040
0.689
0.151
StyleGAN-Faace W
0.878
0.037
0.780
0.190
StyleGAN2-Faac5522 W
0.891
0.048
0.838
0.127
StyleGGAN2-Face256... W
0.869
0.052
0.756
0.159
StyleGANN-Ca1256..W
0.895
0.118
0.539
0.216
WaveGAN_MSEF
0.906
0.022
0.776
0.139
WaveGAN.STFTY
0.809
0.096
0.467
0.285
Table 5: Hessian preconditioning improves GAN inversion The mean and standard error of fitting score (minimum LPIPS distance to target using random initial vectors) are presented the table, which is the same data generating Fig. 3 For each GAN, target dataset pair, 200-300 different target images are used.
Hessian
None
GAN
Target Image
Mean
SEM
Mean
SEM
StyleGANT024
CelebA
0.334
0.002
0.344
0.002
StyleGAN1024
FFHQ
0.231
0.003
0.236
0.003
StyleGANN024
GANgen
0.055
0.002
0.060
0.003
POGAN
CelebA
0.179
0.003
0.207
0.003
PGGAN
FFHQ
0.175
0.003
0.197
0.003
PGGAN
GANgen
0,045
0,003
0,027
0.001
BigGAN
ImageNet
0.162
0.003
0.189
0.003
BigGAN
GANgen 0.144
0.003
0.195
0.003
unchanged. To show how learning affected the geometry of the image manifold, we computed the spectra and the associated metric consistency statistic for weight-shuffled GAN
In Fig. 7, we showed that the shuffled controls exhibited flatter spectra and smaller top eigenvalues.
There, the correlation of metric tensors in shuffled GANs shows an unclear result. In some GANs, there remains strong correlations
the metrics across locations, while in some, the correlation is close to zero. We think the reason is that our statistic for homogeneity (i.e. correlation of action of metric tensors Cifloo) somewhat entangles homogeneity with the anisotropy of the space. That is, when the space has a totally flat spectrum (the map isometric), then the correlation coefficient of action will be zero nan, although the metric tensor will be the same everywhere. Thus the change of anisotropy and the change of homogeneity may interfere with each other, thus shuffling can result in mixed result. We working to develop statistics that will measure the similarity of the Hessian, invariant of anisotropy.
We were unable to obtain sensible spectra for either shuffled or randomly initialized BigBiGAN possibly due its architecture; hut show the comparison for all other models.
18

---

Published as a conference paper at ICLR 2021
DCGAN
FO6GAN
BigGAN
logleigenvalue
-4
-8
-2
-8
-12
40
80
120
1000
2000
3000
4000
50
100
150
200
250
2
StyleGAN Face256 W
2-
StyleGAN2 Face512 W
0
PGGAN Face256
logielgennvalue
0.
-2
-2
-8
-6
200
400
200
400
0
200
400
StyleGANFace2562
StyleGAN2 Face512 Z
0
logleigenvaluee
Spectra Comparison between
GANs and Controls
Original GAN Weight Shuffled GAN
8
-8.
200 Rank eigval
400
200
400
Rank of eigval
Comparing Spectra of Original and Weight Shuffled GANs; Most Shuffled GAN shows a slower spectral decay and a smaller maximum eigenvalue.
Figure 7:
A.9 DETAILED COMPARISON TO PREVIOUS UNSUPERVISED WAYS TO DISCOVER INTERPRETABLE AXES
Here we compare the axes discovered by our method with those from a previous approach. Specif- ically, we applied our method to the same pre-trained GANs used in Voynov & Babenko (2020).
comparing the axes they discovered versus our Hessian structure. Although this method Tollows
quite different approach compared to ours and those of (Härkönen et al. 2020 Shen & Zhou 2020
we thought it would be interesting to determine the interpretable axes discovered in their approach had a relationship with the Hessian structure defined above. If so, this could serve as independent confirmation of the effectiveness of both types of approaches.
In their work, for each generative network G, two additional models were simultaneously trained to discover interpretable axes: a "deformator" M and a "latent shift predictor" P. The "defor- mator" M learned to propose vectors {v:} to alter the image, which were used to create im- ages pairs (G(z), G(2 + vz)) using random reference vectors 2; the "latent shift predictor"' p took in the images pairs and learned to classify the direction in which the latent code shifted é = P(G(2), G(z +.)). The axes learned by the deformator M were subsequently annotated and subset was selected by humans.
19

---

Published as a conference paper at ICLR 2021
Using their code, we compared these annotated axes with the Hessian structure computed on their GANs (PGGAN512, BigGAN noise and StyleGAN Face). In PGGAN512, we found that their discovered axes had significantly larger e Hu (i.e. approximate rate of image change) than random vectors in that space; in other words, their axes were significantly more aligned with the top eigen space (P < 0.05 for all axes). Further, we wanted to investigate whether their axes aligned with individual eigenvectors identified by our Hessian or whether their axes randomly mixed with our top eigen space. To achieve this goal, we search for the power coefficients that are significantly higher than expected from projection of unit random vector. In fact, for each and every of the discovered axes, we found 1-3 eigenvectors that they are significantly (p < 5 x 10-4) aligned to. Moreover, these strongly aligned eigenvectors are all in our top 60 eigen dimensions, in fact, 3 of their axes aligned with eigenvector 11 and of their axes aligned with our eigenvector 6. (Fig 8 A) Moreover, we "purify" their axes by a) retaining projection coefficients only in top 60 eigenvectors, or b) retaining the coefficients only in the 1-3 strongly aligned eigen vectors and set all the other 500+ coefficients to zero, and compared their effect on same set of reference vectors, using the same step size. We found that by project out coefficients in the lower space, the image transformation is perceptually very similar (Fig B,C). If we only retains the eigenvectors that it highly aligns to, the image transform will be more different, but the annotated semantics in the transform seems to keep (Fig 8 D.E). Thus, their method also discovered that the top eigenspace of PGGAN contained informative transformations, and further confirmed that optimizing interpretability may improve alignment with individual eigen vectors rather than mixing all the eigen dimensions.
Note, as we project out coefficients, the resulting vector has a smaller than unit norm, thus we are moving smaller distance in latent space using the same step size (Fig88B-F title). If renormalize the vector to unit norm we will need to take smaller step size to achieve the same transform. This confirming our predictions in Sec. + v. which add direction e in the lower eigenspace or null space of the GAN. These axes encode the same transforms but the speeds of image change on them are different. this sense, the top eigenspace could be used to provide "purer" version of the interpretable axes discovered elsewhere.
Each interpretable axis u has a family of equivalent axes
Although both types of approaches are promising, by removing the need to train additional net- works, our method can be viewed as a more efficient way to identify informative axes. Further work comparing axes discovered by different methods will elucidate the connection between interpretable axes and the Hessian structure more.
A. DETAILED COMPARISON TO HESSIAN PENALTY
In this section, we compare our work with that of (Peebles et al. 2020), which also focused on the use of a Hessian matrix to "disentangle" directions in generator latent space. This Hessian penalty approach devised a stochastic finite difference estimator of the non- non-diagonal elements of the Hessian, using only forward pass, resulting in an efficient regularization of Hessian diagonality during GAN training. This clever approach led to several benefits, including increases in smoothness in latent interpolation, and improvements in interpretability. However, there clear contrasts between the Hessian penalty approach and ours.
At the highest conceptual level, both works relied on the analysis of the Hessian matrix. Our frame- work is motivated by a geometric picture of the image manifold and latent representations, while their work is motivated by the idea of increasing the independence of latent factors. The Hessian matrices involved are also different: we computed the Hessian of the squared distance metric d² in image space, while they they computed the Hessian of every pixel in the image.
At the implementation level, the methods are also different. We computed the exact Hessian matrices or their low-rank approximations based on backpropagation and the Lanezos iteration. They devised a stochastic fnife-difference estimator of the total non-diagonal elements of the Hessian, using only a forward pass, which they termed the Hessian penalty. This remarkable achievement enabled them to efficiently regularize Hessian diagonality during GAN training. As another point of contrast, the Hessian penalty does not provide an explicit geometric picture as the exact Hessian matrices do in work. Specifically, their penalty did not reveal the spectral structure of the Hessian matrices, which is encoded in the diagonal elements of the matrix. Because of this, the anisotropy can be explicitly demonstrated in our work.
20

---

Published as a conference paper at ICLR 2021
A3GAN512Annotateee Axes Proj Hess Frame
A
LatentDiscovery Axes
Aligned
Annotaked Axes
5
10
15
20
25
30
35
Eigenvector
11.38 38 60
11
6, 11,25 40
43
29
eigen rank Random Control
10
15
20
25
30
35
40
0.06
random vestors
0.05
0.04
0.03
0.01
eigen rank
Original Axes 19, Annotation: Hair
Original Axes 20, Znnamagion-Zoaming
Top 60 Eigen Projection, Vec 0.51
Top Eigen Projection, Norm 0.49
D
Highly Aligned Only, VecNorm 0,236
E
Highly Aligned Only, Vec Norm 0,292
Figure 8: Analyzing interpretable axes from (Voynov Babenko, 2020) under the Hessian framework. A. Projection power of their annotated interpretable axes and unit norm random vectors on the top 40 eigen vectors. The color code matched. The red line on colorbar denotes p 10 threshold for the power value, and the significantly aligned axes p 5 10 indicated. B.C, Image transforms encoded by projection of of their vectors (19, 20) into the top 60d eigenspace. D.E Similar B,C, but their vectors are projected onto the aligned eigenvectors only. The norms of the projected vectors are noted on title. Panel B,D and C,E share the same reference image and the same step size across each mini column, though the distance travelled along BC and DE is smaller as the vector is shortened.
clear that the Hessian penalty provides a very important and complementary approach to ours. This work showed that regularizing Hessian diagonality during training could promote disentangled latent representation using PGGAN (this provides more detailed evaluations of disentanglement than achieved).
Interestingly, many of the phenomena they observed in busssian-penalizer generative networks were reminiscent to phenomena eigen-frame rotation of the latent space. As a first example, they found that after applying the Hessian penalty, many factors stopped affecting the image (termed "latent space shrinkage" in their work). We also found this phenomenon normal pre-trained GANs, i.e. they showed a large bottom eigenspace with close eigenvalues, in which the eigenvectors generated small no changes in
observed in normally trained generative networks with Hessian
21

---

Published as a conference paper at ICLR 2021
the image (termed "null space" in work; Sec. 4p. Thus if we performed a Hessian eigen- -rrame generators will exhibit similar behavior as theirs do.
rotation, Hesssan-
As second example, they also found that enforcing the Hessian penalty middle layers is helpful in regularizing the Hessian diagonality in the image space, which is reminiscent of our finding that the Hessian aignnfrrames are usually well aligned across the layers of generator (Sec. 5), though the spectra get shaped throughout the layers.
The Hessian penalty raises interesting question: what makes diagonal Hessian matrix special? Because the Hessian is a real symmetric matrix, at each point 2, it can be diagonalized by a rotation of the latent space. However, to achieve a diagonal matrix across the latent space, |Pebles et al. (2220) had to (implicitly) enforce each point to share the same rotation (i.e. Hessian Sigenfrrang Given the training involved, this encouraged the homogeneity or flatness of latent space as identified in our framework. However, as shown in our work, most GANs exhibit homogeneity or flatness in their latent space even without the Hessian penalty. So it would be interesting to compare generators trained with Hessian penalty against those trained without the penalty but with a post hoc rotation of the latent space using the global eigen-frame. We expect that their training will exhibit flatter geometry than the post hoc rotated latent space. However, even if this is not true, it would still be interesting if this flat geometry can emerge from modern GAN training i.c. fitting the natural image distribution.
Finally, aside from regularizing GAN training, Peebles et al. (2020) also explored the use of the Hessian penalty in finding interpretable directions in pre-trained BigGAN (BigGAN--22). Similar to Voynov & Babenko (2020). the axes they discovered showed a striking correspondence to those identified in our approach (Fig. D. We showed that when computed the Hessian few points in the noise space of the generator, the interpretable axes they found aligned well with our top eigen- vectors, with a one-to-one or one-to-two matching. For example. their reported interpretable axes 5, 6 and 8 (for the golden retriever class) had 0.998, 0.964, 0.990 of their power concentrated in our top 10-dimensional eigenspace. Moreover, they aligned with single eigenvectors power 0.988, 0.499, 0.852 respectively. Due to the presence of close by eigenvalue, eigen vectors can mix into each other, resulting in the phenomenon that the axes identified in PPeblees et al. (2020) can correspond to few adjacent eigenvectors (e.g. their axes 5 correspond to our eigenvectors 4 and 5, with corresponding eigenvalues 8.3 and 7.4; as reference, eigenvalue 3 = 17.6, eigenvalue 6 = 5.6, a much larger gap.). We found that power concentration within the top eigenspace is great indicator of the interpretability of their reported axes: all but one of the reported axes showed over 0.95 power concentrated in top 10d eigenspace, while the axes they found not be interpretable showed power 0.103 0.141 (mean+-std) in the top 10d eigenspace. One advantage of our approach is computational efficiency. As the alternative method required learning (i.e. iterative optimization of a mixing matrix), it took us 40 minutesx500 50 epochs to finish the training on 6GB GTX 1060 GPU. In contrast, the present method directly computes the Hessian matrices one- or a few points, taking 12 seconds to compute a full Hessian matrix (5-50 points usually suffice). The reason for this difference that their method optimizes for a basis that diagonalizes the Hessian matrix based on noisy estimate of diagonality- the Hessian penalty; in contrast, our approach directly computes and diagonalizes the matrix at given points. Finally, our method orders the axes by the eigenvalues, facilitating focus on the top eigenspace, and thus alleviating the need to go through all the axes to find interpretable ones.
and 2 with
In summary, we believe that stochastic regularizer of the Hessian matrix, the Hessian penalty is a valuable and complementary approach to ours. Our methods provide additional value by ac- curately estimating the top eigenvectors and eigenspeetrum, suitable for analyzing geometry post hoc. However, unlike the Hessian penalty, direct use of our method to regularize training may be inefficient. It would be interesting explore whether there is a middle ground that incorporates the advantages of both their estimator and our more precise calculation.
22

---

Published as a conference paper at ICLR 2021
Golden Retriever Class, Full
Fine scale
Mid scale
Hessin Egenn vedtoss
Hesian Egeevectorn 100 block), top
Hosdon Bgen Bgen vectors c4 bloog
10
15
20
5
10
15
10
Porsian Phe wayy Leant Kes
08
0.6
0.6
Io
15
15
15
02
0.0
Heastan Bion vectorsall
Hostion Parskyy Pardkyyytty Leumttss
12
10
0.8
04
12
Q2
00
N=1
N
N=1D
N=25
N-50
Nnnotored Interpictable fucs HP
www.lmteepeeaablee by projecting
Penalty
Penilty
Figure 9: Analyzing interpretable axes from (Peebles al., 2020) under the Hessian frame- work. A. Projection power of Hessian Hessian-penalty- penalty-identified axes on the top 25 eigen vectors. B,C. For the axes corresponding fine or mid-scale changes, we performed eigen decomposition the corresponding 40d diagonal block of the averaged full Hessian. We then computed the projection power with their axes. D. Comparison of the alignment with the Hessian eigen frame averaged over different number of points (N 1, 5, 10, 25, 50, 100) in the full space. Although the specific matched eigen id varies between N, those interpretable axes all remain our top eigenspace, with power over 0.95, E. Visualize the axes in PPeblee et al. (2200) with good alignment with our top 10d eigenspace (axes 0 and 9, boxes marked in green), other than those annotated in their paper (boxes marked in red). They arguably interpretable as well.
23

---

Published as a conference paper at ICLR 2021
ProgGrowGAN
StyleGAN
StyleGANZ Face256
StyleGAN2 Face512
StyleGAN2 Face1024
Figure 10: Similar transforms encoded in the top eigendimension of GANs trained on face dataset. Linear exploration along top 20 eigenvectors from origin latent space are showed for each GAN. Linear equi-distance sampling on each eigenvector occupies column and their eigenvalues are sorted descending order from left to right. Step size along each vector is adjusted according to its eigenvalue for best continuity.
24

---

Published as a conference paper at ICLR 2021
StyleGAN2 Cat Eig1
StyleGAN Face Eig1
+
StyleGAN2 Cat Eig4
StyleGAN Face Eig3
StyleGAN2 Cat 10
StyleGAN Face Eig5
Figure 11: Top eigenvectors encode similar transforms around different reference images. Lin- car equidistant explorations from six randomly chosen reference images along the eigenvectors of averaged Hessians. These show qualitatively similar transforms to images for example, proxim- ity of Cat face (Eig1), proximity and cat number (Eig4), fur color darkening (Eig10) in StyleGAN2 Cat; face direction (Eigl), masculine vs feminine (Eig3), child vs adult (Eig5) in StyleGAN Face.
25


Reviews:
[{"review_id": "voPCTUJeFO", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "_Summary_:\nThis work intends to explore the geometry of the latent space and proposes to define the distance in latent space as the distance between the corresponding generated images and use the Hessian of that squared distance as metric tensor to define the manifold. Using Learned Perceptual Image Patch Similarity (LPIPS), they show that the Hessian can either be computed through backpropagation or if that is not efficient, it is sufficient to iteratively compute the eigenvectors corresponding to the largest eigenvalues. With the proposed method, the empirical observations showed 1) the impact of those eigenvectors through examples, 2) consistent geometric local changes over different positions in the latent space, and 3) the impact of top eigenvectors on particular layers. Further, the authors discuss three areas of possible application (gradient-based GAN inversion, gradient-free image search, interpretable axes discovery).\n\n_Strengths_:\n- The paper addresses an interesting topic (understanding the latent space of GANs) and proposes a straightforward method.\n- The empirical observations demonstrates the advantages of using an image similarity metric as latent space distance and the correlation between eigenvalues corresponding to the largest eigenvalues and image perception.\n\n_Weaknesses_:\n- The method is depending heavily on the distance metric employed, however, does not discuss in what way this could influence the outcome. In this paper, only LPIPS was considered as metric. What are the advantages and disadvantages? How would the analysis and outcome change when other distance metrics are used? These are questions that might benefit the work to discuss.\n- The actual method was barely discussed in the paper, but rather moved to the appendix. I think the authors may want to restructure the paper to include how to compute the eigenvectors are computed.\n- The actual application of the proposed method was rather discussed than actually performed. I do think this is a very interesting proposal, however, with only showing empirical observations, I believe the scope of this paper is too little. It would have been better if the authors picked one of the applications and applied their method.\n\nFor more details see below.\n\n_Overall assessment_: \nOverall, I find the proposed method very interesting and the empirical observations compelling. However, I find the scope of the paper not sufficient as it would have been nice to see that one of the application with the usage of the Hessian metric as distance function would have worked. \n\n_Detailed questions and comments_:\n- Learned Perceptual Image Patch Similarity (LLIPS, Section 3 \"Numerical Method\"): As this is the main distance metric being used, would it be possible to briefly introduce and define it in the paper?\n- Requirement for understanding the latent space (abstract): The abstract mentions inversion and interpretability as requirement for understanding the latent space. It was not clear to me how the method presented is addressing each of the requirement. Further, the abstract also claims \"This geometric understanding unifies previous results of GAN inversion and interpretation.\". Can you clarify these claims?\n- Appendix A.2 Methods for computing the hessian: Parts of how the Hessian is computed should be in the main paper as this is the main method for this paper. The explanation in Section 3 \"Numerical Method\" was superficial and I could have not understood the methodology without reading the the appendix.\n- Spectrum Structure of GANs (Section 4, Figure 2): Can you clarify what has been exactly used to plot Figure 2? In the paragraph, Figure 2 was plotted using the mean and 90% confidence interval, Figure 2 y-axis label says $\\log(eig/eigmax)$.\n- Figure 3D: The figure is hard to read, with many data points overlapping each other. I am also confused what the two lines crossing each data point represents.\n- \"Then we explored linearly the latent space along the eigenvector\" (Section 4): Why is the exploration linearly? Does this conform with the manifold? How is $\\mu_i$ defined? The footnote also says that spherical linear exploration is used some spaces. Can you elaborate more on how you performed exploration?\n- Top Eigenvectors Capture Perceptual Relevant Changes: Would it be able to quantify this? As only four samples were shown, how would we know that this generalizes for all samples?\n- [1]-[4] might be also relevant to the topic of latent space exploration. They are not necessarily w.r.t. to manifold learning, however, with respect to applications in Section 6, they look very close. \n\n_Post-rebuttal_:\nI do highly appreciate the authors trying to answer all our questions and adding more details and experiments. However, after also reading through [Chiu et al.,  SIGGRAPH 2020] I do find that this paper has a large overlap with the one mentioned. Therefore, agreeing with Reviewer #2, the contribution is reduced to applying the idea to GANs. Therefore I am keeping my recommendation.\n\n[1] Lipton, Z.C. and Tripathi, S., 2017. Precise Recovery of Latent Vectors from Generative Adversarial Networks. ICLR 2017 workshop.\n[2] Albright, M. and McCloskey, S., 2019, May. Source Generator Attribution via Inversion. In CVPR Workshops (Vol. 7).\n[3] Webster, R., Rabin, J., Simon, L. and Jurie, F., 2019. Detecting overfitting of deep generative networks via latent recovery. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 11273-11282).\n[4] Bojanowski, P., Joulin, A., Lopez-Pas, D. and Szlam, A., 2018, July. Optimizing the Latent Space of Generative Networks. In International Conference on Machine Learning (pp. 600-609).", "extra_scores": {}, "extra_text": {}}, {"review_id": "fq72SEFNxaB", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper performs the analysis of the GAN latent spaces from the geometric perspective, inducing a metric tensor in the latent space from the LPIPS distance in the image space. The main authors' finding is that under such metric, the latent spaces of typical GANs are highly anisotropic, which can be exploited for more effective GAN inversion. Furthermore, the authors show that eigen vectors of the metric tensor often correspond to interpretable latent transformations.\n\nPros:\n\n1) The paper is exceptionally well-written and provides a very interesting read. While the performed analysis is simple and natural, it does reveal several interesting findings about typical latent spaces: LPIPS-anisotropy, global consistency of the metric tensor.\n\n2) The authors confirm the usefulness of their analysis by providing immediate practical benefits: more effective GAN inversion, which accounts for the latent space anisotropy.\n\nCons:\n\n1) Missing work on interpretable GAN directions:\n\n[A] The Hessian Penalty: A Weak Prior for Unsupervised Disentanglement, ECCV 2020\n\n2) In my opinion, the authors do not provide enough support for their claim \"This finding unifies previous unsupervised methods that discover interpretable axes in the GAN space\".\n\n- While the proposed method does seem to generalize both Ha ̈rko ̈nen et al., 2020 and Shen & Zhou, 2020, I do not see, how it captures Voynov & Babenko, 2020 and Pebbles et al (see above [A]). Furthermore, I believe that such claims should be supported by the experiments.  Could the authors experimentally confirm that their method results in the same set of directions as the existing methods?\n\nOverall, I am positive about this submission, since the main analysis is both interesting and practically useful. My main criticism is that in terms of discovery of interpretable directions, the methods should be experimentally compared to existing alternatives. If it does provide a super-set of directions, obtained by existing methods, this would make the submission much stronger. Otherwise, the claim about unification should be toned down in my opinion.\n\n======== AFTER REBUTTAL ========\n\nI appreciate the authors' efforts on additional thorough comparison to existing works on interpretable axes discovery. From the updated manuscript, however, it is not clear what method is superior and the authors' approach appears to be a yet another method for this task rather than generalization of previous ones. Overall, I am still on the positive side since the observed findings deliver a clear profit for GAN inversion. But I am not increasing my score given that the \"interpretable axes\" part has become less impressive (in terms of weaker claims and conclusions) and the competing SIGGRAPH work.   ", "extra_scores": {}, "extra_text": {}}, {"review_id": "bsNDI1Q0gav", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a method for finding the axes of largest variation in the latent space of a generative model. This can be leveraged for better generator inversion and explainability. Several experiments are done to evaluate the latent vectors used by the method quantitatively and qualitatively.\n\n##########################################################################\n\nReasons for score: \n\nThe analyses are interesting and the method seems novel. This method could prove useful in analysis of latent space properties. Its full significance and practicality could be clearer, though.\n\n##########################################################################\n\nPros:\n- The problem of understanding the latent space structure better is relevant and timely in the research on generative models equipped with a continuous latent space.\n- Experiments seem to demonstrate successful dimensional truncation, GAN inversion, and that the eigenvectors behave consistently\n\n##########################################################################\n\nCons:\n- I am not sure I see whether each experiment actually supports the corresponding claim. Especially, the claim that interpretable axes are found seems central but it is not much supported in the experiments. It seems mostly represented in Fig. 9 in the Appendix, but I do not fully understand how we are supposed interpret the result in this figure.\n- Fig 2 is hard to parse and clearly too tightly packed. Something should be done about it.\n\n##########################################################################\n\nQuestions during rebuttal period:\n- It is easy to get lost in the math and in the various measurements. Could the authors summarize, in practise and plain English, how their method should be used to find the most intepretable axes of variation for a new generative model, and how to confirm and measure that we have indeed found them?\n\nTypo: on page 7 around \"(Fig 5.)\"\n\n##########################################################################\n\nUpdate after rebuttal discussions:\n- In the light of the considerable overlap with [Chiu et al., 2020] pointed out by the other reviewers, I decreased my score. I have familiarized myself with it and can confirm the said overlap. However, given remaining differences, I do not find it unreasonable to consider this paper as \"complementary\" to [Chiu et al., 2020], *provided that the authors explicitly address the similarities in the final version*.\n- I consider the sum total of contributions of the paper still tilting towards being sufficient for publication.", "extra_scores": {}, "extra_text": {}}, {"review_id": "-uNk-y_ZX9m", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nSummary:\n\nThe paper proposes an analysis way of a latent space of GAN in a GAN architecture agnostic way. They use the Riemannian manifold analysis to investigate image manifold, which leads to a simple algorithm with eigen-decomposition of the Hessian matrix of a local point.\n\nUnfortunately, many ideas and some of the findings (Hessian-based GAN exploration and anisotropy of Hessian in the latent space) are already well explored in [C1] with much higher quality and various modality applications.\nBut still, there are interesting ideas contained in this paper: 1) the latent space is homogeneous (in a sense that the major directions obtained by a latent position are shared at different positions with similar semantic meanings), and 2) extensive eigenvalue analyses of Hessian.\n\nBased on the findings, they show interesting applications of efficient GAN Inversion, gradient-free search in Image space, and unsupervised discovery of interpretable axes. However, these applications are also already explored by [C1] in a visually pleasing way.\n\n[C1] Human-in-the-Loop Differential Subspace Search in High-Dimensional\nLatent Space, SIGGRAPH 2020.\n\nReasons for score: \n\nI like the idea that leverages the findings of the homogeneous property of Hessian and improves the efficiency of GAN inversion with Hessian preconditioning. Also, I appreciate the analysis of eigenvalues and correlations.\nHowever, unfortunately, most of the contributions the authors argued are largely overlapped with [C1], of which citation is missed by the authors. Thus, with these remaining contributions, this reviewer could not vote for the acceptance of this work at this point.\nThis reviewer would like to ask the significant different contribution from [C1] that this reviewer may miss.\n\nPros:\n- The paper shows interesting interpretation and ideas, and its applications.\n- The paper is written very clearly and readily.\n- The authors validate the ideas with enough effort in the experiments.\n\nCons:\n- The GAN architecture agnostic latent space exploration with Riemannian interpretation is already done in [C1] in a more general way. \n\n\nDetail comments:\n- While [C1] presents mainly about the SVD of Jacobian (but they also show Hessian interpretation (metric tensor) in Sec. 4.1 as well), it is essentially the same because of Eq. (3) in this paper. Also, [C1] suggested much more diverse exploration methods with diverse search methods. \nThus, this reviewer would like to listen to the authors' responses about this overlap.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents several analyses on the geometry of GAN generators through the lens of Riemannian geometry: showing interpretability of the leading eigenvectors of the Hessian, homogeneity of the space, and more efficient latent-space inference through preconditioning. Reviewers found the (revised) paper well-written and clear, with a thorough set of experiments to support their main claims. While there were several concerns around the generality of the approach, the authors performed several experiments in the rebuttal period to address many of the reviewer’s concerns (robustness of findings with different image distance functions, inversion on additional GANs and datasets, user study of perceptual properties of axes, and comparison to previous methods for intepretable axes discovery). I found these experiments extensive and convincing, supporting the claims around robustness of the approach to different image distance metrics, GAN architectures, and interpretability of the axes. 

There were also strong concerns around similarity with recent work (Chiu et al., SIGGRAPH 2020 and Peebles et al., ECCV 2020), but both of these papers were published at most 1 month before the ICLR submission deadline, and thus should be considered as concurrent work. 

Given the strong set of additional experiments and interesting empirical observations, I recommend accepting this paper.

There remain concerts around the extent to which the findings “unify” previous approaches on interpretable axes, and we encourage the authors to update the paper before the camera ready to address these and additional reviewer concerns (especially expanding the discussion of the relationship with concurrent work in Chiu et al. and Peebles et al.).
Decision: Accept (Poster)