Do Not Escape From the Manifold: Discovering the Local Coordinates on the Latent Space of GANs
Authors: Jaewoong Choi
Junho Lee
Changyeon Yoon
Jung Ho Park
Geonho Hwang
Myungjoo Kang
Venue: ICLR 2022 Poster (2022)
Keywords: generative adversarial network
disentanglement
semantic factorization
latent space control
image manipulation
grassmannian

Abstract: The discovery of the disentanglement properties of the latent space in GANs motivated a lot of research to find the semantically meaningful directions on it. In this paper, we suggest that the disentanglement property is closely related to the geometry of the latent space. In this regard, we propose an unsupervised method for finding the semantic-factorizing directions on the intermediate latent space of GANs based on the local geometry. Intuitively, our proposed method, called $\textit{Local Basis}$, finds the principal variation of the latent space in the neighborhood of the base latent variable. Experimental results show that the local principal variation corresponds to the semantic factorization and traversing along it provides strong robustness to image traversal. Moreover, we suggest an explanation for the limited success in finding the global traversal directions in the latent space, especially $\mathcal{W}$-space of StyleGAN2. We show that $\mathcal{W}$-space is warped globally by comparing the local geometry, discovered from Local Basis, through the metric on Grassmannian Manifold. The global warpage implies that the latent space is not well-aligned globally and therefore the global traversal directions are bound to show limited success on it.

Published as a conference paper at ICLR 2022
Do NOT ESCAPE FROM THE MANIFOLD: DISCOVER- ING THE LOCAL COORDINATES ON THE LATENT SPACE OF GANS
Jaewoong Choi, Junho Lee; Changyeon Yoon, Jung Ho Park, Geonho Hwang, Myungjoo Kang Scoul National University {chiw1475 joon2003, shinypond, jhpark009, hgh2134 mkang) 8nnu. ac kr
ABSTRACT
The discovery of the disentanglement properties of the latent space in GANs mo- tivated lot of research to find the semantically meaningful directions on it. In this paper, we suggest that the disentanglement property closely related the geometry of the latent space. In this regard, we propose an unsupervised method for finding the semantic-factorizing directions on the intermediate latent space proposed method, called Lo-
GANs based on the local geometry. Intuitively, cal Basis, finds the principal variation of the latent space in the neighborhood the base latent variable. Experimental results show that the local principal vari- ation corresponds to the semantic factorization and traversing along it provides strong robustness image traversal. Moreover, we suggest explanation for the limited success in finding the global traversal directions in the latent space, especially W-space of StyleGAN2. We show that W-space is warped globally by comparing the local geometry, discovered from Local Basis, through the metric on Grassmannian Manifold. The global warpage implies that the latent space is not well- well-aligned globally and therefore the global traversal directions are bound show limited success on
INTRODUCTION
Generative Adversarial Networks (GANs, Goodfellow et al. (014) ) such as ProGAN (Karras al., 2018), BigGAN (Brock et al. 2018), and StyleGANs (Karras et al. 2019 2020b a) have shown tremendous performance generating high-resolution photo-realistic images that are often indistin- guishable from natural images. However, despite several recent efforts (Goetschalekx et al. 2019
Jahanian et al., 2019 Plumerault et al. 2020; Shen et al. 2020) to investigate the disentanglemen properties (Bengio al. (2033) of the latent space in GANs, it still challenging to find meaningful traversal directions in the latent space corresponding to the semantic variation of an image.
The previous approaches to find the factoriinn directions categorized into local and global methods. The local methods (e.g. [Ramesh et al. (2018), Latent Mapper in StyleCL.IP (Patash- nik et al. 2021), and attribute-conditionel normalizing flow in StyleFlow (Abdal et al. (021 suggest a sample-wise traversal direction. By contrast, the global methods, such as GANSpace (Harkönen et 2020) and SeFa (Shen & Zhou 2021), propose global direction for the particular semantics (e.g. gl glasses, age, and gender) that works on the entire latent space, Throughout this pa- per, we refer to these global methods as the global basis. These global methods showed promising results. However, these methods are successful on the limited area, and the image quality is sensi- tive to the perturbation intensity. In fact, if a latent space does not satisfy the global disentanglement property itself, all global methods are bound to show a limited performance on it. Nevertheless, to the best our knowledge, the global disentanglement property of latent space has not been inves- tigated except for the empirical observation of generated samples. In this regard, need local method that describes the local disentanglement property and an evaluation scheme for the global disentanglement property from the collected local information.
" Eqqual contribution

---

Published as a conference paper at ICLR 2022
In this paper, suggest that the semantic property of the latent space in GANs (i.e. disentangle- ment of semantics and image collapse) is closely related to its geometry, because of the sample-wise optimization nature of GANs. In this respect, we propose unsupervised method to find traver- sal direction based on the local structure of the intermediate latent space W, called Local Basis (Fig Tap. We approximate W with its submanifold representing its local principal variation, discovered in terms of the tangent space Tw W. Local Basis is defined as an ordered basis of Tw W corresponding to the approximating submanifold. Moreover, we show that Local Basis is obtained from the simple closed-form algorithm, that is the singular vectors of the Jacobian matrix of the subnetwork. The ge- ometric interpretation of Local Basis provides an evaluation scheme for the global disentanglement property through the global warpage of the latent manifold. Our contributions are follows:
1. We propose Local Basis, a set of traversal directions that can reliably traverse without escaping from the latent space prevent image collapse. The latent traversal along Local Basis corresponds to the local coordinate mesh of seaal--oomtty--essrri
submanifold. 2. We show that Local Basis leads to stable variation and better semantic factorization than global approaches. This result verifies our hypothesis on the close relationship between the semantic and geometric properties the latent space in GANs.
3. We propose Iterative Curve- Traversal method, which is a way to trace the latent space in the curved trajectory. The trajectory of the images with this method shows a more stable variation compared to the linear traversal.
4. We introduce the metries on the Grassmannian manifold to analyze the global geometry of the latent space through Local Basis. Quantitative analysis demonstrates that the W-space of StyleGAN2 is still globally warped. This result provides an explanation for the limited success of the global basis and proves the importance of local approaches.
2 RELATED WORK
based Generators. In recent years, GANs equipped with style-based generators (Karras eet al. 2019 2020b) have shown of-t-ooftthe-an performance in high-fidelity image synthesis. The style-based generator consists of two parts: a mapping network and a synthesis network. The map- ping network encodes the isotropic Gaussian noise z Z an intermediate latent vector E W. The synthesis network takes w and generates an image while controlling the style of the image through w. Here, W-space is well known for providing a better disentanglement property compared to Z (Krrras et al. 2019). However, there is still a lack of understanding about the effect of latent perturbation in a specific direction on the output image.
Latent Traversal for Image Manipulation. The impressive success of GANs in producing high- quality images has led to various attempts to understand their latent space. Early approaches (Rad-) Ford et al. 2016 Upchurch et al. 2017) show that vector arithmetic on the latent space for the se- mantics holds, and StyleGAN (Karras et al. 2019) shows that mixing two latent codes can achieve style transfer. Some studies have investigated the supervised learning of latent directions while as- suming access to the semantic attributes of images (Goetschalckx et al., 2019: Jahanian et al. 2019 Shen et al. 2020 Yang et al., 202T; Abdal et al. (021). In contrast to these supervised methods, some recent studies have suggested novel approaches that do not use the prior knowledge of training dataset, such as the labels of human facial attributes.
Voynov Babenko (2020), an unsupervised optimization method is proposed to jointly learn candidate matrix and corresponding reconstruc- tor, which identifies the semantic direction in the matrix. GANSpace (Härkönen et al. (2200) finds global basis for W in StyleGAN using a PCA, enabling a fast image manipulation. SeFa (Shen & Zhou| 202T) focuses on the first weight parameter right after the latent code, suggesting that it con- tains essential knowledge of image variation. SeFa proposes singular vectors of the first weight parameter as meaningful global latent directions. StyleCLIP (Patashnik et al.) (021) achieves a state- of-the-art performance in the text-driven image manipulation of StyleGAN. StyleCLIP introduces an additional training to minimize the CLIP loss (Radford et 2021).
Jacobian Desitiosition. Some works use the Jacobian matrix to analyze the latent space of GAN (hhu et al. 2021 Wang & Ponce. 2021 Chiu et al.) 2020 Ramesh et al.) 2018). However, these methods focus on the Jacobian of the entire model, from the input noise z to the output image. Ramesh et al. (2018) suggested the right singular vectors of the Jacobian as local disentangled direc-
2

---

Published as a conference paper at ICLR 2022
Tacoktan
Image Space
X
No.
JiGG SVD
Ramoh
Diaections
Weigh Pansactics
Durectons
Intensediate Latent Space W
PCA GANSpace
Devectone
Inpur Space
2
J.G Jaotias
SWID
Local Basis
Directions (Dun)
Local Basis Global Basis
(a) Concept Diagram of Local Basis
(b) Latent Traversal Methods
Figure 1: (a) Concept diagram of Local Basis. The global basis reflects the global variation of latent space. Hence, traversing along the global basis may result in the escape from the latent space (shaded region). On the other hand, Local Basis closely follows the latent space. (b) Comparison of Latent Traversal Methods (Global methods: GANSpace (Härkönen et al. (020) and SeFa (Shen
& Zhou 202Ib,
Local methods:
Ramesh al. 2018 and Local Basis (Ours)
tions in the Z space. Zhu al. 2021) proposed latent perturbation vector that can change only particular area of the image. The perturbation vector is discovered by taking the principal vector of Jacobian to the target area and projecting it into the null space of the Jacobian to the complemen- tary region. On the other hand, our Local Basis utilizes the Jacobian matrix of the partial network, from the input noise to the intermediate latent code w, and investigates the black-box intermedi- ate latent space from it. The top-k Local Basis corresponds to the best-local- best-local-geomeeryy- eecometry-describiee submanifolds. This intuition leads to exploiting Local Basis to assess the global geometry of the intermediate latent space.
3 TRAVERSING A CURVED LATENT SPACE
In this section, we introduce a method for finding a soal-geometry-awarr traversal direction in the intermediate latent space W. The traversal direction is refered as the Local Basis at E W. In addition, we evaluate the proposed Local Basis by observing how the generated image changes as we traverse the intermediate latent variable. Throughout this paper, we assess Local Basis of the W- space in StyleGAN2 (Karras et al. 2020b). However, our methodology is not limited to StyleGAN2. See appendix for the results on StyleGAN Karras et al. 2019 and BigGAN (Brock et al. 2018).
3.1 FINDING A LOCAL BASIS
Given a pretrained GAN model M 2 > X, from the input noise space Z to the image space X, choose the intermediate layer W to discover Local Basis. We refer to the former part of the GAN model the mapping network f Z + W. The image of the mapping network is denoted as W f(2) C W. The latter part, non-linear mapping from W to the image space X, is denoted by G W X. Local Basis at W is defined as the basis of the tangent space Tw W. This basis can be interpreted as a vocal-geometry-awaat linear traversal direction starting from w.
To define the tangent space of the intermediate latent space W properly, assume that W is differentiable manifold. Note that the support of the isotropic Gaussian prior Z = Rdd and the ambient space W = Rdw are already differentiable manifolds. The tangent space at w, denoted by W. is vector space consisting of tangent vectors of curves passing through point Explicitly,
Tw {0(0) 7 (-c,c) + W, (0) = w, for 0).
(1)
Then, the differentiable mapping network f gives linear map dfz between the two tangent spaces T2 Z and Tw W where w = f(z).
dfz : 2,2 - - W, +(0) - (ʃ ا ())
(2)
We utilize the linear map dfa. called the differential of f at z, to find the basis of Tw W. Based on the manifold hypothesis in representation learning, posit that the latent space of the image
3

---

Published as a conference paper at ICLR 2022
space X' in W is a lower-dimensional manifold embedded in W. In this approach, estimate the latent manifold as a lower-dimensional approximation of W describing its principal variations. be obtained by solving the low-rank approximation problem of
The approximation manifold df2. The manifold hypothesis is supported by the empirical distribution of singular values off The analysis is provided in Fn the appendix.
The low-rank approximation problem has an analytic solution defined by Singular Value Decompo- sition (SVD). Because the matrix representation of dfa is a Jacobian matrix (V))) E R2 Local Basis is obtained as the following: For the i-th right singular vector u E Rdz, i-th left
singular vector v.W E R44, and i-th singular value of E IR of (Z) with of 2 ... 2 2
dfz(u,) = of v." for Vi, Local Basis(w w = f()) = [v") Jisisa
(3)
(4)
Then, the k-dimensional approximation of V around is described as the following because Bd (if of > 0). Note that W is submanifold of W corresponding to the & components of Local Basis, i.c Tw svan(v iisisk, k}.
W {
2 Σ 1/ 4...u
14 14E ()) for isisk
(5)
Locally affine mapping network In this paragraph, focus on the locally affine mapping net- work f, which is one of the most widely adopted GAN structures, such as MLP or CNN layers with ReLU leaky-ReLU activation functions. This type of mapping network has several well-suited properties for Local Basis.
f(z) ELEEp (A,z + bp) PEl2
where Q denotes a partition of 2, and Ap and b, are the parameters of the local affine map. With this type of mapping network f, is clear that the intermediate latent space W satisfies differentiable manifold property at least locally on the interior of each p E Q. The region where the property may not hold, the intersection of several closure of p's in Ωt has measure zero in Z.
Moreover, the Jacobian matrix (vz.)(z (z) becomes a locally constant matrix. Then, the approximat- ing manifold W (Eq 5) satisfies the submanifold condition, and is consistent locally for each p, avoiding being defined for each w. In addition, the linear traversal of the latent variable w along v can be described the curve on W (Eq(7) Most importantly, these curves W (Eq 7 starting from w in the direction of Local Basis, corresponds to the local coordinate mesh of W
11 + (7) Traversal (w = f(z). v.) (-c,c) - -2 w, ++ 2 o,
Equivalence Local PCA To provide additional intuition about Local Basis, we prove the fol- lowing proposition. The proposition shows that Local Basis is equivalent to applying a PCA on the samples on W around w.
Proposition 1 (Equivalence to Local PCA). Consider the Local PCA problem around the base latent variable wi = f(z)) on W. Lee. PCA of the latent variable samples w' around Wt-
w' (zb + c. with NO,, I) and for some small >>> 0.
(8)
where (iz( = wa + () - 2b) is the linear approximation of J around Zù- Then, the principal components discovered in the Local PCA problem are equivalent to Local Basis at wh-
3.2 ITERATIVE CURVE-TRANE TRAVERSAA
We suggest natural trvve-traeeral that can keep track of the W-manifold and an iterative method to implement it. We divide the long curved trajectory into small pieces and approximate each piece
Sttricly speaking, We may not satisfy the conditions of the submanifold. The injectivity of dfz on the domain {z + E. uf It E (-G,e)) for 1 < k) is sufficient condition for the submanifold. As described helow, this sufficient condition satisfied under the locally affine mapping network f and of
4

---

Published as a conference paper at ICLR 2022
W
+0
Global basis direction
Local Basis direction
Iterative Curve-Traeessl direction
Figure 2: Illustration of Iterative Traversa (for N 2).
by the local curves using Local Basis. We call this curve-traversal Iterative Curve-Traversal method. Consistent with the linear traversal method, we consider Iterative direction of a Local Basis. Explicitly, for sufficiently large >
departing in the
7 (-c,6) - W, 7(0) - w, +(0) = vx for some S dw
where }r = Local Basis(w). We split the curve-traversal y into N pieces "/n and denote each n-th iterate in W and Z as w, and nn for
< N. The starting point of the traversal is denoted as the O-th iterate w = Wp. z = Zn, and wo = fzz)) (Fig 2 Note that to find Local Basis at W,, need corresponding Zn Z such that w,, - fzz)
Below, we describe the positive party" = 7/oo.c) of Iterative Curve-Traversal. For the negative part, we repeat the same procedure using the reversed tangent vector - The first step Y of Iterative Curve-Traversal method with perturbation intensity is as follows:
गo [0, of - 2 W,
t レ (Zo + t. u ) f(20 + t. +
(10)
21 = 2g +
Zo
W1 = f(z1)
(11)
(N
2o
Note that w is the endpoint of the curve 7/ and 1 (0) = v. We scale the step size in by 1/0 to ensure each piece of curve has a similar length of (I/N). To preserve the variation in semantics during the traversal, the departure direction of y is determined by comparing the similarity between the previous departure direction v and Local Basis at W1. The above process is repeated N-times. (The algorithm for Iterative TraverTra can be found in the appendix.)
(0) v
where
argmnax (v Wo 1sisdw
(12)
3.3 RESULTS OF LOCAL BASIS TRAVERSAL
We evaluate Local Basis by observing how the generated image changes as we traverse W-space in StyleGAN2 and by measuring FID score for each perturbation intensity. The evaluation is based on two criteria: Robustness and Semantic Factorization.
Robustness Fig 3 present the Robustness Test results In Fig 3 the traversal image of Local Basis is compared with those of the global methods (GANSpace (Härkönen et al.) 2020) and SeFa (Shen & Zhou, (021)) under the strong perturbation intensity of 12 along the 1st and 2nd direction of each method. The perturbation intensity is defined as the traversal path length in W. The two global methods show severe degradation of the image compared to Local Basis. Moreover, we perform a quantitative assessment of robustness. We measure the FID score for 10,000 traversed images for each perturbation intensity. In Fig 4 the global methods show the relatively small FID under the small perturbation. But, as we impose the stronger perturbation, the FID scores on the global methods increase sharply, implying the image collapse Fig 3 By contrast, Local Basis achieves much smaller FID scores with and without Iterative Curve-Traversal Traversal.
Ramesh et al. 2018 is not compared hecause it took hours to get traversal direction of an image. See appendix for the Qualitative Rohustness Test results Ramesh al. 2018).
5

---

Published as a conference paper at ICLR 2022
1-B
1-2
1-4
1-6
1-X
TH
1-6
1-N
(a) GANSSace
(b) SeFa
(c) Local Basis (Ours)
(d) Iterative
(Ours)
Figure 3: Qualitative Robustness Test on the W-space of the StyleGAN2 Karras et al. 2020bb
trained on FFHQ Each traversal image is generated by the linear traversal W except for (d) under the strong perturbation intensity of up to 12. The intensity is linearly increased from to 12 for each column. We infer the deterioration of the traversal image along the global method is due
to the escape of the latent traversal from the latent manifold. (See the appendix for the additional Robustness Test results along the first components of Local Basis.)
Without Traverse
140
Without Traverse
100
GANSpace
GANSpacce
Sefa
Sefa
Carve-Travescal
Curve Traversal
60
80
60
40
20
0
0
2
10
12
Perturbation Intensity
Porturbation Intensity
Figure 4: Quantitative Robustness Test on the W-space of the StyleGAN2 (Karras al. 2020b) trained on FFHQ. Fréchet Inception Distance (FID) (Heusel et al.) 2017) is measured for 10,000 traversed images for each perturbation intensity. Left: 1st direction, Right: 2nd direction
We interpret the degradation of image due the deviation of trajectory from W. The theoretical interpretation shows that the linear traversal along Local Basis corresponds to local coordinate axis on W, at least locally. Therefore, the traversal along Local Basis is guaranteed stay close W even under the longer traversal. However, we cannot expect the same property on the global basis because based on the global geometry. Iterative Curve-Traversal shows more stable traversal because of its stronger tracing to the latent manifold. This further supports our interpretation.
Semantic Factorization Local Basis is discovered in terms of singular vectors of dfz- The dis- entangled correspondence, between Local Basis and the corresponding singular vectors in the prior in Local Basis. Fig 5 and 6 presents the semantics of the
space, induces a semantic-factoriizati image discovered by Local Basis. In Fig 5 we compare the semantic factorizations of Local Ba- sis and GANSpace (Härkönen et al. 2020) for the particular semantics discovered by GANSpace. For each interpretable traversal direction of GANSpace provided by the authors, the corresponding Local Basis is chosen by the one with the highest cosine similarity. For a fair comparison. each traversal applied the specific subset of layers the synthesis network (Karras et al. (020b) provided by the authors of GANSpace with the same perturbation intensity. In particular, as we impose the stronger perturbation (from left to right), GANSpace shows the image collapse in Fig 5a and entanglement of semantics (Glasses + Head Raising) in Fig 5d However, Local Basis does not show any of those problems. Froviro additional examples of semantic factorization where the latent traversal is applied to subset of layers predefined in StyleGAN. The subset of the layers is selected as one of four, ie. coarse, middle, fine, or all styles. Local Basis shows decent factorization of semantics such Body Length of car and Age of cat in LSUN Yuu et al. 2015) in Fig

---

Published as a conference paper at ICLR 2022
Local Batis GANSpace
Local Buss GANSpaae
(a) Face length
(b) Open mouth
Local GANSpace
Local Basis GANSpuce
(c) Sunlight on face
(d) Glasses
Figure 5: Comparison of Semantic Factorization between Local Basis and GANSpace pre- trained StyleGAN2-FFHH We compare the aemantic-factorizing directions of GANSpace provided by the authors (Härkönen et al., 2020) with Local Basis of the highest cosine similarity. Local Basis factorizes semantics of image better, notably without collapsing compared to the GANSpace.
Body Length
Age 2nd direction
disection
Rotation
Stand up
direction
5th dircction
Coame
Vehiecle Type
Color
6th direction
1st direction
Middle
Middle
22nd disection Fine
19th direction
Fine
(a) StyleGAN2 LSUN Car
(b) StyleGAN2 LSUN Cat
Figure 6: Additional Semantic Factorization examples of Local Basis. The examples dis- covered by manual inspection due to the unsupervised nature while applying the latent traversal to subset of the layers predefined in StyleGAN (Karras al. 2019): coarse, middle, fine, and all styles. (See the appendix for the additional examples of Semantic Factorization without layer restriction.)
3.4 EXPLORATION INSIDE ABSTRACT SEMANTICS
Abstract semanties of image often consists of several lower-level semantics. For instance, Old can be represented as the correlated distribution Hair color, Wrinkle, Face length, etc. In this section, we show that the adaptation of Iterative Curve-Trave Traversal can explore the variation of abstract semanties, which is represented by the cone-shaped region of the generative factors (Träuble et al.) 2021).
Because of its text-driven nature, we utilize the global basi StyleCLIP (Patashnik al. 2021 corresponding to the abstract semantics of Old. Then, we consider the modification of Iterative Curve-Traversal following the given global basis yloobal- To be specific, the departure direction of each piece of curve in Eq 12 is chosen by the similarity to Vglobol, not by the similarity to previous departure direction. The results for old are provided in Fig [ (See the appendix for other examples.) Step size denotes the length of each piece of curve, i.e. (I/N) in Sec 3.2 For a fair comparison, the overall perturbation intensity I is fixed The linear traversal along the global basis adds only wrinkles to the image and the image collapses shortly. On the contrary, both Iterative Traverra methods obtain the diverse and high-fidelity image manipulation for the target semantics old. In particular, the diversity is greatly increased add stochasticity to the step size. We interpret this diversity as a result of the increased exploration area from the stochastic step size while exploiting the high-fidelit of Iterative Curve-Traversall
by adjusting the number of steps N.
We use the global basis defined W TTv
2021
See the appendix for detail.

---

Published as a conference paper at ICLR 2022
Figure 7: Iterative Curve-Traversee guided by global basis from StyleCLIP for the semantics of old. Left: Linear traversal along global basis. Middle: Iterative Tere-Traveesal of fixed step size (Stepsize = (0.02, 0.04, 0.08, 0.16)). Right: Stochastic Iterative Turve-Traversal (Step size sampled from Uniform Noise on [0.05, 0.15))
4 EVALUATING WARPAGE OF W-MANIFOLD
In this section, we provide an explanation for the limited success of the global basis in W-space of StyleGAN2. See 3 we showed that Local Basis corresponds to the generative factors of data, Hence, the lincar subspace spanned by Local Basis, which is the tangent space WW in Eq describes the local principal variation of image. this regard, we assess the global disentangle- ment property by evaluating the consistency of the tangent space at each E W. We refer to the inconsistency the tangent space as the warpage of the latent manifold. Our evaluation proves that W-manifold is warped globally. In this section, present the quantitative evaluation of the global warpage by introducing the Grassmannian Metric. The qualitative evaluation by observing the subspace traversal is provided in the appendix. The subspace traversal denotes simultaneous traversal in multiple directions.
Grassmannian Manifold Let V be the vector space. The Grassmannian manifold Gr(k, V) (Boothby) 1986) defined as the set of all k-dimensional linear subspaces of V. evaluate the global warpage of W-manifold by measuring the Grassmannian distance between the linear sub- spaces spanned by top-k Local Basis of each w W. The reason for measuring the distance for top-k Local Basis is the manifold hypothesis. The linear subspace spanned by top-k Local Basis corresponds to the tangent space of the k-dimensional approximation of W 5). From this per- spective, large Grassmannian distance means that the k-dimensional local approximation of W severely changes. Likewise, we consider the subspace spanned by the top-k components for the global basis. In this study, two types of metrics (i.e. Projection metric and Geodesic metric) adopted as metrics of the Grassmannian manifold.
Grassmannian Metric First, for two subspaces W, W' E Gr(k, V), ), let the projection into each subspace be Pw and Pw,, respectively Then the Projection Metric Karrasch 2017) on Gr(k, V) is defined as follows.
dproj (W, w') PlPw Pwrll
(13)
where || denotes the operator norm.
Second, let Mw, E dy x be the column- wise orthonormal matrix of which columns span W. W e Gr(k, V), respectively. Then, the Geodesic Metric (Ye & Lim (016) on Gr(k, V). which is induced by canonical Riemannian structure, is formulated as follows.
1/2
(W, dgeo (W, W')
0
(14)
where 0, cos--((s(M) Mw:)) denotes the i-th principal angle between W and W
8

---

Published as a conference paper at ICLR 2022
1.50
Coss W
Rastom
6:96
Ransom
global GAlGeaes
5.4
0:32
Rander O1d1
6:00
Randam
Fandom
Bundam
To (M0b GAN69IcC
6:66
gobal SANGOaae
glocal Sala
12 3Mbbl baha
Sutepace Binonsitn
(a) Projection Metric
(b) Geodesic Metric
Figure 8: Grassmannian metric. The shaded region illustrates (mean + standard deviation) inter- vals of each score. Above all, Random metric is much larger than the Close This means large variation of Local Basis on W, which demonstrates that W-space is globally warped. More- over, the metric result shows the local consistency of Local Basis and the existence of limited global alignment on W. (See Sec 4 for detail)
Evaluation We evaluate the global warpage of the W-manifold by comparing the five distances vary the subspace dimension.
1. Random Od)) Between two random basis of Rdw uniformly sampled from O(dw)
2. Random w: Between two Local Basis from two random w E W
3. Close Between two Local Basis from close w', W (See appendix for the Grassmannian metric with various e Iz' z|.)
w = f(z',, w = f(z) where zz' z 0.1
(15)
4. To global GANSpace: Between Local Basis and the global basis from GANSpace
5. To global SeFa: Between Local Basis and the global basis from SeFa
Fig 8 shows the above five Grassmannian metrics. We report the metric results from 100 samples for the Random (dw) and 1,000 samples for the others. The Projection metric increases in order of Close w, To global GANSpace, Random w, To global SeFa, and Random O(d)). For the Geodesic metric, the order is reversed for Random w and To global SeFa. Most importantly, the Random w metric much larger than Close w. This shows that there is large variation of Local Basis on W, which proves that W-space is globally warped. In addition. Close w metric is always significantly smaller than the others, which implies the local consistency of Local Basis on W. Finally, the metric results prove the existence of limited global disentanglement on W. Random w is smaller than Random (d). This order shows that Local Basis on W not completely random, which implies the existence of global alignment. In this regard, both To global results prove that the global basis finds the global alignment to a certain degree. To global GANSpace lies in between Close and Random To global SeFa does w on the Projection metric. However, the large gap between Close w and both To global implies that the discovered global alignment is limited.
the Geodesic metric and similar Random
CONCLUSION
In this work, we proposed method for finding a meaningful traversal direction based on the local- geometry of the intermediate latent space of GANs, called Local Basis. Motivated by the theoretical explanation of Local Basis, we suggest experiments to evaluate the global geometry of the latent space and an iterative traversal method that can trace the latent space. The experimental results demonstrate that Local Basis factorizes the semantics of images and provides a more stable trans- formation of images with and without the proposed iterative traversal. Moreover, the suggested evaluation of the W-space in StyleGAN2 proves that the W-space is globally distorted. Therefore, the global method can find limited global consistency from W-space.

---

Published as a conference paper at ICLR 2022
ACKNOWLEDGEMENT
This work was supported by the NRF grant [202IR1A2C3010887], the ICT R&D program of MSIT/IITP [2021-0-00077] and MOTIE [P0014715].
ETHICS STATEMENT
The limitations and the potential negative societal impacts our work are that Local Basis would reflect the bias of data. The GANs learn the probability distribution of data through samples from it. Thus, unlike the likelihood- 2014) and Flow-based models (Kingma Dhariwal 2018), the GANs are likely to amplity the dependence between the semantics of data, even the bias of it. Because Local Basis finds meaningful traversal direction based on the local-geometry of latent space, Local Basis would show the bias of data as it is. Moreover, if Local Basis is applied to real-world problems like editing images, Local Basis may amplify the bias of society. However, in order to fix a problem, we have to find a method to analyze it. In this respect, Local Basis can serve as a tool to analyze the bias.
method such as Variational Autoencoder (Kingma & Welling
REPRODUCIBILITY STATEMENT
To ensure the reproducibility of this study, we attached the entire source code in the supplementary material. Every figure can be reproduced by running the jupyter notebooks in notehooks/*. In addition, the proof of Proposition iis included in the appendix.
REFERENCES
Rameen Abdal, Yipeng Qin, and Peter Wonka. Image2stylegan: How to embed images into the stylegan latent space? In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4432 4441, 2019.
Rameen Abdal, Peihao Zhu, Niloy Mitra, and Peter Wonka. Styleflow: Attribute-conditioned exploration of stylegan- images using conditional continuous normalizing flows. ACM Transactions on Graphics (TOG), 40(3):1-21, 2021.
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE transactions on paltern analysis and machine intelligence, 3788)11988-1828 2013.
William M Boothby. An introduction to differentiable manifolds and Riemannian geomelry. Aca- demic press, 1986.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. International Conference on Learning Representations, 2018.
Chia-Hsing Chiu, Yuki Koyama, Yu-Chi Lai, Takeo Igarashi, and Yonghao Yue. nlman--n-tthe-loee differential subspace search in high-dimensional latent space. ACM Transactions on Graphics (TOG), 39(4):85-1, 2020.
Lore Goetschalekx, Alex Andonian, Aude Oliva, and Phillip Isola. Ganalyze: Toward visual defi- nitions cognitive image properties. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 5744 5753, 2019.
Ian J Goodfellow, Jean Pouget-Abadic Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron C Courville, and Yoshua Bengio. Generative adversarial nets. In NIPS, 2014.
Erik Härkönen, Aaron Hertzmann, Jaakko Lehtinen, and Sylvain Paris. Ganspace: Discovering interpretable gan controls. Advances in Neural Information Processing Systems, 33, 2020.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium. Advances in neural information processing systems, 30, 2017.
10

---

Published as a conference paper at ICLR 2022
Ali Jahanian, Lucy Chai, and Phillip Isola. On the" steerability" of generative adversarial networks. In International Conference on Learning Representations, 2019.
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for im- proved quality, stability, and variation. In International Conference on Learning Representations, 2018.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 4401 14110, 2019.
Tero Karras, Miika Aittala, Janne Hellsten, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Training generative adversarial networks with limited data. arXiv preprint arXiv:2006.0666 2020a.
Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyz- ing and improving the image quality stylegan. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 81 101-8119, 19, 2020b.
Daniel Karrasch. An introduction to grassmann manifolds and their matrix representation. 2017.
Diederik P Kingma and Prafulla Dhariwal. Glow: generative flow with invertible 1 convolutions. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, pp. 10236-10245, 2018
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. In Proceedings of the International Conference on Learning Representations (ICLR), 2014.
Or Patashnik, Zongze Wu, Eli Shechtman, Daniel Cohen-Or, and Dani Lischinski. Styleclip: Text- driven manipulation of stylegan imagery. arXiv preprint arXiv:2103. 2021.
David Pfau, Irina Higgins, Aleksandar Botev, and Sébastien Racanière. Disentangling by subspace diffuuion. arXiv preprint 20Xx::2006.19960 2020.
Antoine Plumerault, Hervé Le Borgne, and Céline Hudelot. Controlling generative models with continuous factors of variations. In International Conference on Learning Representations, 2020.
URL /ttl:tttt://sseee //pprrevii//eeeiiieeeeee net/ forum? Hadammmdd--llaaerI
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. In Proceedings the International Conference
Learning Representations (ICLR), 2016.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. Image, 2:T2, 2021.
Aditya Ramesh, Youngduck Choi, and Yann LeCun. spectral regularizer for unsupervised disen- tanglement. arXiv preprint
161, 2018.
Yujun Shen and Bolei Zhou. Closed-fome factorization of latent semantics in gans. In CVPR, 2021.
Yujun Shen, Jinjin Gu, Xiaoou Tang, and Bolei Zhou. Interpreting the latent space of gans for semantic face editing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
2020.
Omer Tov, Yuval Alaluf, Yotam Nitzan, Or Patashnik, and Daniel Cohen-Or. Designing encoder for stylegan image manipulation. ACM Transactions on Graphics (TOG), 40(4): 2021.
Frederik Träuble, Elliot Creager, Niki Kilbertus, Francesco Locatello, Andrea Dittadi, Anirudh Goyal, Bernhard Schölkopf, and Stefan Bauer. On disentangled representations learned from correlated data. In International Conference on Machine Learning, pp. 10401-10412. PMLR, 2021.
Paul Upchurch, Jacob Gardner, Geoff Pleiss, Robert Pless, Noah Snavely, Kavita Bala, and Kilian Weinberger. Deep feature interpolation for image content changes. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7064-7073, 2017.

---

Published as a conference paper at ICLR 2022
Andrey Voynov and Artem Babenko. Unsupervised discovery of interpretable directions in the gan latent space. In International Conference on Machine Learning, pp. 9796. PMLR, 2020.
Binxu Wang and Carlos R Ponce. The geometry of deep generative image models and its applica- tions. arXiv preprint arXiv:2101.06006 2021.
Ceyuan Yang. Yujun Shen, and Bolei Zhou. Semantic hierarchy emerges in deep generative repre- sentations for scene synthesis. International Journal of Computer Vision, pp. 1-16, 2021.
Ke Ye and Lek-Heng Lim. Schubert varieties and distances between subspaces of different dimen- sions. SIAM Journal on Matrix Analysis and Applications, 37(3).1176-1197, 2016.
Fisher Yu, Yinda Zhang, Shuran Song, Ari Seff, and Jianxiong Xiao. Lsun: Construction of large-scale image dataset using deep learning with humans in the loop. arXiv preprini arXiv:1506.03365, 2015.
Jiapeng Zhu, Ruili Feng, Yujun Shen, Deli Zhao, Zhengjun Zha, Jingren Zhou, and Qifeng Chen. Low-rank subspaces in gans, arXiv preprint arXiv: 106. 04488, 2021.
12

---

Published as a conference paper at ICLR 2022
A
PROPOSITION PROOF
Denote (Ve,f) by Then, from w' = Tif(zo c..
w = w + c.Je N(wb, c. JJT)
(16)
The first principal component v is the vector such that v (c. Jc) has the maximum variance.
Var(v] Je)) = _ 11v
(17)
Therefore,
VI argmax Var(vT Je)) = argmax ||JT vll2 |v||==
(18)
||v||==1
Clearly, V1 corresponds to the first right singular vector of JT, i.e. the first left singular vector J, from the linear operator norm maximizing property of singular vectors. Inductively, k-th principal component Vy is the vector such that
VB argmax Var(y Je)) where 1=v
Va 1 (2,. .. Vk-11
(19)
Thus, Vy becomes the k-th left singular vector of J. Therefore, the principal components from the Local PCA problem are equivalent to Local Basis at Ws.
B ALGORITHM
Algorithm Local Basis
Input:
Rds is the input code.
2: f IR + Rdw is the mapping network.
Output: LOCALBBASS((E,/
3: w + f(2)
4: / E Rdwxxz + JACOBIAN(Z, w)
5: U,S,V + SVD(J)
6: return {U,S,V}
Algorithm 2 Iterative Curve-Traversal along positive direction
Input:
Rd2 is the input code.
2: f IRdz + Rdw is the mapping network.
3: k min (dz, dw]] is the ordinal number of direction traverse.
4: I is the total perturbation intensity.
5: N 2 1 the number of iterations.
Output: TTERATTIETTRAVERSALLL...ALLL
6: 20
7: c c ones(dz,1
8: i [0, N) do
9:
U.S, V + LOCAL
if à 0 then
11:
C UT ui-1 arg max((c))
The row number most similar to the previously selected basis.
13:
end if
Us 1i t sign (ck Uk. (ig
Aligns with previous orientation
15:
8: t- Sik
2+41 t- 2i + scN Vi
end for
18: return (z0...., FN}
13

---

Published as a conference paper at ICLR 2022
C MODEL AND COMPUTATION RESOURCE DETAILS
Model We evaluate GANSpace (Härkönen et al. 2020), SeFa (Shen & Zhou 2021). and Local
Basis on StyleGAN2 models for FFHQ (Karras al. 2019) and LSUN (Yu al. 2015) provided by the authors (Karras et al. 2020b).
Computation Resource We generated Latent traversal results on the environment of TITAN RTX with Intel(R) Xeon(R) Gold 5220 CPU @ 2.20GHz. However, it requires low computational cost to get a Local Basis. For example, on the environment of GTX 1660 with Ryzen 5 2600, computing Local Basis takes about 0.05 seconds.
D CODE LICENSE
The files models/wrappers.p notebooks/ganspace.uk pu and notebooks/notebook.uig utills.pyuilppyyy..... are derivative of the GANSpace, and are provided under the Apache 2.0 license. The directory netdissect is a derivative of the GAN Dissection project, and is provided under the MIT license. The directories models/biggan and models/stylegan2 are provided under the MIT license.
14

---

Published as a conference paper at ICLR 2022
E DISTRIBUTION OF SIGULAR VALUES OF JACOBIAN
433
Jac24br
200
133
122
(a) Singluar values of S o $
(b) All Singluar values of
Figure 9: Histogram of the Singular Values of of df₂ for three random and the random matrix. The random matrix is sampled from the Gaussian distribution, then transformed to have the mean and standard deviation of the 100 Jacobian matrix. The sharp peak around zero demonstrates that most of the linear perturbation from z collapses. This observation proves our manifold hypothesis. To better represent the sparsity of singular values, provide the histogram of sigular values of with 0 < of < 2 separately.
F GRASSMANNIAN METRIC
1.0
0.8
eps_0.01
Metric
0.6
Metric
eps 25__
eps_0.1
0.4
ep5_0.5
To global GANSpace
eps_0.00
0.2
eps_0.1
eps, 0525
eps. 0.5
0.0
To global GANSpace
0
10
20
30
40
50
10
20
30
40
50
Subspace
Subspace Dimension
(a) Projection Metric
(b) Geodesic Metric
we vary c. We denote c 1z' z| where w' = f(z'), = f(z). As expected, the Grassmannian metric monotonically increases as we increase e. However, even for case smaller than To global GANSpuce. Therefore, regardless of e, every metric for Close supports our claim for the global warpage W-space. In the main text, we present only case of 0.1. The reported Grassmannian metries, Fig 10 in the supplementary material and Fig 8 in the main text,
Figure 10: Grassmannian metric between two close w, w'
= 0.5, evaluated metric is much
evaluated on the SytleGAN2 model trained on FFHQ.
15

---

Published as a conference paper at ICLR 2022
G MORE LATENT TRAVERSAL EXAMPLES
(a) GANSpace
(b) SeFa
Ramesh 2018)
(d) Local Basis (Ours)
(c) Iterative Curve- Traversal (Ours)
Figure 11: Enlarged figure for Fig 3 Each row represents latent traversal the W-space of the StyleGAN2-FTHO, except for (c). Ramesh et al. (2018) provides the local traversal directions on Z. Except for (e), each traversal image generated by the linear traversal. The latent code is perturbed up to 12 along the 1st and 2nd direction of the corresponding method. The perturbation intensity linearly increased from 0 12 for each column. Since Ramesh al. (2018) is defined on 2, we downscaled the perturbation intensity by the singular values from Local Basis for fair comparison. In the case the existing methods, the quality of the image gets severely degraded as perturb stronger. On the other hand, Local Basis shows relatively stable traversal.
16

---

Published as a conference paper at ICLR 2022
(a) Local basis (Ours)
(b) Iterative Curve-Travense (Ours)
(c) GANSpace
(d) SeFa
Figure 12: Additional Robustness Test results of each Latent Traversal methods along the first 10 components. For each traversal methods, each row corresponds to latent traversal of perturbation up to 12. Compared the global methods, GANSpace and SeFa, even Local Basis with linear traversal (Fig 12a shows more stable traversal on images. Moreover, Local Basis with Iterative 12b) rarely shows any collapse under the latent traversal of 12 along the curve.
17

---

Published as a conference paper at ICLR 2022
(a) StyleGAN2 FFHQ
(b) StyleGAN2 FFHQ
(c) StyleGAN2 LSUN Cat
(d) StyleGAN2 LSUN Car
Figure 13: Additional examples of Semantic Factorization without layer restriction. i.c. Latent traversal along the first 10 components of Local Basis with moderate perturbation of up to 5. Local Basis finds diverse and natural- natural-loooing semantic variations on each dataset.
18

---

Published as a conference paper at ICLR 2022
Figure 14: Iterative Curve-Travere guided by global basis only at departure from StyleCIIP for the semantics of old. Contrary to Fig 7 Iterative Curve-Traversal follows the global basis only at departure. After that, the departure direction is chosen by the similarity to the previous departure direction. Left: Linear traversal along global basis. Middle: Iterative Curve-Traversal of fixed stepsize (Stepsize = (0.02, 0.04, 0.08, 0.16)). Right: Stochastic Iterative Curve-Travents (Stepsize sampled from Uniform Noise on [0.05, 0.15))
H IMPLEMENTATION DETAILS FOR SEC 3.4
In Sec 3.4, we utilize the global basis from StyleCLIP Patashnik et al. 2021 defined on W+, the layer-wise extension of W introduced in (Abdal et al. 2019 Patashnik al. 202T). To be more specific, since the synthesis network in StyleGAN has 18 layers, we obtain an extended latent code w W defined by the concatenation latent codes W, W of dimension 512 for each i-th layer and it can be described as follows:
w (w W2 Wis) E 512x18
(20)
Note that our Iterative Curve-Traversan originally defined on W has a canonical extension W+ without additional changes in structures or methodologies.
For implementing the stochastic Iterative Curve-Traversal introduced in See 3.4, we firstly find global basis W using StyleCLIP (Patashnik et al.) 2021) which implies given semantic attribute in the form of text (e.g. old). Now denote the global basis by vt global which can be
represented as follows:
v+
global = qyoloba
v22 globbll
vis globbll
(21)
Then we perform the (extended) Iterative Curve-Traversal following v+ stochastic movement for each step. In practice, we consider two options to choose a traversal di- rection for each step; First, follow the direction most similar to the previously selected basis (as Algorithm 2], except for the first iteration. Note that we compute the similarity between the local basis and the global basis only once when choosing the first traversal direction. Second, follow direction most similar to the given global basis. This slightly different from our Algorithm 2, however, we empirically verify that setting the exploration in that way leads to more desirable image change.
global equipped with
Fig 14 shows that the first method still preserves the image quality well, but does not guarantec that the desired direction image change, namely "old'. We speculate the reason why such phenomenon occurs is that most of the information contained in the meaningful global basis disappears after the first step (a unique, direct comparison to the global basis), although our methodology guarantees that escape from the manifold and achieve high image quality. Nevertheless, Fig 15 shows that the second method for the stochastic Iterative Curve- Traversal can change a given facial image in a very high quality and various ways.
the latent code does
19

---

Published as a conference paper at ICLR 2022
Figure 15: Additional Examples of Stochastic Iterative Curve Traversal guided by the global basis from StyleCLIP for the semantics of Old. Left: Linear traversal along global basis. Right: Stochastic Iterative Surve-Traversal
20

---

Published as a conference paper at ICLR 2022
(a) GANSpace
(b) ScFa
(c) Local Basis (Ours)
Figure 16: Subspace traversal with two directions W-space of the StyleGAN2. The horizontal (red box) and vertical (green box) axes correspond to the 1st and 2nd directions of each method.
1 SUBSPACE TRAVERSAL
In Section we proved that the W-space in StyleGAN2 is warped globally. Specifically, the sub- space of traversal direction generating principal variation in the image changes severely as we vary the starting latent variable To verify the claim further, visualize the subspace traversal on the latent space W. The subspace traversal denotes a simultaneous traversal in multiple directions. In this paper, visualize the two-dimensional traversal,
Subspace Truversall.,)x..0 v) G
y N
(22)
N
where w = fz) and denotes a subnetwork the given GAN model from W the images space X. Since the disentanglement into the linear subspace implies the commutativity of transformation (Píau et al.] 2020), the subspace traversal can be a more challenging version of linear traversal experiments.
Fig 16d Fig 17 17 show results of the subspace traversal for the global basis and Local Basis. Starting from the center, the horizontal and vertical traversals correspond to the 1st and 2nd directions of each method. The same perturbation intensity per step is applied for both directions. When restricted to the linear traversal (red and green box), the GANSpace shows relatively stable traversals. However, the traversal image deteriorates at the corner of the subspace traversal. By contrast, Local Basis shows stable variation during the entire subspace traversal. This result proves that the global basis is not well-aligned with the local-geometry of the W manifold.
21

---

Published as a conference paper at ICLR 2022
(a) GANSpace
(b) ScFa
(c) Local Basis (Ours)
Figure 17: Subspace traversal with two directions on W-space of the StyleGAN2. The horizontal (red box) and vertical (green box) axes correspond the 1st and 2nd directions of each method.
22

---

Published as a conference paper at ICLR 2022
J LOCAL BASIS ON OTHER MODELS
(a) GANSpace
(b) Local Basis (Ours)
(c) GANSpacc
(d) Local Basis (Ours)
Figure 18: Comparison of GANSpace and Local Basis on StyHeGAN-FFHQ Karras et al., 2019) Each traversal image is generated along the first 10 components of each method with per- turbation of up to 5.
23

---

Published as a conference paper at ICLR 2022
(a) GANSpace
(b) Local Basis (Ours)
(c) GANSpace
(d) Local Basis (Ours)
Figure 19: Comparison of GANSpace and I Local Basis on BigGAN-512 (Brock et al. 2018) Each traversal image is generated along the first 10 components of each method with perturbation of up to 3.
24


Reviews:
[{"review_id": "X3QIRorDDLb", "reviewer": "Reviewer_v9kk", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The idea is simple and convincing. The key is to use the principal subspace of the Jacobian to extract  attribute representations. The novelty lies in two aspects: 1) the authors study the Jacobian of the mapping network, as opposed to previous works and 2) the editing path is iteratively formed by taking the most similar direction  of adjacent tangent spaces (principal subspaces of the Jacobians). \n\nA very relevant NeurIPS work is missed.\n\nLow-Rank Subspaces in GANs\nhttps://arxiv.org/abs/2106.04488\n\nIn this paper, the authors reviewed the related work using the Jacobians. Please cite those related papers. In this work, the authors show that the principal subspace of the Jacobian may fail to capture disentangled attribute directions and result in artifacts.  They use the null space of the Jacobian with respect to complementary region  to improve the performance. Please discuss this problem in your paper.", "summary_of_the_paper": "The authors proposed a geometric method for image editing with pretrained StyleGAN2. The idea is simple. Editing is performed by constraining the editing path on the manifold. The simplest way of making this is to use the tangent space. A natural realization of the tangent space is the principal subspace of the Jacobian. The authors harness the Jacobian regarding the mapping from $z$ to $w$. The extensive experimental results verify the effectiveness of the method.", "summary_of_the_review": "A simple and effective algorithm supported by geometry. But the author did not seriously discuss the drawback revealed by the relevant paper. This part should be improved.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "RTF401FlfG", "reviewer": "Reviewer_Pg5m", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Disentanglement of a representation often stems from intuitions gathered from linear models, namely that we should expect that different aspects of data may change as we move along a chosen Euclidean coordinate axis (similar to principal components). This paper challenges this view and argue that we should only locally follow a specific direction, and that this direction should come from the subspace spanned by the top singular vectors of the Jacobian of the generator (the specific paper investigate GANs, but the idea is general). It is trivially seen (Proposition 1) that if we consider samples from a 'small' Gaussian in latent space, then looking at the directions of the top singular vectors of the Jacobian amounts to local PCA in observation space (follows from Taylor's theorem).\n\nThis analysis is sound and reasonable. It is, however, not particularly novel. In the context of VAEs (rather than GANs), there is a large body of work that equip the latent space with the Riemannian pull-back metric associated with the generator, i.e. $G(z) = J(z)^T J(z)$, where $J$ is the Jacobian, and $G$ is then the metric tensor. Existing work then study the general identifiability problem associated with latent variable models; this is arguably a more general problem than the disentanglement problem. Riemannian quantities such as geodesics then serve similar purposes to the traversels proposed in the present paper, though geodesics are perhaps better understood. I speculate if indeed the proposed traversel method isn't just a first-order Euler method for solving the initial value problem associated with geodesics.\n\nRelated work in this regard (these groups of authors has published significantly more on the topic, but I just list the most commonly cited papers):\n1. Latent Space Oddity: on the Curvature of Deep Generative Models. G. Arvanitidis, LK. Hansen & S. Hauberg. ICLR 2018. This was the first paper to study the Riemannian geometry of VAEs; a key technical difference to the paper at hand is that Arvanitidis et al. considered stochastic generators, while the current paper considers deterministic generators.\n2. N. Chen, A. Klushyn, R. Kurle, X. Jiang, J. Bayer & P. Smagt. Metrics for deep generative models. AISTATS 2018. Similar to the ICLR paper above except it considers deterministic generators.\n3. H. Shao, A. Kumar & PT. Fletcher. The Riemannian geometry of deep generative models. CVPR Workshops 2018. Similar to paper 2.\n\nThese papers focus on the identifiability problem and not the disentanglement problem, so in that regard the present paper has a novel contribution. This, however, seems rather minor as identifiability is arguable a more general and harder problem.\n\nI have already mentioned that the traversal algorithm seems quite similar to geodesic computations, e.g. akin to the algorithm proposed in paper 3 above. It also seems very similar to the Brownian motion / diffusion sampling proposed in this paper:\n\n4. Variational Autoencoders with Riemannian Brownian Motion Priors. D. Kalatzis, D. Eklund, G. Arvanitidis & S. Hauberg. ICML 2020\n\nThe main difference seems to be that paper 4 solves and SDE (they add a bit of Gaussian noise in each step), while the present seems more like an ODE solver.\n\nThe current paper, like so many other disentanglement papers, has a somewhat ad hoc empirical evaluation strategy. This is fair given the complexity of the topic. Results seems encouraging, but it's really difficult to get a feel for which approaches are better. I don't think this is the authors fault, but it still seems a bit disappointing.\n\nI should also note that I did not really understand the point of the warpage evaluation. It isn't clear to me what the authors aim to achieve in this part of the paper.\n\nAll that being said, I think the paper is nicely presented and the empirical results are generally quite nice (unlike the above papers that are more proof-of-concepts).\n\n*Post-rebuttal*\nI think the authors answered my key concerns during our discussions, and I trust that the paper will be adjusted accordingly. I have therefore bumped my score up a bit to recommend acceptance.", "summary_of_the_paper": "The paper propose to explore disentanglement locally through the top singular vectors of the Jacobian of the generator of a GAN. This contrast the popular view that disentanglement should follow Euclidean coordinate axes in the latent space. An algorithm for traversing the latent space is proposed.", "summary_of_the_review": "The key ideas behind the paper has appeared in a series of papers elsewhere, and these arguably provide a deeper understanding of the problem than what is discussed here. That makes it hard to recommend acceptance. That being said, the paper is well-written and the empirical results are interesting, hence the borderline/reject score.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "FXmWE9JdKP_", "reviewer": "Reviewer_sKqG", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, they did well in this work and are high in potential. \n\nSTRENGTHS:\n- End-to-end, the paper stays focused on a consistent theme via a well-organized summary of their work.\n- Visualizations show concepts well\n- There is a nice distribution of research paper components (e.g., figures, tables, proper intro, background, conclusion, and such).\n- The idea is interesting and relevant.\n\nWEAKNESSES:\n- Figures have text that is difficult to read (text in figures should match the font that is used in the paper's body, in style and size).\n- Equations look okay, but there are several components that should not be variable (i.e., italicized). For instance, \"Local Basis (...)...\" in Eq 4 should be wrapped in \\text{Local Basis}-- there are many instances like this. Make sure only variables are italicized in equations.\n- Fig 1, 4, 6, 8 are low in quality. Increase the resolution and be sure to use vectorized images (e.g., PDF).\n- They do not reference Figure 1 in the text. Also, the first figure referenced in the text is in the supplemental material. Referring to supplemental to further support claims is okay, however, it seems like Fig 9 should be in the major paper (i.e., or they should change the paragraph referring to it not to depend on the supplemental)\n- Figures are not well explained\n   - the conditions for which they interpolated other methods are unclear-- why some cases there are clear variables being swept (e.g., pose of the face), but in others, it is not quite clear. Why does this vary from method to method to the extent it does? Sure, the proposed might work well at staying within photorealistic ranges but is the reason that the other methods do not be that sweeping used was not optimal for each other SOA method\n-- ", "summary_of_the_paper": "The authors proposed Local Basis: a purposeful traversal direction based on the local-geometry of the intermediate latent space of a GAN. To support the theory of Local Basis, the global-geometry of the latent space, along with an iterative traversal method to trace the latent space, moves apart from the space in a piece-wise, step-by-step manner. Their results show that Local Basis decomposes an image's semantics, providing more stability in transforming images via the proposed iterative traversal. Following the suggested evaluation of the W-space of StyleGAN2 reveals that the W-space is distorted at the global level: the global method, thus, limits global consistency while in W-space.\n", "summary_of_the_review": "The paper is a marvelous piece: well put together, professionally presented, relevant and interesting topic, excessive experiments and analysis, and, mostly, well written.\n\nDISCLAIMER: the mathematics in this paper I am not too familiar with-- there appeared to be consistent use in variables, each properly defined, and expressed in the body of text coherently. If another review can validate the math, it would be great (i.e., my review does not include deeply validating the math, for low-rank I have minimal experience with.\n\nMy primary concern is that the analysis is limited (not in figures, but in an explanation). Also, the story could be told better. Note that the two aforementioned concerns go hand in hand: with a better description of the image montages (i.e., what to look for, highlight the pros, also show some failure modes). Share thoughts on why results are better; same with failure modes (which are not currently highlighted).\n\nThus, the figures are sufficient, but the intuitive insight by breaking the figures and plots down is severally lacking (both in-text body, caption, and in the visual themselves). This would likely yield a better story, end-to-end, for it might bring the motivation full circle: as is, I am not seeing it go full circle, as all the best stories do.\n\nWith that, assuming the math is correct (per another reviewer validating), the pros slightly outweigh the cons, so I am neutral (or slightly above, since there is no middle chose :) ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The initial reviews for this paper were diverging. After the rebuttal all reviewers have reached the consensus of recommending the paper's acceptance. Some reviewers have concerns regarding the novelty of the paper, however they appreciate that the paper is ell written and the empirical results are interesting. Following the reviewers recommendation, the meta reviewer recommends acceptance. In the final version of the paper the authors are encouraged to strengthen the weaknesses discussion as requested by one of the reviewers.
Decision: Accept (Poster)