Finding the Global Semantic Representation in GAN through Fréchet Mean
Authors: Jaewoong Choi
Geonho Hwang
Hyunsoo Cho
Myungjoo Kang
Venue: ICLR 2023 poster (2023)
Keywords: generative adversarial network
disentanglement
semantic factorization

Abstract: The ideally disentangled latent space in GAN involves the global representation of latent space using semantic attribute coordinates. In other words,  in this disentangled space, there exists the global semantic basis as a vector space where each basis component describes one attribute of generated images. In this paper, we propose an unsupervised method for finding this global semantic basis in the intermediate latent space in GANs. This semantic basis represents sample-independent meaningful perturbations that change the same semantic attribute of an image on the entire latent space. The proposed global basis, called Fréchet basis, is derived by introducing Fréchet mean to the local semantic perturbations in a latent space. Fréchet basis is discovered in two stages. First, the global semantic subspace is discovered by the Fréchet mean in the Grassmannian manifold of the local semantic subspaces. Second, Fréchet basis is found by optimizing a basis of the semantic subspace via the Fréchet mean in the Special Orthogonal Group. Experimental results demonstrate that Fréchet basis provides better semantic factorization and robustness compared to the previous methods. Moreover, we suggest the basis refinement scheme for the previous methods. The quantitative experiments show that the refined basis achieves better semantic factorization while constrained on the same semantic subspace given by the previous method.

Published as a conference paper at ICLR 2023
FINDING THE GLOBAL SEMANTIC REPRESENTATION IN GAN THROUGH FRÉCHET MEAN
Jaewoong Choi
Geonho Hwang, Hyunsoo Cho, Myungjoo Kang"
Korea Institute for Advanced Study ch chjw1475e k ass kr
Seoul National University {hgh2134, hschol 00, mkang ( ) acc. ac.ke kr
ABSTRACT
The ideally disentangled latent space in GAN involves the global representation of latent space with semantic attribute coordinates. In other words, considering that this disentangled latent space is vector space, there exists the global semantic basis where each basis component describes one attribute of generated images. this paper, propose an unsupervised method for finding this global semantic basis in the intermediate latent space in GANs. This semantic basis represents sample-independeer meaningful perturbations that change the same semantic at- tribute of an image on the entire latent space. The proposed global basis, called Fréchet basis, derived by introducing Fréchet mean to the local semantic pertur- bations in a latent space. Fréchet basis is discovered in two stages. First, the glohal semantic subspace is discovered by the Fréchet mean in the Grassmannian manifold of the local semantic subspaces. Second, Fréchet basis is found by optimizing a basis of the semantic subspace via the Fréchet mean in the Special Orthogonal Group. Experimental results demonstrate that Fréchet basis provides better seman- tic factorization and robustness compared the previous methods. Moreover, we suggest the basis refinement scheme for the previous methods. The quantitative experiments show that the refined basis achieves better semantic factorization while constrained on the same semantic subspace given by the previous method.
INTRODUCTION
Generative Adversarial Networks (GANs, (Goodfellow et al. 2014)) have achieved impressive success in high-fidelity image synthesis, such as ProGAN (Karras et al. 2018). BigGAN (Brock al. (201)), and StyleGANs (Karras af.] 2019 2020a b 2021 Ij. Interestingly, even when a GAN model is trained without any information about the semantics of data, its latent space often represents semantic property of data (Radford et al. 2016 Karras et al. 2019). To understand how GAN models represent the semantics, several studies investigated the disentanglement (Bengio al. 2013 property of latent space in GANs (Goetschalckx et al. 2019 Jahanian et al. 2019 Plumerault et al. 2020 Shen et al. 2020). Here, latent space in GAN is called disentangled if there exists optimal basis of the latent space where each basis coefficient corresponds to one disentangled semantics (generative factor).
One approach to studying the disentanglement property is to find meaningful latent perturbations that induce the disentangled semantic variation on generated images (Ramesh al. 2018: Härkönen et al.) 2020; Shen & Zhou 2021 Choi et al., 2022b). This approach can be interpreted as investigating how the semantics are represented around each latent variable. We classify the previous works on meaningful latent perturbations into local and global methods depending on whether the proposed perturbation is -eepeeedeeeneent or sample-ignorant. In this work, we focus on the global methods (Härkönen et al.) [2200] [Shen& & Zhou (021). If the latent space is ideally disentangled, the optimal semantic basis becomes the global semantic perturbation that represents a change in the same generative factor the entire latent space. In this regard, these global methods are attempts to find the best-possible semantic basis on the target latent space, Throughout this work, the semantic subspace represents the subspace generated by the corresponding semantic basis.
"Corresponding author

---

Published as a conference paper at ICLR 2023
Gr (d2
Tur Wal
Pa
R
sD(dw)
MC Mun
B, M,O
Pio
Figure I: Overview of Fréchet basis. The global semantic subspace S, is defined the Fréchet mean of intrinsic tangent spaces Tw, Wiww in the Grassmannian manifold Gr(dw, R&w). Fréchet basis B, is discovered by selecting the optimal basis of S, using the Fréchet mean in the Special Orthogonal Group.
In this paper, we propose an unsupervised method for finding the global semantic perturbations a latent space GAN, called Fréchet Basis. Fréchet Basis is based on the Fréchet mean on the Riemannian manifold. Fréchet mean is generalization of centroid to the general metric space (Fréchet 1948) and the Riemannian manifold is the metric space (Lee 2013). In particular, Fréchet Basis discovered in two steps (Fig Dp. First, find the global semantic subspace S, of latent space as Fréchet mean in the Grassmannian manifold (Boothby 1986) of the intrinsic tangent spaces. Here, the intrinsic tangent space represent the local semantic subspace (Choi et al. 2022b). Second, Fréchet basis B, is discovered by selecting the optimal basis of S, via Fréchet mean in the Special Orthogonal Group (Lang 2012). Our experiments show that Fréchet basis provides better semantic factorization and robustness compared to the previous unsupervised global methods. Moreover, the second step in finding Fréchet basis provides the basis refinement scheme for the previous global methods. In our experiments, the basis refinement achieves better semantic factorization than the previous methods while keeping the same semantic subspace. Our contributions are as follows:
We propose unsupervised global semantic perturbations, called Fréchet basis. Fréchet basis is discovered by introducing Fréchet mean to the local semantic perturbations.
2. We show that Fréchet basis achieves better semantic factorization and robustness compared to the previous global approaches.
3. We propose the basis refinement scheme, which optimizes the semantic basis on the given semantic subspace. We can refine the previous global approaches by applying the basis refinement on their semantic subspaces.
RELATED WORKS AND BACKGROUND
Latent Perturbation for Image Manipulation The latent space of GANs often represents the semantics of data when the model is trained without the supervision of the semantic attributes. Early approaches to understanding the semantic property of latent space showed that the vector arithmetic on latent space leads to the semantic arithmetic on the image space (Radford al. 2016 Upchurch et al. 2017). In this regard, a line of research has been conducted to find meaningful latent perturbations that perform image manipulation disentangled semanties. We categorize the previous works into local and global methods according to sample dependency. The local method finds meaningful perturbations for each latent variable for one semantic attribute, e.g., Rammesh et al. normalizing
(2018), Latent Mapper in StyleCLIP (Patashnik et al., (021), flow in StyleFlow (Abdal et al. 2021). Local image editing in Zhu et al. al. (2021). and Local Basis (Choi et al.) 2022b). By contrast, the global methods offer the sample-independent meaningful perturbations for each latent space, e.g., Global directions in StyleCLIP (Patashnik et al.] 2021) GANSpace (Härkönen et al. 2020), and SeFa (Shen & Zhou, 2021). I). Among them, StyleCLIP is supervised method requiring text descriptions of each generated image to train CLIP model (Radford) et al. 2021). Throughout this paper, investigate unsupervised methods such GANSpace and
2

---

Published as a conference paper at ICLR 2023
ScFa. GANSpace (Härkönen et al. 2020) suggested the principal components of latent space obtained by performing PCA as global meaningful perturbations. SeFa (Shen & Zhou 2021) proposed the singular vectors of the first weight matrix as global disentangled perturbations.
Unsupervised global disentanglement score The disentanglement latent space is expressed as the correspondence between the semantic attributes of data and the axes of latent space. Because the definition of disentanglement depends on attributes, most of the existing disentanglement metrics for latent spaces are supervised ones, e.g., DCI score (Eastwood (Higgins et al. 2017), and FactorVAE metric (Kim & Mnih 2018). They require the attribute annotations of generated images. This requirement restricts the broad applicability of disentanglement evaluation on real datasets. To address this restriction, [Choi et al. (022a) proposed an unsupervised global disentanglement score, called Distortion. Distortion metric measures the variation of tangent space on the learned latent manifold W. Hence, Distortion metric relies purely on the geometric property the latent space and does not require attribute labels.
Williams 2018), 3-VAE metric
Background Choi al. 2022b) suggested framework for analyzing the semantic property of intermediate latent space by its local geometry. This analysis is performed on the learned latens manifold W f()), where f denotes the subnetwork f from the input noise space 2 to the target latent space. Here, we assume Z is the entire Euclidean space, i.c., Z = Rd for some dz. Note that this is satisfied for the usual Gaussian prior p(z) (O, Láz) ). Choi et al. (2022b) proposed a method for finding the k-dimensional local approximation W of W around w f(z) E W. This local approximation W is discovered by the low-rank approximation problem of dfz and this solution is given by SVD. Then, W is given as follows: For the i-th singular vector U E Rdz, v."" E Rdw, and i-th singular value of E R of dfz with of 2 ... 2 oth and m = min(dz, dw).
df (ff) of v.y" for Vi,
Local Basis(w f(z)) (v.' hisshm
Whth 5
1 E (-G,,), for siisk 2 Σ 4 u
Tw W = span :1 : $i.
Choi et al. (2022b) showed that the codomain singular vectors, called Local Basis (Eq serve as the
local semantic perturbations around w. In this respect, the tangent space at w represents the local semantic subspace because it is spanned the local semantic perturbations (Eq 3)
Upon this framework, Choi et al. (2022a) proposed the intrinsic local dimension estimation scheme for the latent manifold W through the robust rank estimate (Kritchman & Nadler 2008) of dfz- Geometrically, the intrinsic local dimension represents the number of dimensions required to properly approximate the denoised W. [Coi et al. (2022a) showed that this local dimension corresponds to the number of local semantic perturbations. Using this correspondence, Choi et al. (2022a) introduced the unsupervised disentanglement score called Distortion. Distortion metric is defined as the normalized variation of intrinsic tangent space on the latent manifold. The normalized variation is expressed as the ratio of the distance between two tangent spaces at two random E W (Eq 4 to the distance between two tangent spaces at two close w (Eq 5 The distance between tangent spaces is measured by the dimension-normalizee Geodesic Metric dis "go Choi et al. (2022a) in Grassmannian manifold
Boothby (986). Specifically, Distortion of W is defined as Dw Trand/ Ttocat with
Frand Ey,
[dt (.)= (Tw, We Ww Tw2 W for k= = min(ki,k))) [1 (Tw, We, : Tw, W& for 1: = min(ki, k2)
w.=
(4)
Tocat = -p2
(5)
where k; denotes the local dimension estimate at w: = f(z)). Interestingly, although Distortion metric does not exploit any semantic information, Distortion metric provides strong correlation between the supervised disentanglement score and the glbal-basi--compatiblliy (Choi et al. 2022a)
FRÉCHET MEAN GLOBAL BASIS
In this section, we propose unsupervised method for finding global linear perturbation directions that make the same semantic manipulation on the entire latent space, called Fréchet basis. If we have such global meaningful perturhations, the vector space representation of latent space along these
3

---

Published as a conference paper at ICLR 2023
semantic basis provides the global semantic representation of a model. The proposed scheme is based on finding the Fréchet mean (Fréchet, 1948: Karcher 1977) of the local disentangled perturbations. The scheme is in two steps. First, the optimal subspace representing the global semantics is discovered by the Fréchet mean in the Grassmannian manifold (Boothby 1986) of intrinsic tangent spaces (Choi et al. 2022b) the target latent space. Second, the optimal basis is obtained as the Fréchet mean in the Special Orthogonal Group (Lang 2012) of the projected local disentangled perturbations.
3.1 METHOD
Notation Throughout this work, we follow the notation presented in See 2 Let W R be ambient target latent space where we want to find global semantic perturbations. We analyze the learned latent manifold W = f(2) C W embedded in this latent space, which is given as an image of the subnetwork from the input noise 2 to W.
Motivation We investigate the problem of discovering global semantic perturbations through the local geometry of learned latent manifold W. Recently, Choi al. (2022a) discovered that the intrinsic tangent space Tw W at each w E W represents the local semantic variation of the generated image from w. Specifically, the intrinsic local dimension at w. denoted as k in Tw We corresponds to the number of local semantic perturbations. The top-k components of Local Basis Choi et al. 2022b) are these local semantic perturbations and are the basis vectors of T W (Eq 3) Hence, the intrinsic tangent space Tw W describes the local semantic variation of an image because it is spanned by the local meaningful perturbations.
In this regard, we interpret the global semantic variation as the mean of these local semantic variations. One of the most popular methods for defining the mean of subspaces through Fréchet mean (Marrinan et al.] 2014). Fréchet mean is generalization of the mean in vector space to the general metric space. The mean of vectors is the minimizer of the sum of squared distances to each vector. Similarly, Fréchet mean pfrr in the metric space X with metric d is defined as the minimizer of squared metrics, i.c., for I1, 22, Zp E X,
Σ d(, 15i5n
Pff arg min
μEX
In particular, a Riemannian manifold is an example of metric space where we can introduce the Fréchet mean (Lou et al. 2020). In this work, we utilize the Grassmannian manifold (Boothby) 1986) to find the subspace of latent space for the global semantic representation and the Special Orthogonal Group (Lang 2012)
choose the optimal basis on it.
3.1.1 GLOBAL SEMANTIC SUBSPACE
Our goal is to find Riemannian manifold where we can embed these intrinsic tangent spaces (Tw, Wh: w: Jisien describing local semantic variations at each w,. The Grassmannian manifold Gr(k, V) denotes the set of k-dimensional linear subspaces of vector space V (Boothby) 1986). Hence, these tangent spaces can be embedded to one Grassmannian manifold Gr (dw, Rw) by matching their dimensions to the dimension of learned latent manifold dw. Specifically, we match the dimensions of tangent spaces by refining or extending them to the subspaces spanned by the top-dw components of Local Basis (Eq 3.. This is equivalent to approximating W with the dw-ddmeesional local estimate Whw at all w E W (E We estimate the layer-wise dimension dy of learned latent manifold W by averaging local dimensions [kt Mishssnn of n i..d. samples,
Twi Wayy span (v :1sis dw} E Gr (dw, Rdw).
(7)
where vi"h denotes Local Basis at w, (Eq D. Then, we define the global semantic subspace S, of W as the Fréchet mean on Gr (dw, ( with the geodesic metric dyeo (Ye & Lim 2016):
S, = arg min peGrldw IsiS
Σ dgea (M, Tw. Wdw)
(8)
Here, the geodesic metric dgeo is defined dgeo (W, (ΣLn 0?
1/2
for W, W' E Gr(k,
where 0g denotes the i-th principal angle between W and W That is, 0. cos cos- (a, Mw:))
4

---

Published as a conference paper at ICLR 2023
where Mw Rn** denotes the column- concatenation of orthonormal basis for W. For optimization, we used the gradient descent algorithm in the Pymanopt (Townsend et al. 2016).
3.1.2 GLOBAL SEMANTIC BASIS
The aim of this work is to find global meaningful perturbations that represent the disentangled semantics. However, the global semantic subspace S₂ is discovered by solving an optimization problem in the Grassmannian manifold, the set of subspaces. We need an additional step to find a specific basis on S,. In particular, utilize the Fréchet mean the Special Orthogonal Group.
Why Special Orthogonal Group Let the columns of Ms. M's E IR Xdw be the two distinct orthonormal basis of S., Then, there exists an orthogonal matrix O E IRd i.c., O 00T I, which satisfies M's MgO. Therefore, finding the optimal basis of S equivalent to finding the orthogonal matrix O given the initial Mg. The Special Orthogonal Group SO(n) consists of n x n orthogonal matrices with determinant +1 (Lang) 2012). Note that the determinant of an arbitrary orthogonal matrix is +1 or -1. We consider 50(n) instead of the orthogonal matrices for two reasons. First, our task is independent of the flipping 1----mulliplication of each basis component. The positive perturbation along is identical to the negative perturbation along -v. The flipping of a basis component in M's leads to the flipping of the corresponding column in Ó. This results in the 1-1)-mulliplicatiinn may assume that O is a special orthogonal matrix. Second, the Orthogonal Group is disconnected while the Special Orthogonal Group is connected. Hence, the Orthogonal Group is inadequate for finding the Fréchet mean, which is optimized by the gradient descent algorithm.
the determinant of O. Therefore, without loss of generality,
Basis Refinement We propose the optimization scheme for finding the global semantic basis from the global semantic subspace S., Here, we denote the column-wise concatenation of local semantic basis at each w, as Mw, E IR w xdw i.e., each column is the top-dy Local Basis at w E W. Note that the column space of Mw. is the local semantic subspace Tw, Way,, Likewise, Ms refers to an initial orthonormal basis of S.. As an overview, the proposed scheme is as follows:
(i) Project each local semantic basis to the dy--dimensional global semantic subspace S..
Project these projected local semantic basis to (0(d)).
(iii) Find the Fréchet mean O in (O(dw) and embed O back to the ambient space Rdw.
Before the above optimization, we preprocess each local semantic basis at w., i.e., the columns of Mw,- to be positively aligned to each column of Ms, i.e., (Mw, [= :]. Msl:, >> O for all i. (i) As a first step, we project each local semantic basis at w, onto the global semantic subspace S,, i.c.,
M Mw.- (ii) Then, the matrix of projected local semantic basis M Mw. Rdwxdd projected (Oddw)) II The projection
the orthogonal group P. and the proposed projection on the Special
Orthogonal Group P.o can be obtained via SVD. (See the appendix for proof.): Let X = UUVT be SVD of X IRdwxdy Xdw
(x) U diag (1, 1, det (Po(X))) V where Po(X) = UV
(9)
(iii) Finally, we find the optimal orthogonal matrix O, that transforms the initial basis Ms to the global semantic basis Bg, via Fréchet mean of projected local semantic basis {Peo (M3 Mw.)]r C (ddw)).
B, MsO where O
arg min (eSO(dw) isisn
Σ d (p, Poo (MJ Mw.))²²
(10)
Here, the Riemannian metric on (Odw) is defined as d(X1, X2) = Skew (oo( || for X2, E (O(dw) where log denotes matrix logarithm and Skew refers to the skew-symmetric component of matrix, i.c., Skew(X,) utilized the Pymanopt (Townsend et al. 2016) implementation for finding the Fréchet mean on SO(dw). Then, the global semantic basis B, is obtained the embedding of Fréchet mean O to Rdw, i.e., B, = MgO. We call this global semantic basis 15, Fréchet Basis.
1/ (X, X,T). As in the Grassmannian manifold, we
'This projection is underdetermined for the matrix with determinant O hecause of the subspace generated by singular vectors with O. Because these matrixes are measure-zero set, this did happen in practice.
5

---

Published as a conference paper at ICLR 2023
20.0
17.5
0.58
16.0
0.58
12.5
10.0
0.54
7.5
3
0.52
4
5
0.48
9.4
9.6
9.8
10.0
10.2
10.4
10.6
10.8
9.25
9.50
9.75 10.00 10.25 10.55 Distortion
Distortion
(a) StyleGAN2-cat
0.95
(b) StyleGAN2-e p p= 0.91
Figure 2: Correlations of T D.aDDitortion (L) to FID gap ()) and DCI (f) when pre 0.005. L' D'istorrtio shows the correlations of 0.98 to FID-Gap in StyleGAN2-cht and of -0.91 DCI in StyleGAN2-e. (See the appendix for full correlation results in six models.)
3.2 FRÉCHET BASIS AS T T.-DDISTOTION MINIMIZER
The Fréchet basis can be interpreted as the minimizer of the unsupervised disentanglement metric, Distortion (Choi et 2022a). Distortion metric is based on the inconsistency of intrinsic tangent spaces. Specifically, Distortion metric Dw is the ratio between the inconsistency at two random E W and at two close tangent space represents the local semantic variations. From this point of view, the Distortion-basee global semantic subspace Sp would representative of these tangent spaces that minimize inconsistency to each tangent space.
W, i.e., Dw rand/ Ioccl (Eq and 5). Here, the intrinsic
SD arg min (p)) with Igtobat(14) = Exap(()) f()( [dgon (1t, T (w))
(11)
S R"
where y a subspace of WR l refers to the local dimension at w and n² denotes the k-dimennsioal refinement μ. The Fréchet basis assumes that the entire latent manifold W is approximated with dw-dimensional local estimate at all w E W. Under this assumption, (1) becomes
Iglobal(p) (p) (1/v dw)
Egzpp(x) w=f(2) w= [dgeo (M, Tw W(w)] for Gr(dw, R(). (12)
The comparison with Eq 8 shows that the global semantic subspace by Fréchet mean S, can be interpreted as T.2-Distortion minimizer, i.e., de instead of dgee- Although the original T, Disttortti was proven to provide high correlations with the disentanglement score, _ L--Distortion was not tested (Choi et al. 2022a). Therefore, we evaluated whether L'=Distortion is also meaningful metric to verily the validity of minimizing it. Following the experiments Choi et al. (0022)), we assessed the global- sesual-bsiis-ooe masis-commatibilityy by the FID (Heusel et al. 2017) Gap between Local Basis and GANSpace (Härkönen et al. 2020) under the same perturbation intensity. Also, DCI score (Eastwood & Williams) 2018) is adopted as the supervised disentanglement score. We utilized 40 binary attribute classifiers pre-trained on CelebA (Liu et al.) 2015) to annotate the 10k generated images. Figure demonstrates that L2-Distortion achieves high correlations comparable the original Distortion score the global-based compatibility and DCL. These results prove that our framework of minimizing
geo
and the supervised
L L'-Distortion by Fréchet mean is also valid.
4 EXPERIMENTS
4.1 FRÉCHET BASIS AS GLOBAL SEMANTIC PERTURBATIONS
We evaluate the Fréchet basis as the global semantic perturbations on the intermediate layers of the mapping network in various StyleGAN models (Karras al. 2019 2020b). For each StyleGAN model, we used the layers from 3rd to 8th because the local dimension estimate is rather unstable for the 1st and 2nd layers depending the preprocessing hyperparameter épre (Choi et al.) 2022a). We chose these intermediate layers for evaluation because they are diverse and properly disentangled latent spaces. In this manner, the Fréchet basis be tested six latent spaces for each pre-trained

---

Published as a conference paper at ICLR 2023
(a) Bald Fréchet Basis
(b) Bald GANSpace (221. 2-4)
(c) Light Position- Fréchet Basis
(d) Light Position- GANSpace (V28. 8)
Figure 3: Comparison of Semantic Factorization between Fréchet basis and GANSpace. (w;, 112)
denotes the layer-wise edit along the i-th GANSpace component at the /1-12 layers. The image traversals are performed on StyleGAN2-FFHQ (Fig 3a 3b) and EtyleGAN2-LSUN cat (Fig 3e 3d).
StyleGAN model. In this section, all Fréchet basis are discovered using 1,000 i..d. samples of Local Basis with Opee = 0.01. The max iteration is set to 200 when optimizing Fréchet mean with Pymanopt. The evaluation is performed on two properties: Semantic Factorization and Robustness. Fréchet basis is compared with GANSpace (Härkönen et al. 2020) and SeFa (Shen & Zhou 202T)) because these two methods are also unsupervised global basis (Sec Note that GANSpace and Fréchet basis can be applied to arbitrary latent space, but SeFa is only applicable to W-space (Karras) al. 2019 i.e., the last layer of the mapping network in StyleGANs.
Semantic Factorization In Fig 3 and 4 we evaluated the semantic factorization of Fréchet basis. FigureBshows how the image changes as we perturb the latent variable along each global basis. For a fair comparison, we took the annotated basis in GANSpace (Härkönen et al.. 2020) and compared those with Fréchet basis on W-space of three StyleGAN models. Because GANSpace performs layer- wise edits, we matched the set of layers, where the perturbed latent variable is fed, in the synthesis network as annotated. The corresponding Fréchet basis component is selected by the cosine-similarity Each subfigure shows the three images traversed with the same global basis, perturbation intensity. and the set of perturbed layers. The original image is placed at the center. Hence, these subfigures also show the semantic consistency of the global basis. In StyleGAN2 trained on FFHQ. GANSpace shows image failure on the left side and semantic inconsistency on the third row (not representing hairy the left) (bj. In StyleGAN2 trained on LSUN-cat (Yu al., 2015), GANSpace presents entangled semantic manipulation (Fig 3db. The latent traversal along GANSpace changes the light position as annotated, but also darkens the striped pattern of cats. On the other hand, Fréchet basis achieves better semantic factorization without showing those problems (Fig Ba and 3cp. (See the appendix G for additional examples of other attributes and datasets. Also, since Fréchet basis an average of Local Basis, we compared these two methods and GANSpace in the appendix F
For the quantitative comparison of semantic factorization, we compared DCI as in Sec 3.2 DCI is supervised disentanglement metric that assesses the axis-wise alignment of semantics. Hence, we measured DCIs of the latent space representations with two global basis, GANSpace and Fréchet basis. Specifically, we converted latent variable w E Rdw into the dyw-dimensional representation with each global basis: For dw-dimmnsioonal global basis {volobal Jisisaw:
{ Representation of w with global Jisisdw = =(w v. global w. V.2
global
W.
V
globol
dw
(13)
Figure presents the DCI results. StyleGAN2-e denotes the config-e model of StyleGAN2 Karras et al. 2020b), and all three models trained FFHQ (Karras et al. 2019). The intrinsic dimension
7

---

Published as a conference paper at ICLR 2023
0.40
0.35
0.36
0.34
0.34
0.32
0.32
0.30
0.30
0.30
0.20
0.29
0.28
0.28
0.30
0.30
0.34
0.39
0.38
0.40
0.28
0.30
0.32
0.94
0.38
0.38
0.26
0.33
0.11
9
0.34
0.35
GANEpace DC1
GANSpace DOI
GANEpace DCI
(a) StyleGANI-FFHQ
(b) StyleGAN2-c-FFH
(c) StylcGAN2-FFHO
Figure 4: Quantitative Comparison of Semantic Factorization. DCI score (f) is supervised disentanglement metric that evaluates the axis-wise alignment of semantics. Fréchet basis outperforms GANSpace when point located above the black line. (The black line illustrates = x.)
30
Busi F10
25
25
E1000 Bodll
25
80
Frechel
20
20
15
15
10
10
15
20 GANSpace
25
16
15
20
25
30
36
10
15
20
30
GANSSpace FID
G/ASSpace
(a) StyleGANI-FFIIQ
(b) StyleGAN2-e-FFII
(c) StyleGAN2-FFII
Figure 5: Quantitative Comparison of Robustness. The image fidelity under the latent traversal is evaluated by FID ()). Fréchet basis performs better when a point is placed below the black line.
of each latent space is estimated with Ppre 0.01 (Choi et al. 2022a In all latent spaces except for the 5-th layer in StyleGAN2-e, the latent space achieves higher DCI score when represented with Fréchet basis. This quantitative result shows that Fréchet basis provides better semantic factorization along each basis component at the same latent space.
Robustness We tested the robustness of Fréchet basis by comparing the image fidelity under the latent perturbation. For each global basis, we evaluated FID (Heusel et al., (201)) of 50k i.i.d. latent perturbed images. The perturbation direction is selected to be the 1st component for the GANSpace. In Fréchet basis, we chose the component with the highest cosine-similarig GANSpace. The perturbation intensity is in StyleGANI, and StyleGAN2-e and StyleGAN2. provided in Fig (See the appendix for FID
the 1st component of
The FID scores of the latent spaces in three models scores under various perturbations intensity.) We think this higher robustness is because the global semantic subspace S, is the Fréchet mean of the intrinsic tangent spaces of learned latent manifold. The strong robustness of traversing along the tangent space at each latent variable was observed in Choi al. (20226). Therefore, traversing along Fréchet basis can be interpreted traversing along the mean of these locally robust perturbation directions, because Fréchet basis is basis of S..
4.2 BASIS REFINEMENT BY FRÉCHET MEAN
Table 1: Basis Refinement by Fréchet mean.
Fréchet basis consists of two parts: finding the global semantic subspace S, and selecting the global semantic basis B, from the discovered semantic subspace. In par- ticular, the second step can be utilized refine a given global basis from previous methods. We ran this basis re- finement on the subspace generated by the existing global methods, GANSpace and SeFa. This experiment reveals the contribution of the second step.
Global Basis
FID ())
DCI (t)
Fréchet basis
7.49
0.350
GANSpace GANSpace-refine
8,90
0.312
8.63
0.336
SeFa
11.79
0.307
SeFa-refine
7.89
0.337
Let global be the dw-dimensional semantic subspace gen-
erated by the global basis {vglobal) hisisdy- Then, we optimized the constrained-optimal global basis
8

---

Published as a conference paper at ICLR 2023
2.5
To GANSpace
0.350
0.345
2.0
0.340
0.335
1.5
0.330
1.0
0.325
Interpolation
0.5
0.315
GANSpace
0.310
0
1
3
6
3
5
6
Interpalation Index
Interpolation Indox
(a) Grassmannian Distance
(b) DCI score of Interpolation basis B:
Figure 6: Geodesic Interpolation from Fréchet basis (i 1) to GANSpace 7).
Bytobal in Sylobat via Sec 3.1.2 Table 1 shows FID and DCI scores of Fréchet basis, GANSpace, GANSpace refiement, SeFA, and SeFa refinement W-space of StyleGAN2 trained on FFHQ. These two scores are evaluated in the same manner as in Sec 4.1 with the perturbation intensity 2 for FID. Most importantly, Fréchet basis achieves the best FID and DCL. Also, the basis refinement monotonically improves the two scores of the two previous methods, GANSpace and SeFa. The comparison with the previous global method and its refinement proves the contribution basis optimization. Also, the superior performance of Fréchet basis over the two refinements shows the contribution of subspace optimization.
4.3 GEODESIC INTERPOLATION ON GRASSMANNIAN MANIFOLD
We further investigate the optimality of global semantic subspace S, by amalyzing the interpolation from S. to GANSpace subspace Sas. Similar to Sec 4.2 this experiment examines the contribution of the first step in Fréchet basis. In the Grassmannian manifold Gr(k, R"), there exists at least one mingti-mnnmizzing geodesic between any two subspaces in Grk,, Morcover, there is an explicit parametrization
this ength-minimizing geodesic (Eq [4] (Chakraborty & Vemuri 2015). For X. Y.Ye E Gr(k, R. ), let X, Y E Ra be the column-wis concatenation of their orthonormal basis. Then, the length-me minimizing geodesic [ V/ t) from X' y is defined as:
(X),,1) = spant (XV cos(@t) + U sin (e)) VT} for EE [0, 1].
(14) UEVT is the thin SVD, and θ arctan Σ.
where X is non-singular, (Y-X- XX Y) Note that the last V multiplication is unnecessary as the subspace. However, we added it to match the basis X / O. We used this geodesic to perform the interpolation and extrapolation from S, to Sas:
S I (X (i ( 1)/n) for 0, 1, n
(15)
with n = 6. Note that i = 0, n + represent the extrapolations of = (-1/n). 1 (1/n). Because the above interpolation is performed on subspace-scale, we conducted the basis refinement (Sec 4.2) for each interpolation subspace S, to find the interpolation basis B, Figure presennts the geodesic metric dgeo in the Grassmannian manifold for each interpolation subspace GANSpace and Fréchet basis (Fig 6a), and the DCI score evaluated at each interpolation basis B, (Fig 6b) on W-space of StyleGAN2-FFFHO, First, Fig shows that S. performs interpolation from Fréchet basis at 0 to GANSpace at i 7. This interpolation is linear in the Grassmannian metric dgeo- Second, the DCI score at each interpolation basis B, are presented in Fig 6b Fréchet basis at = 0 achieves the best DCI score among the interpolation basis. Note that the DCI score of original GANSpace without refinement is 0.312 (Tab [. which lower than the score after refinement at 7 as in S.c
CONCLUSION
In this paper, we proposed the unsupervised global semantic basis on the intermediate latent space in a GAN, called Fréchet basis. Fréchet basis is discovered by utilizing the Fréchet mean on the Grassmannian manifold and the Special Orthogonal Group. Our experiments demonstrate that Fréchet basis achieves better semantic factorization and robustness than the previous unsupervised global methods. In addition, suggest the basis refinement scheme using the Fréchet mean. Given the same semantic subspace generated by the previous global methods, the refined basis attains better semantic factorization and robustness.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGMENTS
This work was supported by a KIAS Individual Grant [AP087501] via the Center for Al and Natural Sciences at Korea Institute for Advanced Study, the NRF grant [2012R1A2C3310888)] and the MSIT/IITP (17111170933, (2021-0-00077, [No. 2021-0-01343, Artificial Intelligence Graduate School Program(SNU))..
REFERENCES
Rameen Abdal, Peihao Zhu, Niloy J Mitra, and Peter Wonka. Styleflow: Atribute-conditione exploration of -yyleaan-generated images using conditional continuous normalizing flows. ACM Transactions Graphics (TOG), 10,,
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE transactions pattern analysis and machine intelligence, 35(8):1798-1822 2013.
William M Boothby, An infroduction differentiable manifolds and Riemannian geometry. Academic press, 1986.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. International Conference on Learning Representations, 2018.
Rudrasis Chakraborty and Baba C Vemuri. Recursive frechet mean computation on the grassmannian and its applications to computer vision. In Proceedings of the IEEE International Conference on Computer Vision, pp. 4229 4237, 2015.
Jaewoong Choi, Geonho Hwang, Hyunsoo Cho, and Myungjoo Kang. Analyzing the latent space of gan through local dimension estimation. arXiv preprint Xix:22205.13182. 2022a.
Jaewoong Choi, Junho Lee, Changyeon Yoon, Jung Ho Park, Geonho Hwang, and Myungjoo Kang. Do not escape from the manifold: Discovering the local coordinates the latent space of GANs. In International Conference on Learning Representations, 2022b.
Cian Eastwood and Christopher KI Williams. A framework for the quantitative evaluation of disentangled representations. In International Conference on Learning Representations, 2018,
Maurice Fréchet. Les éléments aléatoires de nature quelconque dans un espace distancié. In Annales de I' l'institut Henri Poincaré, volume 10, pp. 215-310, 1948.
Lore Goetschalekx, Alex Andonian, Aude Oliva, and Phillip Isola. Ganalyze: Toward visual defini- tions of cognitive image properties. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 5744-5753, 2019.
Ian J Goodfellow, Jean Pouget-Abadie Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In NIPS, 2014.
Erik Härkönen, Aaron Hertzmann, Jaakko Lehtinen, and Sylvain Paris. Ganspace: Discovering interpretable gan controls. Advances in Neural Information Processing Systems, 33, 2020.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. Beta-VAE: Learning basic visual concepts with constrained variational framework. In International Conference on Learning Representations, 2017.
Ali Jahanian, Lucy Chai, and Phillip Isola. On the" steerability" of generative adversarial networks. In International Conference on Learning Representations, 2019.
10

---

Published as a conference paper at ICLR 2023
Hermann Karcher. Riemannian center of mass and mollifier smoothing. Communications on pure and applied mathematics, 30(5):509 541, 1977.
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for improved quality, stability, and variation. In International Conference on Learning Representations, 2018.
Tero Karras, Samuli Laine, and Timo Aila. style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 4401 44110, 2019.
Tero Karras, Miika Aittala, Janne Hellsten, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Training generative adversarial networks with limited data. arXiv preprint arXiv:2006.06677 2020a.
Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of stylegan. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patlern Recognition, pp. 8110-8119 (10-,199,40 2020b.
Tero Karras, Miika Aittala, Samuli Laine, Erik Härkönen, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Alias-fre free generative adversarial networks. Advances in Neural Information Processing Systems, 34, 2021.
Hyunjik Kim and Andriy Mnih. Disentangling by factorising. Proceedings of the 35th International Conference Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 2659, 2018
Shira Kritchman and Boaz Nadler. Determining the number of components in a factor model from limited noisy data. Chemometrics and Intelligent Laboratory Systems, 94(1):19-32, 2008
Serge Lang. Algebra, volume 211. Springer Science & Business Media, 2012.
John M Lee. Smooth manifolds. In Introduction to Smooth Manifolds, pp. 1-31. Springer, 2013.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes the wild. In Proceedings of International Conference Computer Vision (ICCV), December 2015.
Aaron Lou, Isay Katsman, Qingxuan Jiang, Serge Belongie, Ser-Nam Lim, and Christopher De Sa. Differentiating through the fréchet mean. In International Conference on Machine Learning, pp. 6393.6643. PMLR, 2020.
Tim Marrinan. J Ross Beveridge, Bruce Draper, Michael Kirby, and Chris Peterson. Finding the subspace mean or median to your need. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1082-1089, 2014.
Or Patashnik, Zongze Wu, Eli Shechtman, Daniel Cohen-Or, and Dani Lischinski. Styleclip: Text- driven manipulation of stylegan imagery. arXiv preprint arXiv:2 /7249, 2021.
Antoine Plumerault, Hervé Le Borgne, and Céline Hudelot. Controlling generative models with continuous factors of variations. International Conference on Learning Representations, 2020.
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. Proceedings of the International Conference on Learning Representations (ICLR), 2016.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. International Conference on Machine Learning, pp- 7748-8766 PMLR, 2021.
Aditya Ramesh, Youngduck Choi, and Yann LeCun. spectral regularizer for unsupervised disentan- glement. arXiv preprint arXiv.1812.01611, 2018.
Yujun Shen and Bolei Zhou. Closed-form factorization of latent semantics in gans. In CVPR, 2021.

---

Published as a conference paper at ICLR 2023
Yujum Shen, Jinjin Gu, Xiaoou Tang, and Bolei Zhou. Interpreting the latent space of gans for semantic face editing. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 9243-9252, 2020.
Anuj Srivastava and Eric Klassen. Monte carlo extrinsic estimators of manifold-valued parameters. IEEE Transactions on Signal Processing, 50(2):299-308, 2002.
James Townsend, Niklas Koep, and Sebastian Weichwald. Pymanopt: A python toolbox for opti- mization on manifolds using automatic differentiation. Journal of Machine Learning Research, 17 (137):1-5, 2016.
Paul Upchurch, Jacob Gardner, Geoff Pleiss, Robert Pless, Noah Snavely, Kavita Bala, and Kilian Weinberger. Deep feature interpolation for image content changes. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 7064-7073, 2017.
Ke Ye and Lek-Heng Lim. Schubert varieties and distances between subspaces different dimensions. SIAM Journal on Matrix Analysis and Applications, 37(3):117-1197, 2016.
Fisher Yu, Yinda Zhang, Shuran Song, Ari Seff, and Jianxiong Xiao. Lsun: Construction of a large- scale image dataset using deep learning with humans in the loop. arXiv preprint (rXiv:/506003.0 2015.
Jiapeng Zhu, Ruili Feng, Yujun Shen, Deli Zhao, Zhengjun Zha, Jingren Zhou, and Qifeng Chen. Low-rank subspaces in gans. arXiv preprint arXiv:2 044884488 2021.
12

---

Published as a conference paper at ICLR 2023
A PROJECTION ONTO SPECIAL ORTHOGONAL GROUP
In this section, we provide proof for the projection of an invertible matrix E GL(n) onto the special orthogonal group SO(n). Formally, the set of invertible matrices (L), orthogonal group On)), and special orthogonal group SO(n) are defined as follows:
GL(n): = {A E R=x* det(A) # 0},
(16)
O(n) = {A E R A A = AAT = 13.
(17)
SO(n) (A R=** A A- AAT = I, det(A)) 1}.
(18)
For a non-invertible matrix, the projection onto SO(n) is not uniquely defined because of the subspace generated by singular vectors with o = O. However, the set of non-invertible matrices has measure-zent in the set of matrices Rax= Thus, this did not happen in practice during the Fréchet basis optimization.
Theorem 1. The following optimization problem, i.e., the projection of A E GL(n) onto the Special Orthogonal Group SO(n),
arg min |X - AAll where A GL(n), XESO(m)
(19)
has solution
(s) U diag (I,1,...,,,, det (UVT)) V
(20)
where A = UVVT is the Singular Value Decomposition (SVD) of A with the projection onto orthogonal group Po(A)= = UV (The projection is unique if we assume 0g > 02 > ... > o > where {ai hiisn denote the singular values of A.)
Proof: 1. If Po(A) = UVT E 50Onn,, done. SO(n) ((n)) and det (Po(A)) =))
If (A() UV & (O(n), i.e., det 0,
X - A12- = | - X T A f = 2 tr A) + tr(ATAAA A).
(21)
For any skew-symmetric matrix K, define [(1) as
f(t) -2 tr 2t(AATxe*)) xx for t R.
(22)
Note that XetK (O(n) and tr(AT A) is given. Therefore, if X is the minimizer of ||X we have f'(0) = -2rr(A XK) = 0 for all skew-symmetric K. Thus, A X is symmetric.
Al|r
Without loss of generality, we may assume that A Uo XX U for some Up, Vo SO(n) where En is the diagonal matrix as Σ in SVD but Eo might have the negative elements. This decomposition can be obtained by flipping the singular vectors in U, V of SVD to make Uo, Vo E SO(n) and letting X' = Ug XV.. Explicitly, from del(A) < <0
Uo U diag (I, 1, det (U)).
Vo = V diag (1,..., 1. det (V)), Eo diag (g), ,n on,)
(23)
Then, since X T == x)
x A = Vo (X') VoVo is symmetric and has negative determinant.
(24)
Therefore, (X'T Eo is also symmetric and thus diagonalizable.
|| - _ || x AlE = ||T Vo (X') VoVo 112
|| (x') Eall Σ (1 2x)²,
(25)
where >; denotes the i-th eigenvalue of (X') En with |A > Da 2 ... > Dnpl Note that since XVo E SO(n), the i-th singular value a; of A satisfies a; = [Arl. Moreover, an odd number of signed singular values in Eo is negative because det(A) < 0. Also. det( En) det(XTA) < implies that an odd number of eigenvalues is negative. Hence, || AlEF is minimized when X; = o; for 1 <is this condition, X = U diag (1, I,..,, -1) VT. Lemma proves that when 0 > 02 .... >...>.. on is satisfied, the uniqueness of X' is guaranteed.
= Up X"=
X"
1 and A, -T X' is the diagonal matrix satisfying
13

---

Published as a conference paper at ICLR 2023
Lemma 1. Let X' E (O and Eg diag (d1 (01....,,0 ,n-1, -n) with d1 On-1 On 0.
If = (X') En is symmetric, X' is diagonal matrix.
(26)
Proof. Since Y symmetric, is diagonalizable with the orthogonal matrix P O(n).
= =x(x'² 2o PDP
(27)
We interpret these two decompositions Y = (X') 2i. = PDP as two SVD-like representations of Y because (X')² SO(n) and IT I. The ordered singular vectors in the domain of Y are uniquely determined as the basis for each eigenspace of Y y. Note that if the dimension of eigenspace bigger than 1, there is freedom of choosing basis it.
From Eg. the possible eigenvalues Y are hishh and the eigenvalues of Y are {o? hsisn- Therefore, since the standard basis {eilisisn of R8 the domain singular vectors of Y from Y = (x') Eo and diagonalizable,
((e)) = Toe
for 1 I n- -1
(...>.... (: > ... on- 0)
(28)
(nnnen
for
(: On- on 20)
Hence, the standard basis are also the codomain singular vectors of Y, which implies that X' is diagonal.
B IMPLEMENTATION DETAIL
In this section, we summarize the hyperparameters for Frechet basis presented in the experimental results in Section
Preprocossing hyperparameter 6pre for local dimension estimation Choi et al. (202aa)
pre = 0.01.
Global Semantic Subspace Optimization
Number of samples n 1000.
Max iteration in Frechet mean Optimization using Pymanopt Townsend et al. (016)
1,, 000.
Max time in Frechet mean Optimization using Pymanopt 200.
Global Semantic Basis Optimization
Number of samples 1000.
Max iteration in Frechet mean Optimization using Pymanopt 200.
Max time in Frechet mean Optimization using Pymanopt 10, 000.
14

---

Published as a conference paper at ICLR 2023
C FULL CORRELATION RESULTS OF L'-DISTORTION
20.0
10
FID
10.0
7.
5.0
9.4
9.6
988
10.0 102
104
10.6
10.8
9.25 8.50
9.75 10.00 10.25 10.50 10.75
9.0
9.5
10.0
10.5
Distorion
Distottoon
Distortion
(a) StyleGAN2-Car Cat = 0.95
(b) StyleGAN2-c p 0.75
(c) StyleGAN2 p 0.73
Figure 7: Correlation between L2-Distortion metric (L) and FID gap (L) when Opre 0.005. The correlations are 0.98 for StyleGAN2-cat, 0,73 for StyleGAN2-e, and 0.70 for StyleGAN2 in L*-Distortion.
0.78
0.80
0.74
0.76
0.74
0.72
0.72
0,70
0.70
0.54
0.60
0.68
0.88
0.52
0.66
0.84
8.5
8.0
2.5
10.0
10.6
8.25 8.50 9.75 10.00 10.25 10.50 10.75 Distortion
8.0
9.5
10.0
10.5
Distarton
Distorton
(a) StyleGAN1 p 0.72
(b) StyleGAN2-e - p -0.91 (c) StyleGAN2 p= 0.57
Figure 8: Correlation between L2-Distortion metric (L) and DCI (f) when 0pre 0.005. The correlations are 0.98, 0,73, and 0.70 for StyleGAN2-cat, StyleGAN2-e, and StyleGAN2 for L.1- Distortion.
15

---

Published as a conference paper at ICLR 2023
D QUANTITATIVE COMPARISON OF ROBUSTNESS (FID) FOR VARIOUS PERTURBATION INTENSITY
13
12
18
Fréchel Basie FID
5
Fréchat Basis FID
11
10
7
8
8
9
10
11
12
13
8
10
12
14
16
18
20
GANSpace FID
GANSpace FID
/
(b) / 1.5
30
40
Fréchel Basis FID
25
Frichat Basis FID
30
20
20
10
15
20
25
30
10
15
20
25
30
35
40
GANSpace FID
GANSpace FID
(c) / 2.0
(d) / 2.5
Figure 9: Quantitative Comparison of Robustness between GANSpace and Fréchet basis on StyIGANI-FFHQ (Karras et al. 2019) for various perturbation intensity The perturbation intensity I is measured by the -o-ro on each latent space. The image fidelity under the latent traversal is evaluated by FID (1). The black line indicates where FIDs are equal.
16

---

Published as a conference paper at ICLR 2023
25.0
22.5
Frechel Basis FID
20.0
Frächel Basis
7
15.0
8
-0
12.5
10.0
10
14
16
17.5
22.5
GANSpace
GANSpace
(a) / 3.0
(b) = 4.0
35
40
Frechel Basis FID
Fréchat Basis FID
35
25
30
20
25
15
10
20
25
30
35
15
20
30
35
45
GANSpace FID
GANSpace FID
(c)
(d) 6.0
Figure 10: Quantitative Comparison of Robustness between GANSpace and Fréchet basis on DtyIGAN----FFHS (Krras et al. 2020b) for various perturbation intensity I. The image fidelity under the latent traversal is evaluated by FID (1). The black line indicates where two FIDs are equal.
Fréchel Basis FID
Fréchat Basis FID
6
5
7
10
12
14
GANSpace
GANSpace
(a)
(b) 3.0
22
Frechel Basia FID
20
5
Fréchet Basis FID
18
16
14
20
15
10
12
14
16
18
20
22
24
10
15
25
30
GANSpace
FID
GANSpace FID
(c)
(d) 5.0
Figure 11: Quantitative Comparison of Robustness between GANSpace and Fréchet basis StyIGAN2-FFHQ (Karras et al. 2020b for various perturbation intensity /. The image fidelity under the latent traversal is evaluated by FID (L). The black line indicates where two FIDs are equal.
17

---

Published as a conference paper at ICLR 2023
E ABLATION STUDY ON THE OTHER MEANS OF GRASSMANNIAN MANIFOLD
In this section, we conducted an ablation study on defining the global semantic subspace in the Grassmanifold. particular, compared the Fréchet mean Fréchet 1948); Marrinan et (2014) and extrinsic mean Srivastava & Klassen (2002) of the Grassmannian manifold. The extrinsic mean Je defined as the minimizer of squared extrinsic metries dg, i.e., for 21..... In E Gr(,, R"),
ME arg min PEX
Σ ISi5n
de (H, where dg (1, do (Φ()), ((x)))
(29)
where denotes an appropriate embedding of Gr(k, R"). Following Marrinan et al. (014). we the embedding be the corresponding projection P i.e., ((x+) Pz: M2 Ma, where My. gaxk indicates the column-wise concatenation of an orthonormal basis of 2,. Also, the Frobenius norm is adopted for dæ.
45
0.35
0.34
40
4
0.33
ED
35
5
0.32
3
Frechet Basis
30
7
0.31
4
Frechet
25
8
0.30
0.29
6
20
0.28
15
0.27
0.27
0.28
0.29
0.30
0.31
0.32
0.33
0.34
0.35
10
10
15
20
25
30
35
40
45
Extrinsic Basis DC/
Extrinsic Basis FID
(a) DCI (1)
(b) FID (L) with / = 3.0
Figure 12:
Ablation study on the means Grassmannian manifold on StyleGAN2-FFHCC Extrinsic basis is variant of Fréchet basis where the global semantic basis is discovered by the extrinsic mean of the Grassmannian manifold. In both scores, our Fréchet basis outperforms Extrinsic basis in 5 out 6 intermediate layers in the mapping network.
18

---

Published as a conference paper at ICLR 2023
F COMPARISON TO LOCAL BASIS
F.1 QUANTITATIVE COMPARISON
In this section, we introduced new experiment to quantitatively compare semantic factorization between Fréchet basis and Local Basis Choi et al. (20226). Intuitively, this experiment measures the average of local DCI scores. To be more specific, consider n--ammlees {z: higign C Z of input Gaussian noise. (We set 100) Then, take m-samples from the neighborhood of each z₁ and map them to the target latent space
f(2), where / denotes the subnetwork from Z to W:
Wij = f(z; + G..5
where
G.j ~ (O, and w; = f(;),,
(30)
for sufficiently small a 0. (We set m 000 and o 0.5) Then, we measure DCI score for each neighborhood of w,, i.e., (Wi,) ( ssjsm. after representing these latent variables with each semantic basis as in Eq 13 (Fréchet basis and Local Basis at w.). The averages of these local DCI scores compared between Fréchet basis and Local Basis. Below, we included the evaluated scores W-space of StyleGAN2, StyleGAN2-confgee, and StyleGAN1 trained on FFHQ Note that the overall DCI scores are higher than Fig because it is easier for semantic basis satisfy semantic consistency on the local region than on the entire latent space. As in the qualitative examples, Fréchet basis outperforms Local Basis in the quantitative assessment.
Table Quantitative Comparison of Semantic Factorization by Local DCI (1). Local DCI score is evaluated at W-space of each StyleGAN model.
Model
Fréchet Basis
Local Basis
StyleGAN2
0.740
0.665
StyleGAN2-c
0.726
0.704
StyleGANI
0.733
0.677
F.2 QUALITATIVE COMPARISON
25
(a) GANspace
(b) Fréchet Basis
(c) Local Basis
Figure 13: Comparison of Latent Traversals on StyleGAN2-FFHO. We used the annotated
GANSpace on the semantics of "Bald". The corresponding Fréchet Basis and Local Basis are chosen by the cosine similarity. The traversal images along GANSpace are more deteriorated than the other two bases. The red box indicates where the image deterioration occurred.
19

---

Published as a conference paper at ICLR 2023
(a) GANspace
(b) Fréchet Basis
(c) Local Basis
I Latent Traversals on StyleGAN2-LSSUN Church. We used the annotated GANSpace on the semantics of "Clouds". The corresponding Fréchet Basis and Local Basis are chosen by the cosine similarity. Some traversal examples along Local Basis do not show the annotated semantic variations compared to the other two bases. In the yellow box, no clouds appeared. In the red box, clouds appeared in all images.
Figure 14: Comparison of
(a) GANspace
(b) Fréchet Basis
(c) Local Basis
Figure 15: Comparison Latent Traversals on StyleGAN2-LSUN Horse. We used the annotated GANSpace the semantics of "White horse". The traversal images of GANspace and Local Basis in the yellow box are less affected than that Frechet Basis.
20

---

Published as a conference paper at ICLR 2023
(a) GANspace
(b) Fréchet Basis
(c) Local Basis
Figure 16: Comparison of Latent Traversals on StyleGAN2-LSUN Car. We used the annotated GANSpace the semanties of "Side to Front". The traversal images of Local Basis not affected by perturbations.
21

---

Published as a conference paper at ICLR 2023
G ADDITIONAL SEMANTIC FACTORIZATION COMPARISON
25
PAS
(a) Bald Fréchet Basis
(b) Bald GANSpace (U21. 2-4)
Figure 17: Comparison of Semantic Factorization on StyleGAN2-FFHQ between Fréchet basis and GANSpace. (u,, 11-22) denotes the layer-wise edit along the i-th GANSpace component at the 11-12 layers.
22

---

Published as a conference paper at ICLR 2023
(a) Smile Fréchet Basis
(b) Smile- GANSpace (Vza. 4-5)
(c) Makeup- Fréchet Basis
(d) Makeup GANSpace (Vo, 8-10)
(e) Bright BG FG Fréchet Basis
(f) Bright BG FG GANSpace (U13. 8)
(g) Happy frizzy hair Fréchet Basis
(h) Happy frizzy hair GANSpace (121, 0-4)
Figure 18: Comparison of Semantic Factorization on Sty-eGAN2-FFH between Fréchet basis and GANSpace. (v,, 11-12) denotes the layer-wise edit along the i-th GANSpace component at the 71-12 layers.
23

---

Published as a conference paper at ICLR 2023
(a) Image contrast Fréchet Basis
(b) Image contrast GANSpace (187. 7-15)
(c) Coloring Fréchet Basis
(d) Coloring GANSpace (V11, 5-6)
(e) Fluffiness Fréchet Basis
(f) Fluffiness GANSpace (V27. 2-4)
(g) Sun direction Fréchet Basis
(h) Sun direction GANSpace (U15. 8)
Figure 19: Comparison of Semantic Factorization between Fréchet basis and GANSpace. (v. 41-12) denotes the layer-wise edit along the i-th GANSpace component at the /1-12 layers. The image traversals are performed on StyleGAN2 (L.SUN Car, Horse, Cat, and Church).
24

---

Published as a conference paper at ICLR 2023
(a) Smile Fréchet Basis
(b) Smile GANSpace (2444 3)
(c) Smile- Fréchet Basis
(d) Smile GANSpace (044.
Figure 20: Comparison of Semantic Factorization between Fréchet basis and GANSpace. (vi. 11-12 denotes the layer-wise edit along the i-th GANSpace component at the /1-/2 layers. The image traversals are performed on StylGGANI-FFHO.
(a) Glasses StyleGANI, FFHQ
(b) Smile StyleGANI, FFHQ
Figure 21: Comparison of Supervised method (InterfaceGAN) and Unsupervised methods (GANspace, Fréchet Basis) The first row shows the results of supervised method (InterfaceGAN (Shen al. (0200), the second shows GANspace results, and the last shows our traversing results along Fréchet Basis.
25


Reviews:
[{"review_id": "r2tlgDU0PS", "reviewer": "Reviewer_toHN", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes an unsupervised way to find the global semantic perturbations in a latent space in GAN, named Frechet Basis. Specifically, it is found by using Frechet mean to the local semantic perturbations. Experiments show that the proposed basis can provide better semantics and robustness than the previous unsupervised global methods. The paper also introduces a basis refinement scheme that can refine prior works by running the basis refinement on their subspaces.", "summary_of_the_review": "In general, the paper proposes a straightforward solution for an important problem. However, the experiments seem limited. I am willing to raise my recommendation if more theoretical justification or empirical study are provided.", "strength_and_weaknesses": "### Strength\n* The paper is well-written and easy to follow. Moreover, the paper provides a thorough background introduction and related prior works.\n* The motivation is clear, and the intuition behind the proposed method is straightforward. Interpreting the global semantic variation as the mean of the local semantic variations seems simple but effective.\n* The experiment results seem promising. The proposed method appears effective and provides better semantic factorization and robustness.\n\n### Weaknesses\n* I have some concerns about the design choices in the proposed method. I feel more analysis should be conducted. For example, there are plenty of ways to find the mean of subspaces; why would Frechet mean be the best choice? It would be better to provide an ablation study on such a design.\n* The refinement improvement on GANSpace seems limited. While the refinement on SeFa leads to a near Frechet basis performance. Is there any justification or explanation for the such result?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "lEt5pFkBFc", "reviewer": "Reviewer_hbhj", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents a method to disentangle the semantic space in a GAN model by using Frechet means. The pipeline has two steps. First, the semantic subspace is constructed by the Frechet mean in the Grassmannian manifold of the intrinsic tangent spaces. Then, the Frechet basis of the semantic subspace is constructed by using Frechet mean in the Special Orthogonal Group. The experimental results show that the proposed method produces better semantic factorization than the previous methods. ", "summary_of_the_review": "The paper presents an interesting and technically solid method to enable global semantic factorization of the latent space in GAN models, but the experiments are insufficient to justify the usefulness of the method. Given the current scale of the experiments and the quality of the results, the paper might need another cycle for major revision. I therefore lean toward a clear rejection. ", "strength_and_weaknesses": "** Strength **\n\n- The paper presents a generalization of the intrinsic local tangents introduced by Choi et al. (2022a) by averaging the semantic subspaces. This is an interesting extension with solid theoretical support. \n\n\n** Weakness ** \n\n1) The comparison against unsupervised global basis methods is not sufficient. Since this method is built upon Choi et al. 2022a, I think additional discussions and result comparisons with Choi et al. are needed despite it is a local method. By taking the mean of the local subspaces to discover global directions, does this lead to improvement or similar results?\n\n2) There are not sufficient qualitative results in both the paper and the appendix. Figure 3 only shows the change of two attributes, and it is unclear how the method works for other attributes. \n\n3) Results on additional datasets should be provided too, e.g., LSUN bedroom, cars, etc.\n\n4) More analysis on the factorization of different attributes should be added. For example, consider common attributes such as pose, age, smile, eye gaze, wearing glass, etc., with the proposed unsupervised factorization, how these attributes are disentangled in the discovered global directions? Computing some correlations between these attributes might help. \n\n4) Regarding robustness (Figure 5), it can be seen that Frechet means is not always better. Did the authors have any insight into when we can use GANspace and when to use Frechet means? ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "iFxBLTN8iuh", "reviewer": "Reviewer_McwQ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a way to find global semantic representation in GAN. It is based on looking for an average subspace among the tangent spaces in some layer of the generator. This subspace is then used to generate semantically factorized images in experiments, with both quantitative and qualitative results.", "summary_of_the_review": "Rev: I would remain my score 6 as it would have been better to include the details of i.i.d. sampling in the evaluation of FID scores. As the problem has no uniqueness solution, I find that the impact of the results remains limited. ", "strength_and_weaknesses": "The paper shows that the Frechet basis can achieve better semantic factorization compared to stat-of-the-arts. The paper is well written, and the results are solid. \n- I am just not sure on how the results in Section 4 change when one uses a different set of layers in the considered GAN, in particular how it depends on the size of each layers, and the nature of each layer (e.g. convolutional or fully connected). Could you provide more discussions in Section 4.1 or later?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "dQSIe80N6Bj", "reviewer": "Reviewer_G2MK", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors suggest an unsupervised method for semantically meaningful perturbations of the learned latent W in GAN models such as StyleGAN. They propose to find a global basis called Frechet basis. The basis is discover in two steps: 1. The global semantic subspace is discovered by the Frechet mean in the Grassmannian manifold of the local semantic subspaces. 2.  Frechet basis is found by optimizing a basis of the semantic ´subspace via the Frechet mean in the Special Orthogonal Group. Additionally, the authors suggest a refinement scheme for previous methods.", "summary_of_the_review": "I believe this paper is important to the community and advances the understanding of the semantic meaning of the GANs latent space.", "strength_and_weaknesses": "## Strength\nResults are very good visually compared to the GANSpace method which already gives very good results. The quantitative results are also superior to next best GANSpace. The paper is written in a clear and high quality fashion. The paper suggest nice mathematical insight on the GANs latent space decomposition and semantic structure. The mathematical formulations also are being backed up with the qualitative and qualitative results.\n\n## Weaknesses\nThe paper is mathematically heavy where it is hard to follow (for me) on this translates the latent W (for example the 2D several layers and resolutions latent of StyleGAN2). I would have like to see more comparisons to a supervised method to understand the gap between the supervised and unsupervised methods.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster