Low-Rank Subspaces in GANs
Authors: Jiapeng Zhu
Ruili Feng
Yujun Shen
Deli Zhao
Zheng-Jun Zha
Jingren Zhou
Qifeng Chen
Venue: NeurIPS 2021 Poster (2021)
Keywords: generative adversarial network
network interpretation
local image editing

Abstract: The latent space of a Generative Adversarial Network (GAN) has been shown to encode rich semantics within some subspaces. To identify these subspaces, researchers typically analyze the statistical information from a collection of synthesized data, and the identified subspaces tend to control image attributes globally (i.e., manipulating an attribute causes the change of an entire image). By contrast, this work introduces low-rank subspaces that enable more precise control of GAN generation. Concretely, given an arbitrary image and a region of interest (e.g., eyes of face images), we manage to relate the latent space to the image region with the Jacobian matrix and then use low-rank factorization to discover steerable latent subspaces. There are three distinguishable strengths of our approach that can be aptly called LowRankGAN. First, compared to analytic algorithms in prior work, our low-rank factorization of Jacobians is able to find the low-dimensional representation of attribute manifold, making image editing more precise and controllable.  Second, low-rank factorization naturally yields a null space of attributes such that moving the latent code within it only affects the outer region of interest. Therefore, local image editing can be simply achieved by projecting an attribute vector into the null space without relying on a spatial mask as existing methods do. Third, our method can robustly work with a local region from one image for analysis yet well generalize to other images, making it much easy to use in practice. Extensive experiments on state-of-the-art GAN models (including StyleGAN2 and BigGAN) trained on various datasets demonstrate the effectiveness of our LowRankGAN.

Low-Rank Subspaces in GANs
Jiapeng Zhu' Ruili Feng Yujun Shen' Deli Zhao² Zheng-Jun Zha Jingren Zhou? Qifeng Chen 1x
Hong Kong University of Science and Technology 'Alibaba Group University of Science and Technology of China "ByteDance Inc. {jengzhuO, ruilifenguste, shenyujun0302, zhaodeli)@gmail com cqf@ust.
zhazj @uajjuustcc..... edu. jingren.zhoudan zhauoalibaaa--a inc. com
Abstract
The latent space of Generative Adversarial Network (GAN) has been shown to encode rich semantics within some subspaces. To identify these subspaces, researchers typically analyze the statistical information from collection of synthesized data, and the identified subspaces tend to control image attributes globally (i.e., manipulating an attribute causes the change of an entire image). By contrast, this work introduces low-rank subspaces that enable more precise control of GAN generation. Concretely, given an arbitrary image and a region of interest (e.g., eyes of face images), we manage relate the latent space the image region with the Jacobian matrix and then use low-rank factorization discover stecrable latent subspaces. There three distinguishable strengths of our approach that can be aptly called LowRannGANN First, compared to analytic algorithms in prior work, low-rank factorization of Jacobians is able find the low-dimensional representation of attribute manifold, making image editing precise and controllable. Second, low-rank factorization naturally yields a null space attributes such that moving the latent code within it only affects the outer region of interest. Therefore, local image editing can be simply achieved by projecting an attribute vector into the null space without relying on spatial mask existing methods do. Third, method robustly work with local region from one image for analysis yet well generalize to other images, making it much easy to use in practice. Extensive experiments on state-of-the GAN models (including StyleGAN2 and BigGAN) trained on various datasets demonstrate the effectiveness of our LowRankGAN!
1 Introduction
Generative Adversarial Networks (GANs), which can produce high-fidelity images visually indistinguishable from real ones [18, 19, 4J, have made tremendous progress in many real-world applications [16, 39, 32, 10, 38, 27, 34J. Some recent studies [12, 26, 17] show that pre-trained GAN models spontaneously learn rich knowledge in latent spaces such that moving latent codes towards some certain directions can cause corresponding attribute change in images. In this way, we are able to control the generation process by identifying semantically meaningful latent subspaces.
For latent semantic discovery, one of the most straightforward ways is to first generate a collection of image synthesis, then label these images regarding target attribute, and finally find the latent separation boundary through supervised training [12, 26, 17]. Prior work [30, 25, 14] has pointed that the above pipeline could be limited by the labeling step (e.g., unable identify semantics
denotes the corresponding author.
Code is available at https: /// http//gttttt.t///////////t cww. http://niaaeeggooooeenaaar
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

beyond well-defined annotations) and proposed to find steerable directions of the latent space in an unsupervised manner, such as using Principal Component Analysis (PCA) [14]. However, existing approaches prefer to discover global semantics, which alter the entire image as a whole and fail to relate a latent subspace to some particular image region. Consequently, they highly rely on spatial masks [29, 8, 2] to locally control the GAN generation, which is of more practical usage.
In this work, we bridge this gap by introducing a low-rank subspace of GAN latent space. Our approach, called LowRankGAN, starts with an arbitrary synthesis together with a region of interest, such as the eyes and nose in a face image or the ground in a scene image. Then, we manage to relate the latent space and the image region via computing the Jacobian matrix with the pre-trained generator. After that comes the core part of our algorithm, which to perform low-rank factorization based on the resulting Jacobian matrix. Different from other analytic methods, the low-rank factorization of Jacobians capable of uncovering the intrinsic oowddimesioona representation of attribute manifold, thus helping solve more precise editing direction to the target attribute. Besides, the low-rank factorization naturally yields null space of attributes such that moving the latent code within this subspace mainly affects the remaining image region. With such an appealing property, we can achieve local image control by projecting an attribute vector into the null space and using the projected result to modulate the latent code. This observation is far beyond trivial because the latent code is commonly regarded to act on the entire feature map of GAN generator [13, 4, 18]. We also find that the low-rank subspace derived from local regions of one image is widely applicable to other images. In other words, our approach requires only one sample instead of abundant data for analysis. More astonishingly, supposing that identify changing- colorr boundary one car image, we can use this boundary to change the color of cars from other images, even the cars locate at a different spatial area or with different shapes. This suggests that the GAN latent space is locally semantic-aware Moreover, LowRankGAN is strongly robust to the starting image region for Jacobian matrix computation. For instance, to control the sky in synthesized images, our algorithm does not necessarily rely on a rigid sky segmentation mask but can also satisfyingly work with a casual bounding box that marks the sky region (partially or completely), making it much easy to use in practice.
Our main contributions are summarized as follows. First, we study low-rank subspaces of the GAN latent space and manage to locally control the GAN image generation by simply modulating the latent code without the need for spatial masks. Second, our method can discover latent directions regarding local regions of only one image yet generalize to all images, shedding light on the intrinsic semantic-awar structure of GAN latent space. Third, we conduct comprehensive experiments to demonstrate the effectiveness and robustness of our algorithm wide range of models and datasets.
2 Related Work
Interpreting well-trained GAN models [1, 17, 26] has recently received wide attention because this can help us understand the internal representation learned by GANs [34] and further control the generation process [35, 38]. It is shown that some subspaces in the GAN latent space are highly related the attributes occurring in the output images. To identify these semantically meaningful subspaces, existing approaches fall into two folds, i.e., supervised [12, 27, 17, 22] and unsupervised [30, 14, 25, 28]. On the one hand, supervised approaches seek help from off-the-shelff attribute classifiers [12, 27] or simple image transformations [17, 22] to annotate set of synthesized data and then learn the latent subspaces from the labeled data. On the other hand, unsupervised approaches try to discover steerable latent dimensions by statistically analyzing the latent space [14], maximizing the mutual information between the latent space and the image space [30], or exploring model parameters [25, 28]. However, all of the subspaces detected by these approaches tend to control global image attributes such that the entire image will be modified if we edit a particular attribute. To achieve local image control, common practice is to introduce a spatial mask [29, 8, 2, 34, 37],
which works together with the intermediate feature maps of the GAN generator.
Compared to prior work, our LowRankGAN has the following differences: no spatial masks of objects are needed to perform precise local editing, and generic attribute vectors can be obtained with only one image synthesis and arbitrary local regions of interest without any training or statistical analysis on a large amount of data. To our best knowledge, we are the first to demonstrate the effectiveness of low-rank factorization in interpreting generative adversarial models. Although there are also some works using the Jacobian matrix to analyze the GAN latent space [24, 6, 31], the low-rank subspace enables precise generation control by providing the indispensable null space.
2

---

3 Low-Rank GAN
Let [R and G(.) denote the input latent code and the generator of GAN, respectively. Prior work [12, 26, 17] has observed the relationship between some latent subspaces and image attributes such that we can achieve semantic manipulation on the synthesized sample, G(z), by linearly transforming the latent code
x
adit
G( + an),
(1)
where n denotes an attribute vector within the latent space, and a the editing strength. However, the latent subspaces identified by existing approaches tend to manipulate the entire image whole and fail to control a local image region. Differently, our proposed LowRankGAN confirms that Eq. (1) can also be used to locally control the image generation. The following parts describe how to discover the low-rank subspaces from pre-trained GAN model as well as its practical usage.
3.1 Degenerate Jacobian
First, we give the definition of Jacobian matrix. Let the real image E Rd be in the de di-dimensional space and the input prior z E Rd: be dimension dz. For an arbitrary point 2, the Jacobian matrix
Jz of the generator G(-) with respect to x is defined as (Jz)j = aG(2), where (J=)j, is the dsg
(j, --ttth entry of Jæ Rds xd, By the Taylor series, the first-order approximation the edited result can be written as
G(z on) G(z) aJ.n+ o(a).
An effective editing direction should significantly alter G(z), which can be attained by maximizing the following variance
[G(2 + an) G(=)ll R arg max 2n J J.n.
72 arg max
(3)
2=1
|n]]==1
It is easy to know that the above optimization admits closed-form solution, which is the eigenvector associated with the largest eigenvalue. Therefore, the subspace spanned by editing directions the principal subspace of J J.
For GAN architectures, previous work [11] has proven and demonstrated that the Jacobian matrix J. is degenerate, which is described by the following theorem.
Theorem Assume that the real data X lie in some underlying manifold of dimension dim (x) embedded the pixel space. Let 2 E Rd, denote the latent code of the k-th layer of the generator of GAN over X and Joy be the Jacobian of the generator with respect to zu- Then we have rank(JT 24+1 2441 Jest1) I rank(J], J&)) < d,, where rank(J] Js) denotes the rank of the Jacobian. Further, dii(m) dz, then Ja.
Theorem says that the rank the Jacobian J= might be progressively reduced, and the manifold characterized the generator admits low-dimensional structure under mild condition. Therefore, the attribute subspace must admit the analogous structure, which can he found by low-rank factorization of the Jacobian J J..
3.2 Low-Rank Jacobian Subspace
In practice. the numerical evaluation indicates that the Jacobian is actually of full rank, implying that it is perturbed with noise during the highly nonlinear generation process. If we assume that such perturbation is sparse, then the low-dimensional structure of the attribute manifold can be perfectly revealed by low-rank factorization of J Jz with corruption.
Given low-rank matrix M corrupted with sparse noise, the low-rank factorization can be formulated as M = L + S, where L is the low-rank matrix and S is the corrupted sparse matrix. Directly solving this problem actually very hard due to the non-convexity of the optimization. Hopefully, can reduce tractable convex problem [33]]
min L.S
L||- - S|SS||
s.t. M=L+5
(4)
where ||]] = E, or (M) is the nuclear norm defined by the sum of all singular values, ||S]]] Eng Myyl is the l norm defined by the sum of all absolute values in S, and À is a positive weight
3

---

(a) Jacobian of the object
(b) Low-rank subspace
(c) Null space
Figure 1: Framework of LowRankGAN, which is able to precisely control the image generation of GANs. (a) Given an arbitrary synthesis and a region of interest, i.e., region A under mask, we relate the image region to the latent space with Jacobian matrix. The proposed low-rank subspace (red circle) is derived by performing low-rank factorization on the Jacobian, which naturally yields null space (green circles). (b, c) The low-rank subspace and the null space correspond to the editing of region A and region B, respectively.
parameter to balance the spansity and the low rank. The convex optimization in Eq. (4) is also referred to as Principal Component Pursuit (PCP) [5], which can be solved by the Alternating Directions Method of Multipliers (ADMM) [21, 3]. The detailed introduction of low-rank factorization and ADMM is presented in the Supplementary Material. Suppose that the solution for J J. in Eq. (4) is
M = == -" S",
=
(5)
where L is the low-rank representation of J Jg, which has rank r corresponding to r attributes in images. By singular value decomposition (SVD), we are able to specify those attributes
U U,E.V SVD(L"),
(6)
where E is a diagonal matrix sorted by singular values, U and V [0i,...,, matrices of left and right singular matrices containing attribute vectors associated with attributes. With V. we can simply edit a specific attribute of an image by using (vi..... U,] according to Eq. (1), In this paper, we mainly focus on the local control of an image. So we compute the Jacobian matrix with specified region Megion in an image.
Va are the
3.3 Precise Control via Null Space Projection
Here, precise control means that the pixels in the remaining region change little possible when editing a specific region. For instance, if we want to close the eyes in a face image, the ideal scenario that the other regions, such mouth, nose, hair, and background, will keep unchanged. Previous works [14, 25, 22, 30] could edit some attributes in specific region but without any control on the rest region, which actually results in a global change on the image when editing the region of interest. We solve the problem via algebraic principle and propose universal method to fulfill the precise control by utilizing the null space of the Jacobian.
As shown in Fig. 1, if we want to edit some attributes (e.g., change color) of region A (the car body in the image). the ideal manipulation is that the remaining region B in the image will preserve unedited. natural idea is that we could project the specific attribute vector v; of region A into a space where the perturbation on the attribute direction has effect region B yet has an influence Specifically, we have attained
region A. ra attribute vectors in V that can change the attributes of region A, and the rest d_ Ta vectors in V barely influence the region A. Similarly, we also attain the attribute matrix B = [b..... ba_] for region B via Eq. (6), which has T' attribute vectors changing region B, and the remaining d Tb vectors barely influence the region B but contains some vectors that have some influence on region A. Note that Ta. Tb are the rank of the matrices on regions A and B after performing low-rank factorization, respectively. We let B [B, B2), where B₁ = [b_..... b,,] and B₂ [b..+1.... ..., b Thus, the change in region B can be relieved by projecting the attribute vector 8; of region to the orthogonal complementary space B1, which we formulate as
p (I- B, ) By us,
where I is the identity matrix. For our low-rank case, it is easy know that B, is the null space of the Jacobian of region B, ie., the subspace associated with zero singular values.
However, a complete precise control might restrict the degree of editing in some special circumstances. For instance, the size of eyes will be limited if the eyebrows are kept still during editing. Or the edition of opening the mouth without any change of the chin is nearly impossible. Considering this factor,

---

A
130
(3)
(c)
Illustration of the effect of null spaces. (a, d) The masked regions B and are used to obtain the null spaces through the low-rank decomposition of the Jacobians. (b, e) The images are manipulated by altering the latent code within the derived null spaces. (c. f) The heatmaps of the & loss between (a) and (b), (d) and (e), respectively, suggesting that editing within the null space barely affects the masked region interest.
Figure 2:
propose precision relaxation to alleviate this issue. Recall that the total precise control is performed by projecting v, into the null space of B, where all the singular values are zeros. Hence, we can involve several attribute vectors that have small singular values in order to increase the diversity. We define the Tielax as the number of vectors used for relaxation. So the projection subspace becomes B1 = [bi..... braan and B2 r br.+1.... .... ba_ Obviously, the larger Trelax is, the larger diversity we obtain. Otherwise, a small Frelax will result in little diversity.
Experiments
In this section, conduct experiments two state-of-the-art GAN models, StyleGAN2 [19] and BigGAN [4], to validate the effectiveness of our proposed method. The datasets we use are diverse, including FFHQ [18], LSUN [36] and ImageNet [9]. For metrics, use Fréchet Inception Distance (FID) [15], Sliced Wasserstein Distance (SWD) [23], and masked Mean Squared Error (MSE). For segmentation models, [20] is used for face data, and PixelLib used for LSUN. First, we show the property of the null space obtained by low-rank decomposition and then show the improvement brought by the null space projection. Second, the direction obtain from one image could be easily applied to the other images are verified. Third, the strong robustness to the mask is tested as well, and several -tate-of-the-a
methods are chosen for comparison. At last, the ablation study on the parameter À in Eq. (4) and parameter Frelax in Sec. 3.3 can be found in the Supplementary Material.
4.1 Precise Generation Control with LowRankGAN
Property of Null Spaces. First, show the property of the null space obtained by the low-rank decomposition using a church model. For convenience, we separate the church image into two parts, e.g., the sky and the rest, shown in Fig. 2a and Fig. 2d. According to Sec. 3.2, obtain J= = Mn for the masked region in Fig. 2a. And then, the low-rank decomposition on MB is conducted get the null space of region B. We choose the vector in the null space edit Fig. 2a and get the editing result as shown in Fig. 2b. Fig. 2c shows the f loss heatmap between Fig. 2a and Fig. 2b, in which we could find the null space of region B will affect region A yet could keep region B nearly unchanged. All pixel values of Fig. 2a and Fig. 2b are in range [0, 255] when computing the & loss. The same process conducted on Fig. 2d-f, which we could also find that the null space of region A has an influence on region B but has little influence on region A. Hence, we could project the principal vectors of region into the null space of region to edit region A yet keep region B unchanged, thus accomplishing the precise generation control only by linear operation of latent codes.
Editing with Null Space Projection. Here, we will validate the effectiveness of the null space projection proposed Sec. 3.3 on StyleGAN2 and BigGAN, For convenience, let v; represent the attribute vector obtained by Eq. (6) of a specific region and denote the projected vector through Eq. (7) into the null space. First, we study via StyleGAN2 pre-trained on FFHQ [18]. The eyes and the mouth are chosen as the regions to edit. As shown in Fig. 3, the remaining region changes violently when only using v; control the specific attribute. For example, when closing the eyes of the man, the other attributes such as nose, mouth, and hair have an obvious alteration, and some artifacts appear as well. And when closing the mouth, the size of the face is deformed, and the hair increased as well. On the contrary, when using P: to control the attributes in those images, the editing results are significantly improved, and the changes the other regions are barely perceptible by human eyes. Second, study via BigGAN two categories. As shown in Fig. 3, we select the object center as
5

---

wo Projection
w/ Projection
wo Projection
w Projection
Target
Close Eyes
Target
Close Mouth
Zoom Out Figure Precise generation control via the null space projection on StyleGAN2 [19] and BigGAN [4]. For each image attribute, the original attribute vector identified from the region of interest (under green masks or red boxes) acts as changing the image globally (e.g., the face shape change in the top row and the background change in the bottom row). By contrast, our approach with null space projection leads to more satisfying local editing results.
Target
Zoom In
Target
the region of interest and then obtain the attribute vectors through Meenter Directly using the attribute vector v, will significantly change the background when editing. For instance, the background of the indigo bird will be blurred, and there appear some white dots in the background of the bubble when changing the size. Rather, when using the projected singular vector p,. the surroundings can keep intact as much as possible. All these results are consistent with the phenomenon Fig. 2.
We also give the quantitative results in Tab. la on StyleGAN2. Besides the FID metric, the masked Mean Squared Error (MSE) is also used qualify the change in the rest region when editing specific region. Taking closing eyes as an example, we do not compute the loss using the eyes and their surroundings, shown with green masks in Fig. 3. This metric measures the change in the other region when editing a specific region. Obviously, the low masked MSE means high precision control. As shown in Tab. la, after the null space projection, both FID and MSE metries decrease. Both the qualitative and quantitative results show that after the null space projection, we could achieve more precise control over specific region on StyleGAN2 and BigGAN. More results can be found in the Supplementary Material.
4.2 Generalization from One Image to Others
Results on Versatile GANs, A question here is whether each image needs to compute the corresponding Jacobian for editing since our method requires to compute the Jacobian of the images. The answer no, and we show that the attribute vectors obtained by our algorithm in one image could be used to effectively edit itself or the other images. As shown in Fig. 4, the reference images and the corresponding green masks indicate the images and the regions we use to find the attributes vectors, which are then used to edit the target images. From Fig. 4, we could summarize the following advantages of our method: First, the masks are unnecessary to be aligned between the reference images and target images to manipulate the targets. For example, we obtain an attribute vector that could change the hair color from the masked region the reference image, which can change the hair color of the target images regardless of their hairstyles. And for the car, the body of the target cars varies from the shape, pose, color, etc. However, the attribute vector attained from the masked region of the reference image successfully changed the colors of the car bodies of diverse styles. Second, for conditional GANs like BigGAN, the direction attained from one category can not only be used to edit the category itself but also be used to edit other categories. For instance, we obtain direction from the snowbird that could change its pose, which can change the pose of the other categories such robin, owl, dog, and butterfly. Last but not least, achieve all the local editing by just linearly shifting the latent codes instead of operating in the feature maps, making LowRankGAN easy to use.
Applications. Besides the qualitative results shown in Fig. 4, also give the quantitative results in Fig. 5. And to quantitatively measure the degree the closed eyes or mouth, we count the pixels in the eyes mouth using the segmentation model released by [20]. We first select 10,000 images

---

Target.
Reference
Hair Color
Close Eyes
Close Mouth
Target.
Reference
Ground
Body
Target
Reference
Pose
Figure 4: Generalization of the latent semantic from the local region of one synthesis to other samples StyleGAN2 [19] faces, StyleGAN2 cars, and BigGAN [4]. Our approach does not require the target image to have the same masked region as the reference image. For example, cars can have different shapes and locations. The results BigGAN from one category can be applied to other categories.
suggest that the pose semantic identified
that have open eyes when they have pixels of more than 200. After collecting images, use the attribute vector (closing eyes) obtained from the image shown in Fig. 4 to edit those images. After editing, the number of pixels the eyes each image is reported in Fig. 5a. For the closed mouth, the process is exactly the same as the closed eyes, and the result is reported in Fig. 5b. We also perform statistics on the number pixels the eyes and mouth of the real data and synthesized data and report them in Fig. 5. Interestingly, in real data synthesized data, only a few images have few
7

---

(a) Number of Pixels in Eyes
(b) Number of Pixels in Mouth
Real Data
Spathesized Data
Edited Data
Beal Date
Syntheszzed Date
Edood Data
9016
9034 9332
9058 2970
63
49
61.0
410
6330 505
6U7
319
0-9
10-49
30 99
50-000
0-9
10:r49
50
2c100
Figure 5: Comparison among the real data distribution, synthesized data distribution, and the shifted data distribution with our Low LowRankGAN, regarding closing-eyes and closing- mouth attributes. We use the number of pixels from a segmented object (i.e., eyes or a mouth) to evaluate whether the eyes and mouth closed. By analyzing only one image, people close their eyes or mouths, yielding an application to augment the real data with rare samples.
make around 90%
Table 1:
Quantitative comparison results on (a) the effect of using null space projection, and (b) the image quality from various image editing approaches [14, 25].
(a) Comparison on whether use projection.
(b) Comparison with other methods.
Methods
Close Eyes
Close Mouth
Methods
FID SWDI
User Study
FID MSEJ
FIDJ MSEJ
GANSpace [14]
7.91
10.46
5.4%
w/o Projection
9.14
0.0014
9.47
0.00063
SeFa [25]
7.57
7.50
37.6%
w/ Projection
8.01
0.00099
7.69 0.00020
LowRankGAN (Ours)
7.30
6.70
56.0%
pixels in the eyes (61 and 49 for the real and synthesized data, respectively). This means that the images with closed eyes are rather scarce in real or synthesized data. Thereby, a potential application of our algorithm could be data augmentation. A detailed data augmentation experiment can be found in the Supplementary Material.
4.3 Robustness of LowRankGAN to Regions of Interest
What surprises us is that our method is extremely robust to the mask (or to the region). which further enables our method could be easily used even if we do not have segmentation model contrary the previous works [29, 2, 34, 37]. Ás shown in Fig. 6, for each group, the mask varies from the size, position, and shape, but the attribute vectors same role when editing the target image. For instance, if we want to find some eye-related semantic, like closing eyes in Fig. we could either use segmentation model [20] to get the precise pixels of eyes or just use some coarse masks that users specify as the green masks shown in Fig. 6. The attribute vector of closing the eyes of the target can be derived from any Jacobian of those masked regions. For the church and car, we could either use the segmentation masks or the random masks as the green regions shown in Fig. 6. The same editing results can be achieved by any attribute vectors from the Jacobians of those masked regions, adding clouds to the sky of the church or changing the color of the car to the target image. One of the possible reasons for this robustness might be that the semantic in a specific region will lie in the same latent subspace. Therefore, the subspace for sub-region of this specific region could coincide with the global one, and moving in this subspace will affect the whole region. Note that in the experiments of car and church, for the small mask regions, relatively larger Trelax needed. Here we set Trelax 20.
solve from those masked regions play nearly the
4.4 Comparison with Existing Alternatives
Now we compare our method with the state-of-the-art algorithms. We compare our method with GANSpace [14] and SeFa [25] on StyleGAN2, which are the two state-of-the-art unsupervised approaches to image editing. We find the most relevant vectors that can control the smile and hair in GANSpace and SeFa, according to their papers. As shown in Fig. 7, both GANSpace and SeFa have some degree of influence on the other region when editing specific region. For example, when adding the smile, the identity of GANSpace changed as well as hairstyle. When changing the hair color, GANSpace just has little activation on this attribute, while SeFa has an obvious change in the background. Meanwhile, the eyes of the women are altered as well. Instead, our method has negligible change other attributes when adding the smile changing the hair color.
8

---

Reference
Target
Reference
Target
shutterst
shhutterst
shutterst
shutterst aa
shutterst
Shuttest
Reference
Target.
Robustness of LowRankGAN the region of interest for analysis. For example, to edit the color of car, our algorithm does not necessarily require a rigid segmentation mask but can also work well with an offhand bounding box and give similar results.
Figure 6:
Add Smile
Hair Color
Target
GANSeace
SeFa
LOWRaREGAN (Ows))
GANSpace
SeFa
LowRankGAN (Ous))
Figure 7:
Comparison on image local editing with GANSpace [14] and SeFa [25]. Our
LowRankGAN can better preserve the information beyond the target region.
Besides the qualitative results shown in Fig. 7, we also give the quantitative results in Tab. 1b smiling. About user study, twelve students are invited to perform the evaluation. All students have a computer vision background. Each one is assigned fifty original images and the associated editing results (adding smile) obtained from different methods. Then they are asked to discriminate the image-editing quality of different methods. They are 12 x 50 600 votes. We get 595 valid votes because, in some cases, some students cannot tell which method performs best. The result the user study is reported in Tab. lb. We could see that our method surpasses the others for all metrics.
Conclusion and Limitation
In this work, we propose low-rank subspaces to perform controllable generation in GANs. The low-rank decomposition of the Jacobian matrix established between an arbitrary image and the latent space yields null space, which enables image local editing by simply altering the latent code with need for spatial masks. We also find that the low-rank subspaces identified from the local

---

region of one image can be robustly applicable to other images, revealing the internal semantic-awand structure of the GAN latent space. Extensive experiments demonstrate the powerful performance of our algorithm on precise generation control with well-trained GAN models.
As for the limitation, the proposed RowRanGGAN is able to precisely control the local region of GAN synthesized images. As discussed in Sec. 4.3, our method is robust to the region of interest. For example, to edit the eyes of faces, we do not need accurate eye segmentation yet only require rough region around the eyes. This yields a shortcoming of our approach, which is that it almost fails to achieve pixel-level control. More concretely. our LowRankGAN can only perform editing by treating eyes as a whole, but fails to edit every individual pixel of eyes. We also find that our algorithm tends control both eyes simultaneously. As result, is hard to close only one eye by keeping the other open.
References
[1] D. Bau, J.-Y. Zhu, H. Strobelt, B. Zhou, J. B. Tenenbaum, W. Freeman, and A. Torralba. GAN dissection: Visualizing and understanding generative adversarial networks. In Int. Conf. Learn. Represent. 2019. 2
[2] D. Bau. H. Strobelt, W. Peebles, J. Wulff. B. Zhou, J.-Y. Zhu, and A. Torralba. Semantic photo manipulation with generative image prior. ACM Trans. Graph., 2020. 2,
[3] S. Boyd, N. Parikh, and F. Chu. Distributed optimization and statistical learning via the alternating direction method of multipliers. Now Publishers Inc, 2011. 4
[4] A. Brock, J. Donahue, and K. Simonyan. Large scale GAN training for high fidelity natural image synthesis. In Int. Conf. Learn. Represent. 2019. 1, 2, 5, 6,7
[5] E. J. Candès, X. Li, Y. Ma, and J. Wright. Robust principal component analysis? Journal of the ACM (JACM), 2011. 4
[6] C.-H. Chiu, Y. Koyama, Y. Lai, T. Igarashi, and Y. Yue. Human-in-the--oog in-the-loop differential subspace search in high-dimensional latent space. ACM Trans. Graph., 2020. 2
[7] Y. Choi, Y. Uh, J. Yoo, and J. Ha. StarGAN v2: Diverse image synthesis for multiple domains. In /EEE Conf. Comput. Vis. Pattern Recog., 2020.
[8] E. Collins, R. Bala, B. Price, and Susstrunk. Editing in style: Uncovering the local semantics of GANs. In IEEE Conf. Comput. Vis. Pattern Recog., 2020.
[9] Deng, W. Dong. R. Socher, L.-J. Li, K. Li, Fei-Fei. ImageNet: A large-scale hierarchical image database. In IEEE Conf. Comput. Vis. Pattern Recog., 2009. 5
[10] Donahue and Simonyan. Large seale adversarial representation learning. In Adv. Neural Inform. Process. Syst., 2019.
[11] R. Feng, D. Zhao, and Zha. On noise injection in generative adversarial networks. In Int. Conf. Mach. Learn., 2021. 3
[12] I.. Goetschalekx, A. Andonian, A. Oliva, and P. Isola. GANalyze: Toward visual definitions of cognitive image properties, In Int, Conf. Comput. Vis., 2019. 2,
[13] I. Goodfellow, J. Pouget-Abadie, M. Mirza, Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. Generative adversarial networks. In Adv. Neural Inform. Process. Syst., 2014.
[14] E. Härkönen, A. Hertzmann, Lehtinen, and S. Paris. GANSpace: Discovering interpretable GAN controls. In Adv. Neural Inform. Process. Syst. 2020. 1, 2, 4, 8,9
[15] M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter. GANs trained by two time-scale update rule converge local nash equilibrium. In Adv. Neural Inform. Process. Syst., 2017. 5
[16] X. Huang and S. Belongie. Arbitrary style transfer in real-time with adaptive instance normalization. In Int. Conf. Comput. Vis., 2017.1 1
[17] A. Jahanian, L. Chai, and P. Isola. On the" steerability" of generative adversarial networks. In Int. Conf. Learn. Represent., 2020. 1, 2.3
[18] T. Karras. S. Laine, and T. Aila. A style generator architecture for generative adversarial networks. In IEEE Conf. Comput. Vis. Pattern Recog., 2019. 2,
10

---

[19] T. Karras, S. Laine. M. Aittala, J. Hellsten, J. Lehtinen, and T. Aila. Analyzing and improving the image quality of StyleGAN. In IEEE Conf. Comput. Vis. Pattern Recog, 2020. 1, 5. 6,
[20] C.-H. Lee, 7. Liu, I.. Wu, and P. Luo. MaskGAN: Towards diverse and interactive facial image manipulation. In IEEE Conf. Comput. Vis, Pattern Recog., 2020. 5, 6,
[21] Z. Lin, M. Chen, and Y. Ma. The augmented lagrange multiplier method for exact recovery of corrupted low-rank matrices. arXiv preprint arXiv:1009.5005: 2010. 4
[22] A. Plumerault, H. I., Borgne, and C. Hudelot. Controlling generative models with continuous factors of variations. In Inf. Conf. Learn. Reprexent., 2020. 2.
[23] J. Rabin, G. Peyré, J. Delon, and M. Bernot. Wasserstein barycenter and its application to texture mixing.
In International Conference Scale Space and Variational Methods in Computer Vision, 2011. 5
[24] A. Ramesh, Y. Choi, and Y. LeCun. A spectral regularizer for unsupervised disentanglement. In Int. Conf. Mach. Learn., 2019. 2
[25] Y. Shen and B. Zhou. Closed-form factorization of latent semantics in GANs. In IEEE Conf. Compur. Vis. Pattern Recog,, 2021. 2, 4, 8,9
[26] Y. Shen, J. Gu, X. Tang, and B. Zhou. Interpreting the latent space of GANs for semantic face editing. In IEEE Conf. Comput. Vis. Pattern Recog., 2020. 1, 3,
[27] Y. Shen, C. Yang, X. Tang, and B. Zhou. InterFaceGAN: Interpreting the disentangled face representation Iearned by GANs. IEEE Trans. Pottern Anal. Mach. Intell., 2020. 1,
[28] N. Spingarn-Eliezer Eliezer, R. Banner, and T. Michaeli. GAN steerability without optimization, In Int. Conf. Learn. Represent., 2021. 2
[29] R. Suzuki, M. Koyama, T. Miyato, T. Yonetsuji, and H. Zhu. Spatially controllable image synthesis with internal representation collaging. arXiv preprint arXiv: 1811.1015,, 2018.
[30] A. Voynov and A. Babenko. Unsupervised discovery of interpretable directions in the GAN latent space. In Int. Conf. Mach. Learn., 2020. 1, 2,2,4
[31] B. Wang and C. R. Ponce. The geometry of deep generative image models and its applications. In Int. Conf. Learn. Represent., 2021. 2
[32] T.-C. Wang, M.-Y. Liu, J.-Y. Zhu, G. Liu, A. Tao, J. Kautz, and B. Catanzaro. Video-to-video synthesis. In Adv. Neural Inform. Process. Syst., 2018.
[33] J. Wright and Y. Ma. High- Digeessinnsi Data Analysis with Low-Dimenn Do--Dmimnsinn Models: Principles, Computation, and Applications. Cambridge University Press, 2021. 3
[34] Y. Xu, Y. Shen, J. Zhu, C. Yang, and B. Zhou, Generative hierarchical features from synthesizing images. In IEEE Conf. Comput. Vis. Pattern Recog., 2021. 1,
[35] C. Yang, Y. Shen, and B. Zhou. Semantic hierarchy emerges in deep generative representations for scene synthesis. Int. J. Comput. Vis., 2021.
[36] F. Yu, A. Seff, Y. Zhang. S. Song, T. Funkhouser, and J. Xiao. LSUN: Construction of a large-scale image dataset using deep learning with humans in the loop. arXiv preprint arXiv: 033000000335.. 2015. 5
[37] C. Zhang, Y. Xu, and Y. Shen. Decorating your bedroom: Locally controlling image generation with generative adversarial networks. arXiv preprint urXiv:2105.0822 2021. 2,
[38] J. Zhu, Y. Shen, D. Zhao, and B. Zhou. In-domain GAN inversion for real image editing. Eur: Conf. Comput. Vis., 2020. 1.2
[39] J.-Y. Zhu, T. Park, P. Isola, and A. A. Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. Int. Conf. Comput. Vis., 2017.


Reviews:
[{"review_id": "fPKmfyjruw", "reviewer": "Reviewer_HHnz", "summary": "This paper proposes using low-rank factorization of the Jacobian of generator’s selected output region with respect to its latent noise source in order to perform precise edits of generated images. Specifically, a latent vector in the top-subspace (imply maximium edit impact) of the output region is identified, after which, it is then projected onto the null-space (no edit impact) of the negative region (the region not selected).\n\nContributions:\n1) Examine the low-rank subspaces of of the generator output w.r.t. the latent space and show how to perform edits in an unsupervised manner.\n2) Demonstrate that this process need not be repeated on each image individually, but instead can be performed just once and generalize to many images due to the semantic organization of the generator’s latent space.\n3) Perform several experiments on various image datasets, comparing against other state of the art models, both qualitatively and quantitatively.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality:\nTo my knowledge, this approach is novel and also appears to be much simpler than previous approaches.\n\nQuality: The paper is of good quality.\n\nClarity: There are some small typos in places, but otherwise the paper is clearly written.\n\nSignificance: Precise editing of generated / synthetic images is of great important to the computer vision and graphics communities.\n\n\nOverall, I liked the simplicity of the approach and the results in several cases are impressive. To improve the paper, I would suggest:\n- Figure 2 is visually a bit disappointing relative to the rest of in the paper. What are the units on the $l_1$-loss heatmaps? Is it the sum of the absolute values of the differences across the rgb channels $[0, 255]$? I can see the cloud is slightly larger in (b) than (a), but I can’t visually tell the any difference between (d) and (e). Please comment on any visual difference we should be noticing.\n- Figure 3 (bubble): In order to confirm that the “white dots” were not there before, we need to see a zoomed-out version of the target image.\n- Could you include Figure 4 repeated but with $v$ (the original attribute vector) instead of $p$ (the vector projected into the null-space)?\n- Figure 6 is interesting but reveals a downside to the approach. The fact that different masks lead to similar edits means a lack of precise control of edits. Can you please discuss this limitation?\n\n\nMinor:\n- I don’t have a counterexample, but this sentence raised alarm bells for me: “first to demonstrate effectiveness of low-rank factorization in deep generative models”. It seems like an overstatement.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I included a suggestion for discussion of a specific limitation in my review above. The authors might also want to add a comment on the use of GAN editing for producing fake news among other things.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1.5"}, "extra_text": {}}, {"review_id": "Kv75zfQBXe_", "reviewer": "Reviewer_6ack", "summary": "The authors propose a method to exploit the low-rank structure of the Jacobian of a deep generative model to perform precise image editing. The Jacobian corresponding to a region of an image is constructed and its null space is computed. By varying the input to the generator in the null space along the eigenvectors, only one attribute can be changed at a time, leaving the rest of the image unchanged. Experiments show that the method outperforms strong baselines in terms of semantic editing. ", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I think this paper has many strengths:\n- The overall method is quite simple and easy to implement.\n- The method is unsupervised which is attractive.\n- That finding the relevant directions in the latent space using only image, and using it for other images is very interesting and could be useful.\n- Both the qualitative and quantitative results are very promising. The experiments have been well done in general, the user study in particular is an important addition to understand image quality. Editing is shown on many different attributes on several datasets.\n- The paper is well-written and easy to understand.\n\nQuestions and suggestions for improvement:\n- I understand the algorithm and why the eigenvectors are calculated. However, what is the guarantee that every eigenvector corresponds to a single semantic attribute? I can easily imagine a case where the eigenvector could change both the hair color and the eye color, for example, depending on the correlations in the dataset. In other words, what are the assumptions made on the dataset used to train the generator?\n\n- What is the step size taken in the direction of the eigenvectors? Can the authors show results where the step size goes from a larger negative number to a larger positive number in an experiment involving zooming, pose changes and eye opening/closing? \n\n- The experimental results in Figure 5 are very interesting and the author mention that this suggests that the method could be used for data augmentation. Would it be possible for the authors to substantiate this claim with a small experiment. This can make the paper even stronger. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I am not that the authors have discussed limitations of their work. Perhaps they can outline it in the rebuttal and add them to the paper. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5 hours"}, "extra_text": {}}, {"review_id": "9oTwMSc2b7", "reviewer": "Reviewer_xDMm", "summary": "The authors propose an approach to identify meaningful latent directions in a GAN latent space. First, they factorize the Jacobian of a generator map $G: z \\to \\mathrm{RGB}$, treating its principal axis as the desired directions. Then they fine-tune them in a supervised regime to guarantee only the regions of the interest being edited. Though some of the results look intriguing, a lot of relevant comparisons with prior works is missed and the experiments are not sufficient.", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- some of the results look promising, especially, the transferability of the directions discovered by a single image to the whole dataset.\n\nWeakness:\n- As for me, it was really hard to read the paper. Its structure is unclear, the introduction gives just a vague idea of the contribution. The experiments have no implementation details which makes them unreproducible as the authors do not supply the code.\n- Though the authors state that one of the main contributions is the locality of the editings produced by their directions, the comparison with the prior works is unfair as they use additional supervision provided by either segmentation masks or region boundaries. It would be reasonable to provide the editing comparison of the proposed approach with e.g. [1, 2].\n- It remains unclear how a direction discovered for a particular $z$ is translated to the whole latent space. Do the authors use always the same z to discover the interpretable directions?\n- On line 85 the authors state that \"to our best knowledge, we are the first to demonstrate the effectiveness of low-rank factorization...\". The very close idea to utilize a generator's Hessian eigenvectors as interpretable directions was first proposed in [3] which is not cited. A close idea to use SVD or Hessian-based editable weights decomposition was also discussed in [4].\n- line 157: it is unclear, how exactly the authors determine if a particular vector influences region $A$.\n- The only quantitive evaluation is presented in Table 1. The authors perform a comparison with unsupervised approaches ignoring supervision approaches. It is also unclear, how do they scale the magnitudes of the latent shifts for different methods. For a particular latent direction $h$, I would be happy to see a plot of points [ $\\mathrm{FID}(G(z + \\alpha \\cdot h))$, $\\mathbb{E}~ \\mathrm{LPIPS}(G(z), G(z + \\alpha \\cdot h))$ ] with different shift scales $\\alpha$. The first coordinate is the FID of the shifted latent distribution and the second coordinate is the expectation of an image deformation in terms of the lpips-distance.\n- line 251: \"Thereby, a potential application of our algorithm could be data augmentation.\" it would be preferable if the authors could demonstrate this experimentally.\n\n[1] Editing in Style: Uncovering the Local Semantics of GANs, Edo Collins et al.\n\n[2] Mask-Guided Discovery of Semantic Manifolds in Generative Models, Mengyu Yang et al.\n\n[3] The Hessian Penalty: A Weak Prior for Unsupervised Disentanglement, William Peebles et al.\n\n[4] Navigating the GAN Parameter Space for Semantic Image Editing, Anton Cherepkov et al.\n\n---------\nUPD: I raise my score 3 -> 4\nI still think that the quantitive evaluation is insufficient as it consists of a table in the main text with only two attributes and one figure in the response for a single attribute. In my opinion, some of the baselines (though, with higher supervision) should also be reported.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "There is no limitations section.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper used the Jacobean of the generator and Robust PCA (on covariance of Jacobean) to find directions for editing a region A that keep a region B in an image intact. By basically projecting the robust PCA eigenvectors of region A on the null space of robust PCA in region B. The idea is intuitive and makes sense and works in practice. 

After reading the reviews and the paper. I have few suggestions to clarify the paper. Although reviewers said the paper is easy to follow, and I am familiar with works in the topic  I found the description of the method to be quite unclear and not precise. 

Please include a sketch of an algorithm in the paper that makes it clear to the reader what is your method, and please also make your code available since one can not reproduce this paper in its current version. 

is  your method  I) 
1. The method starts by finding $z_A$ and $z_{B}$ , to produce $x_{A},x_{B}$  and compute Robust PCA of $J^{\top}J$ at $z_{A}$ and $z_{B}$ get $V_{A}, V_{B}$
2. Project $V_{A}$ on null space of $V_{B}$, get $P_{A}$
3. $G(z_{A}+ \alpha P_{A,j})$  and merge with $x_{B}$ to produce new images

or II) 
1. Find $z$ to reproduce image then compute robust PCA on $JG_{A}(z)^{\top}JG_{A}(z)$ , by restricting the covariance to pixels in region A , same for region B
2. Continue with steps 2 and 3 from above using latent $z$
 

I am left with a question to the authors that you should clarify in the final manuscript . I may have overlooked it in appendix but I did not see it. The method works because of projection to the null space of region B or because of  the use of Robust PCA. 
What if you replaced in your method Robust PCA with just PCA , I think this ablation  should be included in the paper.

I recommend a weak accept but I ask the authors to include a sketch of clear procedure of how their method is applied, open source their code, and to do the ablation robust PCA versus PCA. 

Decision: Accept (Poster)