Low-Rank Subspaces in GANs

1 Introduction . Generative Adversarial Networks ( GANs ) , which can produce high-fidelity images visually indistinguishable from real ones [ 18 , 19 , 4 ] , have made tremendous progress in many real-world applications [ 16 , 39 , 32 , 10 , 38 , 27 , 7 , 34 ] . Some recent studies [ 12 , 26 , 17 ] show that pre-trained GAN models spontaneously learn rich knowledge in latent spaces such that moving latent codes towards some certain directions can cause corresponding attribute change in images . In this way , we are able to control the generation process by identifying semantically meaningful latent subspaces . For latent semantic discovery , one of the most straightforward ways is to first generate a collection of image synthesis , then label these images regarding a target attribute , and finally find the latent separation boundary through supervised training [ 12 , 26 , 17 ] . Prior work [ 30 , 25 , 14 ] has pointed out that the above pipeline could be limited by the labeling step ( e.g. , unable to identify semantics ∗ denotes the corresponding author . 1Code is available at https : //github.com/zhujiapeng/LowRankGAN/ . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . beyond well-defined annotations ) and proposed to find steerable directions of the latent space in an unsupervised manner , such as using Principal Component Analysis ( PCA ) [ 14 ] . However , existing approaches prefer to discover global semantics , which alter the entire image as a whole and fail to relate a latent subspace to some particular image region . Consequently , they highly rely on spatial masks [ 29 , 8 , 2 ] to locally control the GAN generation , which is of more practical usage . In this work , we bridge this gap by introducing a low-rank subspace of GAN latent space . Our approach , called LowRankGAN , starts with an arbitrary synthesis together with a region of interest , such as the eyes and nose in a face image or the ground in a scene image . Then , we manage to relate the latent space and the image region via computing the Jacobian matrix with the pre-trained generator . After that comes the core part of our algorithm , which is to perform low-rank factorization based on the resulting Jacobian matrix . Different from other analytic methods , the low-rank factorization of Jacobians is capable of uncovering the intrinsic low-dimensional representation of attribute manifold , thus helping solve more precise editing direction to the target attribute . Besides , the low-rank factorization naturally yields a null space of attributes such that moving the latent code within this subspace mainly affects the remaining image region . With such an appealing property , we can achieve local image control by projecting an attribute vector into the null space and using the projected result to modulate the latent code . This observation is far beyond trivial because the latent code is commonly regarded to act on the entire feature map of GAN generator [ 13 , 4 , 18 ] . We also find that the low-rank subspace derived from local regions of one image is widely applicable to other images . In other words , our approach requires only one sample instead of abundant data for analysis . More astonishingly , supposing that we identify a changing-color boundary on one car image , we can use this boundary to change the color of cars from other images , even the cars locate at a different spatial area or with different shapes . This suggests that the GAN latent space is locally semantic-aware . Moreover , LowRankGAN is strongly robust to the starting image region for Jacobian matrix computation . For instance , to control the sky in synthesized images , our algorithm does not necessarily rely on a rigid sky segmentation mask but can also satisfyingly work with a casual bounding box that marks the sky region ( partially or completely ) , making it much easy to use in practice . Our main contributions are summarized as follows . First , we study low-rank subspaces of the GAN latent space and manage to locally control the GAN image generation by simply modulating the latent code without the need for spatial masks . Second , our method can discover latent directions regarding local regions of only one image yet generalize to all images , shedding light on the intrinsic semantic-aware structure of GAN latent space . Third , we conduct comprehensive experiments to demonstrate the effectiveness and robustness of our algorithm on a wide range of models and datasets . 2 Related Work . Interpreting well-trained GAN models [ 1 , 17 , 26 ] has recently received wide attention because this can help us understand the internal representation learned by GANs [ 34 ] and further control the generation process [ 35 , 38 ] . It is shown that some subspaces in the GAN latent space are highly related to the attributes occurring in the output images . To identify these semantically meaningful subspaces , existing approaches fall into two folds , i.e. , supervised [ 12 , 27 , 17 , 22 ] and unsupervised [ 30 , 14 , 25 , 28 ] . On the one hand , supervised approaches seek help from off-the-shelf attribute classifiers [ 12 , 27 ] or simple image transformations [ 17 , 22 ] to annotate a set of synthesized data and then learn the latent subspaces from the labeled data . On the other hand , unsupervised approaches try to discover steerable latent dimensions by statistically analyzing the latent space [ 14 ] , maximizing the mutual information between the latent space and the image space [ 30 ] , or exploring model parameters [ 25 , 28 ] . However , all of the subspaces detected by these approaches tend to control global image attributes such that the entire image will be modified if we edit a particular attribute . To achieve local image control , a common practice is to introduce a spatial mask [ 29 , 8 , 2 , 34 , 37 ] , which works together with the intermediate feature maps of the GAN generator . Compared to prior work , our LowRankGAN has the following differences : no spatial masks of objects are needed to perform precise local editing , and generic attribute vectors can be obtained with only one image synthesis and arbitrary local regions of interest without any training or statistical analysis on a large amount of data . To our best knowledge , we are the first to demonstrate the effectiveness of low-rank factorization in interpreting generative adversarial models . Although there are also some works using the Jacobian matrix to analyze the GAN latent space [ 24 , 6 , 31 ] , the low-rank subspace enables precise generation control by providing the indispensable null space . 3 Low-Rank GAN . Let z ∈ Rdz and G ( · ) denote the input latent code and the generator of a GAN , respectively . Prior work [ 12 , 26 , 17 ] has observed the relationship between some latent subspaces and image attributes such that we can achieve semantic manipulation on the synthesized sample , G ( z ) , by linearly transforming the latent code xedit = G ( z + αn ) , ( 1 ) where n denotes an attribute vector within the latent space , and α is the editing strength . However , the latent subspaces identified by existing approaches tend to manipulate the entire image as a whole and fail to control a local image region . Differently , our proposed LowRankGAN confirms that Eq . ( 1 ) can also be used to locally control the image generation . The following parts describe how to discover the low-rank subspaces from a pre-trained GAN model as well as its practical usage . 3.1 Degenerate Jacobian . First , we give the definition of Jacobian matrix . Let the real image x ∈ Rdx be in the dx-dimensional space and the input prior z ∈ Rdz be of dimension dz . For an arbitrary point z , the Jacobian matrix Jz of the generator G ( · ) with respect to z is defined as ( Jz ) j , k = ∂G ( z ) j∂zk , where ( Jz ) j , k is the ( j , k ) -th entry of Jz ∈ Rdx×dz . By the Taylor series , the first-order approximation to the edited result can be written as G ( z + αn ) = G ( z ) + αJzn+ o ( α ) . ( 2 ) An effective editing direction should significantly alter G ( z ) , which can be attained by maximizing the following variance n = argmax ‖n‖2=1 ‖G ( z + αn ) −G ( z ) ‖22 ≈ argmax ‖n‖2=1 α2nTJTz Jzn . ( 3 ) It is easy to know that the above optimization admits a closed-form solution , which is the eigenvector associated with the largest eigenvalue . Therefore , the subspace spanned by editing directions is the principal subspace of JTz Jz . For GAN architectures , previous work [ 11 ] has proven and demonstrated that the Jacobian matrix JTz Jz is degenerate , which is described by the following theorem . Theorem 1 Assume that the real data X lie in some underlying manifold of dimension dim ( X ) embedded in the pixel space . Let zk ∈ Rdz denote the latent code of the k-th layer of the generator of GAN over X and Jzk be the Jacobian of the generator with respect to zk . Then we have rank ( JTzk+1Jzk+1 ) ≤ rank ( JTzkJzk ) ≤ dz , where rank ( JTz Jz ) denotes the rank of the Jacobian . Further , if dim ( X ) < dz , then rank ( JTzkJzk ) < dz . Theorem 1 says that the rank of the Jacobian JTz Jz might be progressively reduced , and the manifold characterized by the generator admits a low-dimensional structure under a mild condition . Therefore , the attribute subspace must admit the analogous structure , which can be found by low-rank factorization of the Jacobian JTz Jz . 3.2 Low-Rank Jacobian Subspace . In practice , the numerical evaluation indicates that the Jacobian is actually of full rank , implying that it is perturbed with noise during the highly nonlinear generation process . If we assume that such perturbation is sparse , then the low-dimensional structure of the attribute manifold can be perfectly revealed by low-rank factorization of JTz Jz with corruption . Given a low-rank matrix M corrupted with sparse noise , the low-rank factorization can be formulated as M = L + S , where L is the low-rank matrix and S is the corrupted sparse matrix . Directly solving this problem is actually very hard due to the non-convexity of the optimization . Hopefully , it can reduce to a tractable convex problem [ 33 ] : min L , S ‖L‖∗ + λ‖S‖1 s.t . M = L+ S , ( 4 ) where ‖L‖∗ = ∑ i σi ( M ) is the nuclear norm defined by the sum of all singular values , ‖S‖1 =∑ ij |Mij | is the ` 1 norm defined by the sum of all absolute values in S , and λ is a positive weight parameter to balance the sparsity and the low rank . The convex optimization in Eq . ( 4 ) is also referred to as Principal Component Pursuit ( PCP ) [ 5 ] , which can be solved by the Alternating Directions Method of Multipliers ( ADMM ) [ 21 , 3 ] . The detailed introduction of low-rank factorization and ADMM is presented in the Supplementary Material . Suppose that the solution for JTz Jz in Eq . ( 4 ) is M = JTz Jz = L ∗ + S∗ , ( 5 ) where L∗ is the low-rank representation of JTz Jz , which has rank r corresponding to r attributes in images . By singular value decomposition ( SVD ) , we are able to specify those attributes U , Σ , V T = SVD ( L∗ ) , ( 6 ) where Σ is a diagonal matrix sorted by singular values , U and V = [ v1 , . . . , vr , . . . , vdz ] are the matrices of left and right singular matrices containing r attribute vectors associated with r attributes . With V , we can simply edit a specific attribute of an image by using [ v1 , . . . , vr ] according to Eq . ( 1 ) . In this paper , we mainly focus on the local control of an image . So we compute the Jacobian matrix with a specified region Mregion in an image .

Summary: The authors propose an approach to identify meaningful latent directions in a GAN latent space. First, they factorize the Jacobian of a generator map $G: z \to \mathrm{RGB}$, treating its principal axis as the desired directions. Then they fine-tune them in a supervised regime to guarantee only the regions of the interest being edited. Though some of the results look intriguing, a lot of relevant comparisons with prior works is missed and the experiments are not sufficient.