Low-Rank Subspaces in GANs

1 Introduction . Generative Adversarial Networks ( GANs ) , which can produce high-fidelity images visually indistinguishable from real ones [ 18 , 19 , 4 ] , have made tremendous progress in many real-world applications [ 16 , 39 , 32 , 10 , 38 , 27 , 7 , 34 ] . Some recent studies [ 12 , 26 , 17 ] show that pre-trained GAN models spontaneously learn rich knowledge in latent spaces such that moving latent codes towards some certain directions can cause corresponding attribute change in images . In this way , we are able to control the generation process by identifying semantically meaningful latent subspaces . For latent semantic discovery , one of the most straightforward ways is to first generate a collection of image synthesis , then label these images regarding a target attribute , and finally find the latent separation boundary through supervised training [ 12 , 26 , 17 ] . Prior work [ 30 , 25 , 14 ] has pointed out that the above pipeline could be limited by the labeling step ( e.g. , unable to identify semantics ∗ denotes the corresponding author . 1Code is available at https : //github.com/zhujiapeng/LowRankGAN/ . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . beyond well-defined annotations ) and proposed to find steerable directions of the latent space in an unsupervised manner , such as using Principal Component Analysis ( PCA ) [ 14 ] . However , existing approaches prefer to discover global semantics , which alter the entire image as a whole and fail to relate a latent subspace to some particular image region . Consequently , they highly rely on spatial masks [ 29 , 8 , 2 ] to locally control the GAN generation , which is of more practical usage . In this work , we bridge this gap by introducing a low-rank subspace of GAN latent space . Our approach , called LowRankGAN , starts with an arbitrary synthesis together with a region of interest , such as the eyes and nose in a face image or the ground in a scene image . Then , we manage to relate the latent space and the image region via computing the Jacobian matrix with the pre-trained generator . After that comes the core part of our algorithm , which is to perform low-rank factorization based on the resulting Jacobian matrix . Different from other analytic methods , the low-rank factorization of Jacobians is capable of uncovering the intrinsic low-dimensional representation of attribute manifold , thus helping solve more precise editing direction to the target attribute . Besides , the low-rank factorization naturally yields a null space of attributes such that moving the latent code within this subspace mainly affects the remaining image region . With such an appealing property , we can achieve local image control by projecting an attribute vector into the null space and using the projected result to modulate the latent code . This observation is far beyond trivial because the latent code is commonly regarded to act on the entire feature map of GAN generator [ 13 , 4 , 18 ] . We also find that the low-rank subspace derived from local regions of one image is widely applicable to other images . In other words , our approach requires only one sample instead of abundant data for analysis . More astonishingly , supposing that we identify a changing-color boundary on one car image , we can use this boundary to change the color of cars from other images , even the cars locate at a different spatial area or with different shapes . This suggests that the GAN latent space is locally semantic-aware . Moreover , LowRankGAN is strongly robust to the starting image region for Jacobian matrix computation . For instance , to control the sky in synthesized images , our algorithm does not necessarily rely on a rigid sky segmentation mask but can also satisfyingly work with a casual bounding box that marks the sky region ( partially or completely ) , making it much easy to use in practice . Our main contributions are summarized as follows . First , we study low-rank subspaces of the GAN latent space and manage to locally control the GAN image generation by simply modulating the latent code without the need for spatial masks . Second , our method can discover latent directions regarding local regions of only one image yet generalize to all images , shedding light on the intrinsic semantic-aware structure of GAN latent space . Third , we conduct comprehensive experiments to demonstrate the effectiveness and robustness of our algorithm on a wide range of models and datasets . 2 Related Work . Interpreting well-trained GAN models [ 1 , 17 , 26 ] has recently received wide attention because this can help us understand the internal representation learned by GANs [ 34 ] and further control the generation process [ 35 , 38 ] . It is shown that some subspaces in the GAN latent space are highly related to the attributes occurring in the output images . To identify these semantically meaningful subspaces , existing approaches fall into two folds , i.e. , supervised [ 12 , 27 , 17 , 22 ] and unsupervised [ 30 , 14 , 25 , 28 ] . On the one hand , supervised approaches seek help from off-the-shelf attribute classifiers [ 12 , 27 ] or simple image transformations [ 17 , 22 ] to annotate a set of synthesized data and then learn the latent subspaces from the labeled data . On the other hand , unsupervised approaches try to discover steerable latent dimensions by statistically analyzing the latent space [ 14 ] , maximizing the mutual information between the latent space and the image space [ 30 ] , or exploring model parameters [ 25 , 28 ] . However , all of the subspaces detected by these approaches tend to control global image attributes such that the entire image will be modified if we edit a particular attribute . To achieve local image control , a common practice is to introduce a spatial mask [ 29 , 8 , 2 , 34 , 37 ] , which works together with the intermediate feature maps of the GAN generator . Compared to prior work , our LowRankGAN has the following differences : no spatial masks of objects are needed to perform precise local editing , and generic attribute vectors can be obtained with only one image synthesis and arbitrary local regions of interest without any training or statistical analysis on a large amount of data . To our best knowledge , we are the first to demonstrate the effectiveness of low-rank factorization in interpreting generative adversarial models . Although there are also some works using the Jacobian matrix to analyze the GAN latent space [ 24 , 6 , 31 ] , the low-rank subspace enables precise generation control by providing the indispensable null space . 3 Low-Rank GAN . Let z ∈ Rdz and G ( · ) denote the input latent code and the generator of a GAN , respectively . Prior work [ 12 , 26 , 17 ] has observed the relationship between some latent subspaces and image attributes such that we can achieve semantic manipulation on the synthesized sample , G ( z ) , by linearly transforming the latent code xedit = G ( z + αn ) , ( 1 ) where n denotes an attribute vector within the latent space , and α is the editing strength . However , the latent subspaces identified by existing approaches tend to manipulate the entire image as a whole and fail to control a local image region . Differently , our proposed LowRankGAN confirms that Eq . ( 1 ) can also be used to locally control the image generation . The following parts describe how to discover the low-rank subspaces from a pre-trained GAN model as well as its practical usage . 3.1 Degenerate Jacobian . First , we give the definition of Jacobian matrix . Let the real image x ∈ Rdx be in the dx-dimensional space and the input prior z ∈ Rdz be of dimension dz . For an arbitrary point z , the Jacobian matrix Jz of the generator G ( · ) with respect to z is defined as ( Jz ) j , k = ∂G ( z ) j∂zk , where ( Jz ) j , k is the ( j , k ) -th entry of Jz ∈ Rdx×dz . By the Taylor series , the first-order approximation to the edited result can be written as G ( z + αn ) = G ( z ) + αJzn+ o ( α ) . ( 2 ) An effective editing direction should significantly alter G ( z ) , which can be attained by maximizing the following variance n = argmax ‖n‖2=1 ‖G ( z + αn ) −G ( z ) ‖22 ≈ argmax ‖n‖2=1 α2nTJTz Jzn . ( 3 ) It is easy to know that the above optimization admits a closed-form solution , which is the eigenvector associated with the largest eigenvalue . Therefore , the subspace spanned by editing directions is the principal subspace of JTz Jz . For GAN architectures , previous work [ 11 ] has proven and demonstrated that the Jacobian matrix JTz Jz is degenerate , which is described by the following theorem . Theorem 1 Assume that the real data X lie in some underlying manifold of dimension dim ( X ) embedded in the pixel space . Let zk ∈ Rdz denote the latent code of the k-th layer of the generator of GAN over X and Jzk be the Jacobian of the generator with respect to zk . Then we have rank ( JTzk+1Jzk+1 ) ≤ rank ( JTzkJzk ) ≤ dz , where rank ( JTz Jz ) denotes the rank of the Jacobian . Further , if dim ( X ) < dz , then rank ( JTzkJzk ) < dz . Theorem 1 says that the rank of the Jacobian JTz Jz might be progressively reduced , and the manifold characterized by the generator admits a low-dimensional structure under a mild condition . Therefore , the attribute subspace must admit the analogous structure , which can be found by low-rank factorization of the Jacobian JTz Jz . 3.2 Low-Rank Jacobian Subspace . In practice , the numerical evaluation indicates that the Jacobian is actually of full rank , implying that it is perturbed with noise during the highly nonlinear generation process . If we assume that such perturbation is sparse , then the low-dimensional structure of the attribute manifold can be perfectly revealed by low-rank factorization of JTz Jz with corruption . Given a low-rank matrix M corrupted with sparse noise , the low-rank factorization can be formulated as M = L + S , where L is the low-rank matrix and S is the corrupted sparse matrix . Directly solving this problem is actually very hard due to the non-convexity of the optimization . Hopefully , it can reduce to a tractable convex problem [ 33 ] : min L , S ‖L‖∗ + λ‖S‖1 s.t . M = L+ S , ( 4 ) where ‖L‖∗ = ∑ i σi ( M ) is the nuclear norm defined by the sum of all singular values , ‖S‖1 =∑ ij |Mij | is the ` 1 norm defined by the sum of all absolute values in S , and λ is a positive weight parameter to balance the sparsity and the low rank . The convex optimization in Eq . ( 4 ) is also referred to as Principal Component Pursuit ( PCP ) [ 5 ] , which can be solved by the Alternating Directions Method of Multipliers ( ADMM ) [ 21 , 3 ] . The detailed introduction of low-rank factorization and ADMM is presented in the Supplementary Material . Suppose that the solution for JTz Jz in Eq . ( 4 ) is M = JTz Jz = L ∗ + S∗ , ( 5 ) where L∗ is the low-rank representation of JTz Jz , which has rank r corresponding to r attributes in images . By singular value decomposition ( SVD ) , we are able to specify those attributes U , Σ , V T = SVD ( L∗ ) , ( 6 ) where Σ is a diagonal matrix sorted by singular values , U and V = [ v1 , . . . , vr , . . . , vdz ] are the matrices of left and right singular matrices containing r attribute vectors associated with r attributes . With V , we can simply edit a specific attribute of an image by using [ v1 , . . . , vr ] according to Eq . ( 1 ) . In this paper , we mainly focus on the local control of an image . So we compute the Jacobian matrix with a specified region Mregion in an image .

Summary: This paper proposes using low-rank factorization of the Jacobian of generator’s selected output region with respect to its latent noise source in order to perform precise edits of generated images. Specifically, a latent vector in the top-subspace (imply maximium edit impact) of the output region is identified, after which, it is then projected onto the null-space (no edit impact) of the negative region (the region not selected).  Contributions: 1) Examine the low-rank subspaces of of the generator output w.r.t. the latent space and show how to perform edits in an unsupervised manner. 2) Demonstrate that this process need not be repeated on each image individually, but instead can be performed just once and generalize to many images due to the semantic organization of the generator’s latent space. 3) Perform several experiments on various image datasets, comparing against other state of the art models, both qualitatively and quantitatively.