On Noise Injection in Generative Adversarial Networks

1 INTRODUCTION . Noise injection is usually applied as regularization to cope with overfitting or facilitate generalization in neural networks ( Bishop , 1995 ; An , 1996 ) . The effectiveness of this simple technique has also been proved in various tasks in deep learning , such as learning deep architectures ( Hinton et al. , 2012 ; Srivastava et al. , 2014 ; Noh et al. , 2017 ) , defending adversarial attacks ( He et al. , 2019 ) , facilitating stability of differentiable architecture search with reinforcement learning ( Liu et al. , 2019 ; Chu et al. , 2020 ) , and quantizing neural networks ( Baskin et al. , 2018 ) . In recent years , noise injection1 has attracted more and more attention in the community of Generative Adversarial Networks ( GANs ) ( Goodfellow et al. , 2014a ) . Extensive research shows that it helps stabilize the training procedure ( Arjovsky & Bottou , 2017 ; Jenni & Favaro , 2019 ) and generate images of high fidelity ( Karras et al. , 2019a ; b ; Brock et al. , 2018 ) . In practice , Fig . 1 shows significant improvement in hair quality due to noise injection . Particularly , noise injection in StyleGAN ( Karras et al. , 2019a ; b ) has shown the amazing capability of helping generate sharp details in images , shedding new light on obtaining high-quality photo-realistic results using GANs . Therefore , studying the underlying principle of noise injection in GANs is an important theoretical work of understanding GAN algorithms . In this paper , we propose a theoretical framework to explain and improve the effectiveness of noise injection in GANs . Our framework is motivated from a geometric perspective and also combined with the results of optimal transportation problem in GANs ( Lei et al. , 2019a ; b ) . Our contributions are listed as follows : • We show that the existing GAN architectures , including Wasserstein GANs ( Arjovsky et al. , 2017 ) , may suffer from adversarial dimension trap , which severely penalizes the property of generator ; • Based on our theory , we attempt to explain the properties that noise injection is applied in the related literatures ; • Based on our theory , we propose a more proper form for noise injection in GANs , which can overcome the adversarial dimension trap . Experiments on the state-of-the-art GAN architecture , StyleGAN2 ( Karras et al. , 2019b ) , demonstrate the superiority of our new method compared with original noise injection used in StyleGAN2 . 1It suffices to note that noise injection here is totally different from the research field of adversarial attacks raised in Goodfellow et al . ( 2014b ) . To the best of our knowledge , this is the first work that theoretically draws the geometric picture of noise injection in GANs . 2 RELATED WORKS . The main drawbacks of GANs are unstable training and mode collapse . Arjovsky et al . ( Arjovsky & Bottou , 2017 ) theoretically analyze that noise injection directly to the image space can help smooth the distribution so as to stabilize the training procedure . The authors of Distribution-Filtering GAN ( DFGAN ) ( Jenni & Favaro , 2019 ) then put this idea into practice and prove that this technique will not influence the global optimality of the real data distribution . However , as the authors pointed out in ( Arjovsky & Bottou , 2017 ) , this method depends on the amount of noise . Actually , our method of noise injection is essentially different from these ones . Besides , they do not provide a theoretical vision of explaining the interactions between injected noises and features . BigGAN ( Brock et al. , 2018 ) splits input latent vectors into one chunk per layer and projects each chunk to the gains and biases of batch normalization in each layer . They claim that this design allows direct influence on features at different resolutions and levels of hierarchy . StyleGAN ( Karras et al. , 2019a ) and StyleGAN2 ( Karras et al. , 2019b ) adopt a slightly different view , where noise injection is introduced to enhance randomness for multi-scale stochastic variations . Different from the settings in BigGAN , they inject extra noise independent of latent inputs into different layers of the network without projection . Our theoretical analysis is mainly motivated by the success of noise injection used in StyleGAN ( Karras et al. , 2019a ) . Our proposed framework reveals that noise injection in StyleGAN is a kind of fuzzy reparameterization in Euclidean spaces , and we extends it into generic manifolds ( section 4.3 ) . 3 THE INTRINSIC DRAWBACKS OF TRADITIONAL GANS . 3.1 OPTIMAL TRANSPORTATION AND DISCONTINUOUS GENERATOR . Traditional GANs with Wasserstein distance are equivalent to the optimal transportation problem , where the optimal generator is the optimal transportation map . However , there is rare chance for the optimal transportation map to be continuous , unless the support of Brenier potential is convex ( Caffarelli , 1992 ) . Considering that the Brenier potential of Wasserstein GAN is determined by the real data distribution and the inverse map of the generator , it is highly unlikely that its support is convex . This means that the optimal generator will be discontinuous , which is a fatal limitation to the capacity of GANs . Based on that , Lei et al . ( Lei et al. , 2019a ) further point out that traditional GANs will hardly converge or converge to one continuous branch of the target mapping , thus leading to mode collapse . They then propose to find the continuous Brenier potential instead of the discontinuous transportation map . In the next paragraph , we show that this solution may not totally overcome the problem that traditional GANs encounter due to structural limitations of neural networks . Besides , it suffices to note that their analysis is built upon the Wasserstein distance , and may not be directly applied to the Jenson-Shannon divergence or KL divergence . We refer the readers to Lei et al . ( 2019a ) ; Caffarelli ( 1992 ) for more detailed analysis . 3.2 ADVERSARIAL DIMENSION TRAP . In addition to the above discontinuity problem , another drawback is the relatively low dimension of latent spaces in GANs compared with the high variance of details in real-world data . Taking face images as an example , the hair , freckles , and wrinkles have extremely high degree of freedom , which make traditional GANs often fail to capture them . The repetitive application of non-invertible CNN blocks makes the situation even worse . Non-invertible CNN , which is a singular linear transformation , will drop the intrinsic dimensions of feature manifolds ( Strang et al. , 1993 ) . So during the feedforward procedure of the generator , the dimensions of feature spaces will keep being dropped . Then it will have a high chance that the valid dimension of the input latent space is lower than that of the real data . The relatively lower dimension of the input latent space will then force the dimension of the support with respect to the distribution of generated images lower than that of the real data , as no smooth mappings increase the dimension . However , the discriminator , which measures the distance of these two distributions , will keep encouraging the generator to increase the dimension up to the same as the true data . This contradictory functionality , as we show in the theorem bellow , incurs severe punishment on the smoothness and invertibility of the generative model , which we refer as the adversarial dimension trap . Theorem 1 . 2 For a deterministic GAN model and generator G : Z → X , if the dimension of the input latent Z is lower than that of data manifold X , then at least one of the two cases must stand : 1. the generator can not be Lipschitz ; 2. the generator fails to capture the data distribution and is unable to perform inversion . Namely , for an arbitrary point x ∈ X , the possibility of G−1 ( x ) = ∅ is 1 . The above theorem stands for a wide range of GAN loss functions , including Wasserstein divergence , Jenson-Shannon divergence , and other KL-divergence based losses . Notice that this theorem implies much worse situation than it states . For any open sphere B in the data manifold X , the generator restricted in the pre-image of B also follows this theorem , which suggests bad properties of nearly every local neighborhood . This also suggests that the above consequences of Theorem 1 may both stand . As in some subsets , the generator may successfully capture the data distribution , while in some others , the generator may fail to do so . The first issue in section 3.1 can be addressed by not learning the generator directly with continuous neural network components . We will show how our method addresses the second issue . 4 FUZZY REPARAMETERIZATION . The generator G in the traditional GAN is a composite of sequential non-linear feature mappings , which can be denoted as G ( z ) = fk ◦ fk−1 ◦ · · · ◦ f1 ( z ) , where z ∼ N ( 0 , 1 ) is the standard Gaussian . Each feature mapping , which is typically a single layer convolutional neural network ( CNN ) plus non-linear activations , carries out a certain purpose such as extracting multi-scale patterns , upsampling , or merging multi-head information . The whole network is then a deterministic mapping from the latent space Z to the image space X . We propose to replace f i ( x ) , 1 ≤ i ≤ k , with gi ( x ) = µi ( x ) + σi ( x ) , ∼ N ( 0 , 1 ) , x ∈ gi−1 ◦ · · · ◦ g1 ( Z ) . ( 1 ) We call it as Fuzzy Reparameterization ( FR ) as it in fact learns fuzzy equivalence relation of the original features , and uses reparameterization to model the high-dimensional feature manifolds . We believe that this is the proper form of generalization of noise injection in StlyeGAN , and will show the reasons and benefits in the following sub-sections . 2As the common practice in the manifold learning community , our theorems and discussions are based on Riemannian manifolds . Proofs to all the theorems are included in the supplementary material . It is not hard to see that our proposed method can be viewed as the extension of the reparameterization trick in VAEs ( Kingma & Welling , 2013 ) . While the reparameterization trick in VAEs serves to a differentiable solution to learn through random variables and is only applied in the latent space , our method is a type of deep noise injection in feature maps of each layer to correct the defect in GAN architectures . Therefore , the purposes of using reparameterization in these two scenarios are different , thus leading to thoroughly different theories that are presented in the next sub-section . 4.1 HANDLING ADVERSARIAL DIMENSION TRAP WITH NOISE INJECTION . As Sard ’ s theorem tells us ( Petersen et al. , 2006 ) , the key to solve the adversarial dimension trap is to avoid mapping low-dimensional feature spaces into high-dimensional ones , which looks like a pyramid structure in the generator . However , we really need the pyramid structure in practice because the final output dimension of generated images is much larger than that of the input space . So the solution could be that , instead of mapping into the full feature spaces , we choose to map only onto the skeleton of the feature spaces and use random noise to fill up the remaining space . For a compact manifold , it is easy to find that the intrinsic dimension of the skeleton set can be arbitrarily low by applying Heine–Borel theorem to the skeleton ( Rudin et al. , 1964 ) . By this way , the model can escape from the adversarial dimension trap . Now we develop the idea in detail . The whole idea is based on approximating the manifold by the tangent polyhedron . Assume that the feature spaceM is a Riemannian manifold embedded in Rm . Then for any point µ ∈ M , the local geometry induces a coordinate transformation from a small neighborhood of µ inM to its projection onto the tangent space TµM at µ by the following theorem . Theorem 2 . Given Riemannian manifoldM embedded in Rm , for any point µ ∈M , we let TµM denote the tangent space at µ . Then the exponential map Expµ induces a smooth diffeomorphism from a Euclidean ball BTµM ( 0 , r ) centered at O to a geodesic ball BM ( µ , r ) centered at µ inM . Thus { Exp−1µ , BM ( µ , r ) , BTµM ( 0 , r ) } forms a local coordinate system ofM in BM ( µ , r ) , which we call the normal coordinates . Thus we have BM ( µ , r ) = Expµ ( BTµM ( 0 , r ) ) = { τ : τ = Expµ ( v ) , v ∈ BTµM ( 0 , r ) } . ( 2 ) Theorem 3 . The differential of Expµ at the origin of TµM is identity I . Thus Expµ can be approximated by Expµ ( v ) = µ+ Iv + o ( ‖v‖2 ) . ( 3 ) Thus , if r in equation ( 2 ) is small enough , we can approximate BM ( µ , r ) by BM ( µ , r ) ≈ µ+ IBTµM ( 0 , r ) = { τ : τ = µ+ Iv , v ∈ BTµM ( 0 , r ) } . ( 4 ) Considering that TµM is an affine subspace of Rm , the coordinates on BTµM ( 0 , r ) admit an affine transformation into the coordinates on Rm . Thus equation ( 4 ) can be written as BM ( µ , r ) ≈ µ+ IBTµM ( 0 , r ) = { τ : τ = µ+ rT ( µ ) , ∈ B ( 0 , 1 ) } . ( 5 ) We remind the readers that the linear component matrix T ( µ ) differs at different µ ∈ M and is decided by the local geometry near µ . In the above formula , µ defines the center point and rT ( µ ) defines the shape of the approximated neighbor . So we call them a representative pair of BM ( µ , r ) . Picking up a series of such representative pairs , which we refer as the skeleton set , we can construct a tangent polyhedron H ofM . Thus instead of trying to learn the feature manifold directly , we adopt a two-stage procedure . We first learn a map f : x 7→ [ µ ( x ) , σ ( x ) ] ( σ ( x ) ≡ rT ( µ ( x ) ) ) onto the skeleton set , then we use noise injection g : x 7→ µ ( x ) + σ ( x ) , ∼ U ( 0 , 1 ) ( uniform distribution ) to fill up the flesh of the feature space as shown in Figure 2 . However , the real world data often include fuzzy semantics . Even long range features could share some structural relations in common . It is unwise to model it with unsmooth architectures such as locally bounded sphere and uniform distribution . Thus we borrow the idea from fuzzy topology ( Ling & Bo , 2003 ; Zhang & Zhang , 2005 ; Murali , 1989 ; Recasens , 2010 ) which is designed to address this issue . It is well known that for any distance metrics d ( · , · ) , e−d ( µ , · ) admits a fuzzy equivalence relation for points near µ , which is similar with the density of Gaussian . The fuzzy equivalence relation can be viewed as a suitable smooth alternative to the sphere neighborhood BM ( µ , r ) . Thus we replace the uniform distribution with unclipped Gaussian3 . Under this settings , the first stage mapping in fact learns a fuzzy equivalence relation , while the second stage is a reparameterization technique . Notice that the skeleton set can have arbitrarily low dimension by Heine–Borel theorem . So the first-stage map can be smooth and well conditioned . For the second stage , we can show that it possesses a smooth property in expectation by the following theorem . Theorem 4 . Given f : x 7→ [ µ ( x ) , σ ( x ) ] T , f is locally Lipschitz and ‖σ‖∞ = o ( 1 ) . Define g ( x ) ≡ µ ( x ) + σ ( x ) , ∼ N ( 0 , 1 ) ( standard Gaussian ) . Then for any bounded set U , ∃L > 0 , we have E [ ‖g ( x ) − g ( y ) ‖2 ] ≤ L‖x − y‖2 + o ( 1 ) , ∀x , y ∈ U . Namely , the principal component of g is locally Lipschitz in expectation . Specifically , if the definition domain of f is bounded , then the principal component of g is globally Lipschitz in expectation .

Summary: To summarize, this paper proposed a new noise injection method that is easy to implement and is able to replace the original noise injection method in StyleGAN 2. The approach is supported by detailed theoretical analysis and impactful performance improvement on GAN training and inversion. The results show that they are able to achieve a considerable improvement on DCGAN and StyleGAN2.