On Noise Injection in Generative Adversarial Networks
Authors: Ruili Feng
Deli Zhao
Zheng-Jun Zha
Venue:  (2021)
Keywords: Generative Adversarial Networks
StyleGAN
learning theory

Abstract: Noise injection is an effective way of circumventing overfitting  and enhancing generalization in machine learning, the rationale of which has been validated in deep learning as well.  Recently, noise injection exhibits surprising performance when
  generating high-fidelity images in Generative Adversarial Networks (GANs). Despite its successful applications in GANs, the mechanism of its validity is still unclear. In this paper, we propose a geometric framework to theoretically analyze the role of noise injection in GANs. Based on Riemannian geometry, we successfully model the noise injection framework as fuzzy equivalence on geodesic normal coordinates. Guided by our theories, we find that existing methods are incomplete and a new strategy for noise injection is devised. Experiments on image generation and GAN inversion demonstrate the superiority of our method.

Under review as a conference paper at ICLR 2021
ON NOISE INJECTION IN GENERATIVE ADVERSARIAL NETWORKS
Anonymous authors Paper under double-blind review
ABSTRACT
Noise injection is an effective way of cireumventing overfitting and enhancing generalization in machine learning, the rationale of which has been validated deep learning as well. Recently, noise injection exhibits surprising effectiveness when generating high-fidelity images in Generative Adversarial Networks (e.g. StyleGAN). Despite its successful applications in GANs, the mechanism of its validity is still unclear. In this paper, we propose a geometric framework to theoret- ically analyze the role of noise injection in GANs. Based Riemannian geometry, we successfully model the noise injection framework as fuzzy equivalence on geodesic normal coordinates. Guided by theories, we find that existing methods are incomplete and a new strategy for noise injection is devised. Experiments on image generation and GAN inversion demonstrate the superiority of our method.
INTRODUCTION
Noise injection usually applied regularization cope with overfitting or facilitate generalization in neural networks (Bishop 1995 An 1996). The effectiveness of this simple technique has also been proved in various tasks deep learning, such as learning deep architectures (Hinton al.. 2012, Srivastava et al. 2014 Noh et al. (007)) defending adversarial attacks (He et al.) (011)) facilitating stability of differentiable architecture search with reinforcement learning (Liu et al. 2019 Chu et al. (020), and quantizing neural networks (Baskin et al.) 2018). In recent years, noise injection has attracted more and more attention in the community of Generative Adversarial Networks (GANs) (Goodfellow et al. 2014a). Extensive research shows that it helps stabilize the training procedure (Arjovsky Bottou| 2017 Jenni Favaro 2019) and generate images of high fidelity (Karras al.. 20199 b) Brock et al. 2018). In practice, Fig. shows significant improvement
in hair quality due to noise injection.
Particularly, noise injection in StyleGAN (Karras et al. 2019a b) has shown the amazing capability of helping generate sharp details in images, shedding new light on obtaining high-quality photo-realistic results using GANs. Therefore, studying the underlying principle of noise injection in GANs is important theoretical work of understanding GAN algorithms. In this paper, we propose theoretical framework explain and improve the effectiveness of noise injection in GANs. Our framework is motivated from geometric perspective and also combined with the results of optimal transportation problem in GANs (Lei al. 2019a b). Our contributions are listed as follows:
We show that the existing GAN architectures, including Wasserstein GANs (Arjovsky et al. 2017), may suffer from adversarial dimension trap, which severely penalizes the property of generator;
Based on our theory, we attempt to explain the properties that noise injection applied in the related literatures;
Based on our theory, we propose more proper form for noise injection in GANs, which can overcome the adversarial dimension trap. Experiments on the state-of-the-art GAN architecture, StyleGAN2 (Karras et al.] 2019b), demonstrate the superiority of our new method compared with original noise injection used in StyleGAN2.
It suffices to note that noise injection here is totally different from the research field of adversarial attacks raised in Goodfellow al. (0114)).

---

Under review as a conference paper at ICLR 2021
Increasing noise injection depth
Standard Dev
Figure I: Noise injection significantly improves the detail quality of generated images. From left to right, inject extra noise the generator layer by layer. We can see that hair quality is clearly improved. Varying the injected noise and visualizing the standard deviation over 100 different seeds, we can find that the detail information such as hair. parts of background, and silhouettes are most involved, while the global information such as identity and pose is less affected.
To the best of our knowledge, this is the first work that theoretically draws the geometric picture of noise injection in GANs.
2 RELATED WORKS
The main drawbacks of GANs are unstable training and mode collapse. Arjovsky al. (Arjovsky & Bottou| 2017) theoretically analyze that noise injection directly to the image space can help smooth the distribution so as stabilize the training procedure. The authors of Distribution-Fillsn (DFGAN) (Jenni & Favaro 2019) then put this idea into practice and prove that this technique will not influence the global optimality of the real data distribution. However, as the authors pointed out in (Arjovsky & Bottou, 2017), this method depends on the amount of noise. Actually, our method of noise injection is essentially different from these ones. Besides, they not provide theoretical vision of explaining the interactions between injected noises and features.
GAN
BigGAN (Brock et al. 2018) splits input latent vectors into one chunk per layer and projects each chunk to the gains and biases of batch normalization in each layer. They claim that this design allows direct influence on features at different resolutions and levels of hierarchy. StyleGAN (Karras et al.) 2019a) and StyleGAN2 (Karras al. 2019b) adopt slightly different view, where noise injection is introduced to enhance randomness for multi- scale stochastic variations. Different from the settings in BigGAN, they inject extra noise independent of latent inputs into different layers of the network without projection. Our theoretical analysis is mainly motivated by the success of noise injection used in StyleGAN (Karras al.) (2019)). Our proposed framework reveals that noise injection in StyleGAN is kind of fuzzy reparameterization in Euclidean spaces, and we extends it into generic manifolds (section 4.3
3
THE INTRINSIC DRAWBACKS OF TRADITIONAL GANS
3.1 OPTIMAL TRANSPORTATION AND DISCONTINUOUS GENERATOR
Traditional GANs with Wasserstein distance are equivalent to the optimal transportation problem. where the optimal generator the optimal transportation map. However, there rare chance for the optimal transportation map to be continuous, unless the support of Brenier potential is convex (Caffarelli 1992). Considering that the Brenier potential of Wasserstein GAN is determined by the real data distribution and the inverse map of the generator, it is highly unlikely that its support is convex. This means that the optimal generator will be discontinuous, which is a fatal limitation to the capacity of GANs. Based on that, Lei et al. (Lei et al.) 2019a) further point out that traditional GANs will hardly converge or converge to one continuous branch the target mapping, thus leading to mode collapse. They then propose to find the continuous Brenier potential instead of the discontinuous
2

---

Under review as a conference paper at ICLR 2021
transportation map. In the next paragraph, show that this solution may not totally overcome the problem that traditional GANs encounter due to structural limitations of neural networks. Besides, it suffices to note that their analysis is built upon the Wasserstein distance, and may not be directly applied to the Jenson-Shannon divergence or KI. divergence. We refer the readers to Lei et al. 2019a); Caffarelli 1992) for more detailed analysis.
3.2 ADVERSARIAL DIMENSION TRAP
In addition to the above discontinuity problem, another drawback the relatively low dimension of latent spaces in GANs compared with the high variance of details in real-world data. Taking face images as an example, the hair, freckles, and wrinkles have extremely high degree of freedom, which make traditional GANs often fail to capture them. The repetitive application of non-invertible CNN blocks makes the situation even worse. Non-invertible CNN, which is a singular linear transformation, will drop the intrinsic dimensions of feature manifolds (Strang et al.) 1993). So during the feedforward procedure of the generator, the dimensions of feature spaces will keep being dropped. Then it will have a high chance that the valid dimension of the input latent space lower than that of the real data. The relatively lower dimension of the input latent space will then force the dimension of the support with respect to the distribution of generated images lower than that of the real data, as no smooth mappings increase the dimension. However, the discriminator, which measures the distance of these two distributions, will keep encouraging the generator to increase the dimension up to the same the true data. This contradictory functionality, as show in the theorem bellow, incurs severe punishment on the smoothness and invertibility of the generative model, which we refer as the adversarial dimension trap.
Theorem 1. For a deterministic GAN model and generator G 2 - X, if the dimension of the input latent is lower than that of data manifold X, then at least one of the two cases must stand:
1. the generator cannot be Lipschitz;
2. the generator fails to capture the data distribution and is unable to perform inversion. Namely, for an arbitrary point x: E X, the possibility of (x) = 1.
The above theorem stands for wide range of GAN loss functions, including Wasserstein divergence, Lenson-Shannon divergence, and other KL.-divergence based losses. Notice that this theorem implies much worse situation than it states. For any open sphere B in the data manifold X, the generator restricted in the pre-image of also follows this theorem, which suggests bad properties of nearly every local neighborhood. This also suggests that the above consequences of Theorem I may both stand. As in some subsets, the generator may successfully capture the data distribution, while in some others, the generator may fail to do so.
The first issue in section 3.1 can be addressed by not learning the generator directly with continuous neural network components. We will show how our method addresses the second issue.
4 FUZZY REPARAMETERIZAATION
The generator G' in the traditional GAN is composite of sequential non-lineat feature mappings, which can be denoted as G(z) f fk-1 oo-...... f (z), where z A 1) is the standard Gaussian. Each feature mapping, which is typically single layer convolutional neural network (CNN) plus non-linear activations, carries out a certain purpose such as extracting multi-scale patterns, upsampling, or merging multi-head information. The whole network is then deterministic mapping from the latent space to the image space X. We propose to replace f'(x), 1 sis k, with
g' (x) (x) + o² (x)c, 1), g o.... o
We call it as Fuzzy Reparameterization (FR) as in fact learns fuzzy equivalence relation of the original features, and uses reparameterization to model the high-dimensional feature manifolds. We believe that this is the proper form of generalization of noise injection in StlyeGAN, and will show the reasons and benefits in the following sub-sections.
2 As the common practice in the manifold learning community, theorems and discussions are based on Riemannian manifolds. Proofs to all the theorems are included in the supplementary material.
3

---

Under review as a conference paper at ICLR 2021
T.M
11
(a) Skeleton of the manifold.
(b) Representative pair.
Figure 2: Illustration of the skeleton set and representative pair. The blue curve (a) is the skeleton. In (b), the dashed sphere in M is the geodesic ball, while the solid sphere in Tr.M is its projection onto the tangent space. The normal vector ń determines the final affine transformation into the Euclidean space.
is not hard to see that our proposed method can be viewed as the extension of the reparameterization trick in VAEs (Kingma & Welling, 2013). While the reparameterization trick in VAEs serves to differentiable solution to learn through random variables and is only applied in the latent space, our method is a type of deep noise injection in feature maps of each layer correct the defect in GAN architectures. Therefore, the purposes of using reparameterization in these two scenarios are different, thus leading to thoroughly different theories that are presented in the next sub-section.
4.1 HANDLING ADVERSARIAL DIMENSION TRAP WITH NOISE INJECTION
As Sard's theorem tells us (Petersen et al. 2006), the key solve the adversarial dimension trap is to avoid mapping low-dimensional feature spaces into high-dimensional ones, which looks like a pyramid structure in the generator. However, really need the pyramid structure in practice because the final output dimension of generated images is much larger than that of the input space. So the solution could be that, instead of mapping into the full feature spaces, we choose map only onto the skeleton of the feature spaces and use random noise to fill up the remaining space. For a compact manifold, it easy find that the intrinsic dimension of the skeleton set can be arbitrarily low by applying Heine- Borel theorem to the skeleton (Rudin et al. 1964). By this way, the model can escape from the adversarial dimension trap.
Now we develop the idea in detail. The whole idea is based approximating the manifold by the tangent polyhedron. Assume that the feature space M a Riemannian manifold embedded in R. Then for any point μ E M, the local geometry induces a coordinate transformation from small neighborhood of μ in M to its projection onto the tangent space Ty.M at μ by the following theorem.
Theorem 2. Given Riemannian manifold M embedded in R any point μ E M. we let T,M denote the tangent space at Then the exponential map Expp induces a smooth diffeomorphism from a Euclidean ball Br, (0, (0,r) r) centered O to a geodesic ball BM r) centered at p in M. Thus Expp1, BM r), BT, (0, r}} forms local coordinate system of M in By(p,r) which call the normal coordinates. Thus we have
BMCP r) = Exp,(BBTT (BT,,,(00)))) = fr: Expp(e), (0,,(
(2)
Theorem 3. The differential of Expp at the origin of TyM identity I. Thus Expa can be approximated by
Expμ(t) μ + Iv o(11=12))
(3)
Thus, if in equation 2 is small enough, we can approximate (M(,,T) by
BM(ur)) r) se μ IBT,.((0 = ( u Iu, E (T,)(
(4)
Considering that TyM is affine subspace of Rm", the coordinates (0,, admit an affine transformation into the coordinates on R. Thus equation (4) can be writien as
(x, se p (B,, (T : 4 rT()e, B(0, 1)}.
(5)
We remind the readers that the linear component matrix T(p) differs different μ E M and is decided by the local geometry near μ. In the above formula, a defines the center point and (T()) defines the shape of the approximated neighbor. So call them representative pair of BM( r).
4

---

Under review as a conference paper at ICLR 2021
Picking up series of such representative pairs, which refer the skeleton set, construct a tangent polyhedron H. of M. Thus instead of trying to learn the feature manifold directly, we adopt a two-stage procedure. We first learn map f + [u(z), (x)) (a(x)) rT(u(z))) onto the skeleton set, then we use noise injection x + p(x) + ( U(0,1) 1) (uniform distribution)
to fill up the flesh of the feature space as shown in Figure
However, the real world data often include fuzzy semantics. Even long range features could share some structural relations in common. It is unwise to model it with unsmooth architectures such as locally bounded sphere and uniform distribution. Thus we borrow the idea from fuzzy topology (Ling Bo 2003 Zhang & Zhang, 2005 Murali 1989, Recasens 2010) which designed address this issue. It is well known that for any distance metries di(,;" ") -ddaa) admits fuzzy equivalence relation for points near 14, which is similar with the density of Gaussian. The fuzzy equivalence suitable smooth alternative the sphere neighborhood (M r). Thus
relation can be viewed we replace the uniform distribution with unclipped Gaussiar Under this settings, the first stage mapping in fact learns a fuzzy equivalence relation, while the second stage is reparameterization technique. Notice that the skeleton set can have arbitrarily low dimension by Heine- Morel theorem. So the first-stage map can smooth and well conditioned. For the second stage, we can show that possesses smooth property in expectation by the following theorem.
[n(æ)), o(x)] I is locally Lipschitz and ||0|| = o(1). Define g(x) = {((x) + (x)e, ~ NNO, (standard Gaussian). Then for any bounded set U, 31. we have ([]lg(x)= [ s L|| yll2 + o(1), Vx,y U. Namely, the principal component of g is locally Lipschitz in expectation. Specifically, if the definition domain of f is bounded, then the principal component of g is globally Lipschitz in expectation.
Theorem 4. Given J
4.2 PROPERTIES OF NOISE INJECTION
As we have discussed, traditional GANs face two challenges: the discontinuous optimal generator and the adversarial dimension trap. Both of the two challenges will lead to an unsmooth generator: Theorem M alss implies an unstable training procedure because the gradient explosion that may occur on the generator. Besides, the dimension reduction in GAN will make it hard to fit high-variance details as information keeps compressed along channels in the generator. With noise injection in the network of the generator, however, we can theoretically overcome such problems if the representative pairs are constructed properly to capture the local geometry. In this case, our model does not need to fit the discontinuous optimal transportation map, nor the image manifold with higher dimension than that the network architecture can handle. Thus the training procedure will not encourage the unsmooth generator, and can proceed more stably. Also, the extra noise can compensate the loss of information compression so as to capture high-variance details, which has been discussed and illustrated in StyleGAN (Karras et al. 2019a). We will evaluate the performance of our method from these aspects in section 5
4.3 CHOICE OF p(x) AND (x)
As μ stands for particular point in the feature space, we simply model it by the traditional deep CNN architectures. (xx) designed to fit the local geometry of p(x)). According our theory, the local geometry should only admit minor differences from u(x). Thus we believe that (x) should be determined by the spatial and semantic information contained ((x),, and should characterize the local variations of the spatial and semantic information. The deviation of pixel-wise sum along channels of feature maps in StyleGAN2 highlights the semantic variations like hair, parts of background, and silhouettes,
the standard deviation map over sampling instances shows in Fig. I. This observation suggests that the sum along channels identifies the local semantics we expect to reveal. Thus it should be directly connected to (xx) map = DCNN(2) (x: from the deep CNN, which is specific point in the feature manifold, the sum along its channels
pursuing here. For given feature
Pajk
Pijk:
'A detailed analysis about why unclipped Gaussian should be applied is offered in the supplementary material.
5

---

Under review as a conference paper at ICLR 2021
where i enumerates all the feature maps of μl, while j. k enumerate the spatial footprint of μ in its h rows and w columns, respectively. The resulting f is then a spatial semantic identifier, whose variation corresoonds to the local semantic variation. We then normalize μ to obtain spatial semantic coefficient matrix s with
mean(
Pyke
h
k=1
8 jt meam(p).
(7)
max((s)) 1si$,
max
mar([s|)
Recall that the standard deviation of over sampling instances highlights the local variance in semantics. Thus a can be decomposed into two independent components: 8m that corresponds the main content of the output image, which is almost invariant under changes of injected noise; S, that associated with the variance that is induced by the injected noise, and is nearly orthogonal to the main content. We assume that this decomposition can be attained by an affine transformation such that
Sd A s b = Sm Su, μ O,
where * denotes element-wisg wise matrix multiplication, and O denotes the matrix whose all elements zeros. To avoid numerical instability, we add 1 whose all elements are ones to the above decomposition, such that its condition number will not get exploded,
s' asd + (1 a),
The regularized 9m component is then used to enhance the main content in μ. and the regularized sy component is then used to guide the variance of injected noise. The final output o is then calculated as
ra μl + ra * c, ~ NN 1).
(10)
In the above procedure, A, b.r. and a are learnable parameters. Note that in the last equation, we do not need decompose s' into S, and 8m, as s, is designed to be nearly orthogonal to μi, and will automatically drop the 8, component, and
8m is nearly invariant. Thus adding an invariant bias to the variance of injected noise. There are alternative forms for μ and a with respect to various GAN architectures. However, modeling μ by deep CNNs and deriving through the spatial and semantic information of μ are universal for GANs, as they comply with our theorems. We further conduct ablation study verify the effectiveness of the above procedure. The related results can be found in the supplementary material.
amounts to
Using our formulation, noise injection in StyleGAN2 can be written as follows:
# DCNN μ
No,
(11)
where r a learnable scalar parameter. This can be viewed as a special case of our method, where T(u) in (5) is set to identity. Under this settings, the local geometry is assumed to be everywhere identical among the feature manifold, which suggests globally Euclidean structure. While our theory supports this simplification and specialization, and more usual occasions, where the feature manifolds are non-Fuclidean. We denote this fashion of noise injection additive noise injection, and will extensively study its performance compared with our choice in the following section.
choice of p(z)) and (z)) can suit broader
5 EXPERIMENT
We conduct experiments on benchmark datasets including FFHQ faces. LSUN objects, and CIFAR- 10. The GAN models we use are the baseline DCGAN (Radford et al. 2015) (originally without noise injection) and the state-of-the-an StyleGAN2 (Karras et al. 2019b) (originally with additive noise injection). For StyleGAN2, performance with respect to Path Perceptual Length (PPL) score. Besides, we apply the experimental settings from StyleGAN2.
config-e e in the original paper due to that config-e achieves the best

---

Under review as a conference paper at ICLR 2021
Table 1: Comparison for different generator architectures.
FFHQ
LSUN-Church
GAN arch
PPL ())
FID (4)
PPL (1)
FID (4)
DCGAN
2.97
45.29
33.30
51.18
DCGAN + Additive noise
3.14
44.22
22.97
54.01
DCGAN FR (Ours)
2.83
40.06
22.53
46.31
Bald StyleGAN2
28.44
6.87
425.7
6.44
StyleGAN2
16.20
7.29
123.6
6.80
EyyeGAAN-NNPPahhee + FR (Ours)
16.02
7.14
178.9
5.75
StyleGAN2 + FR (Ours)
13.05
7.31
119.5
6.86
FFHQ
LSUN-Church
Bald StyleGAN2
StyleGAN2
StyleGANN2-oPathhkgg FR
StyleGAN2 FR
Figure 3: Synthesized images of different StyleGAN2- models.
Image synthesis. PPL (Zhang et al. 2018) has been proven an effective metric for measuring structural consistency of generated images (Karras et al. 2019b). Considering its similarity to the expectation of the Lipschitz constant of the generator, it can also be viewed as a quantification of the smoothness of the generator. The path length regularizer is proposed in StyleGAN2 generated image quality by explicitly regularizing the Jacobian of the generator with respect to the intermediate latent space. We first compare the noise injection methods with the bald StyleGAN2, which remove the additive noise injection and path length regularizer in StyleGAN2. As shown Table [] we can find that all types of noise injection significantly improve the PPL scores. is worth noting that our method without path length regularizer can achieve comparable performance against the standard StyleGAN2 on the FFHQ dataset, and the performance can further improved if combined with path length regularizer. Considering the extra GPU memory consuming of path length regularizer in training, we think that our method offers a computation-friendj) alternative to StyleGAN2 as we observe smaller GPU memory occupation of our method throughout all the experiments. Another benefit is that our method accelerates the convergence to the optimal FID scores, illustrated in Figure 4 This superior convergence be explained with theorem. The underlying reason that our method offers an architecture that more consistent with the intrinsic geometry of the feature space. Thus is easier for the network to fit.
improve
7

---

Under review as a conference paper at ICLR 2021
14-
StyleGAN2
StyleGAN2 StyleGANN-NoPathReg
12
13-
StyleGANZ-NoPathiRgg FR
12
FID
10
9-
0.6
0.8
1.
1.2
1.4
1.6
0.6
0.8
1.0
1.2
14
1.6
1.8
Iterations (x 107)
Iterations (x (x10²)
(a) FFHQ
(b) LSUN-Church
Figure 4: Comparison of FID curves. All curves are terminated by the optimal FIDs in 25M training iterations.
CIFAR-10
DCGAN
DCGAN Additive noise PPL=77.9, FID=84.8. IS=4.73
DCGAN FR PPL=69.9, 83.2. IS=4.64
PPL=101 4.1.4..ID88...8 IS=4.46
Cat-Selected
StyleGAN2
StyleGAN2 + FR PPL=106. MC=0.686, TTMC=1.45, FID=13.4
PPL=115. MC=0.725, TTMC=1.54 FID=12.7
Figure Image synthesis on CIFAR-10 and L.SUN cats.
For the LSUN-Church dataset, we observe obvious improvement FID scores compared with StyleGAN2. We believe that this is because the LSUN-Church data are scene images and contain various semantics multiple objects, which are hard fit for the original StyleGAN2 that is more suitable for single object synthesis. So our FR architecture offers more degrees of freedom to the generator to fit the true distribution of the dataset. In cases, our method is superior to StyleGAN2 in both PPL and FID scores. This proves that our noise injection method is one used in StyleGAN2. For DCGAN, as does not possess the intermediate latent space, we cannot facilitate it with path length regularizer. So only compare the additive noise injection with FR method. Through all the cases we can find that our method achieves the best performance PPL and FID scores.
powerful than the
Table 2: Conditions for different GAN architectures. MC and TTMC (Top Thousand Mean Condition) are mean condition and mean value of the largest 1000 conditions at 50000 randomly sampled points in the input space, respectively. The intermediate latent space taken as the input space.
FFHQ
LSUN-Church
GAN arch
MC (L)
TTMC (L)
MC (L)
TTMC (L)
Bald StyleGAN2 StyleGAN2 EyyleGAN2-NoPathee FR
0.943
2.81
2.31
6.31
0,666
1.27
0.883
1.75
0.766
2.39
1.71
4.74
StyleGAN2 + FR
0.530
1.05
0.773
1.51
8

---

Under review as a conference paper at ICLR 2021
We also study whether our choice for p(z) and (2)) can be applied to broader occasions. We further conduct experiments on a cat dataset which consists of 100 thousand selected images from 800 thousand LSUN- Cat images by PageRank algorithm (Zhou et al. 2004). For DCGAN, conduct extra experiments on CIFAR-10 to test whether our method could succeed in multi-class image synthesis. The results are reported in Figure[5] We can that our method still outperforms the compared methods in PPL scores and the FID scores are comparable, indicating that the proposed noise injection is more favorable of preserving structural consistency of generated images with real ones.
Numerical stability. As we have analyzed before, noise injection should be able to improve the numerical stability of GAN models. To evaluate it, we examine the condition number of different GAN architectures. The condition number of a given function f is defines as Horn & Johnson (013)
[I(xx) - f(x + Ax)1//
Cfnd(f) = lim sup 6-50 ||
(12)
|
S4
measures how sensitive a function is to changes or errors in the input. A function with a high condition number is said to be ill-conditioned. Considering the numerical infeasibility of the sup operator in the definition of condition number, we resort to the following alternative approach. We first sample batch of 50000 pairs of (Input. Perturbation) from the input distribution and the perturbation Ar ~ (0, le-4), and then compute the corresponding condition numbers. We compute the mean value and the mean value of the largest 1000 values of these 50000 condition numbers as Mean Condition (MC) and Top Thousand Mean Condition (TTMC) respectively to evaluate the condition of GAN models. We report the results in Table 2 where we can find that noise injection significantly improves the condition of GAN models, and our proposed method dominates the performance.
GAN inversion. StyleGAN2 makes use of latent style space that is capable of enabling control- lable image modifications. This characteristic motivates us to study the image embedding capability of our method via GAN inversion algorithms (Abdal et al. 2019) as it may help further leverage the potential of GAN models. From the experiments, we find that the StyleGAN2 model is prone to work well for full-face, non-blocking human face images. For this type of images, we observe comparable performance for all the GAN architectures. We think that this is because those images are close to the 'mean' face of FFHQ dataset (Karras et al. (209a)), thus easy to learn for the StyleGAN- models. For faces of large pose or partially occluded ones, the capacity of compared models differs significantly. Noise injection methods outperform the bald StyleGAN2 by a large margin, and our method achieves the best performance. The detailed implementation and results are reported in the supplementary material.
6 CONCLUSION
In this paper, we propose theoretical framework to explain the effect of noise injection technique in GANs. We prove that the generator can easily encounter difficulty unsmoothness, and noise injection is an effective approach to addressing this issue. Based on our theoretical framework, we also derive a more proper formulation for noise injection. We conduct experiments on various datasets to confirm its validity. Despite the superiority compared with the existing methods, however, it is still unclear whether our formulation is optimal and universal for different networks. In future work, we will further investigate the realization of noise injection, and attempt to find more powerful way to characterize local geometries of feature spaces.
REFERENCES
Rameen Abdal, Yipeng Qin, and Peter Wonka. Image2StyleGAN: How to embed images into the StyleGAN latent space? In Proceedings of the IEEE International Conference on Computer Vision, pp. 4432-4441, 2019.
Guozhong An. The effects of adding noise during backpropagation training on a generalization performance. Neural computation, 8(3):643-67) 667, 1996.

---

Under review as a conference paper at ICLR 2021
Martin Arjovsky and Léon Bottou. Towards principled methods for training generative adversarial networks. arxiv e-prints, art. arXiv preprint 00000.044222222 2017.
Martin Arjovsky, Soumith Chintala, and Léon Bottou. arXiv:1701.07875 2017.
Wasserstein GAN. arXiv preprint
Chaim Baskin, Natan Liss, Yoav Chai, Evgenii Zheltonozhskii, Eli Schwartz, Raja Giryes, Avi Mendelson, and Alexander M Bronstein. Nice: Noise injection and clamping estimation for neural network quantization. arXiv preprint frXiv1/81000012,, 2018.
Christopher M. Bishop. Training with noise is equivalent to tikhonov regularization. Neural computation, 7(1): 1995.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large-scale GAN training for high fidelity natural image synthesis, 2018.
Luis Caffarelli. The regularity of mappings with a convex potential. Journal of the American Mathematical Sociery, 5(1):99-104, 1992.
Xiangxiang Chu, Bo Zhang, and Xudong Li. Noisy differentiable architecture search. arXiv preprint 0:Xiv:2005.03566, 2020.
Klaus Deimling. Nonlinear functional analysis. Courier Corporation, 2010.
lan J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks, 2014a.
lan Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv PreprintarXivv1412..20
2014b.
Zhezhi He, Adnan Siraj Rakin, and Deliang Fan. Parametric noise injection: Trainable randomness to improve deep neural network robustness against adversarial attack. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 588-5977777 537,597740 2019.
Geoffrey F. Hinton, Nitish Srivastava, Alex Krizhevsky, Ilya Sutskever, and Ruslan R. Salakhutdinov. Improving neural networks by preventing co-adaptation of feature detectors. arXiv preprint arXiv:1207.0580 2012.
Roger Horn and Charles R. Johnson. Matrix Analysis. Cambridge University Press, 2013.
Simon Jenni and Paolo Favaro. On stabilizing generative adversarial training with noise. In Pro- ceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 12145-12153, 2019.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 4401-4410, 2019a.
Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila, Analyzing and improving the image quality of StyleGAN. arXiv preprint arXiv: (9t2::9922044 04955,, 2019b.
Diederik P Kingma and Max Welling. arXiv:1312.6114, 2013.
Auto-encoding variational
bayes.
arXiv preprint
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Na Lei, Yang Guo, Dongsheng An, Xin Qi, Zhongxuan Luo, Shing-Tung Yau, and Xianfeng Gu. Mode collapse and regularity of optimal transportation maps. arXiv preprint arXiv: 02934. 2019a.
Na Lei, Kehua Su, Li Cui, Shing-Tung Yau, and Xianfeng David Gu. A geometric view of optimal transportation and generative model. Computer Aided Geometric Design, 68:1-21, 2019b.
Zhang Ling and Zhang Bo. Theory of fuzzy quotient space (methods of fuzzy granular computing). 2003.
10

---

Under review as a conference paper at ICLR 2021
Hanxiao Liu, Karen Simonyan, and Yiming Yang. Darts: Differentiable architecture search. Interna- tional Conference of Representation Learning (ICLR), 2019.
V Murali. Fuzzy equivalence relations. Fuzzy sets and systems, 30(2): 1989.
Hyeonwoo Noh, Tackgeun You, Jonghwan Mun, and Bohyung Han. Regularizing deep neural networks by noise: Its interpretation and optimization. Advances in Neural Information Processing Systems (NeurIPS), 2017.
Peter Petersen, S Axler, and KA Ribet. Riemannian geometry, volume 171. Springer, 2006.
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv: 2015.
Jordi Recasens. Indistinguishabiiiy operators: Modelling fuzzy equalities and fuzzy equivalence relations, volume 260. Springer Science & Business Media, 2010.
Walter Rudin et al. Principles of mathematical analysis, volume 3. McGraw-hill New York, 1964.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, 15(1):1929-199 1958,,99988111 2014.
Gilbert Strang, Gilbert Strang, Gilbert Strang, and Gilbert Strang. Introduction to linear algebra, volume 3.
Press Wellesley, MA, 1993.
Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. Lsun: Construction of large-scale image dataset using deep learning with humans in the loop. arXiv preprint arXiv:/5006 506.033653 2015.
Ling Zhang and Bo Zhang. Fuzzy reasoning model under quotient space structure. Information Sciences, 173(4):353-364, 2005.
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 586-595, 2018
Deli Zhao and Xiaoou Tang. Cyclizing clusters via zeta function of a graph. In Advances in Neural Information Processing Systems, pp. 1953 1960, 2009.
Dengyong Zhou, Jason Weston, Arthur Gretton, Olivier Bousquet, and Bernhard Schölkopf. Ranking on data manifolds, In Advances neural information processing systems, pp. 169-176, 2004.
Appendices
A PROOF TO THEOREMS
A.1 THEOREM
Proof. Denote the dimensions of G(2) and X as de and dx, respectively. There are two possible cases for G: de is lower than dx, or dg is higher than equal to dx.
For the first case, a direct consequence that, for almost all points in X, there are no pre-images under G. This means that for an arbitrary point E X. the possibility of G-) 0 is 1, as {z E X: (x) + 6) c G(Z) which is zero measure set in X. This also implies that the generator is unable to perform inversion. Another consequence is that, the generated distribution P, can never get aligned with real data distribution P,. Namely, the distance between P, and P, cannot be zero for arbitrary distance metrics. For the KI. divergence, the distance will even approach infinity.
11

---

Under review as a conference paper at ICLR 2021
For the second case, de dxx dz. We simply show that Lipschitz-continuonn function cannot map zero measure set into positive measure set. Specifically, the image of low dimensional space of a Lipschitz-continun function has measure zero. Thus if dg 2 dx, G cannot be Lipschitz.
Now we prove our claim.
Suppose that f R" + Rm f(R") has measure zero in R As R" is zero measure subset of R.m,, by the Kirszbraun theorem (Deimling, 2010), has extension to Lipschitz function of the same Lipschitz constant RT For convenience, we still denote the extension as ʃ. Then the problem reduces to proving that maps zero measure of balls [Bx} J& of radius тк such that IR" C Ug and Σn m(BB) < E in R" where m(.) is the Lebesgue measure in R" But f(Bx) is contained m(f(R")) < L Σg m (B)) < L which means that it is zero measure set in IR". For the mapping between manifolds, using the chart system can turn into the case we analyze above, which completes our proof.
m, f is Lipschitz with Lipschitz constant L. We show that
measure
For every >> O, can find countable union
ball with radius Lrk- Thus have
We want to remind the readers that, even if the generator suits one of the cases in Theorem 1, the other still occur. For example, G could succeed in capturing the distribution of certain parts of the real data, while it may fail in the other parts. Then for the pre-image of those successfully captured data, the generator will not have finite Lipschitz constant.
A.2 THEOREMS 2 &
Theorems & 3 are classical conclusions in Riemannian manifold. We refer readers to section 5.5 of the book written by Petersen al. (2006) for detailed proofs and illustration.
A.3 THEOREM
Proof.
B[llg(+) s < ||((x) - ((v)ll2 + (()))
(13)
-ppl yll 2C L,ll= vll2 + o(1),
(14)
where C is a constant related to the dimension of the image space of a and Ln is the Lipschitz constant of μ.
B WHY GAUSSIAN DISTRIBUTION?
We first introduce the notion of fuzzy equivalence relations.
Definition 1. t-norm is a functionn T: :0, 1] [o, 1] + [0, 1] which satisfies the following properties:
1. Commutativiry: b(a,b) T(b,
2. Monotonicity: T(a,b) b) < T d), a S and b s d.
= 3. Associativity: T(a, ((((a)) ((T(a,)),
4. The number I acts as identity element: T(a, 1)
Definition 2. Given a t-norm T, T-equivalence relation on a set X is a relation E on and satisfies the following conditions:
1. E(x, x) 1, E X (Reflexivity).
2. F(x, v) = Fly. x), Va, y X (Symmetry).
3. T(E(x,)) v). Ely, 2)) < E(x, va v. E X (T-transitivity).
Then it easy check that T(x v) = ry t-norm, and E(x, y) e d(x.v) is T-equivalence for any distance metric d on X, as
T(F(x,)) v). Fly, 2)) (d(x.)) d(l.))))) F(xx2))
12

---

Under review as a conference paper at ICLR 2021
Table GPU environments for all experiments in this work.
Experiment
Environment
StyleGAN2 based GAN model training
NVIDIA Tesla V100-SXM2-16GB GPUs (DGX-1 station)
DCGAN based GAN model training
4 TITAN Xp GPUs
Metrics measurement
8 GeForce GTX 1080Ti GPUs
GAN inversion
TITAN Xp GPU
Considering that we want to contain the fuzzy semantics of real world data in our local geometries of feature manifolds, natural solution will be that sample points from the local neighborhood of with different densities on behalf of different strength of semantic relations with μ. Points with stronger semantic relations will have larger densities to be sampled. A good framework to model this process is the fuzzy equivalence relations we mention above, where the degrees of membership are used the sampling density. However, our expansion of the exponential map Exp, carries an errоr term of o[I (11v)) We certainly do not want the local error to be out of control, and we also wish to constrain the sampling locally. Thus we accelerate the decrease density when points depart from the center μ4, and constrain the integral of E to be identity, which turns E to the density of standard Gaussian.
C DATASETS
(FFHQ) (Karras al. (019a) is a high-quality image dataset of human faces, originally created as a benchmark data for generative adversarial networks (GANs). The dataset consists of 70,000 high-quality PNG images and contains considerable variations in terms of age, pose, expression, hair style, ethnicity and image backgrounds. It also covers diverse accessories such as eyeglasses, sunglasses, hats, etc.
FFHQ
LSUN-Church and Cat-Selected LSUN-Church is the church outdoor category of LSUN dataset (Yu et al., 2013), which consists of 126 thousand church images of various styles. Cat- Selected contains 100 thousand cat images selected by ranking algorithm (Zhou et al.) (2044) from the LSUN cat category. The plausibility of using PageRank to rank data was analyzed in Zhou et al. (2004). We also used the algorithm presented in Zhao & Tang (2099) construct the graph from the cat data.
CIFAR-10 The CIFAR-10 dataset (Krizhevsky al. 2009 consists 60,000 images of size 32x32. There are all 10 classes and 6000 images per class. There are 50,000 training images and 10,000 test images.
D IMPLEMENTATION DETAILS
D. MODELS
We illustrate the generator architectures of StyleGAN2 based methods in Figure 6 For all those models, the discriminators share the same architecture as the original StyleGAN2. The genera- architecture of DCGAN based methods are illustrated in Figure 7 For all those models, the discriminators share the same architecture as the original DCGAN.
E EXPERIMENT ENVIRONMENT
All experiments are carried out by TensorFlow 1.14 and Python 3.6 with CUDA Version 10.2 and NVIDIA-SMI 440.64.00. We basically build our code upon the framework of NVIDIA official StyleGAN2 code, which is available at tttt //ttts:://ittbb variety of servers to the experiments reported in Table
http///labs/sstyleeee We use
13

---

Under review as a conference paper at ICLR 2021
const 4 x 512
consE x 4 512
Mod
Mod
Demod
Cozy 3c3
Dettod
Cosv 3x3
Desse
Dense
Dense
Dense
Bias
Bias
*
Nouse
Dense
Dense
Dense
Dense
Mod
Upamppe
Dense
Mod
Upaamele
D Pense
Demod
Cony 3r3
Dense
Demad
Cony 3x3
Dense
Denso
Bits
Bits
Nouse
W '07
Med
Mod
Demod
Cony 3r3
Demad
Cony 3r3
x
(a) Bald StyleGAN2.
StyleGAN2.
const 4 512
Mod
Derood
Cone 363
Diess
Diciz
Xd(())
Dieise
Denta
Mod
Upsemple
Dince
Demod
Comr 2x3
Diess
Xd(())
NEW
Mod
Demoo
Com: 2x3
Xa(i)
(c) Fuzzy Reparameterizzation.
Figure 6: Generator architectures of StyleGAN2 based models. (a) The generator of bald StyleGAN2. (b) The generator of StyleGAN2. (c) The generator of StyleGAN2 + FR and StyleGANN-NoPathReg FR. 'Mod' and 'Demod' denote the weight demodulation method proposed in section 2.2 of StyleGAN2 Karras et al. 2019h). A denotes learned affine transformation from the intermediate latent space W.
F IMAGE ENCODING AND GAN INVERSION
From a mathematical perspective, well behaved generator should be easily invertible. In the last section, we have shown that our method is well conditioned, which implies that it could be easily invertible. We adopt the methods in Image2StyleGAN (Abdal al., 2019) to perform GAN inversion and compare the mean square error and perceptual loss on a manually collected dataset of 20 images. The images shown in Figure and the quantitative results provided in Table 4 For FR
14

---

Under review as a conference paper at ICLR 2021
Conv
Conv
Conv
Batch Norm
Batch Norm
Batch Norm
ReLu
ReLu
ReLu
1
x(())
Noise
FR
Noise
Conv
Conv
Conv
Batch Norm
Batch Norm
Batch Norm
ReLu
ReLu
ReLu
X
x O())
Noise
FR
Noise
(a) DCGAN.
(b) Additive Noise Injection.
(c) Fuzzy Reparameterization.
Figure 7: Generator architecture of DCGAN based models. (a) The generator of DCGAN. (b) The generator of DCGAN + Additive Noise. (c) The generator of DCGAN + FR.
Table Image inversion metrics for different StyleGAN2 based models. The perceptual loss the mean square distance of VGG16 features between the original and projected images as in Abdal et al.) (0019)
Overall
Hard Cases
GAN arch
MSE (L)
Perceptual Loss (L)
MSE (L)
Perceptual Loss (L)
StyleGAN2 StyleGAN2
1.34
5.42
2.86
11.34
1.24
4.86
2.58
9.82
StyleGANN-NoPathRRe NoPathReg FR
1.24
5.11
2.70
10.49
StyleGAN2 + FR
1.13
4.52
2.23
8.47
methods, further optimize the parameter in Eq. in section 4.3, which fine-tunes the local
geometries of the network to suit the new images that might not be settled in the model. Considering (a*)*
that a is limited to j0, 1). we use (I-)<+(I-a+)) replace the original and optimize t. The initial value t set to 1.0 and a is constant with the same value as a in the converged FR models.
During the experiments, non-blocking human face images. For this type images (which we refer as regular case in Figure 9. observe comparable performance for all the GAN architectures. We think that this is because those images are closed to the 'mean' face of FFHQ dataset (Karras et al., 2019a), thus easy to learn for the StyleGAN based models. For faces of large pose or partially blocked ones, the capacity of different models differs significantly. Noise injection methods outperform the bald StyleGAN2 by large margin, and our method achieves the best performance.
find that the StyleGAN2 model is prone to work well for full-face,
G ABLATION STUDY OF FR
In Tab, 5] we perform the ablation study of the proposed FR method on the FFHQ dataset. We test different choices of FR implementation and compare their FID and PPL scores after convergence.
15

---

Under review as a conference paper at ICLR 2021
Figure 8: Manually collected 20 images for GAN inversion.
Origin
Bald GAN2
Style-
StyleGAN2
StyleGAN2- NoPathReg FR
StyleGAN2 FR
Regular case
Hard case
Figure 9: Projected Images to the intermediate latent spaces of StyleGAN2 based models.
1. No normalization: this setting we remove the normalization of f in Eq. 7 and use the unnormalized replace in the following equations. The network comes minimum FID of 23.77 after training on 1323 thousand images, and then quickly falls into mode collapse after that.
16

---

Under review as a conference paper at ICLR 2021
Table 5: Ablation study of different noise injection methods FFHQ The values of PPI. scores in the first two methods suggest mode collapse.
Method
FID
PPL.
No normalization
628.94
No stabilization
184.30
No decomposition
6.48
18.78
CNN
22.54
14.53
FR
7.31
13.05
2. No stabilization: in this setting we remove the stabilization technique in Eq. The network comes to a minimum FID of 50.27 after training on 963 thousand images, and then quickly falls mode collapse after that.
3. No decomposition: in this setting remove the decomposition in Eq. The network successfully converges, but admits a large PPL score.
4. CNN: this setting we use convolutional neural network to replace the procedure that we get in section 4.3 Namely, we take CNN(u). The network successfully converges, but admits a very large FID score.
The zero PPL scores 'No normalization' and 'No stabilization' suggest that the generator output invariant to small perturbations, which means mode collapse. We can find that the stabilization and normalization in the FR implementation in section 4.3 is necessary for the network to avoid numerical instability and mode collapse. The implementation of FR method reaches the best performance in PPL score and comparable performance against the 'no decomposition' method FID score. As analyzed in StyleGAN (Krrra et al. 2019a) and StyleGAN2 (Karras et al. (201b)), for high fidelity images, PPL is more convincing than the FID score in measuring the synthesis quality. Therefore, the FR implementation is the best among these methods.
17


Reviews:
[{"review_id": "gKHxvgSzYzI", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper the authors highlight two major drawbacks of GANs. 1) The optimal Generator is discontinuous and 2) the 'adversarial dimension trap' caused by the relatively lower dimension of the latent space compared to the real-world data which makes the generator not Lipschitz and/or the generator fails to capture the real-world data distribution and is not invertible. \nBoth issues could lead to an unsmooth generator.\n\nSecondly, the authors provide a form of generalization of noise injection in GANs called fuzzy reparameterization, which leads to a solution by letting the generator map onto an arbitrarily low dimensional skeleton of the feature spaces and filling up the remaining space with random noise. Therefore, the difference to the latent space dimensionality is minimized addressing issue 2). The solution consists of two stages, first a map from feature space onto the skeleton set is learned followed by noise injection adapted to the local geometry of the orginal feature manifold. \n\nExperiments:\n\nExperiments were done on FFHQ faces, LSUN objects, and CIFAR-10 datasets with models DCGAN, StyleGAN2, and bald StyleGAN2 which is StyleGAN2 without noise injection and path length regularizer. On DCGAN the proposed fuzzy reparameterization (FR) outperforms DCGAN with and without additive noise on FFHQ, CIFAR10, LSUN-Church measured with the Path Perceptual Length (PPL) and the FID. Also StyleGAN2 with FR outperforms StyleGAN2 with additive noise and bald StyleGAN2 on FFHQ and LSUN objects. To test numerical stability, condition numbers of 50000 Input, Pertubation pairs were computed for StyleGAN2 models with, without additive noise and with FR and path length regularization. StyleGAN2 + FR outperforms both on the mean and top-1000 mean condition number indicating that FR improves numerical stability. StyleGAN2 + FR also outperforms on the image inversion experiments.\n\nPros: This paper provides a theoretical framework for noise injection for GANs which is novel and interesting for the GAN community. The experimental results are extensive and convincing and support the theoretical analysis.\n\nCons: In section 4.3 the algorithm eq. 6-8 is not very clear to me. E.g. what are the parameters A,b, alpha and r and how are they motivated? PixSum is over the feature maps? A more detailed description with comments would be helpful for the reader. I could not find the FR implementation in the supplementary file, it looks like it contains only the original StyleGAN(2), DCGAN and DCGAN with additive noise models.", "extra_scores": {}, "extra_text": {}}, {"review_id": "eaRSziRFwd0", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "To summarize, this paper proposed a new noise injection method that is easy to implement and is able to replace the original noise injection method in StyleGAN 2. The approach is supported by detailed theoretical analysis and impactful performance improvement on GAN training and inversion. The results show that they are able to achieve a considerable improvement on DCGAN and StyleGAN2.\n\nMeanwhile, this reviewer did not fully understand the theoretical part and also has some questions regarding the implementation and results.\n\nBased on my understanding, the fuzzy reparameterization technique realizes something that StyleGAN2 cannot achieve, and resolves some fundamental limitations of StyleGAN2. However, the improvement is not contiguous in Table 1, where we see the vanilla StyleGAN2 still outperforms the proposed architecture. What could be the reason?\n\nFR seems to bring more parameters (Eq 6, 7, 8, 9). How many more compared to the additive noise implementation? Could the number of parameters be the reason that the proposed method performs better? Since FR can be seen as a generalization of StyleGAN2 noise injection, we would naturally expect that the proposed method should perform better than StyleGAN2. However, this is not always the case in Table 1. I guess more ablation studies can also be done on $\\sigma$, such as interpolating between StyleGAN2 implementation and FR implementation, or a linear layer with the same number of additional parameters but has no constraint as in Eq. 6, 7, 8, 9.\n\nFor Figure 8, do we have the reconstruction visualization? Is the inversion done in the z space, w space or the w+ space?  I am curious to see how better this method performs in terms of inverting real images in the wild. I also believe the inversion in z space allows me to appreciate more about the inversion improvement.\n\nOverall, I vote to accept this paper due to its good performance improvement over prior standard noise injection implementation. Meanwhile, I hope the theoretical analysis can be made easier to understand for a researcher that lacks the related background.\n\n[Update after reading authors' comments]\nBased on the authors' and other reviewers' comments,  I keep the score unchanged.", "extra_scores": {}, "extra_text": {}}, {"review_id": "GRSM9ubBojE", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Pros:\n\nThis work introduces the problem of adversarial dimension trap, which leads to punishment on the smoothness and invertibility of GANs. \n\nThis work proposes to learn fuzzy equivalence relation of the features and uses reparameterization trick to model the high-dimensional feature manifolds.\n\nA novel form of noise injection is proposed to overcome the adversarial dimension trap. Prior noise injection methods can be explained as a special case with certain hyper-parameters. This method is universal for the families of GANs, including WGAN, DCGAN, etc.\n\nExperiments on three datasets are conducted. The results of both image synthesis and GAN inversion are desirable with plausible texture details.\n\n\nCons:\n\nMy major concern is about the experiments. In Table 1, it seems that the reported FID and PPL differ from the scores reported by StyleGAN2 [1]. For the FFHQ dataset, [1] report that the FID is 3.31 (config E), but the FID of the baseline in this paper is 7.14 (the same setting). Such a huge discrepancy is wired. Recall that [1] improve the FID from 4.40 (StyleGAN v1) to 2.84, but the baseline, which should be the same, performs even worse than StyleGAN v1. The authors need to explain these contradictions.\n \nThe differences between the PPL scores reported in this paper and [1] are even more significant. I notice it is 13.05 (the best in this paper) versus 122.5 (the best in [1]). Why is the PPL about ten times better (even without the proposed method)?\n\nThe authors need to provide more details to explain how they calculated the FIDs and PPL. It seems that the authors calculate these scores in a non-standard manner. Besides, I suggest the author evaluate the Precision and Recall [2] on FFHQ. I wonder whether these metrics will be consistent.\n\n[1] Karras, Tero, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. \"Analyzing and improving the image quality of stylegan.\" CVPR, pp. 8110-8119. 2020.\n\n[2] Kynkäänniemi, Tuomas, Tero Karras, Samuli Laine, Jaakko Lehtinen, and Timo Aila. \"Improved precision and recall metric for assessing generative models.\" NeurIPS, pp. 3927-3936. 2019.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "55QT618mp1M", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper analyzes the theoretical properties of noise injection in StyleGAN-like networks, and proposes an extension to in particular to StyleGAN2 that results in somewhat improved metric scores. Unfortunately the paper is rather confusingly written and hard to follow. To highlight what I mean, I will try to paraphrase my understanding of the paper in the following.\n\nThe theoretical treatment begins by framing the problem around optimal transport, but later seems to mostly drop this viewpoint. While the cited OT/GAN work presents interesting and relevant viewpoints about the difficulties in GAN training, I am not sure if it is particularly more relevant here than any number of other theoretical works. It may be noted that StyleGAN and DCGAN are not even formulated as to minimize a Wasserstein divergence.\n\nThe paper then coins a term \"adversarial dimension trap\", which I am not exactly sure why this terminology was chosen. The gist of the observation seems to be mostly well known, i.e. the generator can only cover a zero-measure region of the data space whereas the data is more spread out. That said, I am not thoroughly familiar with previous theoretical work on GANs and the particular formulation here may be novel. The paper then introduces a fairly general form of stochastic noise injection into the network layers and calls this fuzzy reparametrization. Here some connections to are drawn to \"fuzzy equivalence relations\" which (apparently?) are an existing concept, however as far as I see there is no citation to discuss these and little insight is given about why this is relevant.\n\nThen, the key theory is developed. If I understand correctly, the key idea here is that the zero-measure manifold is \"puffed up\" with random distributions centered around points on it, with spread that depends on the point. This makes sense as a principle but overall I am confused about whether something fundamental was discrovered or proved, or whether this is just an introduction to the reasoning behind the practical algorithms.\n\nThis then leads to a proposal of a practical algorithm. If I understand correctly, it basically generalizes the StyleGAN noise injection in such a way that not only the mean, but also the stdev of each feature is predicted by the network. This plausibly allows for more flexibility. Unfortunately the description here is again rather confusing. Apparently formulas 6-8 are not really equations, but rather some kind of imperative pseudocode with variable assignments. It would be better to spell this out as an algorithm listing. As for the content of these formulas, I am not sure if I understand what the operations or the reasoning behind them is. Why the pixsum here? Apparently it produces a single value per feature map? After this there seems that these numbers are transformed by some global matrix(?) A and bias b, however I'm not sure what the convex combination with a matrix(?) I means here given that the first half of the formula is a vector (?). Then the result seems to be normalized again (why?) And finally the means are transformed by this standard deviation? There may well be good reasons to use these steps, but they are not explained so it ends up looking like an arbitrary heuristic. Here it would be important to make a strong connection to the insights derived from theory.\n\nThe presentation is further made confusing by the language. I understand that the authors may not be native speakers, but the readability is much below the usual standard of ICLR papers and the paper would benefit from improving this.\n\nAs for the results, it does appear that there is some improvement in some of the metrics, and the proposed method may in principle be useful. It is not hard to believe that adding some extra flexibility to the noise injection might improve the results, at least in a limited number of scenarios. In this sense the paper may be on to something. \n\nWhat is the meaning of using PageRank to reduce the number of LSUN-Cat images? How is PageRank related to choosing images and what's the difference between that and just taking the first 100k pictures in the set? And for that matter, I am not sure if we learn anything from randomly limiting the set to 100k images, when we don't know how it worked for the full set. For the inversion experiments, the table in the appendix does show improvement and this may well be the case. In figure 9, though, it's hard to see much of a difference between any of the methods, perhaps in part because the images shown are very low resolution and do not correspond to anywhere near the SG2 output image size -- any differences in details are completely hidden by this.\n\nThe architecture figures 6-7 are unnecessarily low detail. They contain a black box \"FR\" node precisely at the place where you'd want to know more. Perhaps this node could be expanded into its own architecture diagram as well, given that there is no shortage of space in the appendix.\n\nIn summary the paper might contain useful bits -- this is somewhat hard to judge -- but whether or not that is the case, it is not in an acceptable condition without some significant rewriting, and I would recommend rejection at this time.\n\n_UPDATE AFTER REBUTTAL_\n\nThe authors have improved the paper somewhat by expanding and clarifying the discussion on some key parts. While I think there is still much room for improvement in the paper, the general consensus seems to be towards acceptance. I will not oppose if that is the decision, and have increased my score accordingly. However I remain very borderline and I am not sure if I am fully convinced by all the claims.\n\nOne specific issue: I think the authors should make it more clear in the paper that the experiments are done in 128 pixel resolution, in light of R1's questions. It is important that the reader be aware of this, as the noise inputs arguably become much more important in high resolutions where there is more stochastic detail. I personally did not realize this when writing my review, and now wonder how the results would be at e.g. 256 or 512 resolution. If possible I would suggest the authors still run such experiments. This also probably explains my comment above on the lack of apparent visual differences in inversion results.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper studies the role of “noise injection” in GANs with tools from Riemannian geometry, and derives a new noise injection approach that aims to learn a fuzzy coordinate system to model non-Euclidean geometry. The new noise injection approach is shown to improve over StyleGANv2 noise injection on lower-resolution 128x128 FFHQ, LSUN, and 32x32 CIFAR-10 images. 


Some reviewers found the experimental results a “considerable improvement on DCGAN and StyleGANv2” (R3), “extensive and convincing” (R2), while others had concerns around the experimental setup using lower resolution images (R1, R4).  While reviewers were mostly positive about the experimental wins of the paper, there was confusion (R3) and several concerns (R4) around the theory and the relationship between the theory and the practical noise injection algorithm. I additionally had several concerns around the presentation and relation to prior work on generative models. Thus in the current state I cannot recommend this paper for acceptance. Below I highlight concerns that should be addressed in future revisions.


1. My biggest concern is the tremendous gap between the theoretical claims and the practical implementation. When training a GAN with the new form of noise injection, does it learn the skeleton and fuzzy equivalence relationships you claim? This paper is missing any kind of toy experimenting showing that training a GAN with fuzzy reparameterization discovers these relationships or coordinates. Such an experiment would greatly strengthen the paper and help to answer the question of why this new method works (i.e. it’s not just more parameters, a slightly better architecture, or better hyperprameters as mentioned by R3 and R4). There’s also no discussion of what happens theoretically when you have multiple layers of fuzzy reparameterization, and the claims that StyleGAN2’s noise injection limits to Euclidean geometry is false in this case (and thus StyleGAN2’s noise injection can also overcome the “adversarial dimension trap”).
 
2. Theoretical setting: As mentioned by R4, there is much prior work on the difficulties in fitting a lower-dimensional model manifold to a higher-dimensional data manifold (e.g. WGAN). Theorem 1 highlights the impossibility of exactly fitting the data manifold with (smooth) neural networks, but the resulting solutions of increasing the dimensionality of the latent space is well-known and commonly used (e.g. StyleGAN). This paper also doesn’t discuss the alternative of *approximately* fitting the data manifold with a lower-dimensional structure, which is what is often studied in practice. 


3. Clarity: The term “noise injection” is overloaded in the literature, and the current presentation of the paper does not sufficiently describe the method. There’s also no discussion of “instance noise” that is another solution to this problem that adds noise to inputs of the discriminator to yield finite f-divergences (Sonderby et al., 2016, Roth et al., 2017). The work on instance noise is very related to the approach here, but only adds noise to the output of the generator, not at all levels. 
There's also no discussion of how adding noise is just expanding the generative model with additional latent variables, a standard approach that is often discussed in the context of hierarchical generative models. The authors mention the relation to reparameterization trick in VAEs, but argue it is doing something fundamentally different. However, modern VAE architectures (IAF-VAE, Very Deep VAE), use a very similar form of modulation at multiple levels in the hierarchy. 


4. Experiments: There are no error bars in experimental results, and many results are presented in a new experimental setting defined by the authors (lower resolution than prior work even if using prior code). Rerunning experiments in more standard settings on full resolution images would greatly improve the confidence that the new noise injection strategy is effective.

Decision: Reject