NICE: NoIse-modulated Consistency rEgularization for Data-Efficient GANs
Authors: Yao Ni
Piotr Koniusz
Venue: NeurIPS 2023 poster (2023)
Keywords: Image Generation
limited dataset
Generative Adversarial Networks

Abstract: Generative Adversarial Networks (GANs) are powerful tools for image synthesis. However, they require access to vast amounts of training data, which is often costly and prohibitive. Limited data affects GANs, leading to discriminator overfitting and training instability. In this paper, we present a novel approach called NoIse-modulated Consistency rEgularization (NICE) to overcome these challenges. To this end, we introduce an adaptive multiplicative noise into the discriminator to modulate its latent features. We demonstrate the effectiveness of such a modulation in preventing discriminator overfitting by adaptively reducing the Rademacher complexity of the discriminator. However, this modulation leads to an unintended consequence of increased gradient norm, which can undermine the stability of GAN training. To mitigate this undesirable effect, we impose a constraint on the discriminator, ensuring its consistency for the same inputs under different noise modulations. The constraint effectively penalizes the first and second-order gradients of latent features, enhancing GAN stability. Experimental evidence aligns with our theoretical analysis, demonstrating the reduction of generalization error and gradient penalization of NICE. This substantiates the efficacy of NICE in reducing discriminator overfitting and improving stability of GAN training. NICE achieves state-of-the-art results on CIFAR-10, CIFAR-100, ImageNet and FFHQ datasets when trained with limited data, as well as in low-shot generation tasks.

NICE: Nolse-modulated Consistency rEgularization for Data-Efficient GANs
Yao Nit, Piotr Koniusz
The Australian National University $Data61 farsttame.lastmamnn @anu.edu.an
Abstract
Generative Adversarial Networks (GANs) are powerful tools for image synthesis. However, they require access to vast amounts of training data, which is often costly and prohibitive. Limited data affects GANs, leading to discriminator overfitting and training instability. In this paper, we present a novel approach called Nolse- modulated Consistency r'gullrization (NICE) to overcome these challenges. To this end, we introduce an adaptive multiplicative noise into the discriminator to modulate its latent features. We demonstrate the effectiveness of such a modulation in preventing discriminator overfitting by adaptively reducing the Rademacher complexity of the discriminator. However, this modulation leads unintended consequence of increased gradient norm, which can undermine the stability of GAN training. To mitigate this undesirable effect, impose constraint the discriminator, ensuring its consistency for the same inputs under different noise modulations. The constraint effectively penalizes the first and second-order gradients of latent features, enhancing GAN stability. Experimental evidence aligns with theoretical analysis, demonstrating the reduction of generalization error and gradient penalization of NICE. This substantiates the efficacy of NICE in reducing discriminator overfitting and improving stability of GAN training. NICE achieves state-of-ther the-art results on CIFAR-10. 10, LIFAR-100 ImageNet and FFHQ datasets when trained with limited data, as well as low-shot generation tasks.
1 Introduction
The remarkable advancements and breakthroughs deep learning [31] can be largely attributed to the extensive utilization of vast amounts of training data. This abundance of data has driven the progress across various domains of deep learning. Among notable recent advancements are Generative Adversarial Networks (GANs) [13], popular in industry and academia. GANs have proven their high-quality image generation abilities and achieved high generation speeds [47], establishing them as versatile tool for wide range of applications, such text-to-image generation [20, 47, 55], destylization [48, 49, 50], image-lo-image translation [27, 44], and 3D generation [52, 53, 65].
Despite the impressive capabilities of state-of- oftte-of-the-arn art GANs in generating diverse and Nighhquaaityy images [7, 25], their effectiveness heavily relies on large volumes of training data. The acquisition of such large datasets helps GANs attain the desired adversarial equilibrium. However, under limited training data regime, GANs encounter challenges associated with discriminator overfitting and unstable training [22, 68, 57, 11, 17].
To address the aforementioned challenges, recent studies have approached the problem from three perspectives. The first perspective involves the utilization of extensive differentiable data augmenta- tion techniques, aimed at expanding the distribution of the available data [22, 68, 17, 32, 63, 64, 69]. The second perspective leverages knowledge gained from large-scale models trained on large datasets [9, 70, 29]. While these approaches are effective, they also carry inherent risks, such the leakage of augmentation clues [64. 22, 40] or pre-trained knowledge [30].
The corresponding author.
Code: https: h http:///ttuuu.eoooaaeeelleeeeaee
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

The third perspective focuses on the regularization of the discriminator, aiming to weaken its learning ability [11, 19] or increase the overlap between real and fake support sets [57]. In line with this perspective, our paper introduces a novel approach to improve GAN generalization. We propose to modulate the hidden features of discriminator via adaptive multiplicative noise, which strikes a balance in maintaining certain level of discrimination ability while also regularizing the Rademacher complexity [6] of the discriminator. This reduction in Rademacher complexity, which quantifies the capacity of model to fit random variables, narrows the generation gap between the training and unseen data, resulting in enhanced GAN generalization [4, 56, 18].
Nevertheless, training the discriminator with adaptive noise unintentionally amplifies second-orde gradient derivative of latent features corresponding to real images. This elevated gradient leads to abrupt gradient changes near the real sample points, potentially causing instability in the feedback to the generator. This issue aligns with the findings in works [26, 36, 56, 11], which emphasize the importance of penalizing gradients for both real and fake samples to promote convergence and stability of GAN training. To address these challenges, we propose a constraint on the discriminator that ensures consistency for the same inputs under different noise modulations. While our idea is simple, our theoretical analysis reveals that this constraint effectively penalizes the first and second- order gradients of the latent features. Consequently, the gradient provided to the generator becomes more stable, resulting in improved training stability and generalization.
Our comprehensive experiments confirm the effectiveness of NICE in penalizing gradients and reducing the generalization gap. Despite the simple design, NICE significantly improves the training stability and generalization of GANs, outperforming alternative approaches in preventing discrimina- tor overfitting. NICE achieves superior results on challenging limited data benchmarks, including CIFAR- 10/100, ImageNet, FFHQ, and low-shot image generation tasks.
Our contributions can be summarized as follows:
i. We limit discriminator overfitting by using the adaptive multiplicative noise to modulate the latent features of the discriminator, resulting in enhanced generalization of GANs.
ii We introduce NICE, technique that enforces the discriminator to be consistent with the same inputs under different noise modulations, implicitly penalizing the first and second-ordes gradients of the latent features during GAN training, promoting the training stability. iii. We show that NICE, both in theory and practice, effectively prevents discriminator overfitting and achieves superior performance in image generation under limited data setting.
2 Related Work
Improving GANs. Generative Adversarial Networks [13] are powerful generative models that excel in image generation [21, 24, 25], text-to-image generation [20, 47, 55], image-to-image translation [27, 44], and 3D generation [52, 65, 53]. However, GANs commonly encounter challenges such as training instability [26]. mode collapse [45], and discriminator overfitting [22]. Researchers have investigated different GAN loss functions [2, 42, 71], architectures [21, 24, 25, 23], and regularization
strategies [38, 33. 14]. The f-GAN [42] generalizes GANs to f-divergences. WGAN [2] adopts the Earth-Mover distance. OmniGAN [71] extends conditional GANs to multi-label softmax loss. StyleGANV1-3 [24, 25, 23] enhances the architecture of generator. Approaches [14, 26, 36, 56, 11] propose explicit penalty of gradient of discriminator. SNGAN [38] enforces a Lipschitz constraint on the discriminator, and approach [33] regularizes the spectral norm of the discriminator. In this study, we propose novel regularization strategy applicable to diverse GAN loss functions and architectures, specifically tailored for training GAN models with limited data.
Image generation with limited data. Collecting data is both laborious and expensive. Conse- quently, generating images in limited data settings presents significant challenges, primarily due to discriminator overfitting. Previous approaches have tackled this issue through data augmentation techniques [22, 68, 17, 32, 63] and leveraging knowledge transfer from large-scale pre-trained models [9, 29, 70]. Differentiable data augmentation methods [22, 68, 17] "extend" the data distribution, while methods [63, 32] explore contrastive learning within GANs. Approaches [9, 29, 70] employ pre- rre-trained models to guide training of discriminator. However, both GAN types suffer issues, including leakage of augmentation clues [64, 22] or pre-trained knowledge [30]. An alternative approach involves regularizing the discriminator. LeCamGAN [57] suggests reducing the output gap of discriminator between real and fake distributions. ICSAGAN [40] employs manifold techniques to project discriminator features onto manifold and decrease the capacity of discriminator. APA [19]

---

uses fake images as real images to increase overlap between the real and fake distributions. DigGAN [11] minimizes the gradient norm gap between real and fake images. Inspired by such an approach. we propose to use an adaptive noise to modulate the latent features of discriminator to reduce the generalization error of discriminator as a means of regularization.
Consistency regularization. CTGAN [61] introduces the enforcement of a Lipschitz constraint by ensuring the consistency of the response of discriminator to real images. CAGAN [41] and GenCo [8] enforce consistency among multiple discriminators. R-Drop [62] applies consistency regularization to transformers [58] equipped with dropout for natural language processing tasks. Augmentation based consistency regularization GANs [64, 69] enforce consistency by considering different augmented views of the same image, although they may inadvertently leak augmentation clues to the generator [22]. Despite the effectiveness of consistency regularization demonstrated by these previous works, their success is primarily empirical, lacking theoretical analysis. In our study, we go beyond prior works by providing a theoretical analysis of consistency regularization in GANs. By delving deeper into the underlying principles, we develop a more comprehensive understanding of the mechanism behind its effectivvness.
Generalization of GANs. Arora et al. [4, 56, 67, 18] have contributed to the understanding and improvement of generalization of GANs by showing the importance of achieving adversarial equi- librium. reducing discriminator discrepancy, penalizing gradients, and bounding the generalization error. Our motivation aligns with these works in the pursuit of enhancing generalization capabilities. However, our approach implicitly and adaptively reduces the generalization gap while penalizing gradients of latent features of discriminator. Such setting provides an efficient and effective means of preventing discriminator overfitting and improving generalization.
3 Method
To boost the generalization of GAN, we start by analyzing their generalization error, bounding it with the Rademacher complexity of discriminator, and linking with the weight norm. By incorporating multiplicative noise, we demonstrate its regularization benefits on the weight norm. However, such a strategy induces large gradient norm destabilizing training of GAN, To this end, we introduce NICE to penalize the gradients of discriminator. Finally, we provide NICE and showcase it use in GANs.
3.1 Generalization error of GANs and complexity of neural network
The primary goal of GAN is to minimize the integral probability metric [39], assuming access to infinite real and fake data during optimization, i.e., the infinite real and generated distributions (n, v/ as discussed in [67]. In practice, we often have limited access to a finite dataset fh, of size n. Consequently, our optimization is restricted to the empirical loss:
inf wEG
(d v) == sup (Em-A. [h(=)] 1h)))())))
(1)
NEH
Function sets of discriminator and generator, H and G, are typically parameterized in GAN as neural network classes Han = {h(x; 02) Oa θ₂} and G.n = {g(2; 0,) 0 E O₂} where z ~ p. serves as the random noise input to the generator. The associated term dx is referred to as the neural network distance [4]. The discriminator network D == 4p consists of real/fake prediction head yp and feature extractor f. As the loss function <b(-) varies across tasks, architectures or choice of divergence type, we compose it with D [2, 67, 4] to simplify the analysis and notation, ie., h(-) := (D))). Thus, the alternative optimization of discriminator and generator becomes:
LD min Ea (a;;; 0a)] E [xx:;; 0z)].
Gi LG min 0,
E Ex-pa 1h(sl ( 0,))].
(2)
where assume Vn minimizes dx v) up to precision 2 0, meaning that dze(ān vn) S infweg dx (fn, v) + As we are interested in how close the generator distribution Vn. is to the unknown infinite distribution μ1, we refer to the lemma [67] on the generalization error of GAN:
Lemma (Theorem 3.1 of [67]) Assume that the discriminator set H is even (h E H. implies -h E H) and all discriminators are bounded by |1fi]loc I A. Let p.n be an empirical measure of an i.i.d. sample of size n drawn from μ. Assume (fn, va) - iinfveg d' (Pn, v) S c. Then with probability at least - 8, we have:
2 2log(1/8)
dx du(u, vn)) inf du(u, v) < 2 sup
E, - Ep., [h]
+E S (H))+A
+ e, (3)
vEg
hCH
3

---

7 )(x((())) measures how well the function h fits the Rademacher random variable 7(0) with prob(r(0) 1) - prob(r())
where the Rademacher complexity 16], R R E| supnen 2 E
-1) given samples (+) Pr.
Lemma provides crucial insight that one can assess the generalization error of GAN by comparing the output discrepancy of discriminator between training data and unseen data, and such an error is influenced by the Rademacher complexity of the discriminator. To enhance the performance of generator while reducing the generalization error, we have two possibilities: 1) increase the quantity of real data, which provides a stronger foundation for training a better generator; 2) reduce the Rademacher complexity of the discriminator. However, this reduction must be carefully controlled, as overly simplified discriminator may struggle to effectively distinguish real and fake data.
To manage the Rademacher complexity of the discriminator, we leverage theorem from approach [5] to establish an upper bound on the Rademacher complexity of the neural network.
Lemma 2 (Eq. 1.2 in [5], Theorem 5.20 in [35]) Consider fully-connected neural network vo(x=) = W.o WWo (...oWW 12))..)) where W; are linear weights at i-th layer and a is a 1 1-Lipschitz activation function. Suppose that Vi E [I,...,Th. ||zo I Let Wallnn be the
lipschitz constant of W and I/W? Il2,1 be the sum of the l2 norm of columns in W,, Let V = {vo- ilpp k2 [WT 112,1 S bi]. we have the Rademacher complexity:
(
2/3 3/2
RC (V) S [II Kg
(4)
2/13
Lemma 2 links the Rademacher complexity of a neural network to the Lipschitz constant and the (2.1)-norm of its weights, providing essential insights for controlling the complexity of discriminator.
3.2 Improving generalization by feature modulation with multiplicative noise
Taking into account Lemmas and 2, one can effectively manage the generalization error of GAN by controlling the Rademacher complexity of the discriminator. Such a control requires managing the Lipschitz constant and the norm of the weights of discriminator. Typically, controlling the Lipschitz constant can be achieved by the spectral normalization [38] or the gradient penalty [14]. In contrast, we propose novel approach: regularizing the norm of the weights by modulating the latent features of the discriminator with multiplicative noise.
We build on the analysis (Prop. in Appendix of [3) of Dropout regularization [54] by exploring the regularization effect of the Gaussian multiplicative noise within the context of deep regression.
Theorem (Regularization by the Gaussian multiplicative noise in deep regression) Consider a regression task two-layer neural network. Let X CR and y 11-1,,,11 1j be the input and output spaces, and D = x.. Let examples {xx ()) ~ Let fw :X x be parameterized by {w: W, E RRRdi xdo W₂ E Rd2xd1) } and fmn z) W2( o o (W 12)) where is I-Lipschitz activation function, @ denotes element-wise multiplication and z 2~, = -E, [a( âu 20 denotes the k-th element of Let 2222,,11 - ||wk//2 where we is the k-th column vector in W2. The regression task with the l2 loss leads to the weight regularization as:
Let a o 1x),
Lnoose = E.
W2(2 ① a )0) E. [1lg - Wea 11 32
Σ @ (5)
A. The proof can be found in SC.1. Theorem reveals that modulation with Gaussian multiplicative noise in the regression task implicitly regularizes the internal weight norm, with the regularization strength determined by the variance of the noise /32 and the magnitude of the features.
Despite analysis in a simplified two-layer system, the implicit weight regularization by the noise modulation applies to multi-layer and convolutional neural networks, which can be expressed as a combination of two-layer nets, and convolutional layers are a type of linear layer [37, 60, 3]. Assuming latent features in deep neural networks follow a Gaussian distribution [34], we can equate the Bernoulli noise modulation (Dropout) with Z ~ B(3), whose values are set to be 1/(1 - 33) with probability 3 and otherwise, to the Gaussian multiplicative noise modulation in subsequent layers, making the regularization effects from Theorem applicable to the Bernoulli noise as well.
Theorem illustrates that one can modulate the latent features in the discriminator by the multi- plicative noise to adaptively regularize the norm of its weights, leading to the reduced Rademacher complexity in Lemma 2 and improved GANs generalization in Lemma 1.

---

3.3 Consistency regularization
Although the noise modulation can reduce the Rademacher complexity, incorporating the noise increases the gradient norms of latent features and the inputs to the discriminator, making training of GAN unstable [56] and difficult to converge [36]. To understand the gradient- related challenges in the
discriminator, adopt the Taylor expansion with focus on the pivotal first- and second-orde order terms, following standard practice [1, 12, 66].
Define f := ft o f2 as the feature extractor of the discriminator from the second layer onward. Let ā = fă) and a fi(x)) be the latent features of the first layer for fake and real images, respectively. The discriminator is defined as h(()) = (()((())) with up as the prediction head and è as the loss function. Introduce a multiplicative noise = ~ (1,, 3²1")) modulating solely the first layer. Let (1(6)) (a) be the k-th diagonal entry of the Hessian matrix of h at a and ag be the k-th element of a. Applying Taylor expansion to h(-), the GAN loss can be expressed as follows:
Proposition 1
(h)
min LT 0a
EzE (h(2 ① [a]]= Ex [h(2 2 aa]]
-FAa [M(a]] E. [M)] + of (Ea Ex az H
(1)
(a)]
Ea of H
(1 kk (a))),
(6)
kk
min LAN := -
8 E Eg h12 2 a]] ~ Ea [h(a]] - 2
Ea
E
(M o2 RK (a)]
02
Prop. (the proof is in $C.2) shows that introducing the noise modulation during the training of the discriminator amplifies the value of diagonal entry of Hessian matrix at real images, causing abrupt changes in their gradient and potentially increase of the gradient norm of real images. Eq. 7 further shows that as the generator is optimized to produce images resembling real images, the gradient norm of the fake images also experiences amplification. Consequently, gradient norms for both real and fake images increase. Empirically, we observe an increase in the gradient norm of latent features before the prediction head, resulting in larger gradient norm of samples.
To mitigate this problem, we propose a consistency regularization for f. promoting invariance of f to the same input under varying noise modulations. For simplicity, we take f R. R, noting that this analysis readily extends to the consistency loss for f with a multidimensional output.
Theorem Using notations from Prop. I and with J IT= R, let (a) and jHj (a))² denote the squares of the k-th gradient element and kytt entry of Hessian matrix of f at a, respectively.
Given 2 22, 1d) ) enforcing invariance of f under different noise modulations yields:
PNICE
2 (()) + +'' EE (H (a))² (8)
Theorem 2 (the proof is in $C.3) shows that the consistency loss essentially penalizes the first- and second-order gradient of f at a, with the strength adaptively controlled by the noise variance 8² and modulated by a. This results in reweighted gradient squared norms modulation, with terms in expressions ||
(af(), and (a))l: =E, discriminator in Eq. 6. NICE regularizes the first-
inffuenced
by ag 2 0. In contrast the and second-order gradients of layers ahead of the classification head. Despite the gradient from the classification head is large. NICE ensures the gradient reaching the input stays small, resulting in a stable training and improved generalization and convergence [4, 26, 67].
3.4 Implementation of NICE
The amalysis above highlights practical ways to enhance the generalization and stability of GANs. Firstly, we apply the adaptive multiplicative noise to the latent features of the discriminator, striking balance between discrimination and generalization, as highlighted in [67]. The consistency regular- ization is then applied ensure invariance of the discriminator the same inputs under varied noise modulations, penalizing the gradient and stabilizing the training process.
GAN with the adaptive noise modulation (AN). Figure shows the discriminator composed of blocks B1, Bg.... By, where each block B, contains multiple convolution weights e (C1, C2 Cs). To achieve a trade-off between generalization and discrimination, we modulate the latent features X E IR Dad' xx"" " from convolutional layers of the discriminator using adaptive noise Z E R Ded by:
AN(
X @ Z,
(9)
5

---

Discriminator with adaptive noise (AN)
Nolse-modulated Consistency rEigularization (NICE)
x
B1
B2
B1 -
head
DjG loss
Bg
By.
head
D/G loss
noise modulation
sign(r 6c) 2-1)
shared weights
f1()
AN
C2
AN
Overfit ?
X
B1
B
X
non-shured noises
Cs
AN
(N) = X where (1,(1,
B
- BL
f2
(I - fO
Figure 1: Our discriminator pipeline. NICE is applied to both real a and fake a: images.
where @ represents the operation that expands Z into x xd x d shape and performs element- wise multiplication with X. The noise is carefully controlled through an adaptive 13. In the case of BigGAN and OmniGAN, we control the variance of Z using 33 as Z ~ (N1,, / 3²)"") ). For StyleGAN2, we control the noise through Z B(3), where with probability 3+ and with probability / it takes O.
it takes the value
Below we introduce a mechanism to control the noise via the meta parameter /3 by detecting potential overfitting in the discriminator. Firstly, we compute the expectation over the discriminator output r(x)= = Elsign(D(
w.r.t. real samples and evaluate & = sign(r(z) (<)>>) (-1, 0, 0,11 where a fixed threshold. value greater than the threshold indicates potential overfitting [22]. We apply Be-1 B, + A3 to update 3 with smaller Ag to control Z. Denote the modified discriminator with our adaptive noise as lAN; the objective of GAN with the adaptive noise modulation becomes:
LD min Ex-y, [ (z; 0a)] Ex-a, [aN((;; 0a)], L 8,
(10)
min EEzp. ( 0g))]
GAN with the noise-modulated consistency regularization (NICE). To deal with the increased gradient norm due to the adaptive noise modulation, we introduce the consistency regularization (NICE) to promote invariance of the discriminator to the same samples under different noise modula- tions. Our regularization implicitly penalizes the gradient of the latent features in the discriminator by comparing outputs f.) and f2(-) of two feature extractors (with shared parameters) that are subjected to different noise modulations in Figure 1. The objective for GAN with NICE is:
I
NICE min En [ 0a) + y NICE (2)] + Em-aa h- (x; 02) + o (ICCE())) (x)),
NICE
NICE
min E EE-P, 6,
0a
h- 0g)) + NICE (g(2;0g));
(11)
where (NICE(-)= 11-( and 7= A,,3 3 is a meta-parameter that controls the strength of the consistency regularization and adaptively determined by B.
Efficient NICE. our experiments, we observed that applying the noise modulation the later blocks of the discriminator often outperformed applying noise in the earlier blocks. To avoid unnecessary computations (the early blocks do not use modulation), we divide the discriminator into modulation-fice -ree part (BI.... ....... Bí-1) and the starting block of noise modulation. For efficient calculation of the consistency loss, we input two copies of X (output from the first part) into the second part, bypassing the need for redundant calculations to maintain a low computational overhead compared to standard approaches.
part (Bi.... B1), where indicates
Experiments
We conduct experiments CIFAR-10/100 [28] using BigGAN [7] and OmniGAN [71], as well as on ImageNet [10] using BigGAN for conditional image generation. We also evaluate our method on low-shot datasets [68], which include 100-shot Obama/Panda/Grumpy Cat and AnimalFace Dog/Cat [51], and FFHQ [24] using StyleGAN2 [25]. We compare our method against several strong
baselines, including DA [68], ADA [22], DigGAN [11], MaskedGAN [17], KDDLGAN [9]. LeCam
[57], GenCo [8], InsGen [63], FakeCLR [32] and TransferGAN [59]. For fair comparison, we denote
methods using massive augmentation "MA", which include DA and ADA.
Datasets. CIFAR-10 has 50K/10K training/testing images with resolution of 32 32 from 10 categories, whereas CIFAR-100 has 100 classes. FFHQ contains 70K human face images at 256

---

pixels. Lhw-shot datasets contain 100-shot Obama/Panda/Grumpy Cat images, AnimalFace (160 cats and 389 dogs) images at 256 256 resolution. ImageNet has 1.2M/50K training/valiiatiinn images with IK categories. Following [17, 9], we center-crop and downscale its images to 64 x 64 resolution. The implementation details can be found in $D. The generated images can found in ŞJ.
Evaluation metrics. We generate 50K images per dataset to compute the commonly used Inception Score [46] and Fréchet Inception Distance (FID) [15]. We report tFID, computed between 50K generated images and all training images. For CIFAR-10/100 we also compute vFID between 10K generated images and 10K real testing images. For low-shot datasets, FID between 5K generated images and the entire dataset. For FFHQ. we calculate FID between 50K fake images and the entire training set. For ImageNet, we follow [17, 9] and generate 10K images for computing the IS and FID, where the reference distribution is the entire training set. Following [68, 11, 32], we run trails for methods using NICE, and report the mean of the results. Given that all standard deviations fall below the 1% relative, we omit them for clarity.
follow [68] and compute
4.1 Results on CIFAR-10 and CIFAR-100 for BigGAN and OmniGAN
Tables and demonstrate that NICE consistently outperforms baselines such as BigGAN, LeCam+DA, OmniGAN and OmniGAN+ADA CIFAR-10 and CIFAR-100, firmly establishing its superiority. NICE also outperforms LeCam+DA+KDDLGAA in the majority scenarios without any knowledge integration from large-scale models, underscoring its efficiency and effectiveness.
Table Comparison w/ and w/o NICE on CIFAR-10 given different percentage training data.
100% data
20% data
10% data
Method
MA
ISt
LFID
vFID
1St
(FID
vFID.
1St
(FID)
vFID
BigGAN(d 256))
9.21
5.48
9.42
8.74
16,20
20.27
8.24
31.45
35.59
+LeCam
9.45
4.27
8.29
8.95
11.34
15.25
8.44
28.36
33,65
+DigGAN
5.33
9.35
8.81
13.28
17.25
8.32
18.54
22.45
+NICE
X
9.50
4.19
8.24
8.96
8.51
12.54
8.73
13.65
17.75
+LeCam+DA
9.45
4.32
8.40
9.01
8.53
12.47
8.81
12.64
16.42
J.CCam+DA-KDDLLAN
8.19
11.15
13.86
+LCCam+DA+NICE
9.52
3.72
7.81
9.12
6.92
10.89
8.99
9.86
13.81
AmniGAN(d"=1022
X 10.01
6.92
10.75
8.64
36.75
41.17
6.69
53.02
57.68
+DA
10.13
4.15
8.06
9.49
13.45
17.27
8.99
19.45
23.48
+ADA
10.24
4.95
9.06
9.41
27.04
30.58
7.86
40.05
44.01
+NICE
10.21
2.72
6.79
9.86
6.06
9.87
9.78
6.40
10.37
+NICE+ADA
10.38
2.25
6.32
10.18
4.39
8.42
10.08
5.49
9.42
Table 2: Comparison w/ and w/o NICE on CIFAR- 100 given different percentage of training data.
100% data
20% data
10% data
Method
MA
ISt
IFID
vFID
1St
(FID
vFID.
1St
(FIDJ
vFID
BigGAN(d' = 256)
11.02
7.86
12.70
9.94
25.83
30.79
7.58
50.79
55.04
+LeCam
x
11.41
6.82
11.54
10.05
20.81
25.77
8.14
41.51
46.43
+DigGAN
11.15
8.13
13.06
9.98
16.87
21.59
9.04
23.10
27.78
+NICE
10.99
6.31
11.08
10.32
13.17
17.80
8.96
19.53
24.33
+LeCam+DA
11.25
6.45
11.26
10.12
15.96
20.42
9.17
22.75
27.14
ALCCAmHDAAKKDL.ANN
10.12
18.70
22.40
+LeCam+DA+NICE
11.28
5.72
10.40
10.54
10.02
14.93
9.35
14.95
19.60
OmniGAN(d'=100)
12.73
8.36
13.18
10.14 4
40.59
44,92
6.91
60.46
64.76
+DA
12.94
7.41
12.08
11:35
17.65
22.37
10.01
30.68
34,94
+ADA
13.07
6.12
10.79
12.07
13.54
18.20
8.95
44.65
49.08
+NICE
13.77
3.83
8.61
12.57
8.68
13.53
11.97
14.53
19.22
+NICE+ADA
13.82
3.78
8.59
12.75
6.28
10.92
12.04
9.32
14.18

---

4.2 Results on low-shot generation on StyleGAN2
Table 3 showcases the superior performance of NICE in comparison leading methods, significantly improving baselines and achieving tatee-o--theart results. NICE also surpasses KDDLGAN, despite KDDLGAN leverages the large-scale CLIP model [43]. Figure shows realistic images generated by NICE under scarce data training.
Figure 2: Images generated using NICE+ADA StyleGAN2; see Figure 14 in $J for more examples and comparisons for this low-shot generation setting.
Table 3: FID + scores for unconditional image generation with StyleGAN2 100-shot Obama/Grumpy cat/Panda and AnimalFace-Cat/Dog datasets. indicates using generator pre- trained on the full FFHQ dataset. means using the CL.PP [43] model pre-trained on large scale data.
100-shot
Animal Face
Method
MA
Pre-trained
Obama
Grumpy Cat
Panda
Cat
Dog
StyleGAN2 [25]
80,20
48.90
34.27
71.71
131.90
StyleGAA22-SGAAN-LA [16]
X
79.88
38.42
28.6
78.78
109.91
StyleGAN2+NICE
24.56
18.78
8.92
25.25
46.56
ADA [22]
45.69
26.62
12.90
40.77
56.83
DA [68]
46.87
27.08
12.06
42.44
58.85
DigGAN [1]]
36.38
25.42
11.54
35.67
59.98
LeCam [57]
X
33.16
24.93
10.16
34.18
54.88
GenCo [8]
32.21
17.79
9.49
30.89
49.63
InsGen [63]
V
32.42
22.01
9.85
33.01
44.93
MaskedGAN [17]
33.78
20.06
8.93
FakeCIR [32]
26.95
19.56
8.42
26.34
42.02
TransferGAN [59]
39.85
29.77
17.12
49.10
65.57
LcCam-KDDL.GAN [9]
29.38
19.65
31.89
50.22
ADA+NICE
X
20.09
15.63
8.18
22.70
28.65
4.3 Results on FFHQ for StyleGAN2 and ImageNet for BigGAN
Results for unconditional image generation FFHQ are in Table 4, and for conditional image generation on ImageNet 64 x 64 in Table 5. We limit FFHQ to 100, 1K, 2K, 5K real images and follow [17, for ImageNet setting. NICE showeases superior performance FFHQ and ImageNet.
Table 4: FID J. scores on FFHQ using StyleGAN2. ADA-Linear is introduced [63].
Table 5: Comparison with and w/o NICE on ImageNet given different percentage of training data.
10% data
5% data
2.5% data
Method
MA
Method
MA
FFHQ
IS f
FID 1
IS
FID 4
1St
FID J
100
2K
5K
BigGAN
X
10.94
38.30
6.13
91.16
3.92
133.80
StyleGAN2
179
100.16
54
49.68
ADA
12.67
31.89
9.44
43.21
8.54
56,83
ADA
85.8
21.29
15.39
10.96
DA
12.76
32.82
9.63
56.75
8.17
63.49
ADA-Linear
82
19.86
13.01
9.39
MaskedGAN
13.34
26.51
12.85
35.70
12.68
38.62
InsGen
45.75
18.21
11.47
7.83
KDDLGAN
14.14
20.32
14.06
22.35
14.65
28.79
FakeCLR
42.56
15.92
9.90
7.25
NICE
X
14.18
21.44
13.96
24.72
13.32
31.45
ADA+NICE
38.42
14.57
8.85
6.48
ADA+NICE
14.58
18.29
14.10
20.07
13.92
24.41
4.4 Analysis NICE
Analysis of the stabilizing effect of NICE. In our 10% CIFAR-10 experiments with OmniGAN(d' 256), Figure: provides compelling evidence supporting our theory. Figure 3a shows that Omni- GAN+AN and OmniGAN+NICE achieve lower weight norms than OmniGAN, validating Theorem

---

Weight Norm
Real images
Fake umages
OmniGAN
OmniGAN+AN
OmniGAN+NICE
[|]]]]]
11
2y
dr
4.6c2
30-4
3e2
9c-3
4.3e2
2c-4
2c2
6e-3
4.0e2
1o-4
1e2
3e-3
3.7e2
0e-4
0e2
0e-3
50 100 150 195
50
100
150 195
50
100
150 195
50
100
150 195
iterations (x 1000)
iterations (x 1000)
iterations (x1000)
itcrations (x 1000)
(a) Weight Norm
(h) P
0s (c) ||
(d) || BLD
Figure 3: Weight and gradient norms of the discriminator on 10% CIFAR-I0 with OmniGAN
(d' 256). (a) total discriminator weight norms, (b) gradient norm the layer before classification head, (c) gradient norm of f w.r.t. input, and (d) gradient norm of discriminator loss w.r.t input.
Real images
Fake images
Test images
tFID
[(z)
D
[(ar]
45
OmaiGAN
18
OmniGAN+AN
9-
3
OMniGAN+NICE
25
9
15
118
5
50
100
150
195
50 100
150
195
50 100
150
195
100
200
300
390
iterations (x 1000)
iterations (x 1000)
iterations (x 1000)
iterations (x 1000)
(a) OmniGAN
(b) OmniGAN+AN
(c) OmmiGAN+NICE
(d) LFID curves
Figure 4: The discriminator output w.r.t. real, fake and test images of (a) OmniGAN, (b) Omni-
GAN+AN, (c) OmniGAN+NICH along with (d) (FID curves on 10% CIFAR-I0 using OmniGAN
256). The shaded region represents the standard deviation. Note we scale the y-axis in (b) and (c) for visual clarity. In (d), training time is doubled to evaluate the endurance of methods under prolonged training conditions.
1 that the multiplicative noise modulation reduces the weight norm of discriminator, thus lowering the Rademacher complexity and improving generalization.
Figure 3b shows gradient surge at the latent layer before the classification head, especially for real images in OmniGAN+AN and OmniGAN+NICE, surpassing gradients of OmniGAN, This observation aligns with our prediction in Prop. 1 that introducing noise amplifies gradients due the maximization effect on real images when training discriminator and on fake images when training generator, causing undesired gradient issues.
Figure 3c illustrates smaller squared gradient norm |0f/0z for OmniGAN+NICE compared to OmniGAN+AN and OmniGAN, showcasing the effectiveness of consistency regularization in penalizing the gradient of discriminator, in line with the theoretical derivation Theorem 2.
Figure 3d demonstrates that OmniGAN+NICE achieves a lower gradient norm of the discriminator loss w.r.t. the input than both OmniGAN and OmniGAN+AN, affirming ability of NICE counteract the negative effects of large gradient norms caused by noise modulation. The resulted smaller gradients at the input validates efficiency of NICE in stabilizing training.
Analysis of the ability of NICE to enhance generalization. Figure 4 visualizes the discriminator outputs for real, fake, and test images along with tFID eurves. In contrast OmniGAN, Omni- GAN+AN and OmmiGAN+NICE balance discrimination and generalization [67]. maintaining a steady discrepancy between real and fake images. This equilibrium ensures smaller discrepancy between real seen images and unseen samples, effectively lowering the generation error as outlined in Eq. 3 of Lemma 1. As result, the generalization of GANs is enhanced, leading superior performance for image generation, as illustrated by the tFID curves in Figure 4d.
4.5 Ablation studies
Ablation of components in NICE. We examine the impact of various components of NICE: (i) introducing adaptive noise (AN) to the discriminator, (ii) applying consistency regularization to

---

Table 6: Ablation studies on components (AN, NICED,, NICED,, NICEG, in NICE.
Method
10%
CIFAR-10 10
10%
CIFAR-100
Obama
AN
NICEd, NICED, NICEE,
1St
tFID
vFID
1S1
tFID
vFID 4
FID 4
8.49
22.24
26.33
8.19
45.41
50.33
80.26
9.16
10.14
13.80
11.22
23.76
28.34
44.68
9.17
8.88
12.69
11.38
20.09
24.62
32.14
9.16
12.59
11.19
18.80
24.13
29.95
9.23
7.45
11.25
11.62
18.73
23.42
27.93
9.26
7.23
11.08
11,50
16.91
21.56
24.56
real images (NICEp,), (iii) applying consistency regularization fake images (NICEp,) during discriminator training, and (iv) enforcing consistency on fake images while training the generator (NICEG,). Table 6 shows evaluations across 10% CIFAR-I0//00 using OmniGAN (d' = 256) and the Obama on StyleGAN2. The optimal performance is achieved when NICE is applied to both real and fake images in the training of both the discriminator and generator.
Impact of different factors in NICE. For an in-depth analysis on how different factors NICE contribute its performance, readers are directed to $E.
Comparison with alternatives for enhanced generalization and stability. Table shows that NICE outperforms DA, ADA, AWD (adaptive weight decay), AN+AGP (adaptive noise with adaptive gradient penalization) and NICEadd (with additive noise). Detailed implementation of these variants can be found in $D.2. Figure 5 shows that NICE also enhances performance with increasing network size, while ADA and DA exhibit decline. As augmentation- methods such as DA, ADA, and AACR, may leak augmentation cues to the generator (Figure 9 of Ş1), NICE mitigates this drawback.
10% CIFAR-10
10% CIFAR-100
OmniGAN
DA
ADA
Method
IFID
tFID
1St
tFID vFID
ISt
tFID] vFID
OmniGAN
8.49
22.24
26.33
8.14
45.41 50.33
38
50
40
+AWD
8.56
18,28
22.12
9.64
37.68
42.01
+AN+AGP
8.98
11.55
15.25
10.72
27.73
32.15
20
30
+NICFadd
8.64
17.94
21.59
9.34
28.59
33.02
13
22
+DA
8.84
12.90
16.67
10.16
24.50
28.96
10
+ADA
9.67
13.86
17.70
11.23
23.11
27.58
8-
17
+AACR
9.63
10.93
14.73
11.37
21.42
25.76
6-
=56
512
768
1024
256
512
768
1024
+NICE
9.26
7.23
11.08
11.50
16.91
21.56
10% CIFAR-10
10%
CIFAR-100
Table
Various methods+OmniGAN 256))
Figure 5: tFID + w.r.t. different network sizes d'.
Table 7 Fig. 5: Various methods on OmniGAN, Tab. 7: AWD is Adaptive weight decay. AN+AGP:
Adaptive noise adaptive gradient penalty. NICEaad: NICE with additive noise. AACR: Adaptive augmentation- based consistency reg. Fig. 5: tFID | w.r.t. different d' on 10% CIFAR-10/100.
Rationalizing Advantages of NICE. Refer to $F and &G for a comprehensive explanations of why NICE outperforms other alternatives, enhancing generalization and stability. Given mild increase in the computational load ($H), substantial performance gains are achieved.
Conclusions
Our newly proposed approach for GAN training, Nolse-modulated Consistency rEgularization (NICE), improves generalization of various GAN models and their performance under limited data training. Through rigorous theoretical analysis, we have shown that NICE reduces Rademacher complexity and penalizes the gradient of the latent features within the discriminator. Our experimental results match theoretical findings, illustrating that NICE not only improves generalization but also enhances stability of GAN training- By hamnessing these two key advantages, NICE works with various network backbones and consistently outperforms existing methods. These exceptional results firmly establish NICE as powerful solution for preventing the pervasive issue of discriminator overfitting in GANs. Limitations and border impact of our work are discussed in SA.
10

---

Acknowledgements
We thank to Moyang Liu, whose discussions and constant encouragement significantly shaped this work. YN is supported by China Scholarship Council (202008520034). PK is supported by CSIRO's Science Digital. We are grateful for the valuable feedback provided by all anonymous reviewers.
References
[1] Guillaume Alain and Yoshua Bengio. What regularized auto-encoders learn from the data-generating
distribution. JMLR, 15(1)33563-3593, 2014. 5
[2] Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. In ICML, pages 214-223, 2017. 2.
[3] Raman Arora, Peter Bartlett Poorya Mianjy, and Nathan Srebro. Dropout: Explicit forms and capacity control. In ICML. pages 351-361. PMLR, 2021. 4
[4] Sanjeev Arora, Rong Ge. Yingyu Liang. Tengyu Ma, and Yi Zhang. Generalization and equilibrium in generative adversarial nets (gans). In ICML, pages 224-232. PMLR, 2017. 2, 3.
[5] Peter L. Bartlett, Dylan J Foster, and Matus J Telgarsky. Spectrally-normaiizee margin bounds for neural networks. NewrIPS, 30, 2017, 4
[6] Peter L Bartlett and Shahar Mendelson, Rademacher and gaussian complexities: Risk bounds and structural results. JMLR, 38N4v):463-48, 2002. 2,
[7] Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale GAN training for high fidelity natural
image synthesis. In ICLR, 2019. 1, 6
[8] Kaiwen Cui, Jiaxing Huang. Zhipeng Luo, Gongjie Zhang, Fangneng Zhan, and Shijian Lu. Genea: generative co-training for generative adversarial networks with limited data. In AAAI, volume 36, pages 499-507, 2022. 3, 6, 8
Kaiwen Cui, Yingehen Yu, Fangneng Zhan, Shengeai Liao, Shijian Lu, and Eric Xing. Kd-dlgan: Data
limited image generation via knowledge distillation. CVPR, 2023. 1, 6, 7,8
[10] Jia Deng. Wei Dong, Richard Socher, Li-Jia Li. Kai Li, and Li Fei-Fei. Imagenet: large-scale hierarchical image datahase. In CVPR, pages 248-255, 2009.
[11] Tiantian Fang, Ruoyu Sun, and Alex Schwing. DigGAN: Discriminator gradient gap regularization for GAN training with limited data. In NeurlPS, 2022. 1,2,3, 7,
[12] Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-aware minimization for efficiently improving generalization. In ICLR, 2021. 5
[13] lan Goodfellow, Jean Pouget-Absas Abadie, Mehdi Mirza, Bing Xu, David Fardee-rrley,.a Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In NeurlPS, 2014. 1,2
[14] Ishaan Gulrajani. Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron Courville. Improved training of wasserstein gans. In NeurlPS. pages 5769 5779, 2017. 2,4
[15] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge to a local nash equilibrium. NeurIPS, volume 30, pages 6626 6637, 2017. 7
[16] Liang Hou, Huawei Shen, Qi Cao, and Xueqi Cheng. Self-supervised gans with label augmentation. NeurIPS, 3019--13,,,,,,, 2021.
[17] Jiaxing Huang, Kaiwen Cui, Dayan Guan, Aoran Xiao, Fangneng Zhan, Shijian Lu, Shengeai Liao, and Eric Xing. Masked generative adversarial networks are data-efficient generation learners. NeurIPS, 35:2154-2167, 2022. 1, 2. 6. 7,8
[18] Kaiyi Ji, Yi Zhou, and Yingbin Liang. Understanding estimation and generalization error of generative adversarial networks. IEEE Transactions on Information Theory, 67(5):3114-311 2021. 2.
[19] Liming Jiang, Bo Dai, Wayne Wu, and Chen Change Loy. Deceive d: adaptive pseudo augmentation for
gan training with limited data. NeurIPS, 34:21655-21667 2021. 2
[20] Minguk Kang, Jun-Yan Zhu, Richard Zhang, Jacsik Park, Eli Shechtman, Sylvain Paris, and Tacsung Park.
Scaling up gans for text-to-image synthesis. CVPR, 2023. 1,
[21] Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of GANs for improved
quality, stability, and variation. In /CLR, 2018. 2
[22] Tero Karras, Miika Aittala, Janne Hellsten. Samuli Laine, Jaakko Lehtinen. and Timo Aila. Training generative adversarial networks with limited data. NeurIPS, 33, 2020. 1, 2, 3, 6, 8, 20, 25, 28, 29
[23] Tero Karras, Miika Aittala, Samuli Laine, Erik Härkönen, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Alias-free generative adversarial networks. NewrIPS, 34:852-863. 863, 2021.
[24] Tero Karras, Samuli I Laine, and Timo Aila. A style-based generator architecture for generative adversarial
networks. In CVPR, 2019. 2, 6
[25] Tero Karras. Samuli Laine, Mila Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of stylegan. In CVPR, pages 11111018111 2020. 1, 2. 6, 2,
11

---

[26]
Naveen Kodali, Jacob Abemneth, James Hays, and Zsolt Kira. On convergence and stability of gans.
urXiv:1705.07215, 2017. 5.5
[27]
Lingke Kong, Chenyu Lian, Detian Huang, Yanle Hu, Qichao Zhou, et al. Breaking the dilemma of medical image-to-image translation. NeurlPS, 34:196419978, 2021. 1.2
[28]
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers features from tiny images. 2009, 6 [29] Nupur Kumari, Richard Zhang, Eli Shechtman, and Jun-Yan Zhu. Ensembling off-the-shelf models for gan training. In CVPR, pages 10651-10662, 2022. 1,2
[30] Tuomas Kynkäänniemi, Tero Karras, Mika Aittala, Timo Aila, and Jaakko Lehtinen. The role of imagenet classes in fréchet inception distance. In ICIRR, 2023. 1,2
[31] Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. nature, 521(7553)4436 2015.
[32] Ziqiang Li. Chaoyue Wang. Heliang Zheng. Jing Zhang. and Bin Li. Fakecir: Exploring contrastive learning for solving latent discontinuity in data-efficient gans. In ECCV, pages 598-615, 2022. 1, 2, 6, 7,
8, 23
[33] Kanglin Liu, Wenming Tang, Fei Zhou, and Guoping Qiu. Spectral regularization for combating mode collapse in gans. In ICCV, pages 6382-6390, 2019.
[34] Wenjie Luo, Yujia Li, Raquel Urtasun, and Richard Zemel. Understanding the effective receptive field in deep convolutional neural networks. NeurlPS, 29, 2016. 4
[35] Tengyu Ma. Lecture Notes for Machine Learning Theory (CS229M/STA7S214), 4
[36] Lars Mescheder, Andreas Geiger, and Sebastian Nowozin. Which training methods for gans do actually
converge? In ICML, pages 3481-3490, PMLR, 2018. 5,5
[37] Poorya Mianjy and Raman Arora. On dropout and nuclear norm regularization. In ICML, pages 4575-4584.
PMLR, 2019.
[38] Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for
generative adversarial networks. In ICLR, 2018, 2, 4
[39] Alfred Müller. Integral probability metrics and their generating classes of functions. Advances in applied probability, 29(2):429-443, 1997. 3
[40] Yao Ni, Piotr Koniusz, Richard Hartley, and Richard Nock. Manifold learning benefits gans. In CVPR,
pages 11265-1 1274, 2022. 1,2
[41] Yao Ni, Dandan Song, Xi Zhang, Hao Wu, and Lejian Liao. Cagan: Consistent adversarial training enhanced gans. In IJCAI, pages 2588-2594, 2018. 3
[42] Sebastian Nowozin, Botond Cseke, and Ryota Tomioka. f-gan: Training generative neural samplers using variational divergence minimization. NeurlPS, 29, 2016. 2
[43] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, pages 8748-8763, 2021. 8
[44] Elad Richardson, Yuval Alaluf, Or Patashnik, Yotam Nitzan, Yaniv Azar, Stav Shapiro, and Daniel Cohen- Or. Encoding in style: a stylegan encoder for image-to-image translation. In CVPR, pages 2287-2296,
2021. 2,2
[45] Kevin Roth, Aurelien Lucchi, Sebastian Nowozin, and Thomas Hofmann. Stabilizing training of generative
adversarial networks through regularization. NeurlPS, 30, 2017. 2
[46] Tim Salimans, Ian Goodfellow, Wojeiech Zaremba, Vicki Cheung. Alec Radford, Xi Chen, and Xi Chen.
Improved techniques for training gans. In NeurlPS. pages 2234-2242. 2016.
[47] Axel Sauer. Tero Karras, Samuli Laine, Andreas Geiger, and Timo Aila. StyleGAN-T: Unlocking the power of GANs for fast large-scale text-to-image syathesis. In ICML, pages 30105-30118 18, 2023. 1,2
[48] Fatemeh Shiri, Xin Yu, Piotr Koniusz, and Fatih Porikli. Face destylization. In DICTA, pages 1-8. IEEE,
2017. 1
[49] Fatemch Shiri, Xin Ya, Fatih Porikli, Richard Hartley, and Piotr Koniusz. from portraits. In WACV, 2018. 1
face recovery
[50] Fatemeh Shiri, Xin Ya, Fatih Porikli, Richard Hartley, and Piotr Koniusz. Identity- face recovery from stylized portraits. IJCV, 127(6-7):8633-8) (8333-83,222 2019.
[51] Zhangzhang Si and Song-Chun Zhu. Learning hybrid image templates (hit) by information projection.
TPAMI, 345)) 2012.
[52] Ivan Skorokhodov, Aliaksandr Siarohin, Yinghao Xu, Jian Ren, Hsin-Ying Lee, Peter Wonka, and Sergey
Tulyakov. 3D generation on ImageNet. In ICLR, 2023. 1,2
[53] Ivan Skorokhodov, Sergey Tulyakov, Yiqun Wang, and Peter Wonka, EpiGRAF: Rethinking training of 3d GANs. In NewrIPS, 2022. 1,2
[54] Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: simple way to prevent neural networks from overfitting, JMLR, 15(1):1929-1958 2014. 4
[55] Ming Tao, Bing-Kun Bao, Hao Tang, and Changsheng Xu. Galip: Generative adversarial clips for text-to-image synthesis. CVPR, 2023. 1,2
[56] Hoang Thanh- Tung, Truyen Tran, and Svetha Venkatesh. Improving generalization and stability of generative adversarial networks. In ICLR, 2019. 3.5
12

---

[57] Hung- Yu Tseng, Lu Jiang, Ce Liu, Ming-Hssan Yang. and Weilong Yang. Regularing generative adversarial networks under limited data. In CVPR, 2021. 2, 6.
[58]
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. NeurlPS, 30, 2017. 3
[59]
Yaxing Wang, Chenshen Wu, Luis Herranz, Joost Van de Weijer, Abel Coonalez-Garcinn and Bogdan Raducanu. Transferring gans: generating images from limited data. In ECCV, pages 218-234, 2018. 6,
[60] Colin Wei, Sham Kakade, and Tengyu Ma. The implicit and explicit regularization effects of dropout. In ICML, pages 10181-10192. PMLR, 2020.
[61]
Xiang Wei, Beqing Gong, Zixia Liu, Wei Lu, and Liqiang Wang. Improving the improved training of wasserstein gans: A consistency
and its dual effect. arXiv preprini 10xx:1803.01444 2018. Lijum Wu, Juntao Ii, Yue Wang, Qi Meng, Tao Qin, Wei Chen, Min Zhang, Tic- Yan Liu, et al. R-drop: Regularized dropout for neural networks. NeurIPS, 34: 14:10890-100 119800-0995, 2021.
[62]
[63] Ceyuan Yang. Yujun Shen, Yinghao Xu, and Bolei 7hou. Data-efficient instance generation from instance discrimination. NeurlPS, 34:9378-9339,, 2021. 1, 2, 6, 8, 23
[64] Han Zhang, Zizhao Zhang, Augustus Odena, and Honglak Lee. Consistency regularization for generative adversarial networks. In ICLR, 2020. 1,
[65] Jianfeng Zhang. Zihang Jiang, Dingdong Yang, Hongyi Xu, Yichun Shi, Guoxian Song. Zhongeong Xu, Xinchao Wang. and Jiashi Feng. Avatargen: 3d generative model for animatable human avatars. In
ECCV, pages 668 (88-685.666 Springer, 2023. 1.2
[66]
Linjun Zhang. Zhun Deng. Kenji Kawaguchi, Amirata Ghorbani, and James Zou. How does mixup help
with robustness and generalization? In ICLRR, 2021.
[67]
Pengchuan Zhang, Qiang Liu, Dengyong Zhou, Tao Xu, and Xiaodong He. On the discrimination-
generalization tradeoff in GANs. In ICLRR 2018. 3,5, 5,
[68] Shengyu Zhao, Zhijian Liu, Lin, Jun- Yan Zhu, and Song Han. Differentiable augmentation data-
efficient gan training. NewrlPS, 33, 2020. 2, 7, 8, 20, 25
[69] Zhengli Zhao, Sameer Singh, Honglak Lee, Zizhao Zhang, Augustus Odena, and Han Zhang. Improved
consistency regularization for gans. In AAAI, volume 35, pages 1033-11041, 2021. 1,
[70] Yong Zhong, Hong Tao Liu, Xiaodong Liu, Fan Bao, Weiran Shen, and Chongxuan Li. Deep generative modeling on limited data with regularization by nontransferable pre-trained models. In ICI.R, 2023. 1, [71] Peng Zhou, Lingxi Xie, Bingbing Ni, Cong Geng, and Qi Tian. Omni-gan: On the secrets of egans and beyond. In ICCV, pages 14061-14071, 2021. 2, 6, 25
13

---

NICE: Nolse-modulated Consistency rEgularization for Data-Efficient GANs (Supplementary Material)
Yao Nit, Piotr Koniusz* *:5.
Thee Australian National University $Data61CCSIRO
lastmmm @an..edu.an
A Broader impact and limitations
NICE is a solution to limiting the extensive image usage in GAN training. Conventional GANs heavily rely on massive training data, which can jeopardize data privacy during large-scale data collection. NICE is an innovative method for data-limited image generation. By successfully training GANs with just 100 images, NICE only minimizes the dependency on large-scale training datasets, but also significantly reduces the training cost. NICE decreases the risk of privacy violations (using smaller datasets equals to the lesser risk data mismanagement), NICE also provides a cost-effective alternative for training GANs. As the data required for training is lesser compared with large-scale models, NICE also energy-consume consumtonsn friendly. Of course, with small-scale dataset comes limitation on diversity. The model cannot generate instances for which the data is completely lacking. Thus, special care needs to be taken regarding gender, race and other biases generated images.
NICE advances GANs with Theorems 1 and 2, along with Corollaries and 2. It offers novel insights the weight regularization. These findings, driven by the multiplicative and additive noise modulation, along with the gradient regularization due to the consistency regularization, pave the way for further future research.
B
Notations
Below, we explain the notations used in this work.
Scalars: Represented by lowercase letters (e.g., 8).
Vectors: Bold lowercase letters (e.g., v. u. m))
Matrices: Bold uppercase letters (e.g., W. H).
Vector/Matrix elements: d4K and represent k-th element of and a. His denotes the element located at the j-th row and k-th column of matrix H.
Functions: Letters followed by brackets (e.g., (-)) A(-), V.()), H().
Function sets: Calligraphic uppercase letters are used, but note that / and B specifically denote Gaussian and Bernoulli distributions, respectively (e.g., H, F).
Probability measures: Denoted by letters μ, and ps-
Dataset samples: Expressed as (+)), (()) (()) ((),
Mutippicatio//Coppsttio:: C represents element-wise multiplication and denotes function
composition.
Expectation//epirrcaal expectation: E[-] represents the average or expected value of random variable while B[:] denotes the empirical expectation calculated over observed data samples.
'The corresponding author.
Code: pestps://githee http://hhbb.co/aaaeeell//l//////
14

---

C Proofs
We define ||o||2= v² v as the squared of vector Note the commutative property of the dot product, v u== u v, and the distributive property over matrices, v² A + u² A = (o - = A. With these definitions and properties established, we now prove Theorem 1, Proposition and Theorem 2. We begin with simplified scalar- derivation, ensuring a rapid derivation of the final results, followed by comprehensive proofs encompassing vector and matrix calculus for holistic understanding.
C.1 Proof of Theorem
Theorem (Regularization by the Gaussian multiplicatiiv noise in deep regression) Consider regression task on two-layer neural network. Let X CR and ) [-1, 1]á be the input and output spaces, and D X x y. Let examples {(x!1) ) ---- ~D" Let f :X + be parameterized by {w: WIE E IR xdo W 226 E Rdxxd and fm (xr 2) 22 @ a (W 1x)) where a is I-Lipschitz funtttoionfunnctinn,, @ denotes element-wise multiplication and 2 (1, (3 Let a= (W w à = =E, ( (.(a())aa(()) iR 20 denotes the k-th element of â. Let || W2l12,1=22
activation function,
||wk|l2 where we is the k-th column vector W2. 2. The regression task with the lo loss leads to the weight regularization as:
Lmoise = E, [1 (8) W2(2 O )1 = E. [1l W (9) [2 + 83 Σ au ww
We commence with simplified derivation using scalars, which can quickly lead to the final results:
E,E. (2) w2(za ))) ) E, (v - wya w2(z - 1)) y0 10222 - 1)a()) + (wz(z - 1)
EE, (1 9 - 22) - -( - w2a(() )
[1y( uya!!)) 9 + w E 1)
(1)( ) +
Proof:
Lnaise (w) = E, 1gy (2 a (0) E, W 20 W2 1) o
(1y(( "2 - (0_ W 2a (
I (
(1) W2a.0 11]] 22, (1) W a "22'0 + E,E, W ((=
-B,1 y
W 101 E W W2 1) o (*)
(12)
Let m (O, "1"),, with my and a representing the k-th elements of vectors m and a (
respectively. Using these definitions, we derive the second term of Eq. 12 as follows:
E m
1) о 1 = Ê,E, W W2(mmmm (
= E.
E.
[m @ (1) ( W W
W @ ())
di
di
E,
(m;a, (w?
(maa
(13)
j=
Given that the elements in are independent, for a l we have:
E, [m (W W (m
(2)) =0.
(14)
Observe that (W (W2W2)as 224 || represents the squared norm of k-th column vector in W2. Let denote the element-wise mean square of ( i.e., à= E, (a(() @ a0)) We leverage the distribution property B|2 o² + 1² for z NNu, o²), simplifying to E[2²] = o² when μ 0. This lets us
15

---

reformulate Eq. 13 explicitly as:
di
E, E.
(mja) (3) (W? W W2))s
=R, Bm
( (1)
(W? W (2)
di
di || 1w 10 11 =33E, (+)
Bg Em
a
(1)2)
Σ
k=1
= 32
(15)
Substituting the second term in Eq. 12 with Eq. 15, we arrive at:
Enoise (w) = E, W2 o a 1a = [1 (2
+5)+ ax 1
Σ &
(16)
Integrating the Gaussian multiplicative noise into the optimization of regression task leads to an adaptive regularization B² >x fx 1/ on W2, effectively reweighting the squared norm ||w l of each column in accordance with åg 2 0. This mechanism, driven by the variance 3 of noise enhances the robustness of model and its generalization capability. It is worth highlighting that this process relates closely to the (2.1)-norm of W2, defined as [22ll2,1 = Ex|ww
Corollary Utilizing the additive noise injection where the l2 loss yields weight regularization
N(0, 32 1² the regression task with
Lost noise (w)
B, [19) - WW + aa
[B, W2a (1)
Σ k
w 12 (17)
C.2 Proof of Proposition
Proposition 1 Define f= o. as the feature extructor of the discriminator from the second layer onward. Let ā = fā) and = fx) be the latent features of the first layer for fake and real images, respectively. The discriminator is defined h(-) (-)), with ip the prediction head and & as the loss function. Introduce a multiplicative noise z ~ N(1, (1, 1 modulating solely the first layer. Let kkk (a) be the k-th diagonal entry of the Hessian matrix of h at a and ag be the k-th element of a. Applying Taylor expansion to h((),, the GAN loss can be expressed as follows:
min LDN 0g
:xE [1( 2 a)] Ex [1 O a))
BRa [haa]] F.g [Maa]] + (Ra a (ā)]
E. EA af Hf
min LA = E.E [1 a)] Ehalhaa 8 Ea E a H (a) 0
We provide a succinct proof for the discriminator using scalar values to quickly arrive at the final result, where (-) and (-) are the first- and second-order derivatives, respectively.
E.h(zä) F.g((aa E.h (ā (2 1)ā) E. + (= - (1-1)a)
[E.[M(ā)++ 1)āh 1x-1)²
(x-1)
/² (2))((a) - E. (x( (z)1)a
2 a (a)())
2
(z 1)2
=h(āa h(aa) + Ee
(a) aahh
2
82
=h(āa h(aa + a (a) a2hh"(aaaa))) d2h)(aa):
2
Proof: Consider a multivariate normal distribution m ~ ~ ~ Let Vh(a) and H (a)
represent the first-order derivative and the (j, k) entry of the Hessian matrix of h at a, respectively,
16

---

implying that H = H due to the symmetry of the Hessian matrix. With these notations established, we proceed as follows:
D = Ex h(ā =)] - E. h(a 3 2)
ExEE (2-) a) Eaby
(z 1)
EaEm h [a] - B. h(a f
EalEm
I(ā) + (m (m)))+mm@)))//)) V/(ā)) + m o (a)"
Em h(a) + (m VV(aa + (m @maa)" (q(mma))
-Eah(a) Eah(a)
+ Em (m o V\()) Em (m a) Vh((a
(18)
F.a Fm (m@a) (a) F. F. (m@a) H ( (a)(moa).
(19)
Addressing the first-order gradient term presented in Eq. 18, we notice the following:
Em (m @ a)" [a)]] EgE. (m V)"aa( ( Vh(a)) - E. (0 2 a) Vh(a)] = O.
-Ea (0 a)
(20)
Examining the second-order gradient in Eq. 19, and leveraging the independence of elements in m, we apply the result from Eq. 14. With âg and ag representing the k-th elements in à and a. respectively, we derive:
Ea Km
m O (1 o aa] - EalEm m m aa) H (h (a) e Ga)l
-Ea Bm
mg (1) (a)maā
Em
m (h) (a)a(aa)m)) (a)maas)
32
Ea
az (
(a)
Ea
Σ
az 22 (a))
(21)
2
&
&
By substituting Eq. 18 with the results from Eq. 20, and Eq. 19 with Eq. 21, arrive at the following expression:
8²
(a))
DN Ea [M(a]] E. [h(a)]
Ea
Σ
a
H Rk
(a)
E,
Σ
a H
(22)
kk
Utilizing the same derivation process before for LAN, arrive the following result:
32 2 Ea
LAN ~ -a
KR (a))
(23)
Introducing the multiplicative Gaussian noise during training of GAN produces the gradient max- imization effect on the real samples in the discriminator training, specially through the term 8² 2 o (a). This term intertwines the second-order partial derivative (diagonal entry H((h) (a) of the Hessian matrix) with the non-negative factor g2 0 2 O, and is related to the trace of the
Hessian matrix given by trace (a,)) = E, RA (a). Employing similar reasoning for the fake samples when training the generator reveals that the noise-modulated discriminator training of GAN tends to amplify the the diagonal entry of the second-order partial derivative in the Hessian matrix at both the real and fake points, influencing the optimization stability.
C.3 Proof of Theorem 2
Theorem Using notations from Prop. and with F: let (af(a) and (H (a))² denote the squares of the k-th gradient element and ki, entry of Hessian matrix of J at a, respectively. Given 2 22 (1, B rd enforcing invariance of f under different noise modulations yields:
QNICE [,=,,,,,((((]) a) (1te::aaaa((00ll00) 0l) 28 (f
SI (a))²
(a))²
17

---

A concise derivation employs scalar values for the Taylor expansion, swiftly leading to results using f-) and ('))) for first and second-order derivatives, respectively.
22 (f(az) f(az))) 2 == (S(a + (z1 1)a) - f(a + (z2 11)))²
=E (2(a)+(=11)aa(())) (a)+ (2) (21-1)² (a) f(a)= - (22 (1) (22-1)² a² f (a)
2
= 3s (11- 22) + (21 - 11-11-- -22 1)² a² {²3"((a) 2
m - m2 fr (a)
- Mmm,mannN(u.8 (m - m2)af (a)
[mī m2)²
=Emi, ma (ml f2 (m, (m) m f -282 a² (12 + 3 (
frz (a)
4
Proof: Let m1, my ~ NO, 8²1 Note 21 and 23 are independent of each other, as well as independent of and (a). The same independence holds for ø2. By taking expectations 21 and 22, and applying Taylor expansion, we analyze the consistency loss of f(a):
NIIE (a) =, (f(aa=))
z2))
-Emy,my (I(a+ [f(a+mi mm) I(a m2 oa))
~ ~Bmi, m2 (I(a) + (ml ① a) (a(a)
2 (mi Ha) oa)
-f(a) - (m2 @ a VVf(a)
mm @ a) aa)" (a) oa))
=Em,mm ((m 2 (a)
(m1 @ a) (0) (a)(m @a)
1
mm 4 a)" H (a) @ a))
(m2 2 Oa)"V
(24)
2
Defining e == my and u == m2 @ with my and m2 are independently sampled from (O, 3² ) we get u ~ (O, diag a @ a)). Here diag(a @ a) is a diagonal matrix with of on the diagonal. Letting ō = V f(a) and H (a), we can rewrite Eq. 24 as follows:
1 u Hu)
Ev, (0= 8 + x" HU u= 8
2
1 DT Ho
Bu,, ((v-u)")
u T
Hu)
2
E... ((u-u)²
(25)
+ Ee,u
((v - =)) (u HU u Hu)
(26)
+ Ev [(0" Hu) + Eu [(u Hu)²]
(27)
Bu (0o Hu)(u" Hu)].
(28)
Next, succinctly derive the four terms: the first (Eq. 25), second (Eq. 26), third (Eq. 27), and fourth (Eq, 28) term, respectively.
The first term (Eq. 25):
Leveraging the property Er, 12.102((11 - -1 as follows:
= 20² for 21, 22 ~ -N(0, we can simplify (Eq. 25)
d'
Ev,, ((v 100-mu) -1))))² E.p,a
Σ
u 00000
E [(0
u)]]0
2 Σ 22 (29)
k=1
k=1
18

---

The second term (Eq. 26):
Utilizing F|(²)= n² + 3 for (s,, )²), and noting that E|>0] when f and elements in v are independent, Eq. 26 is derived as:
E,,u ((v u) o) HU u² Hu)
-E. (v"o)(""Hu)) Eu (u"))(""HH) Hu)] -B,, [10 (u])]
Hu]] Bo, (u'd) Hb)]
(H)))
-2B ((v"))( Hu)]
2 Σ uRdnes 21E
Σ
ogdgllRn = 0.
2
(30)
The third term (Eq. 27):
Eu (v Hb)
Eu [u Hu)*]
Ev [(v" Hu) - Zb ΣΣΣΣ Ug o
En
(31)
Given the independence elements n, only terms with an element repeated two or four times contribute non-zero results, leading to four distinct, non-overlapping cases. Using R[=²] = o² 11
and [=[=1] = fa 6 + 30 for z ((u,o²), and simplifying to R[2²] = d² and B[eA] 3 when μ 0, have:
Case 1: j k + p q, the independence of vj and up simplifies our calculation, leading to:
E Σ
Σ EIg 84 j.pデj
Σ
HzyHuuazaf
(32)
p#1
J.
Case 2: For j = P # k
given the independence of and UR. we have:
Eu Σ
o HH
E j.kys
H3 It Σ
(33)
skys
Case 3: For j Q
* k p. leveraging the independence of vy and us well as the symmetry
HGK = Haj. we obtain:
E.
2 H H = Σ
E = 8t Σ
H2
(34)
jkps
Case 4: For j = q = k
p. using |[ 3 where NN(0, 02), deduce:
Eg Σ
by Hy Hソ Σ
Euf 8r Σ 3H3
(35)
Combining Cases 1-4 together, we arive at for Eq. 27:
g4
Σ 3H3
Σ
Hag 2032
(36)
iktj
The forth term (Eq. 28):
Ep [00 Hv) (u Hu))
Exy
Σ H 03) Ex Σ H
2
Σ
H El (19]] Σ HAREEUR]
2
1
81
Σ
H3jag + Σ
H
(37)
2
)
j.bdj
19

---

Substitute Eq. 29, 30, 36, 37 back into Eq. 25, 26, 27, 28, we have the final results:
[f(2) @a) 1(2200))
2=
0 +0
g
+ 2
Σ
3H aj + Σ
HHy + +2Hz))aaak
HHzja -
23
Σ
8a
[Eizja}+
Σ Hijazak)
jkij
23
Σ
06 8t ΣΣ H
Hzg0zak
233 Σ (af(a) + 8t
(a))²
(38)
Enforcing the consistency regularization implicitly penalizes the first- and second gradients of w.r.t a. with an adaptive penalty controlled / and modulated by a. This modulation recalibrates the squared norms of gradient: the squared norm of the first-order gradient, [ E. [a) and the Frobenius norm of the Hessian, (IH() (a) = E, Ex (HV) (a))². (a))² Specifically, each term is reweighted by ak 2 O, ensuring that the penalty applied each gradient component is scaled proportionally to its associated magnitude of feature.
Corollary Enforcing invariance of f under different additive noise injections, where 21, 22 ~ (O ffr rd leads to the gradient regularization as:
E2,,,, [((() + a) f(22 + +)) 2,3² Σ Vif(a)+ ΣΣ H
(a)) (a))
(39)
D Implementation details for Experiments
Following previous benchmarks [68, 22], augmented all datasets with the simple z- z-flips augmen- tation.
D.1 Network architecture and hyperparameters
CIFAR-10/100. We experiment with OmniGAN (d' 256, 1024) and BigGAN (d' - with the batch size of 32. We follow [68] and train the OmniGAN and BigGAN for 1K epochs on the full data and 5K epochs on 10%/20% data setting. We equip the discriminator with the adaptive noise modulation after convolution weights c E {C1, C2, Cs} at all blocks E 2, 3, 4}. We set Ag 0.001, 0.5, A, 10. Features before loss function are used for NICE on BigGAN.
ImageNet. We experiment with BigGAN with the batch size of 512. We use learning rate of lc-4 for generator and 4e-4 for discriminator. The noise modulation is placed after convolution layers e E (C1, C2, Cs} blocks I E {3, 4, 5). As 0.001, 0.5, A, = Features before loss function are used for NICE.
Low-shot images. We build our NICE upon StyleGAN2 with batch size of 64 and train the networks until the discriminator had seen 25M images. We apply noise modulation with Bernoulli noise after convolutions E {C1, C2) at blocks I E [3, 4, 5, 6). Ag 0.0001, 0.9, A,, 0.05
We experiments on StyleGAN2 with batch size of 64 and train the networks until the dis- criminator had seen 25M images. We apply noise modulation with Bernoulli noise after convolutions e E (C1, C2} at blocks E [3,4, 6). Ag = 0.0001, = 0.6, A, 0.05
FFHQ.
D.2
Implementation details for AWD, A AN+AGP, and AACR
In Figure 5, we compare NICE with three different approaches: adaptive weight decay (AWD), adaptive noise + adaptive gradient penaulty (AN+AGP) and adaptive augmentation-ban consistency regularization (AACR). For AWD, we dynamically control the weight decay using BAwD. We search
20

---

the AwD E (2e-4, le-4, 5e-5, 2e-5, le-5, 5e-6) and obtain the best result at 5e-5. For AN+AGP, we employ the adaptive noise modulation and also adaptively penalize the gradient of aL AN
|
|| 8f
dæ 2
az |
with adaptive strength 9AGP- We test different values AGP E {0.2, 0.5, 2, 5, 10 and obtain the best results with = For AACR, incorporate the adaptive differentiable augmentation (ADA) to provide two different views for the same images, and enforce the discriminator to be consistent to
the two views of the same images. We adaptively control the strength of AACR with BAAACR. We search the AACR {0.2, 0.5, 1,2,5 10} and obtain the best result with AAACR 2.
E Impact of various factors
Below, we analyze the importance of different factors NICE by conducting ablation experiments. The results are presented in Figure 6. In Figure 6a, we observe that placing the noise modulation module after C1 and C2 improves performance, and the best results are obtained when the noise is placed after all convolution layers. Figure 6b indicates that placing the noise at deeper layers leads better tFID scores, and the best performance achieved when the noise is applied at all blocks. Figure 6e shows that A, 10 is good choice for NICE. Figure 6d illustrates = 0.5 produce overall good performance. Figure 6e plots the results for consistency regularization among M feature extractors f modulated with noise. We use M 2 (i.e., f1 and f2) in the main paper. Figure 6e indicates there is no need to use more feature extractors, as fi(-) and fz(-) for consistency regularization produce the best results. Figure 6f presents the results for consistency regularization at every N discriminator steps, indicating that regularizing the discriminator during each iteration preferable, larger intervals result in performance.
FFD
FIID
FFD
18
8.8
15
15
8.2
12
9
7.6
7.0
C. C C. CCs CCs cc ccCs
2 4 12 34
123 234 1234
2
5
10
15
20
47
25
(a) e (C,,O2,C0 Cs}
(b) / {1...4}
(c)
A
IFID
8.8
8.8
8.8
8.2
8.2
8.2
7.6
7.0
0.2 0.3
0.4 0.5 0.6 0.7
0.8
0.9
3
4
5
8
10
2
6
8
12
16
(d) 1
(c) M-time
(f) N-step
Figure tFID under different factors. Ablation studies on 10% CIFAR-10 with OmniGAN 256) w.r.t. different places, different blocks, 4y, M-time and N-step for NICE.
F Rationale for choosing multiplicative noise over direct weight regularization
For the multiplicative modulation with the noise, we have:
i Lm puise
E. W2(z a 1
(, W Wea 10 + 8² Σ àg 11
where z O (*) is element- wise multiplication of z with () and à = (a(a()) o ()) This implies that while the direct penalty of network weights, Σx / 112, is possible, our approach enjoys dynamic modulation 32 where variance 3 is adapted based on the discriminator decisions and āy 2 0 depends on the magnitude of feature vectors.
The importance of such multiplicative modulation is that semantic contents is not added by noise z to the modulated feature vector (*) i.e., only "active" channels a (4) فf 0 are modulated: they can be suppressed magnified according to variance 32.
21

---

In contrast, the additive noise modulation results in the standard penalty 32 >x 122wl1
Lade noise -i.22-Nio..nnlll W2( + a (*))) M1
= "Billg"") W W2a + 3² > |1 11
While obvious, B Σ [Wwkll does not enjoy the dynamic penalty term related to feature norms. This also means (based on the derivation) that the additive noise may change feature semantics, ie., z + (4) may "activate" channels which are non-active a (features (6) 0). Therefore, our multiplicative modulator not only controls the Rademacher Complexity (RC), but controls it feature
in a meaningful manner for discriminator. Only feature semantics that are present vector (that describe an object) are modulated while helping control RC. In contrast, the additive noise introduces semantics that are not present for given image or object.
While directly regularizing the network using 3 "EEle / does not introduce the additive noise, the explicit regularization effect is similar to introducing the additive noise. Thus, we connect the direct weight regularization to the additive noise, it will also have the effect of changing the semantics, which will negatively impact the classification accuracy.
We provide experimental comparisons in Table 8. In addition, Figure 7 shows that AN achives higher classification accuracy than AAN and AWR, demonstrating that multiplicative noise preserves the feature semantics better than the additive noise and the weight regularization.
Table 8: Results for different methods on 10% CIFAR- 10/100 using OmniGAN (d' 256))56). AAN (adaptive additive noise), AWR (adaptive weight regularization), AN (our adaptive multiplicative noise). Please note the consistency loss and dual branch from Figure are not used here as that would result in additional penalties on gradient norms.
10% CIFAR-10
10% CIFAR- 100
Method
Equation
IS f
IFID J
vFID 4
IS 1
(FID 1
vFID
OmniGAN
8.49
22.24
26.33
8.19
45.41
50.33
+AAN
a = a (0
8.52
20.12
24.65
9.64
37.68
42.01
+AWR
32 Σ b. Wk
8.44
18.42
22.56
9.80
32.05
36.53
+AN
(1)) = a 2: N
((1 11)
9.16
10.14
13.80
11.22
23.76
28.34
Acc
OmniGAN
+AN
0.7
+AWR
+AAN
0.6
+NICE
+ALGP
0.5
+AWR+ALGP
+NICE
0.4
0
50
100
150
195
50
100
150
195
iterations (x 1000)
iterations (x 1000)
Figure 7: Classification accuracy of different methods. We input test images into the classifier of discriminator and assess its accuracy in correctly categorizing the images into their true classes (out of 10 categories in CIFAR-10), All other methods have lower accuracy than the baseline OmniGAN, but NICE preserves the classifier accuracy.
22

---

G
Rationale behind choosing NICE over the direct gradient regularization
Consider the regularization term in Theorem and its Taylor expansion. Indeed, this expression penalizes the squared norms of first- and second-order gradients of f(a). However, notice that our expression has specific reweighting effect for penalizing the first- and second-order gradients. The specific gradient penalties
2B²² Σ Vij((a + 3d E anj (a))²
(40)
J.b
emerge only in case of using the multiplicative noise drawn from (1, 3221 for Theorem 2. Impor- tantly, the multiplicative noise z (1,, does not introduce new semantics into feature vector by operation 2 @ a, ie., only feature semantics that are present in feature vector (features of a that are non-zero) that describe object are modulated. Yet. this multiplicative noise does help control the Rademacher Complexity (RC) due to Lemma 2 and Theorem 1. However, imposing generic penalty of form 3( fa)(a) + (H jk (a))² that arises when the additive noise NNO,, 8 is
applied to a (*),, i.e., x + a. Let z (0, 8 I). The consistency loss regularization with the additive noise injection is:
22 + a) f(22 + a)) 2 Σ Vif(a) + Ba ΣΣ
H 12) (a))²
Such a variant means that semantics that are not present in may be "activated" by the noise, drastically altering the meaning of and damaging the information it carries about image/object.
While directly applying gradient penalization does not introduce noise, its connection to the additive noise suggests it will have negative effect semantics. Table 8 evaluates the multiplicative vs. additive noise modulators, and the direct penalties. Table 9 provides experimental results for different gradient regularization variants. Furthermore, Figure 7 shows that NICE obtains best accuracy than other variants, showing that the multiplicative noise modulation preserves the semantics better than other variants, justifying the rationale behind NICE.
Table 9: Results for ALGP (adaptive latent gradient penalization), NICE (consistency regular- ization with additive noise), AWR (adaptive weight regularization) on 10% CIFAR-10/100 using OmniGAN (d' 256).
10% CIFAR-10
10% CIFAR-100
Method
IS f
tFID 4
vFID 1
IS f
tFID
vFID
OmniGAN
8.49
22.24
26.33
8.19
45.41
50.33
+ALGP
8.52
19.15
22.72
9.18
32.98
37.51
+AWR+ALGP
8.72
16.82
20.45
10.14
26.44
30.23
+NICE,dd
8.64
17.94
21.59
9.34
28.59
33.02
+NICE
9.26
7.23
11.08
11.50
16.91
21.56
H
Training overhead with and without NICE
We provide number of parraneters/mmlliily-cccummttt
(MACs) (for both generator and discrimi- nator), number of GPUs and cost of time (seconds per 1000 images, secs/kimg) in Table 10. With the efficient implementation (Figure 1), our NICE only introduces small fraction of cost of time on high-resolute datasets. 15.08% on ImageNet and 18.02% on FFHQ/low-shot datasets. Figure illustrates the FID gain vs. speed, which shows that a slightly increase in training time yields substantial improvements, particularly when compared to recent state-of-the-art approaches like FakeCL. RR [32] and InsGen [63]. We believe the increased computational overhead outweighed by the considerable benefits and the improvements justify the extra fraction of time.
23

---

Table 10: Number of parameters, MACs and secs/kimg for models with vs. without the NICE. Experiments were performed on NVIDA A100 GPUs.
Bascline
+NICE
Dataset
Archtecture
GPUs
#aar.
MACs
sec/kimg
#Par.
MACS
sec/kimg
OmniGAN
CIFAR-10
1
8.51M
2.79G
0.80
8.5
3.87G
1.15
(d'=256)
OmniGAN (d'=256)
CIFAR-I00
8.81M
2.79G
0.81
8.81M
3.87G
1.17
ImageNet
BigGAN
2
115.69M
18.84G
1.79
115.69M
20.43G
2.06
Low-shot & FFHQ
StyleGAN2
48.77M
44.15G
5.66
48.77M
$2.00G
6.68
FID
StyleGANV2
70
55
40
ADA
Ins Gen
25
NICE
FakeCI.
ADA+NICE
15
5
6
7
8
9
10
11
12
sec/kimg
Figure 8: FID sec/kimg of different methods on 5 low-shot datasets. FID are averaged over the low-shot datasets.
I Qualitative comparison among DA, ADA, AACR and NICE+ADA
Figure 9 provides qualitative comparison between augmentation based method DA, ADA, AACR and NICE+ADA. We find DA. ADA and AACR leak augmentation clues the generated images while NICE+ADA prevents the leakage. Figure 10 further demonstrates augmentation-batt although can prevent discriminator overfitting problem by expanding data space through diverse augmentations, has the potential to cause increased gradient norm due to Prop. 1 and the fact ADA involves random additive/muliillicttve noise. However, when combining NICE with ADA, the gradient issues caused by ADA were alleviated.
method,
J Generated images
Figures 11, 12, 13, and 15 provide the generated images on CIFAR-10, ImageNet, CIFAR-100, low-shot datasets and FFHQ with without NICE. We can see NICE improve the image generation quality.
24

---

(a) DA
(b) ADA
(c) AACR
(d) NICE+ADA
Figure 9: Generated images using (a) DA (differentiable augmentation [68]). (b) ADA (adaptive dif-
ferentiable augmentation [22]), (c) AACR (adaptive nugmentation-basee consistency regularization) and (d) NICE+ADA on 10% CIFAR-10 using OmniGAN (d'-256) [71] We use red boxes to bound the images that leak the augmentation. DA leaks the cutout augmentation. ADA and AACR leaks the rotation augmentation. Combining NICE with ADA prevents the leakage of augmentation.
aLD Or
12
Real images
12e-3
Fake images
8e-3
OmniGAN
+ADA
4c-3
+ADA+NICE
0e-3
50
100
150
195
iterations (x 1000)
Figure 10: The gradient norm
öLD
1l2 of ADA and ADA+NICE. dr
(a) DA
(b) ADA
(c) ADA+NICE
Figure 11: Generated images using DA. (b) ADA and (c) NICE+ADA on 10% CIFAR-10 using OmniGAN (d'=1024). Note that DA leaks cutout augmentation and ADA leaks the rotation augmentation.
25

---

(a) ADA
(b) ADA+NICE
Figure 12: Generated images using (a) ADA and (c) ADA+NICE on 2.5% ImageNet using BigGAN, ADA struggles to capture the semanties of images, whereas NICE provides better image quality.
26

---

(a) DA
(b) ADA
(c) ADAANICE
Figure 13: Generated images using (a) DA. (b) ADA and (c) NICE+ADA on 10% CIFAR-100 using OmniGAN (d'=1024). We present the last 40 classes out of the 100 classes 40. Note that DA leaks cutout augmentation clues (row 7 and row 21) and ADA generates images lacking diversity, while NICE+ADA provides better visual quality.
27

---

Obama
Grumpy Cat
Panda
AnimalFace Cat
AnimalFace Dog
(a) StyleGAN2+ADA
(b) SYYYAGANNZ-DAANNICE
Figure 14: Qualitative comparison between ADA [22] and ADA+NICE on 100-shot and AnimalFace datasets. Adding NICE clearly improves the image quality.
28

---

FFHQ-5K
FFHQ-2K
FFHQ-IK
FFHQ- 100
(a) StyleGAN2+ADA
(b) SOVVEGAN2-ADA-NICE
Figure 15: Qualitative comparison between ADA [22] and ADA+NICE on FFHQ dataset. Images
generated without trunction [25]. Adding NICE can provide better visual quality.
29


Reviews:
[{"review_id": "wG5JVfKzUw", "reviewer": "Reviewer_fQTZ", "summary": "The authors present a novel approach called NoIse-modulated Consistency rEgularization (NICE) to solve the challenge of training GANs with limited data. The experiment was conducted on reduced small-scale CIFAR-10, CIFAR-100, ImageNet, and FFHQ datasets. Additionally, they applied their method to low-shot generation tasks, and the results demonstrated state-of-the-art performance.\n\nOverall, the proposed methodology is well motivated and supported by theoretical analysis. The flow of the paper and the writing are easy to follow.\n", "questions": "1.\tIn Table 1, Table 2, Table 3, and Table 4, what \"MA\" refers to?\n\n2.\tFigure 2 was plotted based on a specific dataset, but the paper does not clearly state which dataset was used to generate this figure. It would be helpful if the authors provide more information about the dataset used for Figure 2.\n\n3.\tThe notation used in the paper is inconsistent, for instance, with the symbols $\\boldsymbol{x}_{\\text {real }}$ in page 5 row 205 and $\\boldsymbol{x}_r$ in page 5 row 182. Additionally, there is no explanation for the symbols $\\boldsymbol{x}_r$ and $\\boldsymbol{x}_f$. It would be beneficial for the authors to clarify and provide consistent notation throughout the paper to avoid confusion.\n\n4.\tIn Equation 8, it is evident that training with features modulated by Gaussian noise leads to regularization of the internal weight by the (2,1) norm. However, it is not clear why the authors did not directly penalize the (2,1)-norm to reduce the Rademacher complexity. It would be valuable if the paper includes a comparison between the Discriminator with adaptive noise (AN) and directly penalizing the (2,1)-norm with adaptive strength $\\beta$.\n\n5.\tIn Equation 9, the training of the discriminator involves minimizing the second-order gradient at fake images. However, Figure 2 (b) shows that the gradient norm at the layer before the classification head for fake images increases. Furthermore, the ablation study with $\\mathrm{NICE}{D_f}$ demonstrates improved performance, as well as with $\\mathrm{NICE}{G_f}$, but there is no analysis of the gradient norm for the generator.\n", "limitations": "The limitations could be further clarified.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is written in a clear and well-structured manner, making it easy for readers to follow the presented ideas.\n\n2. Through extensive experiments on various reduced small-scale datasets, the method consistently achieves state-of-the-art performance, demonstrating its effectiveness.\n\n3. The paper is well-grounded in theoretical motivation and effectively validates the proposed methodology through empirical analysis.\n", "weaknesses": "Even though the experimental results support their theory, the ablation study and the experiment for gradient analysis are only based on some specific datasets. Evaluating the theory on more datasets would provide more convincing evidence.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Rev. 1 (fQTZ)\n\n***We thank the reviewer** for the constructive review and valuable questions that have helped us improve our work.*\n## 1. What does MA refer to in Tables 1-4?\nWe apologize. MA denotes `massive augmentations' (including DA and ADA) first used in:\n> *Generative co-training for generative adversarial networks*, AAAI'22, Cui et al.\n\nDA and ADA are:\n> *Differentiable augmentation for data-efficient GAN training*, NeurIPS'20, Zhao et al.\n\\\n*Training generative adversarial networks with limited data*, NeurIPS'20, Karras et al.\n\n## 2. Which dataset was used in Fig. 2?\nThank you. This is a 10% data, CIFAR-10, OmniGAN ($d'=256$).\n\n## 3. Improve notations. What are ${\\bf x}_r$ and ${\\bf x}_f$?\nThank you. We have now unified notations as per request. ${\\bf x}_r$ and ${\\bf x}_f$ are the real and fake samples passed to the discriminator. Indeed, ${\\bf x}\\_{real}$ is a redundant symbol as it means ${\\bf x}_r$.\n\n## 4. Why the authors did not directly penalize the (2,1)-norm?\n\n* For **the multiplicative modualtion** with the noise, we have:\n\\\n$\\hat{L}\\_{mul\\\\_noise}:=\\hat{\\mathbb{E}}\\_i\\mathbb{E}\\_{{\\bf z}\\sim\\mathcal{N}({\\bf 1}, \\beta^2\\mathbf{I}) }||{\\bf y}_i-{\\bf W}_2({\\bf z}\\cdot{\\bf a}_i)||^2_2=\\hat{\\mathbb{E}}\\_i||{\\bf y}_i-{\\bf W}_2({\\bf z}\\cdot{\\bf a}_i)||^2_2+\\beta^2(\\hat{\\mathbb{E}}\\_i||{\\bf a}_i||^2_2)||{\\bf W}_2||^2\\_{2,1}$,\nwhere ${\\bf z}\\cdot{\\bf a}_i$ is element-wise multiplication of $\\bf z$ and ${\\bf a}_i$. This implies that while direct penalty of network weights, $||{\\bf W}_2||^2\\_{2,1}$, is possible, our approach has a **dynamic penalty** $\\beta^2(\\hat{\\mathbb{E}}\\_i||{\\bf a}_i||^2_2)$ where variance $\\beta^2$ is adapted based on the discriminator decisions but also $\\hat{\\mathbb{E}}\\_i||{\\bf a}_i||^2_2$ depends on norms of feature vectors. \n\\\n\\\n**The importance of such a multiplicative modulation is that semantic contents is not added by noise ${\\bf z}$ to the modulated feature vector ${\\bf z}\\cdot{\\bf a}_i$**, i.e., only `active' channels ${\\bf a}_i\\neq 0$ are modulated: they can be suppressed or magnified according to variance $\\beta^2$.\n\n* In contrast, the **additive noise modulation** results in the standard penalty $\\beta^2||{\\bf W}_2||^2\\_{2,1}$ which we suspect the reviewer asks about:\n\n  $\\hat{L}\\_{add\\\\_noise}:=\\hat{\\mathbb{E}}\\_i\\mathbb{E}\\_{{\\bf z}\\sim\\mathcal{N}({\\bf 0}, \\beta^2\\mathbf{I}) }||{\\bf y}_i-{\\bf W}_2({\\bf z}+{\\bf a}_i)||^2_2=\\hat{\\mathbb{E}}\\_i||{\\bf y}_i-{\\bf W}_2{\\bf a}_i||^2_2+\\beta^2||{\\bf W}_2||^2\\_{2,1}$,\n  \\\n  \\\n  While obvious, $\\beta^2||{\\bf W}_2||^2\\_{2,1}$ does not have the **dynamic penalty** term related to feature norms. This also means (based on the derivation) that **the additive noise may change feature semantics**, i.e., ${\\bf z}+{\\bf a}_i$ may ``activate'' channels which are non-active in ${\\bf a}_i$ (features ${\\bf a}_i= 0$).  \n  \\\n  Therefore, **our multiplicative modulator not only controls the Rademacher Complexity (RC), but controls it in a meaningful manner for discriminator**. Only feature semantics that are present in feature vector (that describe object) are modulated while helping control RC. Additive noise would likely introduce semantics that are not present for a given image/object.\n  \n* While directly regularizing the network using $\\beta^2||{\\bf W}_2||^2\\_{2,1}$ does not introduce additive noise, it's regularization effect shares similarity with additive noise incorporation. Thus, it's connection to the additive noise suggests that the direct weight regularization has an effect of changing the feature semantics, which will negatively impact the classification accuracy.\n\n* Below we provide **experimental comparisons**. Please note the consistency loss and dual branch from Fig. 1 (right) (main paper) are not used here as that would result in additional penalties on gradient norms:\n\n10\\% CIFAR-10/100 on OmniGAN ($d'=256$), AAN (adaptive additive noise), AWR (adaptive weight regularization), AN (our adaptive multiplicative noise), \n|Method|||10\\% CIFAR-10|||10\\% CIFAR-100||\n|-|-|-:|:-:|:-|-:|:-:|:-|\n| |Eq.| IS|tFID|vFID|IS|tFID|vFID|\n|OmniGAN| |8.49|22.24|26.33|8.19|45.41|50.33|\n|+AAN | ${\\bf a}_i\\:={\\bf a}_i\\\\!+\\\\!{\\bf z}; {\\bf z}\\\\!\\sim\\\\!\\mathcal{N}(0,\\beta^2{\\bf I})$ |8.52|20.12|24.65|9.64|37.68|42.01|\n|+AWR | $\\beta^2\\|\\|{\\bf W}\\|\\|^2\\_{2,1}$ |8.44|18.42|22.56|9.80|32.05|36.53|\n|+AN| ${\\bf a}_i\\:={\\bf a}_i\\\\!\\cdot\\\\!{\\bf z}; {\\bf z}\\\\!\\sim\\\\!\\\\mathcal{N}({\\bf 1},\\beta^2{\\bf I})$ |**9.16**|**10.14**|**13.80**|**11.22**|**23.76**|**28.34**| \n\nIn addition, **Figure 3 in the rebuttal pdf** shows that AN achives higher classification accuracy than AAN and AWR, demonstrating that multiplicative noise preserves the feature semantics better than the additive noise and the weight regularization.\n\n## 5. Fig. 2(b) shows the gradient norm at the layer before the classification head for fake images increases.\nThis is because when we train the generator, we optimize the generator by minimizing: \n$-h\\_\\text{AN}({\\bf a}\\_f)\\approx-h({\\bf a}\\_f)-\\frac{\\beta^2}{2}\\frac{\\partial^2 h}{\\partial f^2}|\\_{{\\bf a}_f}f''({\\bf a}_f){\\bf a}_f^2$\n\nThis means the generator is trained to generate images to the point where the gradient norms of discriminator can become large.  Please also see **Resp. 3 Rev. 3 (LrCf)** for discussing of NICE$\\_{D_f}$ and NICE$\\_{G_f}$ and the **Figure 2 in the rebuttal pdf** for the gradient analysis.\n\n## 6. Results of different ablation models on CIFAR-100 on OmniGAN ($d'=256$)\n\nKindly see below reprint of Figure 4(a) which is a table in the main paper.\n\n|Method||10\\% CIFAR-100||\n|-|-:|:-:|:-|\n||IS|tFID|vFID|\n|OmniGAN|8.19|45.41|50.33|\n|+AWD|9.64|37.68|42.01|\n|+AN+AGP|10.72|27.73|32.15|\n|+NICE$_{add}$|9.34|28.59|33.02|\n|+DA|10.16|24.50|28.96|\n|+ADA|11.23|23.11|27.58|\n|+AACR|11.37|21.42|25.76|\n|+NICE|**11.50**|**16.91**|**21.56**|\n\n Additionally, we provide additional gradient analysis in **Figure 1\\&2 in the rebuttal pdf**.\n"}, {"review_id": "ebii72VYhm", "reviewer": "Reviewer_v2y4", "summary": "This paper proposes a regularization method called noise-modulated consistency regularization (NICE) to train GANs with limited data. In this method, this paper proposes modulating the discriminator's latent features using noise and imposing a constraint on the discriminator so that the middle outputs of the discriminator (particularly, the outputs before the prediction head) for differently modulated data are the same. This paper also provides a theoretical analysis, in which it is shown that the proposed regularization penalizes the first- and second-order gradients of latent features and improves the GAN training stability. The effectiveness of the proposed method was demonstrated using small-scale typically-used datasets, including CIFAR-10, CIFAR-100, ImageNet, and FFHQ datasets, and in low-shot generation tasks.", "questions": "1. What happens when directly regularizing the first- and second-order gradients of latent features? (See weakness 1)\n2. Discuss the calculation cost. (See weakness 2)\n3. Why are some results excluded in Tables 1–3. Provide the scores if possible (See weakness 3)", "limitations": "1. Image synthesis technologies could be misused for synthesizing fake images. In particular, few-shot image synthesis technologies will make it easy to synthesize such images because they reduce the data collection cost. It would be better to discuss the social impact in this aspect.\n2. Although the versatility of the proposed method is demonstrated, I suspect that there may be some previous methods that are not compatible with the proposed method. Discussing this will be useful for future research.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The effectiveness of the proposed method was demonstrated in various scenarios, including evaluation on small-scale typically-used datasets (CIFAR-10, CIFAR-100, ImageNet, and FFHQ datasets) and evaluation in low-shot generation tasks. In many scenarios, the proposed method achieves state-of-the-art performance while comparing with various baselines. Furthermore, the applicability of the proposed method was also demonstrated by applying the proposed method to various GANs (e.g., BigGAN and OmniGAN in Tables 2 and 3, and StyleGAN2 in Table 3) and using the proposed method with orthogonal methods (e.g., LeCam, DA, and ADA). Ablation studies are also conducted.\n\n2. Not only the effectiveness of the proposed method is demonstrated, but also the theoretical analysis is provided. This analysis verifies the proposed method is useful for regularizing the first- and second-order gradients of latent features and improving the GAN training stability. This explanation is reasonable.\n\n3. This paper is well written and easy to read. Although some explanation is slightly too concise, the discussion on related work is thorough.", "weaknesses": "1. Through theoretical analysis, I understand that the proposed method is useful for regularizing the first- and second-order gradients of latent features. However, this analysis raises a question of what happens when regularizing the first- and second-order gradients of latent features directly. As discussed in related work, there are several previous studies that propose gradient regularizations. I guess that the proposed method is better than a direct regularization method in terms of calculation cost; however, I would appreciate it if I could hear the opinion from the authors.\n\n2. I cannot find the discussion on the increase in the calculation cost. I suspect the proposed method increases the calculation cost because the discriminator needs to process data twice, compared to a standard discriminator. For a fair comparison, it would be better to be discussed.\n\n3. Some results are excluded in Tables 1–3 (e.g., IS/tFID for DigGAN in Table 1). I cannot find a clear explanation for why the results are excluded. It seems that DigGAN is one of the comparable baselines; therefore, I would appreciate it if the authors could provide the missing scores.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Rev. 2 (v2y4)\n\n***We thank the reviewer** for the constructive review and valuable questions that have helped us improve our work.*\n## 1. What happens when regularizing the first- and second-order gradients of latent features directly?\n\nThank you. Consider the regularization term in Eq. 10 (main paper) and its Taylor expansion. Indeed, this expression penalizes the squared norms of first- and second-order gradients of $f({\\bf a}_i)$. However, notice that our expression has specific penalty weights $2\\beta^2||{\\bf a}_i||_2^2$ and $\\beta^4||{\\bf a}_i||_2^4$ for penalty $||f'({\\bf a}_i)||^2_2$ and  $||f''({\\bf a}_i)||^2_2$ respectively. In fact, these specific penalty weights with the penalty given by\n \n$2\\beta^2||{\\bf a}_i||_2^2||f'({\\bf a}_i)||^2_2 + \\beta^4||{\\bf a}_i||_2^4||f''({\\bf a}_i)||^2_2$\n\\\nemerge only in case of using multiplicative noise drawn from $\\mathcal{N}({\\bf 1},\\beta^2{\\bf I})$ for Eq. 9 and 10.\n\\\n\\\n**Importantly, see Resp. 4 to Rev. 1 (fQTZ)**. In that response we demonstrate that:\n* **the multiplicative noise ${\\bf z}\\sim\\mathcal{N}({\\bf 1},\\beta^2{\\bf I})$ does not introduce new semantics into feature vector ${\\bf a}_i$ by operation ${\\bf z}\\cdot{\\bf a}_i$**, i.e., only feature semantics that are present in feature vector (features of ${\\bf a}_i$ that are non-zero) that describe object are modulated.\n* yet, **this multiplicative noise** does help control the Rademacher Complexity (RC) due to Lemma 2, Eq. 7 (main paper).\n\nWe suspect that by ``regularizing the first- and second-order gradients of latent features directly'', Rev. 2 (v2y4) means imposing a generic penalty of form $\\beta_1||f'({\\bf a}_i)||^2_2 + \\beta_2||f''({\\bf a}_i)||^2_2$ or its variant  $2\\beta^2||f'({\\bf a}_i)||^2_2 + \\beta^4||f''({\\bf a}_i)||^2_2$ that  arises when the additive noise ${\\bf z}\\sim\\mathcal{N}({\\bf 0},\\beta^2{\\bf I})$ is applied to ${\\bf a}_i$, i.e., ${\\bf z}\\cdot{\\bf a}_i$.\n\nLet ${\\bf z}\\sim\\mathcal{N}({\\bf 0}, \\beta^2{\\bf I})$. we use Taylor expansion to expand the additive noise modulated consistency regularization as follows:\n$\\mathbb{E}_{{\\bf z}_1, {\\bf z}_2}||f({\\bf a}_i+{\\bf z}_1)-f({\\bf a}_i+{\\bf z}_2)||^2\\approx \\mathbb{E} _{{\\bf z}_1, {\\bf z}_2}||({\\bf z}_1-{\\bf z}_2)f'({\\bf a}_i)+\\frac{{\\bf z}_1^2-{\\bf z}_2^2}{2}f''({\\bf a}_i)||^2=2\\beta^2||f'({\\bf a}_i)||^2_2 + \\beta^4||f''({\\bf a}_i)||^2_2$\n\n**Such a variant means that semantics that are not present in ${\\bf a}_i$ may be ``activated'' by the noise, drastically altering the meaning of ${\\bf a}_i$ and damaging the information it carries about image/object.** (see  **Resp. 4 to Rev. 1 (fQTZ)**).\n\\\nWhile directly applying gradient penalization with $\\beta^2||f'({\\bf a}_i)||^2_2 + \\beta^4||f''({\\bf a}_i)||^2_2$ does not introducing noise, It's connection to the additive noise suggests it will have a negative effect on semantics.\n\nKindly **see the table in Resp. 4 to Rev. 1 (fQTZ) that evaluates multiplicative vs. additive noise modulators, and the direct penalties.**\n\nALGP (adaptive latent gradient penalization), NICE$_{add}$ (consistency regularization with additive noise),  AWR (adaptive weight regularization)\n10\\% CIFAR-10/100 on OmniGAN ($d'=256$), \n|Dataset||10\\% CIFAR-10|||10\\% CIFAR-100||\n|-|-:|:-:|:-|-:|:-:|:-|\n||IS|tFID|vFID|IS|tFID|vFID|\n|OmniGAN|8.49|22.24|26.33|8.19|45.41|50.33|\n|+ALGP|8.52|19.15|22.72|9.18|32.98|37.51|\n|+AWR+ALGP|8.72|16.82|20.45|10.14|26.44|30.23|\n|+NICE$_{add}$|8.64|17.94|21.59|9.34|28.59|33.02|\n|+NICE|**9.26**|**7.23**|**11.08**|**11.50**|**16.91**|**21.56**|\n\nKindly see **Figure 3 in the rebuttal pdf**, which plots the classification accuracy of the discriminator on testing images for different variants. AN achieves higher accuracy than AAN (adaptive additive noise) and AWR. NICE obtains best accuracy than other variants, showing that the multiplicative noise modulation preserves the semantics better than other variants. This answers why we not directly penalize the gradient norms or directly regularize the weight norms, as they drastically alter the semantics of the features.\n\n## 2. Is the proposed method better than a direct regularization in terms of calculation cost? What is the increase in the calculation cost?\n\nThe specific penalty weights are important, as elaborated above. The computations are also simpler as we do not have to directly tap into the backpropagation and gradients to penalize them. \n\nKindly notice that only last few blocks of discriminator (blocks $l,\\cdots,L$) use our noise modulator penalty (kindly see Fig. 5 in the supplementary material). This design is also simple and very easy to parallelize. Even without parallelization, **the extra cost compared with the baseline is negligible**, as shown in **Figure 5** in the rebuttal pdf  and in **Table 7** of supplementary material.\n\n\n## 3. Some results are excluded in Tables 1–3. DigGAN is one of the comparable baselines.\n\n* We directly use the code provided by the DigGAN and test the IS, tFID, and vFID, and obtain the results for BigGAN+DigGAN ($d'=256$) on CIFAR10 and CIFAR100 as follows:\n|Dataset||100\\%|||20\\%|||10\\%||\n|-|-|-|-|-|-|-|-|-|-|\n||IS|tFID|vFID|IS|tFID|vFID|IS|tFID|vFID|\n|CIFAR-10 BigGAN+DigGAN|9.28|5.33|9.35|8.81|13.28|17.25|8.32|18.54|22.45|\n|CIFAR-10 BigGAN+NICE|**9.50**|**4.19**|**8.24**|**8.96**|**8.51**|**12.54**|**8.73**|**13.65**|**17.75**|\n|CIFAR-100 BigGAN+DigGAN|**11.15**|8.13|13.06|9.98|16.87|21.59|**9.04**|23.10|27.78|\n|CIFAR-100 BigGAN+NICE|10.99|**6.13**|**11.08**|**10.32**|**13.17**|**17.80**|8.96|**19.53**|**24.33**|\n\n* As the authors did not release the code for StyleGAN2 on the five low-shot datasets, we reproduce their method and swap the hyperparameter ($\\lambda\\in\\\\{10, 20, 50, 100, 150\\\\}$ in their paper) for this tasks. We obtain the best results with $\\lambda=20$ for (DigGAN+ADA) as follows:\n|Dataset|Obama|Grumpy Cat|Panda|AnimalFace Cat|AnimalFace Dog|\n|-|-|-|-|-|-|\n|ADA+DigGAN|36.38|25.42|11.54|35.67|59.98|\n|ADA+NICE|**20.09**|**15.63**|**8.18**|**22.70**|**28.65**|\n\n"}, {"review_id": "YgeAgAgr7u", "reviewer": "Reviewer_LrCf", "summary": "This paper proposes a training approach called NoIse-modulated Consistency rEgularization (NICE) to improve the data-efficiency of generative adversarial networks (GANs) by addressing issues related to limited data. It introduces adaptive multiplicative noise into the discriminator to modulate its latent features, preventing discriminator overfitting. To mitigate the instability of GAN training caused by increased gradient norm, a constraint is imposed on the discriminator to ensure consistency for the same inputs under different noise modulations. The experimental results demonstrate the effectiveness of NICE in reducing discriminator overfitting and improving the stability of GAN training, achieving state-of-the-art results on various datasets and low-shot generation tasks.", "questions": "- Does Eq1 encompass all GAN objective functions, such as the original min-max and non-saturating GANs? It is suggested that the authors can provide detailed explanations, as the theoretical work of this paper is based on Eq1.\n\nSome typos:\n- Lack parentheses in Eq9.\n- f(x) -> f(\\alpha) in Line 185.\n- N(0,\\beta^2 I^d) -> N(1,\\beta^2 I^d) in Line 200.", "limitations": "No, the authors have not addressed the limitations and potential negative societal impact of their work.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper is well-organized and well-written, making it easy to understand its contribution.\n- The paper provides theoretical analysis to understand the connection between introducing multiplicative Gaussian noise to the discriminator and GAN generalization. The authors also uncover the negative impact of simple noise multiplication on gradient norm and propose a noise-modulated consistency regularization with theoretical grounds to improve it.\n- Experimental results presented in the paper validate the effectiveness of the proposed method, surpassing the baseline in most experiments.", "weaknesses": "- The complete objective function of the proposed method should be included in the main text, rather than in the appendix.\n- In the GAN field, it is suggested to conduct multiple experiments and report the results in terms of mean and standard deviation due to the random variation of the results caused by different trials.\n- According to the theoretical analysis in this paper, it is easy to understand the motivation behind the introduction of Eq12, but the motivation behind Eq13 and Eq14 is unclear. Although the ablation experiments in Table 6 empirically demonstrated the benefits of the proposed method, I still expect the authors to provide a more explicit explanation for these equations. Moreover, it is recommended to report the experimental results using only Eq12 and Eq14, as this would further demonstrate the effectiveness of the method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Rev. 3 (LrCf)\n\n***We thank the reviewer** for the constructive review and valuable questions that have helped us improve our work.*\n\n## 1. The complete objective function of the proposed method should be included in the main text, rather than in the appendix.\n\nThank you. Absolutely. We have now combined details from Sections B.2 and B.3 of the supplementary material. In general, we have improved the notation and Eq. 12-14 can be directly injected into the GAN objective by $\\Omega(\\cdot)$. To that end, we have compacted the objective as follows:\n\n$\n\\begin{cases}\nL_D^\\text{AN}=\\min\\limits\\_{{\\boldsymbol\\theta}_d}\\mathbb{E}\\_{\\bf x\\sim\\nu_n}\\[h\\_\\text{AN}({\\bf x}; {\\boldsymbol\\theta}_d)\\\\!+\\\\!\\gamma\\Omega({\\bf x})\\]+\\mathbb{E}\\_{\\bf x\\sim\\hat{\\mu}_m}\\[ -h\\_\\text{AN}({\\bf x}; {\\boldsymbol\\theta}_d)\\\\!+\\\\!\\gamma\\Omega({\\bf x})\\]\\\\\\\\\nL_G^\\text{AN}=\\min\\limits\\_{{\\boldsymbol\\theta}_g}\\mathbb{E}\\_{\\bf z\\sim p\\_z}\\[-h\\_\\text{AN}(g({\\bf z}; {\\boldsymbol\\theta}_g))\\\\!+\\\\!\\gamma\\Omega(g({\\bf z}; {\\boldsymbol\\theta}_g))\\]\n\\end{cases}\\\\;\\text{where}\\\\;\\\\;\\Omega({\\bf x})=||f\\_{1}({\\bf x})-f\\_{2}({\\bf x})||_2^2,\n$\n\nwhere $h\\_\\text{AN}({\\bf x}; {\\boldsymbol\\theta})$ denotes our modified discriminator inclusive of the noise injection,  ${\\boldsymbol\\theta}_d$ and  ${\\boldsymbol\\theta}_g$ are learnable parameters of discriminator and generator, and $f\\_1(\\cdot)$ and $f\\_2(\\cdot)$ are two feature extractors with two different multiplicative noise injections, as per Fig. 5 in the supplementary material. Moreover, $\\nu_n$, $\\hat{\\mu}_m$ and $p\\_z$ represent the finite fake distribution, the empirical training distribution and distribution of the generator input. $\\gamma\\geq 0$ is the regularization hyper-parameter.\n\n## 2. Conduct multiple experiments and report the mean and standard deviation.\n\nThank you. In fact, we have run 5 trials for NICE and NICE+ADA in Tables 1-5 and reported the mean. As the standard deviations were less than 1% relatively, we followed the practice of most GAN papers (listed below) which typically do not report the standard deviations:\n \n >* *FakeCLR: Exploring Contrastive Learning for Solving Latent Discontinuity in Data-Efficient GANs*, ECCV'22, Li et al.\n >* *DigGAN: Discriminator gradIent Gap Regularization for GAN Training with Limited Data*, NeurIPS'22, Fang et al.\n >* *Differentiable Augmentation for Data-Efficient GAN Training*, NeurIPS'20, Zhao et al. \n\nHowever, **Tables 1-4 in the rebuttal PDF contain full results for ADA+NICE with STD** supplemented as per your request.\n\n## 3. The motivation for Eq. 12 is clear but motivation of Eq. 13-14 is unclear.\n\nThank you. This is a very interesting question. \n\\\n\\\nEq. 12 provides regularization for discriminator given the real data. In the same spirit, the discriminator has to be regularized w.r.t. the fake data, as in Eq. 13, to stablizing GAN training. Regularizing both real and fake data was explored in several papers:\n>* *Which Training Methods for GANs do actually Converge?*, ICML'18, Mescheder et al.\n>* *Improving Generalization and Stability of Generative Adversarial Networks*, ICLR'19, Thanh-Tung et al.\n>* *Stabilizing Training of Generative Adversarial Networks through Regularization*, NeurIPS'17, Roth et al.\n>* *DigGAN: Discriminator gradIent Gap Regularization for GAN Training with Limited Data*, NeurIPS'22, Fang et al.\n\nThese works have found that regularizing gradients on both real and fake samples improves convergence and stability of  GAN training. Thus, we also apply NICE on both real and fake samples. In terms of why we use NICE$\\_{G_f}$, it can be explained as reducing first- and second-order gradients of discriminator while optimizing generator parameters ${\\boldsymbol\\theta}\\_g$. Below we compare the Taylor expansions of generator objective without NICE$\\_{G_f}$ *vs.* with NICE$\\_{G_f}$ (in both cases NICE$\\_{D_r}$ and NICE$\\_{D_f}$ are switched on). Let ${\\bf a}\\_f$ be some fake sample from the generator. We optimize parameters ${\\boldsymbol\\theta}\\_g$ of generator by minimizing: \n\n* without NICE$\\_{G_f}$: \n$-h\\_\\text{AN}({\\bf a}\\_f)\\approx-h({\\bf a}\\_f)-\\frac{\\beta^2}{2}\\frac{\\partial^2 h}{\\partial f^2}|\\_{{\\bf a}_f}f''({\\bf a}_f){\\bf a}_f^2$\n* with NICE$\\_{G_f}$:\n$-h\\_\\text{AN}({\\bf a}\\_f)+\\gamma\\Omega({\\bf a}\\_f)\\approx-h({\\bf a}\\_f)-\\frac{\\beta^2}{2}\\frac{\\partial^2 h}{\\partial f^2}|\\_{{\\bf a}_f}f''({\\bf a}_f){\\bf a}_f^2+\\underbrace\\{2\\gamma\\beta^2{\\bf a}_f^2f'^2({\\bf a}_f)+\\gamma\\beta^4{\\bf a}_f^4f''^2({\\bf a}_f)}\\_{\\text{grad. norms}}$,\n\nwhere $f'^2(\\cdot)$ and $f''^2(\\cdot)$ are simply squared norms of first- and second-order gradients (see underbrace in the eq.) of discriminator.\n\\\n\\\nThis means that without NICE$\\_{G_f}$, the generator is trained to generate images to the point where the gradient norms of discriminator can become large. In contrast, with NICE$\\_{G_f}$, for some well-chosen $\\gamma>0$, these first- and second-order gradient norms of discriminator are reduced by optimizing ${\\boldsymbol\\theta}\\_g$ of generator, as they show up in the Taylor expansion.\n\\\n\\\nIn that sense, Eq. 14 is also meaningful because while we are optimizing here the generator, the output of generator is still passed via discriminator. Thus, it makes sense to seek parameters of generator that not only improve the generator to ``fool'' discriminator but also stablizing the GAN training.\n\\\n\\\nTo conclude, stabilizing GAN training can be achieved with Eq. 12-14. **See figure 2 in rebuttal PDF** for the gradient analysis when we ablate the Eq 12-14.\n\n## 4. Report the experimental results using only Eq. 12 and Eq. 14.\nKindly see response in the General Rebuttal (for all reviewers) due to the 6000 characters limited space.\n\n## 5. Does Eq. 1 encompass all GAN objective functions, i.e., the original min-max and non-saturating GANs?\nKindly see response in the General Rebuttal (for all reviewers) due to the limited space.\n"}, {"review_id": "uNmC3OdX1r", "reviewer": "Reviewer_m7vf", "summary": "This paper proposed NICE, a technique that enforces the discriminator to be consistent with the same inputs under different noise modulations. The authors showed us both in theory and practice that NICE is effective at preventing discriminator overfitting and achieves superior performance in image generation under limited data settings.", "questions": "Please refer to “Weaknesses\".\n\nIt seems that authors missed a strong baseline method SSGAN-LA [1] in Table 3 which could achieved a better performance. It is better to compare with missing baselines.\n\n[1] Self-Supervised GANs with Label Augmentation, Neurips 2021.", "limitations": "Please refer to “Weaknesses\" for limitations. The authors adequately addressed the potential negative societal impact of their work.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The method is based on theoretical proof, and the experimental results show the effectiveness.\n\n2. The method achieves competitive performance against many existing methods. The evaluation involves multiple GAN models, and baseline methods.", "weaknesses": "1. The authors show the computational overhead of NICE in appendix, which is not small and it could increase the training time by a large amount. There is no quantitative measure for scalability of NICE. For example, what will happen when the image resolution is 1080p or even higher, or when the dataset is quite large. What is the training cost?\n\n2. NICE introduces some new hyperparameters but it is not clear how they are tuned. What's their sensitivity to different GAN models and data distribution (or datasets)? What's their sensitivity to baselines in Table 3?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Rev. 4 (m7vf)\n\n***We thank the reviewer** for the constructive review and valuable questions that have helped us improve our work.*\n\n## 1. The computational overhead of NICE.\n\nKindly notice **our computational overhead is small**. Firstly, our usage of multiplicative noise modulation as in **Fig. 5 (supplementary material)** and Fig. 1 (main paper), **is restricted to the last 3 or 4 blocks**, i.e.,  $L-l+1$, of the discriminator with $1,\\cdots,L$ blocks. Moreover, **these last few blocks are responsible for handling low resolution feature maps and can be readily parallelized**. Even without parralelization, increase is by 0.35sec per 1000 images on OmniGAN. On StyleGAN2, which is a slower network, the extra time required by NICE was only 18\\% on top of StyleGAN2. Meanwhile, **Fig. 5 in the rebuttal pdf illustrates the FID gain *vs.* speed**, which shows that **a slightly increase in training time yields substantial improvements**, particularly when compared to recent state-of-the-art approaches like FakeCLR and InsGen. We believe the increased computational overhead is outweighed by the considerable benefits and the improvements justify the extra fraction of time.\n\n## 2. What will happen when the image resolution is 1080p or even higher?\n\n**As our module is used only  for the last 3 or 4 blocks of the discriminator**, i.e.,  $L-l+1$, this means they are not exposed to high resolution feature maps. (see Fig. 5 (supplementary material)). **Table below** shows no large speed increase in such a case. \n\nBelow we apply NICE module to the StyleGAN2 for the last 4 blocks, and provide the sec/$k$img given resolutions as follows:\n|Resolution|$512\\times512$|$1024\\times1024$|$2048\\times2048$|\n|-|:-:|:-:|:-:|\n|StyleGAN2|18.65|51.76|143.25|\n|StyleGAN2+NICE|20.74|56.03|147.76|\n|relative increase|11.17\\%|8.25\\%|**3.05\\%**|\n\nAs our NICE is implemented on last few blocks of the discriminator, which are low resolution feature maps, **the relative computation cost increase is actually smaller as the remaining standard network layers dominate compute time**.\n\n\n## 4. What hyper-parameters do you use and what is their sensitivity to different GAN models?\nWe keep the learning hyperparameters of the original GAN backbones untouched. NICE introduces four new hyperparameters: the place to apply NICE within a block $c\\in\\\\{C_1, C_2, C_R\\\\}$, which blocks to use $l\\in\\\\{1,...L\\\\}$, the threshold $\\eta$ and the regularization strength $\\Delta_\\gamma$. $c$ can be simply choosed from $C_1C_2C_R$ and $C_1C_2$ (we use $C_1C_2C_R$ for OmniGAN and BigGAN, and $C_1C_2$ for StyleGAN2). $l$ is often applied for the last 3 or 4 blocks. $\\eta$ is fixed as 0.5 for BigGAN and OmniGAN. On StyleGAN2, we recommand lower $\\eta$ for diverse datasets. we set $\\eta=0.6$ for FFHQ dataset $\\eta=0.9$ for the 5 low-shot dataset. Generally ($c=\\\\{C_1, C_2\\\\}$, $l=\\\\{L-3, ..., L\\\\}$, $\\eta=0.6$, $\\Delta_\\gamma=0.05$) is a good starting point when applied to new dataset.\n\nBelow, we varied the hyper-parameters on the 5 low-shot dataset (Obama, Grumpy Cat, Panda, AnimalFace Cat, Animal Face Dog). **A slight change the hyper-parameters does not make a big difference regarding the results**, showing that our NICE is not too sensitive to hyper-parameters:\n\n|$l\\in\\\\{1...L\\\\}$|Obama|Grumpy Cat|Panda|AnimalFace Cat|AnimalFace Dog|\n|-|:-:|:-:|:-:|:-:|:-:|\n|4,5,6|25.66|20.01|9.25|26.15|48.18|\n|3,4,5,6|24.56|18.78|8.92|25.25|46.56|\n|2,3,4,5,6|26.35|19.34|9.14|25.61|47.68|\n\n|$c\\in\\\\{C_1, C_2, C_R\\\\}$|Obama|Grumpy Cat|Panda|AnimalFace Cat|AnimalFace Dog|\n|-|:-:|:-:|:-:|:-:|:-:|\n|$C_1$|25.77|19.42|8.98|24.97|47.17|\n|$C_1,C_2$|24.56|18.78|8.92|25.25|46.56|\n|$C_1,C_2,C_R$|26.51|19.51|9.56|26.18|46.93|\n\n|$\\Delta_\\gamma$|Obama|Grumpy Cat|Panda|AnimalFace Cat|AnimalFace Dog|\n|-|:-:|:-:|:-:|:-:|:-:|\n|0.01|26.48|19.1|9.09|25.33|47.15|\n|0.05|24.56|18.78|8.92|25.25|46.56|\n|0.1|25.46|19.52|9.01|25.42|47.06|\n|0.2|25.74|19.85|9.13|25.30|47.87|\n\n|$\\eta$|Obama|Grumpy Cat|Panda|AnimalFace Cat|AnimalFace Dog|\n|-|:-:|:-:|:-:|:-:|:-:|\n|0.95|25.20|20.06|9.07|25.20|46.44|\n|0.90|24.56|18.78|8.92|25.25|46.56|\n|0.80|25.71|18.90|9.15|26.05|47.43|\n\n## 5. Authors missed a strong baseline method SSGAN-LA [1] in Table 3. \n\nSSGAN-LA [1] did not perform experiments on the five low-shot datasets, 100-shot Obama/Grumpy cat/Panda\nand Animal Face Cat/Dog datasets. They only released code for BigGAN.\n\\\n\\\nThus, we have followed the authors' code implementation for BigGAN and re-implemented it for StyleGAN2 on the five low-shot datasets. For SSGAN-LA, we tried the multi-hinge loss and the cross_entropy loss provided in the code of SSGAN-LA. We obtained the best the results with the multi-hinge loss for StyleGAN2+SSGAN-LA but our StyleGAN2+NICE is still a stronger performer:\n|Dataset|Obama|Grumpy Cat|Panda|AnimalFace Cat|AnimalFace Dog|\n|-|-|-|-|-|-|\n|StyleGAN2+SSGAN-LA|79.88|38.42|28.6|78.78|109.91|\n|StyleGAN2+NICE (ours)|**24.56**|**18.78**|**8.92**|**25.25**|**46.56**|\n"}, {"review_id": "7UAmEIVLdK", "reviewer": "Reviewer_ijuK", "summary": "The authors propose a noise modulation and regularization scheme for GANs that reduces disciminator overfitting and improves training stability in the low-data scheme. The technique demonstrate consistent improvements when applied to several different network architectures and datasets.", "questions": "I do not understand where the second derivatives come from in equation 9.\nEquation 10 seems to perform a Taylor expansion, is this accurate? If so, the fact should be mentioned for clarity.\n\nIs equation 1 a contribution of this paper or something presented in previous work?\n\nWhat does MA in tables 1-4 mean?\n\nIn figure 2, the y-axis labels are incorrectly formatted (1e^2 instead of 1*10^2 or 1e2)\n\nThe authors claim (line 171) that proposition 1 can be extended to more complicated architectures - is an example of this shown somewhere, such as previous work?\n\nThe datasets used in Figures 2 and 3 don't seem to be mentioned\n\nNICE seems to work best when combined with geometric augmentation techniques, especially ADA. A comment on this would be useful - ADA already minimizes the gap between the real and fake distributions, so what does NICE add in this case?", "limitations": "The limitations are not properly discussed in the main text. Is there a wall clock cost to the proposed method? Does it require changes to training hyper parameters (learning rate etc.)?", "rating": 8, "confidence": 2, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "The paper contains very extensive comparisons to competing previous methods.\nThe theoretical motivation for the proposed technique is very extensive (but also very hard for non-experts to understand).\nThe theoretical analysis sheds some light on the effectiveness of consistency regularization techniques that have shown promise previously.\nThe numerical results are excellent.\nThe paper is very math-heavy, but the authors do a good job of including some intuition of the different propositions and lemmas (e.g. lines 142-147, 156-160, 173-175, 191-196).\nThe inclusion of separate tFIDs and vFIDs is refreshing to see.", "weaknesses": "Not a single generated image is shown in the main paper. Given that some concerns about the reliability of FID have been raised as of late, visual results (e.g. the 100-shot results from the supplemental, Fig. 11) would be appreciated.\n\nThe paper mentions many similar-sounding terms (generation gap, generalization gap, output gap, generalization error, discriminator discrepancy) that might confuse non-experts. A brief description of the effect and its implications on training should be included for non-expert readers.\n\nMany images in the supplemental are of unusably low quality. This is especially true for CIFAR results, which need to be integer-upscaled to preserve sharpness.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Rev. 5 (ijuK)\n\n***We thank the reviewer** for the constructive review and valuable questions that have helped us improve our work.*\n\n## 1. Not a single generated image shown in the main paper.\n\nWe apologize. We have selected now **images on 100-shot and AnimalFace\ndatasets from Figure 11 (supplementary material)** and added them into the main draft.\n\n\n## 2. The paper mentions many similar-sounding terms. \nThank you for bringing our attention to multiple terms requiring disambiguation, which is invaluable in improving the clarity of our work and helping readers. The table below summarizes each the meaning of each term (we have added it now to our revised work):\n|Term|Meaning|\n|-|-|\n|generalization gap|the difference in a model's performance between the training data and unseen testing data|\n|output gap|the difference between the average discriminator output for real data and that for fake data|\n|discriminator discrepancy|the difference in the average output of the discriminator between two distributions|\n|generalization error|the error of the discriminator's predictions on unseen testing data|\n\nAdditionally, we have duly rectified the term from \"generation gap\" to \"generalization gap\". \n\n## 3. Many images in the supplemental are of unusably low quality.\n\nWe apologize. We have now upscaled them accordingly given unlimited supplementary material size.\n\n## 4. Where the second derivatives come from in Eq. 9?\n\nThank you. They come from expanding $h(\\cdot)$ around ${\\bf a}\\_f$ with the Taylor expansion. This sort of analysis and first- and second-order expansion are a typical starting point in several works:\n>* *What Regularized Auto-Encoders Learn from the Data-Generating Distribution?*, Alain & Bengio, JMLR'14\n>* *Sharpness-Aware Minimization for Efficiently Improving Generalization*, Foret et al, ICLR'21\n>* *How Does Mixup Help With Robustness and Generalization?* Zhang et al, ICLR'21\n\nAs higher-order terms usually decay according $\\mathcal{O}(\\frac{1}{o!})$ where $o$ indicates the order, they are negligible.\n\n## 5. Is Eq. 1 a contribution of this paper?\n\nEq. 1 is a standard formulation for analyzing the generalization of GANs, as in:\n> *On the Discrimination-Generalization Tradeoff in GANs*, ICLR'18, Zhang et al.\n\n**We have updated it now with the equation given in Resp. 1 to Rev. 3 (LrCf)**, which is our contribution. We have also included extension of Fig. 5 and explanations from Sections B.2 and B.3 of the supplementary material to complement Fig. 1 in the main paper.\n\n\n## 6. What does MA in Tables 1-4 mean?\n\nWe apologize. MA denotes `massive augmentations' (including DA and ADA) first used in:\n> *Generative co-training for generative adversarial networks*, AAAI'22, Cui et al.\n\nDA and ADA are:\n> *Differentiable augmentation for data-efficient GAN training*, NeurIPS'20, Zhao et al.\n\\\n*Training generative adversarial networks with limited data*, NeurIPS'20, Karras et al.\n\n## 7. Claim that Prop. 1 can be extended to more complicated architectures.\n\nWe believe this is a misunderstanding (we revised now the language). Our aim was to emphasize that the beneficial effect of implicit weight regularization achieved through noise modulation, despite being analyzed within a simplified two-layer system, remains pertinent for networks encompassing multiple layers, including convolutional neural networks (indeed, we use NICE across several layers). This is attributed to the fact that networks with multiple layers can be conceptually treated as an aggregation of two layers, and a convolutional layer is a specialized case of a linear layer.\n\n\nThese propositions find validation in the following references:\n>* *On Dropout and Nuclear Norm Regularization*, Mianjy et al, ICML'19\n>* *The Implicit and Explicit Regularization Effects of Dropout*, Wei et al, ICML'20\n>* *Dropout: Explicit Forms and Capacity Control*, Arora, ICML'21\n\n\n## 8. Datasets used in Figures 2 and 3 do not seem to be mentioned.\n\nWe apologize. In Fig. 2 and 3, we use a 10% data, CIFAR-10, OmniGAN ($d'=256$).\n\n## 9. ADA already minimizes the gap between the real and fake distributions, so what does NICE add in this case?\nThank you for the interesting question. Below we analyze ADA's underlying issues. While both ADA and NICE effectively prevent overfitting, they employ differing strategies. NICE reduces the Rademacher complexity of the discriminator while ADA expands the data space through diverse augmentations. However, **NICE possesses an additional advantage which ADA lacks. It implicitly penalizes gradients** (e.g., minimizing Eq. 10 implies minimizing gradient norms $f'^2(\\cdot)$ and $f''^2(\\cdot)$),  thereby improving the stability of GAN training. \n\\\n\\\nIn contrast, ADA lacks such properties. With augmentation techniques that involve random noise additions or multiplications, an analysis based on Eq. 9 suggests that ADA might in fact lead to an increase in gradient norms. Empirical observations support this supposition: we encountered increase in gradient norms for ADA (**kindly see Figure 4 in the rebuttal PDF**). However, when combining NICE with ADA, the gradient issues caused by ADA were alleviated, giving the best of two worlds (**ADA and NICE are complementary**).\n\n## 10. Limitations, training cost and whether it requires changes to training hyper-parameters (learning rate et.?)\nWe have now moved the limitations addressed in Section I of the supplementary materials to the main paper. While it is true that implementing NICE comes with small added training costs, our design (see Fig. 5, supplementary materials) enables efficient parallelization of the process. Moreover, even without parallelization, the incremental cost compared to the baseline is minimal, as illustrated in Figure 5 in the rebuttal PDF and Table 7 in the supplementary materials. NICE is built on GAN backbone architectures, and we keep all the learning hyper-parameters of original backbones untouched."}]

Author rebuttal:
# General Rebuttal 

We thank the reviewers' for constructive suggestions and in depth analysis helping us refine our work. We are humbled by such a positive response, and we truly appreciate it.
\
\
**Also, kindly refer to the rebuttal PDF** (**at the bottom of this panel**) for additional figures and tables. Individual rebuttals refer to them in more detail.

# Below we clarify remaining issues for Rev. 3 (LrCf)

## 1. Report the experimental results using only Eq. 12 and Eq. 14 (Rev. 3 (LrCf)).

Below are the results as per request. As expected, stablizing GAN training is the best when all steps of GAN strive to regularize the gradient (Eq.12-14):

|Regularization Eq.||10\% CIFAR-10|||10\% CIFAR-100||Obama|
|-|-:|:-:|:-|-:|:-:|:-|:-:|
||IS|tFID|vFID|IS|tFID|vFID|FID|
|Eq. 12 + 14|9.16|8.69|12.59|11.19|18.80|24.13|29.95|
|Eq. 12 + 13 + 14|**9.26**|**7.23**|**11.08**|**11.50**|**16.91**|**21.56**|**24.56**|

**Figure 2 in the rebuttal PDF** also shows gradient norms of discriminator.

## 2. Does Eq. 1 encompass all GAN objective functions, i.e., the original min-max and non-saturating GANs?  (Rev. 3 (LrCf)).

Certainly, Eq. 1 forms the cornerstone for analyzing the generalization of diverse GAN objective functions. Eq. 1 was introduced by Zhang et al. in "On the Discrimination-Generalization Tradeoff in GANs", ICLR'18.

Their study adeptly showcased that Eq. 1 encompasses a wide range of GANs, including the well-known $f$-GANs. These encompass GANs that minimize the $f$-divergence, which notably encompasses the original GAN formulation.

Additionally, insights from  "Non-saturating GAN training as divergence minimization" (arXiv:2010.08029, Shannon et al.) further bolster this applicability. This work discusses how non-saturating GANs can be approximated via minimizing a specific $f$-divergence, aligning with Eq. 1.

These collective contributions highlight that Eq. 1 provides a unifying lens at GANs, encompassing a wide range of models, including the classic min-max and the non-saturating variants.

Meta-review: 1x SA, 1x A, 2x WA, and 1x BA. This paper proposes a noise modulation and regularization scheme for GANs that reduces discriminator overfitting and improves training stability for low data. The reviewers agree on accepting the paper due to its (1) clear presentation, (2) SOTA performance, and (3) thorough theoretical analysis. The rebuttal has addressed their concerns.
Decision: Accept (poster)