Title: NICE: NoIse-modulated Consistency rEgularization for Data-Efficient GANs
Venue: NeurIPS 2023
Accepted: yes
Keywords: Image Generation

limited dataset

Generative Adversarial Networks
Abstract: Generative Adversarial Networks (GANs) are powerful tools for image synthesis. However, they require access to vast amounts of training data, which is often costly and prohibitive. Limited data affects GANs, leading to discriminator overfitting and training instability. In this paper, we present a novel approach called NoIse-modulated Consistency rEgularization (NICE) to overcome these challenges. To this end, we introduce an adaptive multiplicative noise into the discriminator to modulate its latent features. We demonstrate the effectiveness of such a modulation in preventing discriminator overfitting by adaptively reducing the Rademacher complexity of the discriminator. However, this modulation leads to an unintended consequence of increased gradient norm, which can undermine the stability of GAN training. To mitigate this undesirable effect, we impose a constraint on the discriminator, ensuring its consistency for the same inputs under different noise modulations. The constraint effectively penalizes the first and second-order gradients of latent features, enhancing GAN stability. Experimental evidence aligns with our theoretical analysis, demonstrating the reduction of generalization error and gradient penalization of NICE. This substantiates the efficacy of NICE in reducing discriminator overfitting and improving stability of GAN training. NICE achieves state-of-the-art results on CIFAR-10, CIFAR-100, ImageNet and FFHQ datasets when trained with limited data, as well as in low-shot generation tasks.

NICE: NoIse-modulated Consistency rEgularization for Data-Efficient GANs

Abstract

Generative Adversarial Networks (GANs) are powerful tools for image synthesis. However, they require access to vast amounts of training data, which is often costly and prohibitive. Limited data affects GANs, leading to discriminator overfitting and training instability. In this paper, we present a novel approach called NoIsemodulated Consistency rEgularization (NICE) to overcome these challenges. To this end, we introduce an adaptive multiplicative noise into the discriminator to modulate its latent features. We demonstrate the effectiveness of such a modulation in preventing discriminator overfitting by adaptively reducing the Rademacher complexity of the discriminator. However, this modulation leads to an unintended consequence of increased gradient norm, which can undermine the stability of GAN training. To mitigate this undesirable effect, we impose a constraint on the discriminator, ensuring its consistency for the same inputs under different noise modulations. The constraint effectively penalizes the first and second-order gradients of latent features, enhancing GAN stability. Experimental evidence aligns with our theoretical analysis, demonstrating the reduction of generalization error and gradient penalization of NICE. This substantiates the efficacy of NICE in reducing discriminator overfitting and improving stability of GAN training. NICE achieves state-of-the-art results on CIFAR-10, CIFAR-100, ImageNet and FFHQ datasets when trained with limited data, as well as in low-shot generation tasks.

1 Introduction

The remarkable advancements and breakthroughs in deep learning [31] can be largely attributed to the extensive utilization of vast amounts of training data. This abundance of data has driven the progress across various domains of deep learning. Among notable recent advancements are Generative Adversarial Networks (GANs) [13], popular in industry and academia. GANs have proven their high-quality image generation abilities and achieved high generation speeds [47], establishing them as a versatile tool for a wide range of applications, such as text-to-image generation [20, 47, 55], destylization [48, 49, 50], image-to-image translation [27, 44], and 3D generation [52, 53, 65].

Despite the impressive capabilities of state-of-the-art GANs in generating diverse and high-quality images [7, 25], their effectiveness heavily relies on large volumes of training data. The acquisition of such large datasets helps GANs attain the desired adversarial equilibrium. However, under limited training data regime, GANs encounter challenges associated with discriminator overfitting and unstable training [22, 68, 57, 11, 17].

To address the aforementioned challenges, recent studies have approached the problem from three perspectives. The first perspective involves the utilization of extensive differentiable data augmentation techniques, aimed at expanding the distribution of the available data [22, 68, 17, 32, 63, 64, 69]. The second perspective leverages knowledge gained from large-scale models trained on large datasets [9, 70, 29]. While these approaches are effective, they also carry inherent risks, such as the leakage of augmentation clues [64, 22, 40] or pre-trained knowledge [30].

The third perspective focuses on the regularization of the discriminator, aiming to weaken its learning ability [11, 19] or increase the overlap between real and fake support sets [57]. In line with this perspective, our paper introduces a novel approach to improve GAN generalization. We propose to modulate the hidden features of discriminator via adaptive multiplicative noise, which strikes a balance in maintaining a certain level of discrimination ability while also regularizing the Rademacher complexity [6] of the discriminator. This reduction in Rademacher complexity, which quantifies the capacity of model to fit random variables, narrows the generation gap between the training and unseen data, resulting in enhanced GAN generalization [4, 56, 18].

Nevertheless, training the discriminator with adaptive noise unintentionally amplifies second-order gradient derivative of latent features corresponding to real images. This elevated gradient leads to abrupt gradient changes near the real sample points, potentially causing instability in the feedback to the generator. This issue aligns with the findings in works [26, 36, 56, 11], which emphasize the importance of penalizing gradients for both real and fake samples to promote convergence and stability of GAN training. To address these challenges, we propose a constraint on the discriminator that ensures consistency for the same inputs under different noise modulations. While our idea is simple, our theoretical analysis reveals that this constraint effectively penalizes the first and secondorder gradients of the latent features. Consequently, the gradient provided to the generator becomes more stable, resulting in improved training stability and generalization.

Our comprehensive experiments confirm the effectiveness of NICE in penalizing gradients and reducing the generalization gap. Despite the simple design, NICE significantly improves the training stability and generalization of GANs, outperforming alternative approaches in preventing discriminator overfitting. NICE achieves superior results on challenging limited data benchmarks, including CIFAR-10/100, ImageNet, FFHQ, and low-shot image generation tasks.

Our contributions can be summarized as follows:

We limit discriminator overfitting by using the adaptive multiplicative noise to modulate the latent features of the discriminator, resulting in enhanced generalization of GANs.

We introduce NICE, a technique that enforces the discriminator to be consistent with the same inputs under different noise modulations, implicitly penalizing the first and second-order gradients of the latent features during GAN training, promoting the training stability.

We show that NICE, both in theory and practice, effectively prevents discriminator overfitting and achieves superior performance in image generation under limited data setting.

2 Related Work

Improving GANs. Generative Adversarial Networks [13] are powerful generative models that excel in image generation [21, 24, 25], text-to-image generation [20, 47, 55], image-to-image translation [27, 44], and 3D generation [52, 65, 53]. However, GANs commonly encounter challenges such as training instability [26], mode collapse [45], and discriminator overfitting [22]. Researchers have investigated different GAN loss functions [2, 42, 71], architectures [21, 24, 25, 23], and regularization strategies [38, 33, 14]. The f -GAN [42] generalizes GANs to f -divergences. WGAN [2] adopts the Earth-Mover distance. OmniGAN [71] extends conditional GANs to a multi-label softmax loss. StyleGANV1-3 [24, 25, 23] enhances the architecture of generator. Approaches [14, 26, 36, 56, 11] propose explicit penalty of gradient of discriminator. SNGAN [38] enforces a Lipschitz constraint on the discriminator, and approach [33] regularizes the spectral norm of the discriminator. In this study, we propose a novel regularization strategy applicable to diverse GAN loss functions and architectures, specifically tailored for training GAN models with limited data.

Image generation with limited data. Collecting data is both laborious and expensive. Consequently, generating images in limited data settings presents significant challenges, primarily due to discriminator overfitting. Previous approaches have tackled this issue through data augmentation techniques [22, 68, 17, 32, 63] and leveraging knowledge transfer from large-scale pre-trained models [9, 29, 70]. Differentiable data augmentation methods [22, 68, 17] 'extend' the data distribution, while methods [63, 32] explore contrastive learning within GANs. Approaches [9, 29, 70] employ pre-trained models to guide training of discriminator. However, both GAN types suffer issues, including leakage of augmentation clues [64, 22] or pre-trained knowledge [30]. An alternative approach involves regularizing the discriminator. LeCamGAN [57] suggests reducing the output gap of discriminator between real and fake distributions. LCSAGAN [40] employs manifold techniques to project discriminator features onto manifold and decrease the capacity of discriminator. APA [19]

uses fake images as real images to increase overlap between the real and fake distributions. DigGAN [11] minimizes the gradient norm gap between real and fake images. Inspired by such an approach, we propose to use an adaptive noise to modulate the latent features of discriminator to reduce the generalization error of discriminator as a means of regularization.

Consistency regularization. CTGAN [61] introduces the enforcement of a Lipschitz constraint by ensuring the consistency of the response of discriminator to real images. CAGAN [41] and GenCo [8] enforce consistency among multiple discriminators. R-Drop [62] applies consistency regularization to transformers [58] equipped with dropout for natural language processing tasks. Augmentation-based consistency regularization GANs [64, 69] enforce consistency by considering different augmented views of the same image, although they may inadvertently leak augmentation clues to the generator [22]. Despite the effectiveness of consistency regularization demonstrated by these previous works, their success is primarily empirical, lacking theoretical analysis. In our study, we go beyond prior works by providing a theoretical analysis of consistency regularization in GANs. By delving deeper into the underlying principles, we develop a more comprehensive understanding of the mechanism behind its effectiveness.

Generalization of GANs. Arora et al . [4, 56, 67, 18] have contributed to the understanding and improvement of generalization of GANs by showing the importance of achieving adversarial equilibrium, reducing discriminator discrepancy, penalizing gradients, and bounding the generalization error. Our motivation aligns with these works in the pursuit of enhancing generalization capabilities. However, our approach implicitly and adaptively reduces the generalization gap while penalizing gradients of latent features of discriminator. Such a setting provides an efficient and effective means of preventing discriminator overfitting and improving generalization.

3 Method

To boost the generalization of GAN, we start by analyzing their generalization error, bounding it with the Rademacher complexity of discriminator, and linking it with the weight norm. By incorporating multiplicative noise, we demonstrate its regularization benefits on the weight norm. However, such a strategy induces large gradient norm destabilizing training of GAN. To this end, we introduce NICE to penalize the gradients of discriminator. Finally, we provide NICE and showcase it use in GANs.

3.1 Generalization error of GANs and complexity of neural network

The primary goal of GAN is to minimize the integral probability metric [39], assuming access to infinite real and fake data during optimization, i.e ., the infinite real and generated distributions ( µ, ν ) as discussed in [67]. In practice, we often have limited access to a finite dataset ˆ µ n of size n . Consequently, our optimization is restricted to the empirical loss:

Function sets of discriminator and generator, H and G , are typically parameterized in GAN as neural network classes H nn = { h ( x ; θ d ) : θ d ∈ Θ d } and G nn = { g ( z ; θ g ) : θ g ∈ Θ g } where z ∼ p z serves as the random noise input to the generator. The associated term d H nn is referred to as the neural network distance [4]. The discriminator network D := φ ◦ f consists of a real/fake prediction head φ and a feature extractor f . As the loss function ϕ ( · ) varies across tasks, architectures or choice of divergence type, we compose it with D [2, 67, 4] to simplify the analysis and notation, i.e ., h ( · ) := ϕ ( D ( · )) . Thus, the alternative optimization of discriminator and generator becomes:

where we assume ν n minimizes d H (ˆ µ n , ν ) up to precision ϵ ≥ 0 , meaning that d H (ˆ µ n , ν n ) ≤ inf ν ∈G d H (ˆ µ n , ν ) + ϵ . As we are interested in how close the generator distribution ν n is to the unknown infinite distribution µ , we refer to the lemma [67] on the generalization error of GAN:

Lemma 1 (Theorem 3.1 of [67]) Assume that the discriminator set H is even ( h ∈ H implies -h ∈ H ) and all discriminators are bounded by ∥ h ∥ ∞ ≤ ∆ . Let ˆ µ n be an empirical measure of an i.i.d. sample of size n drawn from µ . Assume d H (ˆ µ n , ν n ) -inf ν ∈G d H (ˆ µ n , ν ) ≤ ϵ . Then with probability at least 1 δ , we have:

where the Rademacher complexity [6], R ( µ ) n ( H ) := E [ sup h ∈H 2 n ∑ i τ ( i ) h ( x ( i ) ) ] , measures how well the function h ∈ H fits the Rademacher random variable τ ( i ) with prob ( τ ( i ) = 1) = prob ( τ ( i ) = -1) = 1 2 given samples x ( i ) ∼ ˆ µ n .

Lemma 1 provides a crucial insight that one can assess the generalization error of GAN by comparing the output discrepancy of discriminator between training data and unseen data, and such an error is influenced by the Rademacher complexity of the discriminator. To enhance the performance of generator while reducing the generalization error, we have two possibilities: 1) increase the quantity n of real data, which provides a stronger foundation for training a better generator; 2) reduce the Rademacher complexity of the discriminator. However, this reduction must be carefully controlled, as an overly simplified discriminator may struggle to effectively distinguish real and fake data.

To manage the Rademacher complexity of the discriminator, we leverage a theorem from approach [5] to establish an upper bound on the Rademacher complexity of the neural network.

Lemma 2 (Eq. 1.2 in [5], Theorem 5.20 in [35]) Consider a fully-connected neural network v θ ( x ) = W t σ ( W t -1 σ ( ...σ ( W 1 x ) ... )) where W i are linear weights at i -th layer and σ is a 1-Lipschitz activation function. Suppose that ∀ i ∈ { 1 , ..., n } , ∥ x ( i ) ∥ 2 ≤ q . Let ∥ W i ∥ lip be the lipschitz constant of W i and ∥ W T i ∥ 2 , 1 be the sum of the l 2 norm of columns in W i . Let V = { v θ : ∥ W i ∥ lip ≤ k i , ∥ W T i ∥ 2 , 1 ≤ b i } , we have the Rademacher complexity:

Lemma 2 links the Rademacher complexity of a neural network to the Lipschitz constant and the (2,1)-norm of its weights, providing essential insights for controlling the complexity of discriminator.

3.2 Improving generalization by feature modulation with multiplicative noise

Taking into account Lemmas 1 and 2, one can effectively manage the generalization error of GAN by controlling the Rademacher complexity of the discriminator. Such a control requires managing the Lipschitz constant and the norm of the weights of discriminator. Typically, controlling the Lipschitz constant can be achieved by the spectral normalization [38] or the gradient penalty [14]. In contrast, we propose a novel approach: regularizing the norm of the weights by modulating the latent features of the discriminator with multiplicative noise.

We build on the analysis (Prop. 4 in Appendix of [3]) of Dropout regularization [54] by exploring the regularization effect of the Gaussian multiplicative noise within the context of deep regression.

Theorem 1 (Regularization by the Gaussian multiplicative noise in deep regression) Consider a regression task on a two-layer neural network. Let X ⊆ R d 0 and Y⊆ [ -1 , 1] d 2 be the input and output spaces, and D = X×Y . Let n examples { ( x ( i ) , y ( i ) ) } n i =1 ∼D n . Let f w : X →Y be parameterized by { w : W 1 ∈ R d 1 × d 0 , W 2 ∈ R d 2 × d 1 } and f w ( x , z ) = W 2 ( z ⊙ σ ( W 1 x )) where σ is 1-Lipschitz activation function, ⊙ denotes element-wise multiplication and z ∼N ( 1 , β 2 I d 1 ) . Let a = σ ( W 1 x ) , ˆ a = ˆ E i [ a ( i ) ⊙ a ( i ) ] , ˆ a k ≥ 0 denotes the k -th element of ˆ a . Let ∥ W 2 ∥ 2 , 1 = ∑ k ∥ w k ∥ 2 where w k is the k -th column vector in W 2 . The regression task with the l 2 loss leads to the weight regularization as:

The proof can be found in §C.1. Theorem 1 reveals that modulation with Gaussian multiplicative noise in the regression task implicitly regularizes the internal weight norm, with the regularization strength determined by the variance of the noise β 2 and the magnitude of the features.

Despite analysis in a simplified two-layer system, the implicit weight regularization by the noise modulation applies to multi-layer and convolutional neural networks, which can be expressed as a combination of two-layer nets, and convolutional layers are a type of linear layer [37, 60, 3]. Assuming latent features in deep neural networks follow a Gaussian distribution [34], we can equate the Bernoulli noise modulation (Dropout) with Z ∼ B ( β ) , whose values are set to be 1 / (1 -β ) with probability 1 -β and 0 otherwise, to the Gaussian multiplicative noise modulation in subsequent layers, making the regularization effects from Theorem 1 applicable to the Bernoulli noise as well.

Theorem 1 illustrates that one can modulate the latent features in the discriminator by the multiplicative noise to adaptively regularize the norm of its weights, leading to the reduced Rademacher complexity in Lemma 2 and improved GANs generalization in Lemma 1.

3.3 Consistency regularization

Although the noise modulation can reduce the Rademacher complexity, incorporating the noise increases the gradient norms of latent features and the inputs to the discriminator, making training of GAN unstable [56] and difficult to converge [36]. To understand the gradient-related challenges in the noise-modulated discriminator, we adopt the Taylor expansion with a focus on the pivotal firstand second-order terms, following standard practice [1, 12, 66].

Proposition 1 Define f := f t ◦ . . . ◦ f 2 as the feature extractor of the discriminator from the second layer onward. Let ˜ a = f 1 (˜ x ) and a = f 1 ( x ) be the latent features of the first layer for fake and real images, respectively. The discriminator is defined as h ( · ) = ϕ ( φ ( f ( · ))) , with φ as the prediction head and ϕ as the loss function. Introduce a multiplicative noise z ∼ N ( 1 , β 2 I d ′ ) modulating solely the first layer. Let H ( h ) kk ( a ) be the k -th diagonal entry of the Hessian matrix of h at a and a k be the k -th element of a . Applying Taylor expansion to h ( · ) , the GAN loss can be expressed as follows:

Prop. 1 (the proof is in §C.2) shows that introducing the noise modulation during the training of the discriminator amplifies the value of diagonal entry of Hessian matrix at real images, causing abrupt changes in their gradient and potentially an increase of the gradient norm of real images. Eq. 7 further shows that as the generator is optimized to produce images resembling real images, the gradient norm of the fake images also experiences amplification. Consequently, gradient norms for both real and fake images increase. Empirically, we observe an increase in the gradient norm of latent features before the prediction head, resulting in a larger gradient norm of samples.

To mitigate this problem, we propose a consistency regularization for f , promoting invariance of f to the same input under varying noise modulations. For simplicity, we take f : R d ′ → R , noting that this analysis readily extends to the consistency loss for f with a multidimensional output.

Theorem 2 Using notations from Prop. 1 and with f : R d ′ → R , let ∇ 2 k f ( a ) and ( H ( f ) jk ( a ) ) 2 denote the squares of the k -th gradient element and ( j, k ) -th entry of Hessian matrix of f at a , respectively. Given z 1 , z 2 ∼N ( 1 , β 2 I d ′ ) , enforcing invariance of f under different noise modulations yields:

Theorem 2 (the proof is in §C.3) shows that the consistency loss essentially penalizes the firstand second-order gradient of f at a , with the strength adaptively controlled by the noise variance β 2 and modulated by a . This results in reweighted gradient squared norms modulation, with terms in expressions ∥∇ f ( a ) ∥ 2 2 = ∑ k ∇ 2 k f ( a ) , and ∥ H ( f ) ( a ) ∥ 2 F = ∑ j,k ( H ( f ) jk ( a )) 2 influenced by a 2 k ≥ 0 . In contrast to the noise-modulated discriminator in Eq. 6, NICE regularizes the firstand second-order gradients of layers ahead of the classification head. Despite the gradient from the classification head is large, NICE ensures the gradient reaching the input stays small, resulting in a stable training and improved generalization and convergence [4, 26, 67].

3.4 Implementation of NICE

The analysis above highlights practical ways to enhance the generalization and stability of GANs. Firstly, we apply the adaptive multiplicative noise to the latent features of the discriminator, striking balance between discrimination and generalization, as highlighted in [67]. The consistency regularization is then applied to ensure invariance of the discriminator to the same inputs under varied noise modulations, penalizing the gradient and stabilizing the training process.

GAN with the adaptive noise modulation (AN). Figure 1 shows the discriminator composed of blocks B 1 , B 2 , . . . , B L , where each block B l contains multiple convolution weights c ∈{ C 1 , C 2 , C S } . To achieve a trade-off between generalization and discrimination, we modulate the latent features X ∈ R 1 × d ′ × d H × d W from convolutional layers of the discriminator using adaptive noise Z ∈ R 1 × d ′ by:

AN

AN 𝑿

= 𝑿 ⊚ 𝒁

where 𝒁~𝒩(1,𝛽

!

𝑰

"

!

)

𝐶

#

𝐶

$

𝐶

!

AN

AN

…

𝐵

!

𝐵

%

𝐷/𝐺

loss

sign(𝑟 𝒙 >𝜂)

𝛽

𝐵

#

Discriminator with adaptive noise (AN)

NoIse-modulated Consistency rEgularization (NICE)

Overfit ?

head

𝑿

𝒍,𝟏

…

𝐵

&

𝐵

%

…

𝐵

&

𝐵

%

𝐷/𝐺

loss

𝑓

#

(⋅)

||𝑓

#

⋅ -𝑓

!

⋅

||

!

!

𝑓

!

(⋅)

shared weights

non-shared noises

head

…

𝐵

#

𝐵

&'#

𝒙

no noise modulation

# 𝒙

𝒙

where ⊚ represents the operation that expands Z into 1 × d ′ × d H × d W shape and performs elementwise multiplication with X . The noise Z is carefully controlled through an adaptive β . In the case of BigGAN and OmniGAN, we control the variance of Z using β as Z ∼ N ( 1 , β 2 I d ′ ) . For StyleGAN2, we control the noise through Z ∼ B ( β ) , where with probability 1 -β it takes the value 1 1 -β , and with probability β it takes 0.

Below we introduce a mechanism to control the noise via the meta-parameter β by detecting potential overfitting in the discriminator. Firstly, we compute the expectation over the discriminator output r ( x ) = E [ sign ( D ( x ))] w.r.t. real samples x , and evaluate ε = sign ( r ( x ) >η ) ∈{-1 , 0 , 1 } where η is a fixed threshold. A value greater than the threshold indicates potential overfitting [22]. We apply β t +1 = β t +∆ β · ε to update β with smaller ∆ β to control Z . Denote the modified discriminator with our adaptive noise as h AN, the objective of GAN with the adaptive noise modulation becomes:

GAN with the noise-modulated consistency regularization (NICE). To deal with the increased gradient norm due to the adaptive noise modulation, we introduce the consistency regularization (NICE) to promote invariance of the discriminator to the same samples under different noise modulations. Our regularization implicitly penalizes the gradient of the latent features in the discriminator by comparing outputs f 1 ( · ) and f 2 ( · ) of two feature extractors (with shared parameters) that are subjected to different noise modulations as in Figure 1. The objective for a GAN with NICE is:

where ℓ NICE ( · )= ∥ f 1 ( · ) -f 2 ( · ) ∥ 2 2 and γ =∆ γ β is a meta-parameter that controls the strength of the consistency regularization and is adaptively determined by β .

Efficient NICE. In our experiments, we observed that applying the noise modulation in the later blocks of the discriminator often outperformed applying noise in the earlier blocks. To avoid unnecessary computations (the early blocks do not use modulation), we divide the discriminator into a modulation-free part ( B 1 , . . . , B l -1 ) and a noise-modulated part ( B l , . . . , B L ), where l indicates the starting block of noise modulation. For efficient calculation of the consistency loss, we input two copies of X l -1 (output from the first part) into the second part, bypassing the need for redundant calculations to maintain a low computational overhead compared to standard approaches.

4 Experiments

We conduct experiments on CIFAR-10/100 [28] using BigGAN [7] and OmniGAN [71], as well as on ImageNet [10] using BigGAN for conditional image generation. We also evaluate our method on low-shot datasets [68], which include 100-shot Obama/Panda/Grumpy Cat and AnimalFace Dog/Cat [51], and FFHQ [24] using StyleGAN2 [25]. We compare our method against several strong baselines, including DA [68], ADA [22], DigGAN [11], MaskedGAN [17], KDDLGAN [9], LeCam [57], GenCo [8], InsGen [63], FakeCLR [32] and TransferGAN [59]. For fair comparison, we denote methods using massive augmentation as 'MA', which include DA and ADA.

Datasets. CIFAR-10 has 50K/10K training/testing images with resolution of 32 × 32 from 10 categories, whereas CIFAR-100 has 100 classes. FFHQ contains 70K human face images at 256 × 256

pixels. Low-shot datasets contain 100-shot Obama/Panda/Grumpy Cat images, AnimalFace (160 cats and 389 dogs) images at 256 × 256 resolution. ImageNet has 1.2M/50K training/validation images with 1K categories. Following [17, 9], we center-crop and downscale its images to 64 × 64 resolution. The implementation details can be found in §D. The generated images can be found in §J.

Evaluation metrics. We generate 50K images per dataset to compute the commonly used Inception Score [46] and Fréchet Inception Distance (FID) [15]. We report tFID, computed between 50K generated images and all training images. For CIFAR-10/100, we also compute vFID between 10K generated images and 10K real testing images. For low-shot datasets, we follow [68] and compute FID between 5K generated images and the entire dataset. For FFHQ, we calculate FID between 50K fake images and the entire training set. For ImageNet, we follow [17, 9] and generate 10K images for computing the IS and FID, where the reference distribution is the entire training set. Following [68, 11, 32], we run 5 trails for methods using NICE, and report the mean of the results. Given that all standard deviations fall below the 1% relative, we omit them for clarity.

4.1 Results on CIFAR-10 and CIFAR-100 for BigGAN and OmniGAN

Tables 1 and 2 demonstrate that NICE consistently outperforms baselines such as BigGAN, LeCam+DA, OmniGAN and OmniGAN+ADA on CIFAR-10 and CIFAR-100, firmly establishing its superiority. NICE also outperforms LeCam+DA+KDDLGAN in the majority of scenarios without any knowledge integration from large-scale models, underscoring its efficiency and effectiveness.

4.2 Results on low-shot generation on StyleGAN2

Table 3 showcases the superior performance of NICE in comparison to leading methods, significantly improving baselines and achieving state-of-the-art results. NICE also surpasses KDDLGAN, despite KDDLGAN leverages the large-scale CLIP model [43]. Figure 2 shows realistic images generated by NICE under scarce data training.

心

4.3 Results on FFHQ for StyleGAN2 and ImageNet for BigGAN

Results for unconditional image generation on FFHQ are in Table 4, and for conditional image generation on ImageNet 64 × 64 in Table 5. We limit FFHQ to 100, 1K, 2K, 5K real images and follow [17, 9] for ImageNet setting. NICE showcases superior performance on FFHQ and ImageNet.

4.4 Analysis of NICE

Analysis of the stabilizing effect of NICE. In our 10% CIFAR-10 experiments with OmniGAN( d ′ = 256 ), Figure 3 provides compelling evidence supporting our theory. Figure 3a shows that OmniGAN+AN and OmniGAN+NICE achieve lower weight norms than OmniGAN, validating Theorem

0

50

100

150 195

iterations (

×

1000)

3.7e2

4.0e2

4.3e2

4.6e2

||

W

||

2

2

,

1

0

50

100

150 195

iterations (

×

1000)

0e-4

1e-4

2e-4

3e-4

||

∂LD

∂f

||

2

2

0

50

100

150 195

iterations (

×

1000)

0e2

1e2

2e2

3e2

||

∂f

∂x

||

2

2

0

50

100

150 195

iterations (

×

1000)

0e-3

3e-3

6e-3

9e-3

||

∂LD

∂x

||

2

2

Weight Norm

Real images

Fake images

OmniGAN

OmniGAN+AN

OmniGAN+NICE

(a) Weight Norm

(b)

||

∂L

D

∂f

||

2

2

(c)

||

∂f

∂

x

||

2

2

(d)

||

∂L

D

∂

x

||

2

2

0

50

100

150

195

iterations (

×

1000)

-

18

-

9

0

9

18

D

(

x

)

0

50

100

150

195

iterations (

×

1000)

-

6

-

3

0

3

6

D

(

x

)

0

50

100

150

195

iterations (

×

1000)

-

6

-

3

0

3

6

D

(

x

)

0

100

200

300

390

iterations (

×

1000)

5

15

25

35

45

tFID

OmniGAN

OmniGAN+AN

OmniGAN+NICE

Real images

Fake images

Test images

(a) OmniGAN

(b) OmniGAN+AN

(c) OmniGAN+NICE

(d) tFID curves

1 that the multiplicative noise modulation reduces the weight norm of discriminator, thus lowering the Rademacher complexity and improving generalization.

Figure 3b shows a gradient surge at the latent layer before the classification head, especially for real images in OmniGAN+AN and OmniGAN+NICE, surpassing gradients of OmniGAN. This observation aligns with our prediction in Prop. 1 that introducing noise amplifies gradients due to the maximization effect on real images when training discriminator and on fake images when training generator, causing undesired gradient issues.

Figure 3c illustrates a smaller squared gradient norm ∥ ∂f/∂ x ∥ 2 2 for OmniGAN+NICE compared to OmniGAN+AN and OmniGAN, showcasing the effectiveness of consistency regularization in penalizing the gradient of discriminator, in line with the theoretical derivation in Theorem 2.

Figure 3d demonstrates that OmniGAN+NICE achieves a lower gradient norm of the discriminator loss w.r.t. the input than both OmniGAN and OmniGAN+AN, affirming ability of NICE to counteract the negative effects of large gradient norms caused by noise modulation. The resulted smaller gradients at the input validates efficiency of NICE in stabilizing training.

Analysis of the ability of NICE to enhance generalization. Figure 4 visualizes the discriminator outputs for real, fake, and test images along with tFID curves. In contrast to OmniGAN, OmniGAN+AN and OmniGAN+NICE balance discrimination and generalization [67], maintaining a steady discrepancy between real and fake images. This equilibrium ensures a smaller discrepancy between real seen images and unseen samples, effectively lowering the generation error as outlined in Eq. 3 of Lemma 1. As a result, the generalization of GANs is enhanced, leading to superior performance for image generation, as illustrated by the tFID curves in Figure 4d.

4.5 Ablation studies

Ablation of components in NICE. We examine the impact of various components of NICE: (i) introducing adaptive noise (AN) to the discriminator, (ii) applying consistency regularization to

real images (NICE D r ), (iii) applying consistency regularization to fake images (NICE D f ) during discriminator training, and (iv) enforcing consistency on fake images while training the generator (NICE G f ). Table 6 shows our evaluations across 10% CIFAR-10/100 using OmniGAN ( d ′ =256 ) and the Obama on StyleGAN2. The optimal performance is achieved when NICE is applied to both real and fake images in the training of both the discriminator and generator.

Impact of different factors in NICE. For an in-depth analysis on how different factors in NICE contribute to its performance, readers are directed to §E.

Comparison with alternatives for enhanced generalization and stability. Table 7 shows that NICE outperforms DA, ADA, AWD (adaptive weight decay), AN+AGP (adaptive noise with adaptive gradient penalization) and NICE add (with additive noise). Detailed implementation of these variants can be found in §D.2. Figure 5 shows that NICE also enhances performance with increasing network size, while ADA and DA exhibit a decline. As augmentation-based methods such as DA, ADA, and AACR, may leak augmentation cues to the generator (Figure 9 of §I), NICE mitigates this drawback.

256

512

768

1024

d

′

=

6

8

10

13

20

38

tFID

10% CIFAR-10

256

512

768

1024

14

17

22

30

40

50

tFID

10% CIFAR-100

OmniGAN

DA

ADA

NICE

Table 7 & Fig. 5: Various methods on OmniGAN. Tab. 7: AWD is Adaptive weight decay. AN+AGP: Adaptive noise + adaptive gradient penalty. NICE add : NICE with additive noise. AACR: Adaptive augmentation-based consistency reg. Fig. 5: tFID ↓ w.r.t. different d ′ on 10% CIFAR-10/100.

Rationalizing Advantages of NICE. Refer to §F and §G for a comprehensive explanations of why NICE outperforms other alternatives, enhancing generalization and stability. Given a mild increase in the computational load (§H), substantial performance gains are achieved.

5 Conclusions

Our newly proposed approach for GAN training, NoIse-modulated Consistency rEgularization (NICE), improves generalization of various GAN models and their performance under limited data training. Through rigorous theoretical analysis, we have shown that NICE reduces the Rademacher complexity and penalizes the gradient of the latent features within the discriminator. Our experimental results match theoretical findings, illustrating that NICE not only improves generalization but also enhances stability of GAN training. By harnessing these two key advantages, NICE works with various network backbones and consistently outperforms existing methods. These exceptional results firmly establish NICE as a powerful solution for preventing the pervasive issue of discriminator overfitting in GANs. Limitations and border impact of our work are discussed in §A.

References

Guillaume Alain and Yoshua Bengio. What regularized auto-encoders learn from the data-generating distribution. JMLR , 15(1):3563-3593, 2014. 5

Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. In ICML , pages 214-223, 2017. 2, 3

Raman Arora, Peter Bartlett, Poorya Mianjy, and Nathan Srebro. Dropout: Explicit forms and capacity control. In ICML , pages 351-361. PMLR, 2021. 4

Sanjeev Arora, Rong Ge, Yingyu Liang, Tengyu Ma, and Yi Zhang. Generalization and equilibrium in generative adversarial nets (gans). In ICML , pages 224-232. PMLR, 2017. 2, 3, 5

Peter L Bartlett, Dylan J Foster, and Matus J Telgarsky. Spectrally-normalized margin bounds for neural networks. NeurIPS , 30, 2017. 4

Peter L Bartlett and Shahar Mendelson. Rademacher and gaussian complexities: Risk bounds and structural results. JMLR , 3(Nov):463-482, 2002. 2, 4

Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale GAN training for high fidelity natural image synthesis. In ICLR , 2019. 1, 6

Kaiwen Cui, Jiaxing Huang, Zhipeng Luo, Gongjie Zhang, Fangneng Zhan, and Shijian Lu. Genco: generative co-training for generative adversarial networks with limited data. In AAAI , volume 36, pages 499-507, 2022. 3, 6, 8

Kaiwen Cui, Yingchen Yu, Fangneng Zhan, Shengcai Liao, Shijian Lu, and Eric Xing. Kd-dlgan: Data limited image generation via knowledge distillation. In CVPR , 2023. 1, 2, 6, 7, 8

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR , pages 248-255, 2009. 6

Tiantian Fang, Ruoyu Sun, and Alex Schwing. DigGAN: Discriminator gradient gap regularization for GAN training with limited data. In NeurIPS , 2022. 1, 2, 3, 6, 7, 8

Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-aware minimization for efficiently improving generalization. In ICLR , 2021. 5

Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In NeurIPS , 2014. 1, 2

Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron Courville. Improved training of wasserstein gans. In NeurIPS , pages 5769-5779, 2017. 2, 4

Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. In NeurIPS , volume 30, pages 6626-6637, 2017. 7

Liang Hou, Huawei Shen, Qi Cao, and Xueqi Cheng. Self-supervised gans with label augmentation. NeurIPS , 34:13019-13031, 2021. 8

Jiaxing Huang, Kaiwen Cui, Dayan Guan, Aoran Xiao, Fangneng Zhan, Shijian Lu, Shengcai Liao, and Eric Xing. Masked generative adversarial networks are data-efficient generation learners. NeurIPS , 35:2154-2167, 2022. 1, 2, 6, 7, 8

Kaiyi Ji, Yi Zhou, and Yingbin Liang. Understanding estimation and generalization error of generative adversarial networks. IEEE Transactions on Information Theory , 67(5):3114-3129, 2021. 2, 3

Liming Jiang, Bo Dai, Wayne Wu, and Chen Change Loy. Deceive d: adaptive pseudo augmentation for gan training with limited data. NeurIPS , 34:21655-21667, 2021. 2

Minguk Kang, Jun-Yan Zhu, Richard Zhang, Jaesik Park, Eli Shechtman, Sylvain Paris, and Taesung Park. Scaling up gans for text-to-image synthesis. CVPR , 2023. 1, 2

Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of GANs for improved quality, stability, and variation. In ICLR , 2018. 2

Tero Karras, Miika Aittala, Janne Hellsten, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Training generative adversarial networks with limited data. NeurIPS , 33, 2020. 1, 2, 3, 6, 8, 20, 25, 28, 29

Tero Karras, Miika Aittala, Samuli Laine, Erik Härkönen, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Alias-free generative adversarial networks. NeurIPS , 34:852-863, 2021. 2

Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In CVPR , 2019. 2, 6

Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of stylegan. In CVPR , pages 8110-8119, 2020. 1, 2, 6, 8, 29

Naveen Kodali, Jacob Abernethy, James Hays, and Zsolt Kira. On convergence and stability of gans. arXiv:1705.07215 , 2017. 2, 5

Lingke Kong, Chenyu Lian, Detian Huang, Yanle Hu, Qichao Zhou, et al. Breaking the dilemma of medical image-to-image translation. NeurIPS , 34:1964-1978, 2021. 1, 2

Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009. 6

Nupur Kumari, Richard Zhang, Eli Shechtman, and Jun-Yan Zhu. Ensembling off-the-shelf models for gan training. In CVPR , pages 10651-10662, 2022. 1, 2

Tuomas Kynkäänniemi, Tero Karras, Miika Aittala, Timo Aila, and Jaakko Lehtinen. The role of imagenet classes in fréchet inception distance. In ICLR , 2023. 1, 2

Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. nature , 521(7553):436-444, 2015. 1

Ziqiang Li, Chaoyue Wang, Heliang Zheng, Jing Zhang, and Bin Li. Fakeclr: Exploring contrastive learning for solving latent discontinuity in data-efficient gans. In ECCV , pages 598-615, 2022. 1, 2, 6, 7, 8, 23

Kanglin Liu, Wenming Tang, Fei Zhou, and Guoping Qiu. Spectral regularization for combating mode collapse in gans. In ICCV , pages 6382-6390, 2019. 2

Wenjie Luo, Yujia Li, Raquel Urtasun, and Richard Zemel. Understanding the effective receptive field in deep convolutional neural networks. NeurIPS , 29, 2016. 4

Tengyu Ma. Lecture Notes for Machine Learning Theory (CS229M/STATS214) . 4

Lars Mescheder, Andreas Geiger, and Sebastian Nowozin. Which training methods for gans do actually converge? In ICML , pages 3481-3490. PMLR, 2018. 2, 5

Poorya Mianjy and Raman Arora. On dropout and nuclear norm regularization. In ICML , pages 4575-4584. PMLR, 2019. 4

Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. In ICLR , 2018. 2, 4

Alfred Müller. Integral probability metrics and their generating classes of functions. Advances in applied probability , 29(2):429-443, 1997. 3

Yao Ni, Piotr Koniusz, Richard Hartley, and Richard Nock. Manifold learning benefits gans. In CVPR , pages 11265-11274, 2022. 1, 2

Yao Ni, Dandan Song, Xi Zhang, Hao Wu, and Lejian Liao. Cagan: Consistent adversarial training enhanced gans. In IJCAI , pages 2588-2594, 2018. 3

Sebastian Nowozin, Botond Cseke, and Ryota Tomioka. f-gan: Training generative neural samplers using variational divergence minimization. NeurIPS , 29, 2016. 2

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML , pages 8748-8763, 2021. 8

Elad Richardson, Yuval Alaluf, Or Patashnik, Yotam Nitzan, Yaniv Azar, Stav Shapiro, and Daniel CohenOr. Encoding in style: a stylegan encoder for image-to-image translation. In CVPR , pages 2287-2296, 2021. 1, 2

Kevin Roth, Aurelien Lucchi, Sebastian Nowozin, and Thomas Hofmann. Stabilizing training of generative adversarial networks through regularization. NeurIPS , 30, 2017. 2

Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, Xi Chen, and Xi Chen. Improved techniques for training gans. In NeurIPS , pages 2234-2242. 2016. 7

Axel Sauer, Tero Karras, Samuli Laine, Andreas Geiger, and Timo Aila. StyleGAN-T: Unlocking the power of GANs for fast large-scale text-to-image synthesis. In ICML , pages 30105-30118, 2023. 1, 2

Fatemeh Shiri, Xin Yu, Piotr Koniusz, and Fatih Porikli. Face destylization. In DICTA , pages 1-8. IEEE, 2017. 1

Fatemeh Shiri, Xin Yu, Fatih Porikli, Richard Hartley, and Piotr Koniusz. Identity-preserving face recovery from portraits. In WACV , 2018. 1

Fatemeh Shiri, Xin Yu, Fatih Porikli, Richard Hartley, and Piotr Koniusz. Identity-preserving face recovery from stylized portraits. IJCV , 127(6-7):863-883, 2019. 1

Zhangzhang Si and Song-Chun Zhu. Learning hybrid image templates (hit) by information projection. TPAMI , 34(7):1354-1367, 2012. 6

Ivan Skorokhodov, Aliaksandr Siarohin, Yinghao Xu, Jian Ren, Hsin-Ying Lee, Peter Wonka, and Sergey Tulyakov. 3D generation on ImageNet. In ICLR , 2023. 1, 2

Ivan Skorokhodov, Sergey Tulyakov, Yiqun Wang, and Peter Wonka. EpiGRAF: Rethinking training of 3d GANs. In NeurIPS , 2022. 1, 2

Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. JMLR , 15(1):1929-1958, 2014. 4

Ming Tao, Bing-Kun Bao, Hao Tang, and Changsheng Xu. Galip: Generative adversarial clips for text-to-image synthesis. CVPR , 2023. 1, 2

Hoang Thanh-Tung, Truyen Tran, and Svetha Venkatesh. Improving generalization and stability of generative adversarial networks. In ICLR , 2019. 2, 3, 5

Hung-Yu Tseng, Lu Jiang, Ce Liu, Ming-Hsuan Yang, and Weilong Yang. Regularing generative adversarial networks under limited data. In CVPR , 2021. 1, 2, 6, 8

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. NeurIPS , 30, 2017. 3

Yaxing Wang, Chenshen Wu, Luis Herranz, Joost Van de Weijer, Abel Gonzalez-Garcia, and Bogdan Raducanu. Transferring gans: generating images from limited data. In ECCV , pages 218-234, 2018. 6, 8

Colin Wei, Sham Kakade, and Tengyu Ma. The implicit and explicit regularization effects of dropout. In ICML , pages 10181-10192. PMLR, 2020. 4

Xiang Wei, Boqing Gong, Zixia Liu, Wei Lu, and Liqiang Wang. Improving the improved training of wasserstein gans: A consistency term and its dual effect. arXiv preprint arXiv:1803.01541 , 2018. 3

Lijun Wu, Juntao Li, Yue Wang, Qi Meng, Tao Qin, Wei Chen, Min Zhang, Tie-Yan Liu, et al. R-drop: Regularized dropout for neural networks. NeurIPS , 34:10890-10905, 2021. 3

Ceyuan Yang, Yujun Shen, Yinghao Xu, and Bolei Zhou. Data-efficient instance generation from instance discrimination. NeurIPS , 34:9378-9390, 2021. 1, 2, 6, 8, 23

Han Zhang, Zizhao Zhang, Augustus Odena, and Honglak Lee. Consistency regularization for generative adversarial networks. In ICLR , 2020. 1, 2, 3

Jianfeng Zhang, Zihang Jiang, Dingdong Yang, Hongyi Xu, Yichun Shi, Guoxian Song, Zhongcong Xu, Xinchao Wang, and Jiashi Feng. Avatargen: a 3d generative model for animatable human avatars. In ECCV , pages 668-685. Springer, 2023. 1, 2

Linjun Zhang, Zhun Deng, Kenji Kawaguchi, Amirata Ghorbani, and James Zou. How does mixup help with robustness and generalization? In ICLR , 2021. 5

Pengchuan Zhang, Qiang Liu, Dengyong Zhou, Tao Xu, and Xiaodong He. On the discriminationgeneralization tradeoff in GANs. In ICLR , 2018. 3, 5, 9

Shengyu Zhao, Zhijian Liu, Ji Lin, Jun-Yan Zhu, and Song Han. Differentiable augmentation for dataefficient gan training. NeurIPS , 33, 2020. 1, 2, 6, 7, 8, 20, 25

Zhengli Zhao, Sameer Singh, Honglak Lee, Zizhao Zhang, Augustus Odena, and Han Zhang. Improved consistency regularization for gans. In AAAI , volume 35, pages 11033-11041, 2021. 1, 3

Yong Zhong, Hong Tao Liu, Xiaodong Liu, Fan Bao, Weiran Shen, and Chongxuan Li. Deep generative modeling on limited data with regularization by nontransferable pre-trained models. In ICLR , 2023. 1, 2

Peng Zhou, Lingxi Xie, Bingbing Ni, Cong Geng, and Qi Tian. Omni-gan: On the secrets of cgans and beyond. In ICCV , pages 14061-14071, 2021. 2, 6, 25

NICE: NoIse-modulated Consistency rEgularization for Data-Efficient GANs (Supplementary Material)

Yao Ni † , Piotr Koniusz ∗ , § , †

†

The Australian National University § Data61 CSIRO † firstname.lastname@anu.edu.au

A Broader impact and limitations

NICE is a solution to limiting the extensive image usage in GAN training. Conventional GANs heavily rely on massive training data, which can jeopardize data privacy during large-scale data collection. NICE is an innovative method for data-limited image generation. By successfully training GANswith just 100 images, NICE not only minimizes the dependency on large-scale training datasets, but also significantly reduces the training cost. NICE decreases the risk of privacy violations (using smaller datasets equals to the lesser risk of data mismanagement). NICE also provides a cost-effective alternative for training GANs. As the data required for training is lesser compared with large-scale models, NICE is also energy-consumption friendly. Of course, with small-scale dataset comes a limitation on diversity. The model cannot generate instances for which the data is completely lacking. Thus, special care needs to be taken regarding gender, race and other biases in generated images.

NICE advances GANs with Theorems 1 and 2, along with Corollaries 1 and 2. It offers novel insights into the weight regularization. These findings, driven by the multiplicative and additive noise modulation, along with the gradient regularization due to the consistency regularization, pave the way for further future research.

B Notations

Below, we explain the notations used in this work.

Scalars : Represented by lowercase letters ( e.g ., m , n , δ ).

Vectors : Bold lowercase letters ( e.g ., v , u , m ).

Matrices : Bold uppercase letters ( e.g ., W , X , H ).

Vector/Matrix elements : v k and a k represent k -th element of v and a . H jk denotes the element located at the j -th row and k -th column of matrix H .

Functions : Letters followed by brackets ( e.g ., σ ( · ) , h ( · ) , ∇ f ( · ) , H ( · ) ).

Probability measures : Denoted by letters µ , ν and p z .

Function sets : Calligraphic uppercase letters are used, but note that N and B specifically denote Gaussian and Bernoulli distributions, respectively ( e.g ., H , F ).

Dataset samples : Expressed as x ( i ) , y ( i ) , a ( i ) , τ ( i ) .

Multiplication/Composition : ⊙ represents element-wise multiplication and ◦ denotes function composition.

Expectation/empirical expectation : E [ · ] represents the average or expected value of a random variable while ˆ E [ · ] denotes the empirical expectation calculated over observed data samples.

C Proofs

We define ∥ v ∥ 2 = v T v as the squared norm of vector v . Note the commutative property of the dot product, v T u = u T v , and the distributive property over matrices, v T A + u T A = ( v + u ) T A . With these definitions and properties established, we now prove Theorem 1, Proposition 1 and Theorem 2. We begin with a simplified scalar-based derivation, ensuring a rapid derivation of the final results, followed by comprehensive proofs encompassing vector and matrix calculus for a holistic understanding.

C.1 Proof of Theorem 1

Theorem 1 (Regularization by the Gaussian multiplicative noise in deep regression) Consider a regression task on a two-layer neural network. Let X ⊆ R d 0 and Y⊆ [ -1 , 1] d 2 be the input and output spaces, and D = X×Y . Let n examples { ( x ( i ) , y ( i ) ) } n i =1 ∼D n . Let f w : X →Y be parameterized by { w : W 1 ∈ R d 1 × d 0 , W 2 ∈ R d 2 × d 1 } and f w ( x , z ) = W 2 ( z ⊙ σ ( W 1 x )) where σ is 1-Lipschitz activation function, ⊙ denotes element-wise multiplication and z ∼N ( 1 , β 2 I d 1 ) . Let a = σ ( W 1 x ) , ˆ a = ˆ E i [ a ( i ) ⊙ a ( i ) ] , ˆ a k ≥ 0 denotes the k -th element of ˆ a . Let ∥ W 2 ∥ 2 , 1 = ∑ k ∥ w k ∥ 2 where w k is the k -th column vector in W 2 . The regression task with the l 2 loss leads to the weight regularization as:

We commence with a simplified derivation using scalars, which can quickly lead to the final results:

Proof:

Let m ∼ N ( 0 , β 2 I d 1 ) , with m k and a ( i ) k representing the k -th elements of vectors m and a ( i ) , respectively. Using these definitions, we derive the second term of Eq. 12 as follows:

Given that the elements in m are independent, for j = k we have:

̸

Observe that ( W T 2 W 2 ) kk = ∥ w k ∥ 2 2 represents the squared norm of k -th column vector in W 2 . Let ˆ a denote the element-wise mean square of a ( i ) , i.e ., ˆ a = ˆ E i [ a ( i ) ⊙ a ( i ) ] . We leverage the distribution property E [ z 2 ] = σ 2 + µ 2 for z ∼ N ( µ, σ 2 ) , simplifying to E [ z 2 ] = σ 2 when µ = 0 . This lets us

reformulate Eq. 13 explicitly as:

Substituting the second term in Eq. 12 with Eq. 15, we arrive at:

Integrating the Gaussian multiplicative noise into the optimization of regression task leads to an adaptive regularization β 2 ∑ k ˆ a k ∥ w k ∥ 2 2 on W 2 , effectively reweighting the squared norm ∥ w k ∥ 2 2 of each column in accordance with ˆ a k ≥ 0 . This mechanism, driven by the variance β 2 of noise enhances the robustness of model and its generalization capability. It is worth highlighting that this process relates closely to the (2,1)-norm of W 2 , defined as ∥ W 2 ∥ 2 , 1 = ∑ k ∥ w k ∥ 2 .

Corollary 1 Utilizing the additive noise injection where z ∼ N ( 0 , β 2 I d 1 ) , the regression task with the l 2 loss yields weight regularization as:

C.2 Proof of Proposition 1

Proposition 1 Define f := f t ◦ . . . ◦ f 2 as the feature extractor of the discriminator from the second layer onward. Let ˜ a = f 1 (˜ x ) and a = f 1 ( x ) be the latent features of the first layer for fake and real images, respectively. The discriminator is defined as h ( · ) = ϕ ( φ ( f ( · ))) , with φ as the prediction head and ϕ as the loss function. Introduce a multiplicative noise z ∼ N ( 1 , β 2 I d ′ ) modulating solely the first layer. Let H ( h ) kk ( a ) be the k -th diagonal entry of the Hessian matrix of h at a and a k be the k -th element of a . Applying Taylor expansion to h ( · ) , the GAN loss can be expressed as follows:

We provide a succinct proof for the discriminator using scalar values to quickly arrive at the final result, where h ′ ( · ) and h ′′ ( · ) are the first- and second-order derivatives, respectively.

Proof: Consider a multivariate normal distribution m ∼ N ( 0 , β 2 I d ′ ) . Let ∇ h ( a ) and H ( h ) jk ( a ) represent the first-order derivative and the ( j, k ) -th entry of the Hessian matrix of h at a , respectively,

implying that H ( h ) jk = H ( h ) kj due to the symmetry of the Hessian matrix. With these notations established, we proceed as follows:

Addressing the first-order gradient term presented in Eq. 18, we notice the following:

Examining the second-order gradient in Eq. 19, and leveraging the independence of elements in m , we apply the result from Eq. 14. With ˜ a k and a k representing the k -th elements in ˜ a and a , respectively, we derive:

By substituting Eq. 18 with the results from Eq. 20, and Eq. 19 with Eq. 21, we arrive at the following expression:

Utilizing the same derivation process as before for L AN G , we arrive at the following result:

Introducing the multiplicative Gaussian noise during training of GAN produces the gradient maximization effect on the real samples in the discriminator training, specially through the term β 2 2 ∑ k a 2 k H ( h ) kk ( a ) . This term intertwines the second-order partial derivative (diagonal entry H ( h ) kk ( a ) of the Hessian matrix) with the non-negative factor β 2 2 a 2 k ≥ 0 , and is related to the trace of the Hessian matrix given by trace ( H ( h ) ( a r ) ) = ∑ k H ( h ) kk ( a ) . Employing similar reasoning for the fake samples when training the generator reveals that the noise-modulated discriminator training of GAN tends to amplify the the diagonal entry of the second-order partial derivative in the Hessian matrix at both the real and fake points, influencing the optimization stability.

C.3 Proof of Theorem 2

Theorem 2 Using notations from Prop. 1 and with f : R d ′ → R , let ∇ 2 k f ( a ) and ( H ( f ) jk ( a ) ) 2 denote the squares of the k -th gradient element and ( j, k ) -th entry of Hessian matrix of f at a , respectively. Given z 1 , z 2 ∼N ( 1 , β 2 I d ′ ) , enforcing invariance of f under different noise modulations yields:

A concise derivation employs scalar values for the Taylor expansion, swiftly leading to results using f ′ ( · ) and f ′′ ( · ) for first and second-order derivatives, respectively.

Proof: Let m 1 , m 2 ∼ N ( 0 , β 2 I d ′ ) . Note z 1 and z 2 are independent of each other, as well as independent of a and f ( a ) . The same independence holds for z 2 . By taking expectations over z 1 and z 2 , and applying Taylor expansion, we analyze the consistency loss of f ( a ) :

Defining v := m 1 ⊙ a and u := m 2 ⊙ a with m 1 and m 2 are independently sampled from N ( 0 , β 2 I d ′ ) , we get v , u ∼ N ( 0 , β 2 diag ( a ⊙ a )) . Here diag ( a ⊙ a ) is a diagonal matrix with a 2 k on the diagonal. Letting δ := ∇ f ( a ) and H = H ( f ) ( a ) , we can rewrite Eq. 24 as follows:

Next, we succinctly derive the four terms: the first (Eq. 25), second (Eq. 26), third (Eq. 27), and fourth (Eq. 28) term, respectively.

The first term (Eq. 25):

Leveraging the property E z 1 ,z 2 [( z 1 -z 2 ) 2 ] = 2 σ 2 for z 1 , z 2 ∼ N (0 , σ 2 ) , we can simplify (Eq. 25) as follows:

The second term (Eq. 26):

Utilizing E [ z 3 ] = µ 3 +3 µσ 2 for z ∼ N ( µ, σ 2 ) , and noting that E [ z 3 ] = 0 when µ = 0 and elements in v are independent, Eq. 26 is derived as:

The third term (Eq. 27):

Given the independence of elements in v , only terms with an element repeated two or four times contribute non-zero results, leading to four distinct, non-overlapping cases. Using E [ z 2 ] = σ 2 + µ 2 and E [ z 4 ] = µ 4 +6 µ 2 σ 2 +3 σ 4 for z ∼ N ( µ, σ 2 ) , and simplifying to E [ z 2 ] = σ 2 and E [ z 4 ] = 3 σ 4 when µ = 0 , we have:

Case 1: j = k = p = q , the independence of v j and v p simplifies our calculation, leading to:

̸

̸

̸

̸

Case 2: For j = p = k = q , given the independence of v j and v k , we have:

̸

̸

̸

̸

̸

Case 3: For j = q = k = p , leveraging the independence of v j and v k as well as the symmetry H jk = H kj , we obtain:

̸

̸

̸

Case 4: For j = q = k = p , using E [ z 4 ] = 3 σ 4 where z ∼ N (0 , σ 2 ) , we deduce:

Combining Cases 1-4 together, we arive at for Eq. 27:

̸

The forth term (Eq. 28):

̸

Substitute Eq. 29, 30, 36, 37 back into Eq. 25, 26, 27, 28, we have the final results:

̸

̸

̸

Enforcing the consistency regularization implicitly penalizes the first- and second-order gradients of f w.r.t. a , with an adaptive penalty controlled β and modulated by a . This modulation recalibrates the squared norms of gradient: the squared norm of the first-order gradient, ∥∇ f ( a ) ∥ 2 2 = ∑ k ∇ 2 k f ( a ) , and the Frobenius norm of the Hessian, ∥ H ( f ) ( a ) ∥ 2 F = ∑ j ∑ k ( H ( f ) jk ( a )) 2 . Specifically, each term is reweighted by a 2 k ≥ 0 , ensuring that the penalty applied to each gradient component is scaled proportionally to its associated magnitude of feature.

Corollary 2 Enforcing invariance of f under different additive noise injections, where z 1 , z 2 ∼ N ( 0 , β 2 I d ′ ) leads to the gradient regularization as:

D Implementation details for Experiments

Following previous benchmarks [68, 22], we augmented all datasets with the simple x -flips augmentation.

D.1 Network architecture and hyperparameters

CIFAR-10/100. We experiment with OmniGAN ( d ′ =256 , 1024 ) and BigGAN ( d ′ =256 ) with the batch size of 32. We follow [68] and train the OmniGAN and BigGAN for 1K epochs on the full data and 5K epochs on 10%/20% data setting. We equip the discriminator with the adaptive noise modulation after convolution weights c ∈ { C 1 , C 2 , C S } at all blocks l ∈ { 1 , 2 , 3 , 4 } . We set ∆ β = 0 . 001 , η = 0 . 5 , ∆ γ = 10 . Features before loss function are used for NICE on BigGAN.

ImageNet . We experiment with BigGAN with the batch size of 512. We use a learning rate of 1e-4 for generator and 4e-4 for discriminator. The noise modulation is placed after convolution layers c ∈ { C 1 , C 2 , C S } at blocks l ∈ { 3 , 4 , 5 } , ∆ β = 0 . 001 , η = 0 . 5 , ∆ γ = 5 . Features before loss function are used for NICE.

Low-shot images. We build our NICE upon StyleGAN2 with batch size of 64 and train the networks until the discriminator had seen 25M images. We apply noise modulation with Bernoulli noise after convolutions c ∈ { C 1 , C 2 } at blocks l ∈ { 3 , 4 , 5 , 6 } , ∆ β = 0 . 0001 , η = 0 . 9 , ∆ γ = 0 . 05

FFHQ . We experiments on StyleGAN2 with batch size of 64 and train the networks until the discriminator had seen 25M images. We apply noise modulation with Bernoulli noise after convolutions c ∈ { C 1 , C 2 } at blocks l ∈ { 3 , 4 , 5 , 6 } , ∆ β = 0 . 0001 , η = 0 . 6 , ∆ γ = 0 . 05

D.2 Implementation details for AWD, AN+AGP, and AACR

In Figure 5, we compare NICE with three different approaches: adaptive weight decay (AWD), adaptive noise + adaptive gradient penaulty (AN+AGP) and adaptive augmentation-based consistency regularization (AACR). For AWD, we dynamically control the weight decay using βλ WD. We search

the λ WD ∈ { 2e-4, 1e-4, 5e-5, 2e-5, 1e-5, 5e-6 } and obtain the best result at 5e-5. For AN+AGP, we employ the adaptive noise modulation and also adaptively penalize the gradient of ∥ ∥ ∂L AN D ∂ x ∥ ∥ 2 2 or ∥ ∥ ∂f ∂ x ∥ ∥ 2 2 with adaptive strength βλ GP. We test different values of λ GP ∈ { 0 . 2 , 0 . 5 , 1 , 2 , 5 , 10 } and obtain the best results with λ GP =2 . For AACR, we incorporate the adaptive differentiable augmentation (ADA) to provide two different views for the same images, and enforce the discriminator to be consistent to the two views of the same images. We adaptively control the strength of AACR with βλ AACR. We search the λ AACR ∈ { 0 . 2 , 0 . 5 , 1 , 2 , 5 , 10 } and obtain the best result with λ AACR = 2 .

E Impact of various factors

Below, we analyze the importance of different factors in NICE by conducting ablation experiments. The results are presented in Figure 6. In Figure 6a, we observe that placing the noise modulation module after C 1 and C 2 improves performance, and the best results are obtained when the noise is placed after all convolution layers. Figure 6b indicates that placing the noise at deeper layers leads to better tFID scores, and the best performance is achieved when the noise is applied at all blocks. Figure 6c shows that ∆ γ = 10 is a good choice for NICE. Figure 6d illustrates η = 0 . 5 produce overall good performance. Figure 6e plots the results for consistency regularization among M feature extractors f modulated with noise. We use M = 2 ( i.e ., f 1 and f 2 ) in the main paper. Figure 6e indicates there is no need to use more feature extractors, as f 1 ( · ) and f 2 ( · ) for consistency regularization produce the best results. Figure 6f presents the results for consistency regularization at every N discriminator steps, indicating that regularizing the discriminator during each iteration is preferable, as larger intervals result in worse performance.

C

1

C

2

C

S

C

1

C

S

C

2

C

S

C

1

C

2

C

1

C

2

C

S

6

9

12

15

18

tFID

(a)

c

∈ {

C

1

, C

2

, C

S

}

1

2

3

4

12 34 123 234 1234

6

9

12

15

18

tFID

(b)

l

∈ {

1

...L

}

1

2

5

10

15

20

25

∆

γ

7.0

7.6

8.2

8.8

tFID

(c)

∆

γ

0.2 0.3 0.4 0.5 0.6 0.7 0.8

0.9

η

7.0

7.6

8.2

8.8

tFID

(d)

η

2

3

4

5

6

8

10

7.0

7.6

8.2

8.8

tFID

(e)

M

-time

1

2

4

6

8

12

16

7.0

7.6

8.2

8.8

tFID

(f)

N

-step

F Rationale for choosing multiplicative noise over direct weight regularization

For the multiplicative modulation with the noise, we have:

where z ⊙ a ( i ) is element-wise multiplication of z with a ( i ) and ˆ a = ˆ E i [ a ( i ) ⊙ a ( i ) ] . This implies that while the direct penalty of network weights, ∑ k ∥ w k ∥ 2 2 , is possible, our approach enjoys a dynamic modulation β 2 ˆ a 2 k where variance β 2 is adapted based on the discriminator decisions and ˆ a k ≥ 0 depends on the magnitude of feature vectors.

̸

The importance of such a multiplicative modulation is that semantic contents is not added by noise z to the modulated feature vector z · a ( i ) , i.e ., only 'active' channels a ( i ) = 0 are modulated: they can be suppressed or magnified according to variance β 2 .

In contrast, the additive noise modulation results in the standard penalty β 2 ∑ k ∥ w k ∥ 2 2 :

While obvious, β 2 ∑ k ∥ w k ∥ 2 2 does not enjoy the dynamic penalty term related to feature norms. This also means (based on the derivation) that the additive noise may change feature semantics , i.e ., z + a ( i ) may 'activate' channels which are non-active in a ( i ) (features a ( i ) = 0 ). Therefore, our multiplicative modulator not only controls the Rademacher Complexity (RC), but controls it in a meaningful manner for discriminator . Only feature semantics that are present in a feature vector (that describe an object) are modulated while helping control RC. In contrast, the additive noise introduces semantics that are not present for a given image or object.

While directly regularizing the network using β 2 ∑ k ∥ w k ∥ 2 2 does not introduce the additive noise, the explicit regularization effect is similar to introducing the additive noise. Thus, if we connect the direct weight regularization to the additive noise, it will also have the effect of changing the semantics, which will negatively impact the classification accuracy.

We provide experimental comparisons in Table 8. In addition, Figure 7 shows that AN achives higher classification accuracy than AAN and AWR, demonstrating that multiplicative noise preserves the feature semantics better than the additive noise and the weight regularization.

⊙

∼N

0

50

100

150

195

iterations (

1000)

0.4

0.5

0.6

0.7

Acc

50

100

150

195

iterations (

1000)

OmniGAN

+AN

+AWR

+AAN

+NICE

+ALGP

+AWR+ALGP

+NICE

add

×

×

G Rationale behind choosing NICE over the direct gradient regularization

Consider the regularization term in Theorem 2 and its Taylor expansion. Indeed, this expression penalizes the squared norms of first- and second-order gradients of f ( a ) . However, notice that our expression has specific reweighting effect for penalizing the first- and second-order gradients. The specific gradient penalties

emerge only in case of using the multiplicative noise drawn from N ( 1 , β 2 I ) for Theorem 2. Importantly, the multiplicative noise z ∼ N ( 1 , β 2 I ) does not introduce new semantics into feature vector a by operation z ⊙ a , i.e ., only feature semantics that are present in feature vector (features of a that are non-zero) that describe object are modulated. Yet, this multiplicative noise does help control the Rademacher Complexity (RC) due to Lemma 2 and Theorem 1. However, imposing a generic penalty of form β ( ∑ k ∇ 2 k f ( a ) + ∑ j,k ( H ( f ) jk ( a )) 2 that arises when the additive noise z ∼ N ( 0 , β 2 I ) is applied to a ( i ) , i.e ., z + a .

Let z ∼ N ( 0 , β 2 I ) . The consistency loss regularization with the additive noise injection is:

Such a variant means that semantics that are not present in a may be 'activated' by the noise, drastically altering the meaning of a and damaging the information it carries about image/object.

While directly applying gradient penalization does not introduce noise, its connection to the additive noise suggests it will have a negative effect on semantics. Table 8 evaluates the multiplicative vs . additive noise modulators, and the direct penalties. Table 9 provides experimental results for different gradient regularization variants. Furthermore, Figure 7 shows that NICE obtains best accuracy than other variants, showing that the multiplicative noise modulation preserves the semantics better than other variants, justifying the rationale behind NICE.

H Training overhead with and without NICE

We provide number of parameters/multiply-accumulate (MACs) (for both generator and discriminator), number of GPUs and cost of time (seconds per 1000 images, secs/ k img) in Table 10. With the efficient implementation (Figure 1), our NICE only introduces small fraction of cost of time on high-resolution datasets. 15.08% on ImageNet and 18.02% on FFHQ/low-shot datasets. Figure 8 illustrates the FID gain vs . speed, which shows that a slightly increase in training time yields substantial improvements, particularly when compared to recent state-of-the-art approaches like FakeCLR [32] and InsGen [63]. We believe the increased computational overhead is outweighed by the considerable benefits and the improvements justify the extra fraction of time.

5

6

7

8

9

10

11

12

sec/

k

img

15

25

40

55

70

FID

ADA

NICE

ADA+NICE

InsGen

FakeCLR

I Qualitative comparison among DA, ADA, AACR and NICE+ADA

Figure 9 provides qualitative comparison between augmentation based method DA, ADA, AACR and NICE+ADA. We find DA, ADA and AACR leak augmentation clues to the generated images while NICE+ADA prevents the leakage. Figure 10 further demonstrates augmentation-based method, although can prevent discriminator overfitting problem by expanding data space through diverse augmentations, has the potential to cause increased gradient norm due to Prop. 1 and the fact ADA involves random additive/multiplicative noise. However, when combining NICE with ADA, the gradient issues caused by ADA were alleviated.

J Generated images

Figures 11, 12, 13, 14 and 15 provide the generated images on CIFAR-10, ImageNet, CIFAR-100, low-shot datasets and FFHQ with or without NICE. We can see NICE improve the image generation quality.

(a) DA

(b) ADA

(c) AACR

(d) NICE+ADA

0

50

100

150

195

iterations (

1000)

0e-3

4e-3

8e-3

12e-3

||

∂LD

∂x

||

2

2

Real images

Fake images

OmniGAN

+ADA

+ADA+NICE

×

(a) DA

(b) ADA

(c) ADA+NICE

(a) ADA

(b) ADA+NICE

(a) DA

(b) ADA

(c) ADA+NICE

Panda

(a) StyleGAN2+ADA

FFHQ-5K

FFHQ-2K

FFHQ-1K

FFHQ-100

(a) StyleGAN2+ADA

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a noise modulation and regularization scheme for GANs that reduces disciminator overfitting and improves training stability in the low-data scheme. The technique demonstrate consistent improvements when applied to several different network architectures and datasets.\n\nSTRENGTHS:\nThe paper contains very extensive comparisons to competing previous methods.\nThe theoretical motivation for the proposed technique is very extensive (but also very hard for non-experts to understand).\nThe theoretical analysis sheds some light on the effectiveness of consistency regularization techniques that have shown promise previously.\nThe numerical results are excellent.\nThe paper is very math-heavy, but the authors do a good job of including some intuition of the different propositions and lemmas (e.g. lines 142-147, 156-160, 173-175, 191-196).\nThe inclusion of separate tFIDs and vFIDs is refreshing to see.\n\nWEAKNESSES:\nNot a single generated image is shown in the main paper. Given that some concerns about the reliability of FID have been raised as of late, visual results (e.g. the 100-shot results from the supplemental, Fig. 11) would be appreciated.\n\nThe paper mentions many similar-sounding terms (generation gap, generalization gap, output gap, generalization error, discriminator discrepancy) that might confuse non-experts. A brief description of the effect and its implications on training should be included for non-expert readers.\n\nMany images in the supplemental are of unusably low quality. This is especially true for CIFAR results, which need to be integer-upscaled to preserve sharpness.\n\nQUESTIONS:\nI do not understand where the second derivatives come from in equation 9.\nEquation 10 seems to perform a Taylor expansion, is this accurate? If so, the fact should be mentioned for clarity.\n\nIs equation 1 a contribution of this paper or something presented in previous work?\n\nWhat does MA in tables 1-4 mean?\n\nIn figure 2, the y-axis labels are incorrectly formatted (1e^2 instead of 1*10^2 or 1e2)\n\nThe authors claim (line 171) that proposition 1 can be extended to more complicated architectures - is an example of this shown somewhere, such as previous work?\n\nThe datasets used in Figures 2 and 3 don't seem to be mentioned\n\nNICE seems to work best when combined with geometric augmentation techniques, especially ADA. A comment on this would be useful - ADA already minimizes the gap between the real and fake distributions, so what does NICE add in this case?\n\nLIMITATIONS:\nThe limitations are not properly discussed in the main text. Is there a wall clock cost to the proposed method? Does it require changes to training hyper parameters (learning rate etc.)?", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-02"}

comments: SUMMARY:
This paper proposed NICE, a technique that enforces the discriminator to be consistent with the same inputs under different noise modulations. The authors showed us both in theory and practice that NICE is effective at preventing discriminator overfitting and achieves superior performance in image generation under limited data settings.

STRENGTHS:
1. The method is based on theoretical proof, and the experimental results show the effectiveness.

2. The method achieves competitive performance against many existing methods. The evaluation involves multiple GAN models, and baseline methods.

WEAKNESSES:
1. The authors show the computational overhead of NICE in appendix, which is not small and it could increase the training time by a large amount. There is no quantitative measure for scalability of NICE. For example, what will happen when the image resolution is 1080p or even higher, or when the dataset is quite large. What is the training cost?

2. NICE introduces some new hyperparameters but it is not clear how they are tuned. What's their sensitivity to different GAN models and data distribution (or datasets)? What's their sensitivity to baselines in Table 3?

QUESTIONS:
Please refer to “Weaknesses".

It seems that authors missed a strong baseline method SSGAN-LA [1] in Table 3 which could achieved a better performance. It is better to compare with missing baselines.

[1] Self-Supervised GANs with Label Augmentation, Neurips 2021.

LIMITATIONS:
Please refer to “Weaknesses" for limitations. The authors adequately addressed the potential negative societal impact of their work.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a training approach called NoIse-modulated Consistency rEgularization (NICE) to improve the data-efficiency of generative adversarial networks (GANs) by addressing issues related to limited data. It introduces adaptive multiplicative noise into the discriminator to modulate its latent features, preventing discriminator overfitting. To mitigate the instability of GAN training caused by increased gradient norm, a constraint is imposed on the discriminator to ensure consistency for the same inputs under different noise modulations. The experimental results demonstrate the effectiveness of NICE in reducing discriminator overfitting and improving the stability of GAN training, achieving state-of-the-art results on various datasets and low-shot generation tasks.\n\nSTRENGTHS:\n- The paper is well-organized and well-written, making it easy to understand its contribution.\n- The paper provides theoretical analysis to understand the connection between introducing multiplicative Gaussian noise to the discriminator and GAN generalization. The authors also uncover the negative impact of simple noise multiplication on gradient norm and propose a noise-modulated consistency regularization with theoretical grounds to improve it.\n- Experimental results presented in the paper validate the effectiveness of the proposed method, surpassing the baseline in most experiments.\n\nWEAKNESSES:\n- The complete objective function of the proposed method should be included in the main text, rather than in the appendix.\n- In the GAN field, it is suggested to conduct multiple experiments and report the results in terms of mean and standard deviation due to the random variation of the results caused by different trials.\n- According to the theoretical analysis in this paper, it is easy to understand the motivation behind the introduction of Eq12, but the motivation behind Eq13 and Eq14 is unclear. Although the ablation experiments in Table 6 empirically demonstrated the benefits of the proposed method, I still expect the authors to provide a more explicit explanation for these equations. Moreover, it is recommended to report the experimental results using only Eq12 and Eq14, as this would further demonstrate the effectiveness of the method.\n\nQUESTIONS:\n- Does Eq1 encompass all GAN objective functions, such as the original min-max and non-saturating GANs? It is suggested that the authors can provide detailed explanations, as the theoretical work of this paper is based on Eq1.\n\nSome typos:\n- Lack parentheses in Eq9.\n- f(x) -> f(\\alpha) in Line 185.\n- N(0,\\beta^2 I^d) -> N(1,\\beta^2 I^d) in Line 200.\n\nLIMITATIONS:\nNo, the authors have not addressed the limitations and potential negative societal impact of their work.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a regularization method called noise-modulated consistency regularization (NICE) to train GANs with limited data. In this method, this paper proposes modulating the discriminator's latent features using noise and imposing a constraint on the discriminator so that the middle outputs of the discriminator (particularly, the outputs before the prediction head) for differently modulated data are the same. This paper also provides a theoretical analysis, in which it is shown that the proposed regularization penalizes the first- and second-order gradients of latent features and improves the GAN training stability. The effectiveness of the proposed method was demonstrated using small-scale typically-used datasets, including CIFAR-10, CIFAR-100, ImageNet, and FFHQ datasets, and in low-shot generation tasks.\n\nSTRENGTHS:\n1. The effectiveness of the proposed method was demonstrated in various scenarios, including evaluation on small-scale typically-used datasets (CIFAR-10, CIFAR-100, ImageNet, and FFHQ datasets) and evaluation in low-shot generation tasks. In many scenarios, the proposed method achieves state-of-the-art performance while comparing with various baselines. Furthermore, the applicability of the proposed method was also demonstrated by applying the proposed method to various GANs (e.g., BigGAN and OmniGAN in Tables 2 and 3, and StyleGAN2 in Table 3) and using the proposed method with orthogonal methods (e.g., LeCam, DA, and ADA). Ablation studies are also conducted.\n\n2. Not only the effectiveness of the proposed method is demonstrated, but also the theoretical analysis is provided. This analysis verifies the proposed method is useful for regularizing the first- and second-order gradients of latent features and improving the GAN training stability. This explanation is reasonable.\n\n3. This paper is well written and easy to read. Although some explanation is slightly too concise, the discussion on related work is thorough.\n\nWEAKNESSES:\n1. Through theoretical analysis, I understand that the proposed method is useful for regularizing the first- and second-order gradients of latent features. However, this analysis raises a question of what happens when regularizing the first- and second-order gradients of latent features directly. As discussed in related work, there are several previous studies that propose gradient regularizations. I guess that the proposed method is better than a direct regularization method in terms of calculation cost; however, I would appreciate it if I could hear the opinion from the authors.\n\n2. I cannot find the discussion on the increase in the calculation cost. I suspect the proposed method increases the calculation cost because the discriminator needs to process data twice, compared to a standard discriminator. For a fair comparison, it would be better to be discussed.\n\n3. Some results are excluded in Tables 1–3 (e.g., IS/tFID for DigGAN in Table 1). I cannot find a clear explanation for why the results are excluded. It seems that DigGAN is one of the comparable baselines; therefore, I would appreciate it if the authors could provide the missing scores.\n\nQUESTIONS:\n1. What happens when directly regularizing the first- and second-order gradients of latent features? (See weakness 1)\n2. Discuss the calculation cost. (See weakness 2)\n3. Why are some results excluded in Tables 1–3. Provide the scores if possible (See weakness 3)\n\nLIMITATIONS:\n1. Image synthesis technologies could be misused for synthesizing fake images. In particular, few-shot image synthesis technologies will make it easy to synthesize such images because they reduce the data collection cost. It would be better to discuss the social impact in this aspect.\n2. Although the versatility of the proposed method is demonstrated, I suspect that there may be some previous methods that are not compatible with the proposed method. Discussing this will be useful for future research.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors present a novel approach called NoIse-modulated Consistency rEgularization (NICE) to solve the challenge of training GANs with limited data. The experiment was conducted on reduced small-scale CIFAR-10, CIFAR-100, ImageNet, and FFHQ datasets. Additionally, they applied their method to low-shot generation tasks, and the results demonstrated state-of-the-art performance.\n\nOverall, the proposed methodology is well motivated and supported by theoretical analysis. The flow of the paper and the writing are easy to follow.\n\nSTRENGTHS:\n1. The paper is written in a clear and well-structured manner, making it easy for readers to follow the presented ideas.\n\n2. Through extensive experiments on various reduced small-scale datasets, the method consistently achieves state-of-the-art performance, demonstrating its effectiveness.\n\n3. The paper is well-grounded in theoretical motivation and effectively validates the proposed methodology through empirical analysis.\n\nWEAKNESSES:\nEven though the experimental results support their theory, the ablation study and the experiment for gradient analysis are only based on some specific datasets. Evaluating the theory on more datasets would provide more convincing evidence.\n\nQUESTIONS:\n1.\tIn Table 1, Table 2, Table 3, and Table 4, what \"MA\" refers to?\n\n2.\tFigure 2 was plotted based on a specific dataset, but the paper does not clearly state which dataset was used to generate this figure. It would be helpful if the authors provide more information about the dataset used for Figure 2.\n\n3.\tThe notation used in the paper is inconsistent, for instance, with the symbols $\\boldsymbol{x}_{\\text {real }}$ in page 5 row 205 and $\\boldsymbol{x}_r$ in page 5 row 182. Additionally, there is no explanation for the symbols $\\boldsymbol{x}_r$ and $\\boldsymbol{x}_f$. It would be beneficial for the authors to clarify and provide consistent notation throughout the paper to avoid confusion.\n\n4.\tIn Equation 8, it is evident that training with features modulated by Gaussian noise leads to regularization of the internal weight by the (2,1) norm. However, it is not clear why the authors did not directly penalize the (2,1)-norm to reduce the Rademacher complexity. It would be valuable if the paper includes a comparison between the Discriminator with adaptive noise (AN) and directly penalizing the (2,1)-norm with adaptive strength $\\beta$.\n\n5.\tIn Equation 9, the training of the discriminator involves minimizing the second-order gradient at fake images. However, Figure 2 (b) shows that the gradient norm at the layer before the classification head for fake images increases. Furthermore, the ablation study with $\\mathrm{NICE}{D_f}$ demonstrates improved performance, as well as with $\\mathrm{NICE}{G_f}$, but there is no analysis of the gradient norm for the generator.\n\nLIMITATIONS:\nThe limitations could be further clarified.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-26"}