SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Theorem 2 Using notations from Prop. 1 and with  $f: \mathbb{R}^{d'} \to \mathbb{R}$ , let  $\nabla_k^2 f(\pmb{a})$  and  $\left(H_{jk}^{(f)}(\pmb{a})\right)^2$  denote the squares of the  $k$ -th gradient element and  $(j,k)$ -th entry of Hessian matrix of  $f$  at  $\pmb{a}$ , respectively. Given  $z_1, z_2 \sim \mathcal{N}(\mathbf{1}, \beta^2 \mathbf{I}^{d'})$ , enforcing invariance of  $f$  under different noise modulations yields:
$$
\ell^ {N I C E} (\boldsymbol {a}) := \mathbb {E} _ {\boldsymbol {z} _ {1}, \boldsymbol {z} _ {2}} \left[ \left(f \left(\boldsymbol {z} _ {1} \odot \boldsymbol {a}\right) - f \left(\boldsymbol {z} _ {2} \odot \boldsymbol {a}\right)\right) ^ {2} \right] \approx 2 \beta^ {2} \sum_ {k} a _ {k} ^ {2} \nabla_ {k} ^ {2} f (\boldsymbol {a}) + \beta^ {4} \sum_ {j, k} a _ {j} ^ {2} a _ {k} ^ {2} \left(H _ {j k} ^ {(f)} (\boldsymbol {a})\right) ^ {2}. \tag {8}
$$
Theorem 2 (the proof is in §C.3) shows that the consistency loss essentially penalizes the first- and second-order gradient of  $f$  at  $\pmb{a}$ , with the strength adaptively controlled by the noise variance  $\beta^2$  and modulated by  $\pmb{a}$ . This results in reweighted gradient squared norms modulation, with terms in expressions  $\| \nabla f(\pmb{a}) \|_2^2 = \sum_k \nabla_k^2 f(\pmb{a})$ , and  $\| H^{(f)}(\pmb{a}) \|_F^2 = \sum_{j,k} (H_{jk}^{(f)}(\pmb{a}))^2$  influenced by  $a_k^2 \geq 0$ . In contrast to the noise-modulated discriminator in Eq. 6, NICE regularizes the first- and second-order gradients of layers ahead of the classification head. Despite the gradient from the classification head is large, NICE ensures the gradient reaching the input stays small, resulting in a stable training and improved generalization and convergence [4, 26, 67].
3.4 Implementation of NICE
The analysis above highlights practical ways to enhance the generalization and stability of GANs. Firstly, we apply the adaptive multiplicative noise to the latent features of the discriminator, striking balance between discrimination and generalization, as highlighted in [67]. The consistency regularization is then applied to ensure invariance of the discriminator to the same inputs under varied noise modulations, penalizing the gradient and stabilizing the training process.
GAN with the adaptive noise modulation (AN). Figure 1 shows the discriminator composed of blocks  $B_{1}, B_{2}, \ldots, B_{L}$ , where each block  $B_{l}$  contains multiple convolution weights  $c \in \{C_{1}, C_{2}, C_{S}\}$ . To achieve a trade-off between generalization and discrimination, we modulate the latent features  $\mathbf{X} \in \mathbb{R}^{1 \times d' \times d^{H} \times d^{W}}$  from convolutional layers of the discriminator using adaptive noise  $\mathbf{Z} \in \mathbb{R}^{1 \times d'}$  by:
$$
\operatorname {A N} (\boldsymbol {X}) = \boldsymbol {X} \odot \boldsymbol {Z}, \tag {9}
$$
Figure 1: Our discriminator pipeline. NICE is applied to both real  $\pmb{x}$  and fake  $\tilde{\pmb{x}}$  images.
where  $\odot$  represents the operation that expands  $Z$  into  $1\times d^{\prime}\times d^{H}\times d^{W}$  shape and performs elementwise multiplication with  $X$ . The noise  $Z$  is carefully controlled through an adaptive  $\beta$ . In the case of BigGAN and OmniGAN, we control the variance of  $Z$  using  $\beta$  as  $Z\sim \mathcal{N}(\mathbf{1},\beta^{2}\pmb{I}^{d^{\prime}})$ . For StyleGAN2, we control the noise through  $Z\sim \mathcal{B}(\beta)$ , where with probability  $1 - \beta$  it takes the value  $\frac{1}{1 - \beta}$ , and with probability  $\beta$  it takes 0.
Below we introduce a mechanism to control the noise via the meta-parameter  $\beta$  by detecting potential overfitting in the discriminator. Firstly, we compute the expectation over the discriminator output  $r(\pmb{x}) = \mathbb{E}[\mathrm{sign}(D(\pmb{x}))]$  w.r.t. real samples  $\pmb{x}$ , and evaluate  $\varepsilon = \mathrm{sign}(r(\pmb{x}) > \eta) \in \{-1, 0, 1\}$  where  $\eta$  is a fixed threshold. A value greater than the threshold indicates potential overfitting [22]. We apply  $\beta_{t+1} = \beta_t + \Delta_\beta \cdot \varepsilon$  to update  $\beta$  with smaller  $\Delta_\beta$  to control  $\pmb{Z}$ . Denote the modified discriminator with our adaptive noise as  $h_{\mathrm{AN}}$ , the objective of GAN with the adaptive noise modulation becomes:
$$
\left\{ \begin{array}{l} L _ {D} ^ {\mathrm {A N}} = \min  _ {\boldsymbol {\theta} _ {d}} \mathbb {E} _ {\tilde {\boldsymbol {x}} \sim \nu_ {n}} \left[ h _ {\mathrm {A N}} \left(\tilde {\boldsymbol {x}}; \boldsymbol {\theta} _ {d}\right) \right] - \mathbb {E} _ {\boldsymbol {x} \sim \hat {\mu} _ {n}} \left[ h _ {\mathrm {A N}} \left(\boldsymbol {x}; \boldsymbol {\theta} _ {d}\right) \right], \\ L _ {G} ^ {\mathrm {A N}} = \min  _ {\boldsymbol {\theta} _ {g}} - \mathbb {E} _ {\boldsymbol {z} \sim p _ {z}} \left[ h _ {\mathrm {A N}} \left(g \left(\boldsymbol {z}; \boldsymbol {\theta} _ {g}\right)\right) \right]. \end{array} \right. \tag {10}
$$
GAN with the noise-modulated consistency regularization (NICE). To deal with the increased gradient norm due to the adaptive noise modulation, we introduce the consistency regularization (NICE) to promote invariance of the discriminator to the same samples under different noise modulations. Our regularization implicitly penalizes the gradient of the latent features in the discriminator by comparing outputs  $f_{1}(\cdot)$  and  $f_{2}(\cdot)$  of two feature extractors (with shared parameters) that are subjected to different noise modulations as in Figure 1. The objective for a GAN with NICE is:
$$
\left\{ \begin{array}{l} L _ {D} ^ {\text {N I C E}} = \min  _ {\boldsymbol {\theta} _ {d}} \mathbb {E} _ {\tilde {\boldsymbol {x}} \sim \nu_ {n}} \left[ h _ {\mathrm {A N}} \left(\tilde {\boldsymbol {x}}; \boldsymbol {\theta} _ {d}\right) + \gamma \ell^ {\text {N I C E}} (\tilde {\boldsymbol {x}}) \right] + \mathbb {E} _ {\boldsymbol {x} \sim \hat {\mu} _ {n}} \left[ - h _ {\mathrm {A N}} \left(\boldsymbol {x}; \boldsymbol {\theta} _ {d}\right) + \gamma \ell^ {\text {N I C E}} (\boldsymbol {x}) \right], \\ L _ {G} ^ {\text {N I C E}} = \min  _ {\boldsymbol {\theta} _ {g}} \mathbb {E} _ {\boldsymbol {z} \sim p _ {z}} \left[ - h _ {\mathrm {A N}} \left(g \left(\boldsymbol {z}; \boldsymbol {\theta} _ {g}\right)\right) + \gamma \ell^ {\text {N I C E}} \left(g \left(\boldsymbol {z}; \boldsymbol {\theta} _ {g}\right)\right) \right], \end{array} \right. \tag {11}
$$
where  $\ell^{\mathrm{NICE}}(\cdot) = \| f_1(\cdot) - f_2(\cdot)\| _2^2$  and  $\gamma = \Delta_{\gamma}\beta$  is a meta-parameter that controls the strength of the consistency regularization and is adaptively determined by  $\beta$ .
Efficient NICE. In our experiments, we observed that applying the noise modulation in the later blocks of the discriminator often outperformed applying noise in the earlier blocks. To avoid unnecessary computations (the early blocks do not use modulation), we divide the discriminator into a modulation-free part  $(B_{1},\ldots ,B_{l - 1})$  and a noise-modulated part  $(B_{l},\ldots ,B_{L})$ , where  $l$  indicates the starting block of noise modulation. For efficient calculation of the consistency loss, we input two copies of  $X_{l - 1}$  (output from the first part) into the second part, bypassing the need for redundant calculations to maintain a low computational overhead compared to standard approaches.
We conduct experiments on CIFAR-10/100 [28] using BigGAN [7] and OmniGAN [71], as well as on ImageNet [10] using BigGAN for conditional image generation. We also evaluate our method on low-shot datasets [68], which include 100-shot Obama/Panda/Grumpy Cat and AnimalFace Dog/Cat [51], and FFHQ [24] using StyleGAN2 [25]. We compare our method against several strong baselines, including DA [68], ADA [22], DigGAN [11], MaskedGAN [17], KDDLGAN [9], LeCam [57], GenCo [8], InsGen [63], FakeCLR [32] and TransferGAN [59]. For fair comparison, we denote methods using massive augmentation as "MA", which include DA and ADA.
Datasets. CIFAR-10 has 50K/10K training/testing images with resolution of  $32 \times 32$  from 10 categories, whereas CIFAR-100 has 100 classes. FFHQ contains 70K human face images at  $256 \times 256$
pixels. Low-shot datasets contain 100-shot Obama/Panda/Grumpy Cat images, AnimalFace (160 cats and 389 dogs) images at  $256 \times 256$  resolution. ImageNet has 1.2M/50K training/validation images with 1K categories. Following [17, 9], we center-crop and downscale its images to  $64 \times 64$  resolution. The implementation details can be found in §D. The generated images can be found in §J.
Evaluation metrics. We generate 50K images per dataset to compute the commonly used Inception Score [46] and Fréchet Inception Distance (FID) [15]. We report tFID, computed between 50K generated images and all training images. For CIFAR-10/100, we also compute vFID between 10K generated images and 10K real testing images. For low-shot datasets, we follow [68] and compute FID between 5K generated images and the entire dataset. For FFHQ, we calculate FID between 50K fake images and the entire training set. For ImageNet, we follow [17, 9] and generate 10K images for computing the IS and FID, where the reference distribution is the entire training set. Following [68, 11, 32], we run 5 trails for methods using NICE, and report the mean of the results. Given that all standard deviations fall below the  $1\%$  relative, we omit them for clarity.
4.1 Results on CIFAR-10 and CIFAR-100 for BigGAN and OmniGAN
Tables 1 and 2 demonstrate that NICE consistently outperforms baselines such as BigGAN, LeCam+DA, OmniGAN and OmniGAN+ADA on CIFAR-10 and CIFAR-100, firmly establishing its superiority. NICE also outperforms LeCam+DA+KDDLGAN in the majority of scenarios without any knowledge integration from large-scale models, underscoring its efficiency and effectiveness.
Table 1: Comparison w/ and w/o NICE on CIFAR-10 given different percentage of training data.
Method	MA	100% data	20% data	10% data
IS↑	tFID↓	vFID↓	IS↑	tFID↓	vFID↓	IS↑	tFID↓	vFID↓
BigGAN(d' = 256)	×	9.21	5.48	9.42	8.74	16.20	20.27	8.24	31.45	35.59
+LeCam	×	9.45	4.27	8.29	8.95	11.34	15.25	8.44	28.36	33.65
+DigGAN	×	9.28	5.33	9.35	8.81	13.28	17.25	8.32	18.54	22.45
+NICE	×	9.50	4.19	8.24	8.96	8.51	12.54	8.73	13.65	17.75
+LeCam+DA	✓	9.45	4.32	8.40	9.01	8.53	12.47	8.81	12.64	16.42
+LeCam+DA+KDDLGAN	✓	-	-	8.19	-	-	11.15	-	-	13.86
+LeCam+DA+NICE	✓	9.52	3.72	7.81	9.12	6.92	10.89	8.99	9.86	13.81
OmniGAN(d' = 1024)	×	10.01	6.92	10.75	8.64	36.75	41.17	6.69	53.02	57.68
+DA	✓	10.13	4.15	8.06	9.49	13.45	17.27	8.99	19.45	23.48
+ADA	✓	10.24	4.95	9.06	9.41	27.04	30.58	7.86	40.05	44.01
+NICE	×	10.21	2.72	6.79	9.86	6.06	9.87	9.78	6.40	10.37
+NICE+ADA	✓	10.38	2.25	6.32	10.18	4.39	8.42	10.08	5.49	9.42
Table 2: Comparison w/ and w/o NICE on CIFAR-100 given different percentage of training data.
Method	MA	100% data	20% data	10% data
IS↑	tFID↓	vFID↓	IS↑	tFID↓	vFID↓	IS↑	tFID↓	vFID↓
BigGAN(d' = 256)	×	11.02	7.86	12.70	9.94	25.83	30.79	7.58	50.79	55.04
+LeCam	×	11.41	6.82	11.54	10.05	20.81	25.77	8.14	41.51	46.43
+DigGAN	×	11.15	8.13	13.06	9.98	16.87	21.59	9.04	23.10	27.78
+NICE	×	10.99	6.31	11.08	10.32	13.17	17.80	8.96	19.53	24.33
+LeCam+DA	✓	11.25	6.45	11.26	10.12	15.96	20.42	9.17	22.75	27.14
+LeCam+DA+KDDLGAN	✓	-	-	10.12	-	-	18.70	-	-	22.40
+LeCam+DA+NICE	✓	11.28	5.72	10.40	10.54	10.02	14.93	9.35	14.95	19.60
OmniGAN(d' = 1024)	×	12.73	8.36	13.18	10.14	40.59	44.92	6.91	60.46	64.76
+DA	✓	12.94	7.41	12.08	11.35	17.65	22.37	10.01	30.68	34.94
+ADA	✓	13.07	6.12	10.79	12.07	13.54	18.20	8.95	44.65	49.08
+NICE	×	13.77	3.83	8.61	12.57	8.68	13.53	11.97	14.53	19.22
+NICE+ADA	✓	13.82	3.78	8.59	12.75	6.28	10.92	12.04	9.32	14.18
4.2 Results on low-shot generation on StyleGAN2
Table 3 showcases the superior performance of NICE in comparison to leading methods, significantly improving baselines and achieving state-of-the-art results. NICE also surpasses KDDLGAN, despite KDDLGAN leverages the large-scale CLIP model [43]. Figure 2 shows realistic images generated by NICE under scarce data training.
Figure 2: Images generated using NICE+ADA on StyleGAN2; see Figure 14 in §J for more examples and comparisons for this low-shot generation setting.
Table 3: FID ↓ scores for unconditional image generation with StyleGAN2 on 100-shot Obama/Grumpy cat/Panda and AnimalFace-Cat/Dog datasets. † indicates using a generator pretrained on the full FFHQ dataset. ‡ means using the CLIP [43] model pre-trained on large scale data.
Method	MA	Pre-trained	100-shot	Animal Face
Obama	Grumpy Cat	Panda	Cat	Dog
StyleGAN2 [25]	×	×	80.20	48.90	34.27	71.71	131.90
StyleGAN2+SSGAN-LA [16]	×	×	79.88	38.42	28.6	78.78	109.91
StyleGAN2+NICE	×	×	24.56	18.78	8.92	25.25	46.56
ADA [22]	✓	×	45.69	26.62	12.90	40.77	56.83
DA [68]	✓	×	46.87	27.08	12.06	42.44	58.85
DigGAN [11]	✓	×	36.38	25.42	11.54	35.67	59.98
LeCam [57]	✓	×	33.16	24.93	10.16	34.18	54.88
GenCo [8]	✓	×	32.21	17.79	9.49	30.89	49.63
InsGen [63]	✓	×	32.42	22.01	9.85	33.01	44.93
MaskedGAN [17]	✓	×	33.78	20.06	8.93	-	-
FakeCLR [32]	✓	×	26.95	19.56	8.42	26.34	42.02
TransferGAN†[59]	✓	✓	39.85	29.77	17.12	49.10	65.57
LeCam+KDDLGAN‡[9]	✓	✓	29.38	19.65	8.41	31.89	50.22
ADA+NICE	✓	×	20.09	15.63	8.18	22.70	28.65
4.3 Results on FFHQ for StyleGAN2 and ImageNet for BigGAN
Results for unconditional image generation on FFHQ are in Table 4, and for conditional image generation on ImageNet  $64 \times 64$  in Table 5. We limit FFHQ to 100, 1K, 2K, 5K real images and follow [17, 9] for ImageNet setting. NICE showcases superior performance on FFHQ and ImageNet.
Table 4: FID  $\downarrow$  scores on FFHQ using StyleGAN2. ADA-Linear is introduced in [63].
Method	MA	FFHQ
100	1K	2K	5K
StyleGAN2	×	179	100.16	54	49.68
ADA	✓	85.8	21.29	15.39	10.96
ADA-Linear	✓	82	19.86	13.01	9.39
InsGen	✓	45.75	18.21	11.47	7.83
FakeCLR	✓	42.56	15.92	9.90	7.25
ADA+NICE	✓	38.42	14.57	8.85	6.48
Table 5: Comparison with and w/o NICE on ImageNet given different percentage of training data.
Method	MA	10% data	5% data	2.5% data
IS ↑	FID ↓	IS ↑	FID ↓	IS ↑	FID ↓
BigGAN	×	10.94	38.30	6.13	91.16	3.92	133.80
ADA	✓	12.67	31.89	9.44	43.21	8.54	56.83
DA	✓	12.76	32.82	9.63	56.75	8.17	63.49
MaskedGAN	✓	13.34	26.51	12.85	35.70	12.68	38.62
KDDLGAN	✓	14.14	20.32	14.06	22.35	14.65	28.79
NICE	×	14.18	21.44	13.96	24.72	13.32	31.45
ADA+NICE	✓	14.58	18.29	14.10	20.07	13.92	24.41
4.4 Analysis of NICE
Analysis of the stabilizing effect of NICE. In our  $10\%$  CIFAR-10 experiments with OmniGAN  $(d' = 256)$ , Figure 3 provides compelling evidence supporting our theory. Figure 3a shows that OmniGAN+AN and OmniGAN+NICE achieve lower weight norms than OmniGAN, validating Theorem
Figure 3: Weight and gradient norms of the discriminator on  $10\%$  CIFAR-10 with OmniGAN  $(d' = 256)$ . (a) total discriminator weight norms, (b) gradient norm at the layer before classification head, (c) gradient norm of  $f$  w.r.t. input, and (d) gradient norm of discriminator loss w.r.t. input.
(a) OmniGAN
(b) OmniGAN+AN
(c) OmniGAN+NICE
(d) tFID curves
Figure 4: The discriminator output w.r.t. real, fake and test images of (a) OmniGAN, (b) OmniGAN+AN, (c) OmniGAN+NICE, along with (d) tFID curves on  $10\%$  CIFAR-10 using OmniGAN  $(d' = 256)$ . The shaded region represents the standard deviation. Note we scale the  $y$ -axis in (b) and (c) for visual clarity. In (d), training time is doubled to evaluate the endurance of our methods under prolonged training conditions.
1 that the multiplicative noise modulation reduces the weight norm of discriminator, thus lowering the Rademacher complexity and improving generalization.
Figure 3b shows a gradient surge at the latent layer before the classification head, especially for real images in OmniGAN+AN and OmniGAN+NICE, surpassing gradients of OmniGAN. This observation aligns with our prediction in Prop. 1 that introducing noise amplifies gradients due to the maximization effect on real images when training discriminator and on fake images when training generator, causing undesired gradient issues.
Figure 3c illustrates a smaller squared gradient norm  $\| \partial f / \partial x\| _2^2$  for OmniGAN+NICE compared to OmniGAN+AN and OmniGAN, showcasing the effectiveness of consistency regularization in penalizing the gradient of discriminator, in line with the theoretical derivation in Theorem 2.
Figure 3d demonstrates that OmniGAN+NICE achieves a lower gradient norm of the discriminator loss w.r.t. the input than both OmniGAN and OmniGAN+AN, affirming ability of NICE to counteract the negative effects of large gradient norms caused by noise modulation. The resulted smaller gradients at the input validates efficiency of NICE in stabilizing training.
Analysis of the ability of NICE to enhance generalization. Figure 4 visualizes the discriminator outputs for real, fake, and test images along with tFID curves. In contrast to OmniGAN, OmniGAN+AN and OmniGAN+NICE balance discrimination and generalization [67], maintaining a steady discrepancy between real and fake images. This equilibrium ensures a smaller discrepancy between real seen images and unseen samples, effectively lowering the generation error as outlined in Eq. 3 of Lemma 1. As a result, the generalization of GANs is enhanced, leading to superior performance for image generation, as illustrated by the tFID curves in Figure 4d.
Ablation of components in NICE. We examine the impact of various components of NICE: (i) introducing adaptive noise (AN) to the discriminator, (ii) applying consistency regularization to
Table 6: Ablation studies on components (AN,  ${\mathrm{{NICE}}}_{D_r}$  ,  ${\mathrm{{NICE}}}_{D_f}$  ,  ${\mathrm{{NICE}}}_{G_f}$  ) in NICE.
Method	10% CIFAR-10	10% CIFAR-100	Obama
AN	NICEDr	NICEDf	NICEGf	IS↑	tFID↓	vFID↓	IS↑	tFID↓	vFID↓	FID↓
8.49	22.24	26.33	8.19	45.41	50.33	80.26
✓	9.16	10.14	13.80	11.22	23.76	28.34	44.68
✓	✓	9.17	8.88	12.69	11.38	20.09	24.62	32.14
✓	✓	✓	9.16	8.69	12.59	11.19	18.80	24.13	29.95
✓	✓	✓	9.23	7.45	11.25	11.62	18.73	23.42	27.93
✓	✓	✓	✓	9.26	7.23	11.08	11.50	16.91	21.56	24.56
real images  $(\mathrm{NICE}_{D_r})$ , (iii) applying consistency regularization to fake images  $(\mathrm{NICE}_{D_f})$  during discriminator training, and (iv) enforcing consistency on fake images while training the generator  $(\mathrm{NICE}_{G_f})$ . Table 6 shows our evaluations across  $10\%$  CIFAR-10/100 using OmniGAN  $(d' = 256)$  and the Obama on StyleGAN2. The optimal performance is achieved when NICE is applied to both real and fake images in the training of both the discriminator and generator.
Impact of different factors in NICE. For an in-depth analysis on how different factors in NICE contribute to its performance, readers are directed to  $\S E$ .
Comparison with alternatives for enhanced generalization and stability. Table 7 shows that NICE outperforms DA, ADA, AWD (adaptive weight decay), AN+AGP (adaptive noise with adaptive gradient penalization) and  $\mathrm{NICE}_{add}$  (with additive noise). Detailed implementation of these variants can be found in §D.2. Figure 5 shows that NICE also enhances performance with increasing network size, while ADA and DA exhibit a decline. As augmentation-based methods such as DA, ADA, and AACR, may leak augmentation cues to the generator (Figure 9 of §I), NICE mitigates this drawback.
Method	10% CIFAR-10	10% CIFAR-100
IS↑	tFID↓	vFID↓	IS↑	tFID↓	vFID↓
OmniGAN	8.49	22.24	26.33	8.14	45.41	50.33
+AWD	8.56	18.28	22.12	9.64	37.68	42.01
+AN+AGP	8.98	11.55	15.25	10.72	27.73	32.15
+NICEadd	8.64	17.94	21.59	9.34	28.59	33.02
+DA	8.84	12.90	16.67	10.16	24.50	28.96
+ADA	9.67	13.86	17.70	11.23	23.11	27.58
+AACR	9.63	10.93	14.73	11.37	21.42	25.76
+NICE	9.26	7.23	11.08	11.50	16.91	21.56
Table 7: Various methods+OmniGAN  $\left( {{d}^{\prime } = {256}}\right)$  . Figure 5: tFID  $\downarrow$  w.r.t. different network sizes  ${d}^{\prime }$  .
Table 7 & Fig. 5: Various methods on OmniGAN. Tab. 7: AWD is Adaptive weight decay. AN+AGP: Adaptive noise + adaptive gradient penalty.  $\mathrm{NICE}_{add}$ : NICE with additive noise. AACR: Adaptive augmentation-based consistency reg. Fig. 5: tFID ↓ w.r.t. different  $d'$  on 10% CIFAR-10/100.
Rationalizing Advantages of NICE. Refer to  $\S F$  and  $\S G$  for a comprehensive explanations of why NICE outperforms other alternatives, enhancing generalization and stability. Given a mild increase in the computational load ( $\S H$ ), substantial performance gains are achieved.
Our newly proposed approach for GAN training, NoIse-modulated Consistency rEgularization (NICE), improves generalization of various GAN models and their performance under limited data training. Through rigorous theoretical analysis, we have shown that NICE reduces the Rademacher complexity and penalizes the gradient of the latent features within the discriminator. Our experimental results match theoretical findings, illustrating that NICE not only improves generalization but also enhances stability of GAN training. By harnessing these two key advantages, NICE works with various network backbones and consistently outperforms existing methods. These exceptional results firmly establish NICE as a powerful solution for preventing the pervasive issue of discriminator overfitting in GANs. Limitations and border impact of our work are discussed in  $\S A$ .
We thank to Moyang Liu, whose discussions and constant encouragement significantly shaped this work. YN is supported by China Scholarship Council (202008520034). PK is supported by CSIRO's Science Digital. We are grateful for the valuable feedback provided by all anonymous reviewers.
[57] Hung-Yu Tseng, Lu Jiang, Ce Liu, Ming-Hsuan Yang, and Weilong Yang. Regularizing generative adversarial networks under limited data. In CVPR, 2021. 1, 2, 6, 8
[58] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. NeurIPS, 30, 2017. 3
[59] Yaxing Wang, Chenshen Wu, Luis Herranz, Joost Van de Weijer, Abel Gonzalez-Garcia, and Bogdan Raducanu. Transferring gans: generating images from limited data. In ECCV, pages 218-234, 2018. 6, 8
[60] Colin Wei, Sham Kakade, and Tengyu Ma. The implicit and explicit regularization effects of dropout. In ICML, pages 10181-10192. PMLR, 2020. 4
[61] Xiang Wei, Boqing Gong, Zixia Liu, Wei Lu, and Liqiang Wang. Improving the improved training of Wasserstein gans: A consistency term and its dual effect. arXiv preprint arXiv:1803.01541, 2018. 3
[62] Lijun Wu, Juntao Li, Yue Wang, Qi Meng, Tao Qin, Wei Chen, Min Zhang, Tie-Yan Liu, et al. R-drop: Regularized dropout for neural networks. NeurIPS, 34:10890-10905, 2021. 3
[63] Ceyuan Yang, Yujun Shen, Yinghao Xu, and Bolei Zhou. Data-efficient instance generation from instance discrimination. NeurIPS, 34:9378-9390, 2021. 1, 2, 6, 8, 23
[64] Han Zhang, Zizhao Zhang, Augustus Odena, and Honglak Lee. Consistency regularization for generative adversarial networks. In ICLR, 2020. 1, 2, 3
[65] Jianfeng Zhang, Zihang Jiang, Dingdong Yang, Hongyi Xu, Yichun Shi, Guoxian Song, Zhongcong Xu, Xinchao Wang, and Jiashi Feng. Avatargen: a 3d generative model for animatable human avatars. In ECCV, pages 668-685. Springer, 2023. 1, 2
[66] Linjun Zhang, Zhun Deng, Kenji Kawaguchi, Amirata Ghorbani, and James Zou. How does mixup help with robustness and generalization? In ICLR, 2021. 5
[67] Pengchuan Zhang, Qiang Liu, Dengyong Zhou, Tao Xu, and Xiaodong He. On the discrimination-generalization tradeoff in GANs. In ICLR, 2018. 3, 5, 9
[68] Shengyu Zhao, Zhijian Liu, Ji Lin, Jun-Yan Zhu, and Song Han. Differentiable augmentation for data-efficient gan training. NeurIPS, 33, 2020. 1, 2, 6, 7, 8, 20, 25
[69] Zhengli Zhao, Sameer Singh, Honglak Lee, Zizhao Zhang, Augustus Odena, and Han Zhang. Improved consistency regularization for gans. In AAAI, volume 35, pages 11033-11041, 2021. 1, 3
[70] Yong Zhong, Hong Tao Liu, Xiaodong Liu, Fan Bao, Weiran Shen, and Chongxuan Li. Deep generative modeling on limited data with regularization by nontransferable pre-trained models. In ICLR, 2023. 1, 2
[71] Peng Zhou, Lingxi Xie, Bingbing Ni, Cong Geng, and Qi Tian. Omni-gan: On the secrets of cgans and beyond. In ICCV, pages 14061-14071, 2021. 2, 6, 25
NICE: NoIse-modulated Consistency rEgularization for Data-Efficient GANs (Supplementary Material)
Yao Ni†, Piotr Koniusz*,§,†
A Broader impact and limitations