SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Yao Ni†, Piotr Koniusz*§,†
†The Australian National University §Data61♥CSIRO
†firstname_lastname@anu.edu.au
Generative Adversarial Networks (GANs) are powerful tools for image synthesis. However, they require access to vast amounts of training data, which is often costly and prohibitive. Limited data affects GANs, leading to discriminator overfitting and training instability. In this paper, we present a novel approach called NoIsemodulated Consistency rEgularization (NICE) to overcome these challenges. To this end, we introduce an adaptive multiplicative noise into the discriminator to modulate its latent features. We demonstrate the effectiveness of such a modulation in preventing discriminator overfitting by adaptively reducing the Rademacher complexity of the discriminator. However, this modulation leads to an unintended consequence of increased gradient norm, which can undermine the stability of GAN training. To mitigate this undesirable effect, we impose a constraint on the discriminator, ensuring its consistency for the same inputs under different noise modulations. The constraint effectively penalizes the first and second-order gradients of latent features, enhancing GAN stability. Experimental evidence aligns with our theoretical analysis, demonstrating the reduction of generalization error and gradient penalization of NICE. This substantiates the efficacy of NICE in reducing discriminator overfitting and improving stability of GAN training. NICE achieves state-of-the-art results on CIFAR-10, CIFAR-100, ImageNet and FFHQ datasets when trained with limited data, as well as in low-shot generation tasks.
The remarkable advancements and breakthroughs in deep learning [31] can be largely attributed to the extensive utilization of vast amounts of training data. This abundance of data has driven the progress across various domains of deep learning. Among notable recent advancements are Generative Adversarial Networks (GANs) [13], popular in industry and academia. GANs have proven their high-quality image generation abilities and achieved high generation speeds [47], establishing them as a versatile tool for a wide range of applications, such as text-to-image generation [20, 47, 55], destylization [48, 49, 50], image-to-image translation [27, 44], and 3D generation [52, 53, 65].
Despite the impressive capabilities of state-of-the-art GANs in generating diverse and high-quality images [7, 25], their effectiveness heavily relies on large volumes of training data. The acquisition of such large datasets helps GANs attain the desired adversarial equilibrium. However, under limited training data regime, GANs encounter challenges associated with discriminator overfitting and unstable training [22, 68, 57, 11, 17].
To address the aforementioned challenges, recent studies have approached the problem from three perspectives. The first perspective involves the utilization of extensive differentiable data augmentation techniques, aimed at expanding the distribution of the available data [22, 68, 17, 32, 63, 64, 69]. The second perspective leverages knowledge gained from large-scale models trained on large datasets [9, 70, 29]. While these approaches are effective, they also carry inherent risks, such as the leakage of augmentation clues [64, 22, 40] or pre-trained knowledge [30].
*The corresponding author. Code: https://github.com/MaxwellYaoNi/NICE
The third perspective focuses on the regularization of the discriminator, aiming to weaken its learning ability [11, 19] or increase the overlap between real and fake support sets [57]. In line with this perspective, our paper introduces a novel approach to improve GAN generalization. We propose to modulate the hidden features of discriminator via adaptive multiplicative noise, which strikes a balance in maintaining a certain level of discrimination ability while also regularizing the Rademacher complexity [6] of the discriminator. This reduction in Rademacher complexity, which quantifies the capacity of model to fit random variables, narrows the generation gap between the training and unseen data, resulting in enhanced GAN generalization [4, 56, 18].
Nevertheless, training the discriminator with adaptive noise unintentionally amplifies second-order gradient derivative of latent features corresponding to real images. This elevated gradient leads to abrupt gradient changes near the real sample points, potentially causing instability in the feedback to the generator. This issue aligns with the findings in works [26, 36, 56, 11], which emphasize the importance of penalizing gradients for both real and fake samples to promote convergence and stability of GAN training. To address these challenges, we propose a constraint on the discriminator that ensures consistency for the same inputs under different noise modulations. While our idea is simple, our theoretical analysis reveals that this constraint effectively penalizes the first and second-order gradients of the latent features. Consequently, the gradient provided to the generator becomes more stable, resulting in improved training stability and generalization.
Our comprehensive experiments confirm the effectiveness of NICE in penalizing gradients and reducing the generalization gap. Despite the simple design, NICE significantly improves the training stability and generalization of GANs, outperforming alternative approaches in preventing discriminator overfitting. NICE achieves superior results on challenging limited data benchmarks, including CIFAR-10/100, ImageNet, FFHQ, and low-shot image generation tasks.
i. We limit discriminator overfitting by using the adaptive multiplicative noise to modulate the latent features of the discriminator, resulting in enhanced generalization of GANs.
ii. We introduce NICE, a technique that enforces the discriminator to be consistent with the same inputs under different noise modulations, implicitly penalizing the first and second-order gradients of the latent features during GAN training, promoting the training stability.
iii. We show that NICE, both in theory and practice, effectively prevents discriminator overfitting and achieves superior performance in image generation under limited data setting.
Improving GANs. Generative Adversarial Networks [13] are powerful generative models that excel in image generation [21, 24, 25], text-to-image generation [20, 47, 55], image-to-image translation [27, 44], and 3D generation [52, 65, 53]. However, GANs commonly encounter challenges such as training instability [26], mode collapse [45], and discriminator overfitting [22]. Researchers have investigated different GAN loss functions [2, 42, 71], architectures [21, 24, 25, 23], and regularization strategies [38, 33, 14]. The  $f$ -GAN [42] generalizes GANs to  $f$ -divergences. WGAN [2] adopts the Earth-Mover distance. OmniGAN [71] extends conditional GANs to a multi-label softmax loss. StyleGANv1-3 [24, 25, 23] enhances the architecture of generator. Approaches [14, 26, 36, 56, 11] propose explicit penalty of gradient of discriminator. SNGAN [38] enforces a Lipschitz constraint on the discriminator, and approach [33] regularizes the spectral norm of the discriminator. In this study, we propose a novel regularization strategy applicable to diverse GAN loss functions and architectures, specifically tailored for training GAN models with limited data.
Image generation with limited data. Collecting data is both laborious and expensive. Consequently, generating images in limited data settings presents significant challenges, primarily due to discriminator overfitting. Previous approaches have tackled this issue through data augmentation techniques [22, 68, 17, 32, 63] and leveraging knowledge transfer from large-scale pre-trained models [9, 29, 70]. Differentiable data augmentation methods [22, 68, 17] "extend" the data distribution, while methods [63, 32] explore contrastive learning within GANs. Approaches [9, 29, 70] employ pre-trained models to guide training of discriminator. However, both GAN types suffer issues, including leakage of augmentation clues [64, 22] or pre-trained knowledge [30]. An alternative approach involves regularizing the discriminator. LeCamGAN [57] suggests reducing the output gap of discriminator between real and fake distributions. LCSAGAN [40] employs manifold techniques to project discriminator features onto manifold and decrease the capacity of discriminator. APA [19]
uses fake images as real images to increase overlap between the real and fake distributions. DigGAN [11] minimizes the gradient norm gap between real and fake images. Inspired by such an approach, we propose to use an adaptive noise to modulate the latent features of discriminator to reduce the generalization error of discriminator as a means of regularization.
Consistency regularization. CTGAN [61] introduces the enforcement of a Lipschitz constraint by ensuring the consistency of the response of discriminator to real images. CAGAN [41] and GenCo [8] enforce consistency among multiple discriminators. R-Drop [62] applies consistency regularization to transformers [58] equipped with dropout for natural language processing tasks. Augmentation-based consistency regularization GANs [64, 69] enforce consistency by considering different augmented views of the same image, although they may inadvertently leak augmentation clues to the generator [22]. Despite the effectiveness of consistency regularization demonstrated by these previous works, their success is primarily empirical, lacking theoretical analysis. In our study, we go beyond prior works by providing a theoretical analysis of consistency regularization in GANs. By delving deeper into the underlying principles, we develop a more comprehensive understanding of the mechanism behind its effectiveness.
Generalization of GANs. Arora et al. [4, 56, 67, 18] have contributed to the understanding and improvement of generalization of GANs by showing the importance of achieving adversarial equilibrium, reducing discriminator discrepancy, penalizing gradients, and bounding the generalization error. Our motivation aligns with these works in the pursuit of enhancing generalization capabilities. However, our approach implicitly and adaptively reduces the generalization gap while penalizing gradients of latent features of discriminator. Such a setting provides an efficient and effective means of preventing discriminator overfitting and improving generalization.
To boost the generalization of GAN, we start by analyzing their generalization error, bounding it with the Rademacher complexity of discriminator, and linking it with the weight norm. By incorporating multiplicative noise, we demonstrate its regularization benefits on the weight norm. However, such a strategy induces large gradient norm destabilizing training of GAN. To this end, we introduce NICE to penalize the gradients of discriminator. Finally, we provide NICE and showcase it use in GANs.
3.1 Generalization error of GANs and complexity of neural network
The primary goal of GAN is to minimize the integral probability metric [39], assuming access to infinite real and fake data during optimization, i.e., the infinite real and generated distributions  $(\mu, \nu)$  as discussed in [67]. In practice, we often have limited access to a finite dataset  $\hat{\mu}_n$  of size  $n$ . Consequently, our optimization is restricted to the empirical loss:
$$
\inf  _ {\nu \in \mathcal {G}} \left\{d _ {\mathcal {H}} \left(\hat {\mu} _ {n}, \nu\right) := \sup  _ {h \in \mathcal {H}} \left\{\mathbb {E} _ {\boldsymbol {x} \sim \hat {\mu} _ {n}} [ h (\boldsymbol {x}) ] - \mathbb {E} _ {\tilde {\boldsymbol {x}} \sim \nu} [ h (\tilde {\boldsymbol {x}}) ] \right\} \right\}. \tag {1}
$$
Function sets of discriminator and generator,  $\mathcal{H}$  and  $\mathcal{G}$ , are typically parameterized in GAN as neural network classes  $\mathcal{H}_{\mathrm{nn}} = \{h(\pmb{x};\pmb{\theta}_d):\pmb{\theta}_d\in \Theta_d\}$  and  $\mathcal{G}_{\mathrm{nn}} = \{g(z;\pmb{\theta}_g):\pmb{\theta}_g\in \Theta_g\}$  where  $z\sim p_z$  serves as the random noise input to the generator. The associated term  $d_{\mathcal{H}_{\mathrm{nn}}}$  is referred to as the neural network distance [4]. The discriminator network  $D\coloneqq \varphi \circ f$  consists of a real/fake prediction head  $\varphi$  and a feature extractor  $f$ . As the loss function  $\phi (\cdot)$  varies across tasks, architectures or choice of divergence type, we compose it with  $D$  [2, 67, 4] to simplify the analysis and notation, i.e.,  $h(\cdot)\coloneqq \phi (D(\cdot))$ . Thus, the alternative optimization of discriminator and generator becomes:
$$
\left\{ \begin{array}{l} L _ {D} = \min  _ {\boldsymbol {\theta} _ {d}} \mathbb {E} _ {\tilde {\boldsymbol {x}} \sim \nu_ {n}} [ h (\tilde {\boldsymbol {x}}; \boldsymbol {\theta} _ {d}) ] - \mathbb {E} _ {\boldsymbol {x} \sim \hat {\mu} _ {n}} [ h (\boldsymbol {x}; \boldsymbol {\theta} _ {d}) ], \\ L _ {G} = \min  _ {\boldsymbol {\theta} _ {g}} - \mathbb {E} _ {\boldsymbol {z} \sim p _ {\boldsymbol {z}}} [ h (g (\boldsymbol {z}; \boldsymbol {\theta} _ {g})) ], \end{array} \right. \tag {2}
$$
where we assume  $\nu_{n}$  minimizes  $d_{\mathcal{H}}(\hat{\mu}_n,\nu)$  up to precision  $\epsilon \geq 0$ , meaning that  $d_{\mathcal{H}}(\hat{\mu}_n,\nu_n)\leq \inf_{\nu \in \mathcal{G}}d_{\mathcal{H}}(\hat{\mu}_n,\nu) + \epsilon$ . As we are interested in how close the generator distribution  $\nu_{n}$  is to the unknown infinite distribution  $\mu$ , we refer to the lemma [67] on the generalization error of GAN:
Lemma 1 (Theorem 3.1 of [67]) Assume that the discriminator set  $\mathcal{H}$  is even ( $h \in \mathcal{H}$  implies  $-h \in \mathcal{H}$ ) and all discriminators are bounded by  $\| h \|_{\infty} \leq \Delta$ . Let  $\hat{\mu}_n$  be an empirical measure of an i.i.d. sample of size  $n$  drawn from  $\mu$ . Assume  $d_{\mathcal{H}}(\hat{\mu}_n, \nu_n) - \inf_{\nu \in \mathcal{G}} d_{\mathcal{H}}(\hat{\mu}_n, \nu) \leq \epsilon$ . Then with probability at least  $1 - \delta$ , we have:
$$
d _ {\mathcal {H}} (\mu , \nu_ {n}) - \inf  _ {\nu \in \mathcal {G}} d _ {\mathcal {H}} (\mu , \nu) \leq 2 \sup  _ {h \in \mathcal {H}} \left| \mathbb {E} _ {\mu} [ h ] - \mathbb {E} _ {\hat {\mu} _ {n}} [ h ] \right| + \epsilon \leq 2 R _ {n} ^ {(\mu)} (\mathcal {H}) + 2 \Delta \sqrt {\frac {2 \log (1 / \delta)}{n}} + \epsilon , \tag {3}
$$
where the Rademacher complexity [6],  $R_{n}^{(\mu)}(\mathcal{H}) \coloneqq \mathbb{E}\big[\sup_{h\in \mathcal{H}}\frac{2}{n}\sum_{i}\tau^{(i)}h(\pmb{x}^{(i)})\big]$ , measures how well the function  $h \in \mathcal{H}$  fits the Rademacher random variable  $\tau^{(i)}$  with  $prob(\tau^{(i)} = 1) = prob(\tau^{(i)} = -1) = \frac{1}{2}$  given samples  $\pmb{x}^{(i)} \sim \hat{\mu}_n$ .
Lemma 1 provides a crucial insight that one can assess the generalization error of GAN by comparing the output discrepancy of discriminator between training data and unseen data, and such an error is influenced by the Rademacher complexity of the discriminator. To enhance the performance of generator while reducing the generalization error, we have two possibilities: 1) increase the quantity  $n$  of real data, which provides a stronger foundation for training a better generator; 2) reduce the Rademacher complexity of the discriminator. However, this reduction must be carefully controlled, as an overly simplified discriminator may struggle to effectively distinguish real and fake data.
To manage the Rademacher complexity of the discriminator, we leverage a theorem from approach [5] to establish an upper bound on the Rademacher complexity of the neural network.
Lemma 2 (Eq. 1.2 in [5], Theorem 5.20 in [35]) Consider a fully-connected neural network  $v_{\theta}(\boldsymbol{x}) = \boldsymbol{W}_{t}\sigma(\boldsymbol{W}_{t-1}\sigma(\dots\sigma(\boldsymbol{W}_{1}\boldsymbol{x})\dots))$  where  $\boldsymbol{W}_{i}$  are linear weights at  $i$ -th layer and  $\sigma$  is a 1-Lipschitz activation function. Suppose that  $\forall i \in \{1,\dots,n\}$ ,  $\| \boldsymbol{x}^{(i)}\|_{2} \leq q$ . Let  $\| \boldsymbol{W}_{i}\|_{lip}$  be the Lipschitz constant of  $\boldsymbol{W}_{i}$  and  $\| \boldsymbol{W}_{i}^{T}\|_{2,1}$  be the sum of the  $l_{2}$  norm of columns in  $\boldsymbol{W}_{i}$ . Let  $\mathcal{V} = \{v_{\theta} : \| \boldsymbol{W}_{i}\|_{lip} \leq k_{i}, \| \boldsymbol{W}_{i}^{T}\|_{2,1} \leq b_{i}\}$ , we have the Rademacher complexity:
$$
R _ {n} ^ {(\mu)} (\mathcal {V}) \leq \frac {q}{\sqrt {n}} \cdot \left(\prod_ {i = 1} ^ {t} k _ {i}\right) \cdot \left(\sum_ {i = 1} ^ {t} \frac {b _ {i} ^ {2 / 3}}{k _ {i} ^ {2 / 3}}\right) ^ {3 / 2}. \tag {4}
$$
Lemma 2 links the Rademacher complexity of a neural network to the Lipschitz constant and the (2,1)-norm of its weights, providing essential insights for controlling the complexity of discriminator.
3.2 Improving generalization by feature modulation with multiplicative noise
Taking into account Lemmas 1 and 2, one can effectively manage the generalization error of GAN by controlling the Rademacher complexity of the discriminator. Such a control requires managing the Lipschitz constant and the norm of the weights of discriminator. Typically, controlling the Lipschitz constant can be achieved by the spectral normalization [38] or the gradient penalty [14]. In contrast, we propose a novel approach: regularizing the norm of the weights by modulating the latent features of the discriminator with multiplicative noise.
We build on the analysis (Prop. 4 in Appendix of [3]) of Dropout regularization [54] by exploring the regularization effect of the Gaussian multiplicative noise within the context of deep regression.
Theorem 1 (Regularization by the Gaussian multiplicative noise in deep regression) Consider a regression task on a two-layer neural network. Let  $\mathcal{X} \subseteq \mathbb{R}^{d_0}$  and  $\mathcal{Y} \subseteq [-1,1]^{d_2}$  be the input and output spaces, and  $\mathcal{D} = \mathcal{X} \times \mathcal{Y}$ . Let  $n$  examples  $\{(x^{(i)},y^{(i)})\}_{i=1}^n \sim \mathcal{D}^n$ . Let  $f_w: \mathcal{X} \to \mathcal{Y}$  be parameterized by  $\{w:W_1 \in \mathbb{R}^{d_1 \times d_0}, W_2 \in \mathbb{R}^{d_2 \times d_1}\}$  and  $f_w(x,z) = W_2(z \odot \sigma(W_1x))$  where  $\sigma$  is 1-Lipschitz activation function,  $\odot$  denotes element-wise multiplication and  $z \sim \mathcal{N}(1,\beta^2 I^{d_1})$ . Let  $\pmb{a} = \sigma(\pmb{W}_1\pmb{x})$ ,  $\hat{\pmb{a}} = \hat{\mathbb{E}}_i[\pmb{a}^{(i)} \odot \pmb{a}^{(i)}]$ ,  $\hat{a}_k \geq 0$  denotes the  $k$ -th element of  $\hat{\pmb{a}}$ . Let  $\|W_2\|_{2,1} = \sum_k \|w_k\|_2$  where  $w_k$  is the  $k$ -th column vector in  $W_2$ . The regression task with the  $l_2$  loss leads to the weight regularization as:
$$
\hat {L} _ {\text {n o i s e}} (w) := \hat {\mathbb {E}} _ {i} \mathbb {E} _ {\boldsymbol {z}} \left[ \| \boldsymbol {y} ^ {(i)} - \boldsymbol {W} _ {2} (\boldsymbol {z} \odot \boldsymbol {a} ^ {(i)}) \| _ {2} ^ {2} \right] = \hat {\mathbb {E}} _ {i} \left[ \| \boldsymbol {y} ^ {(i)} - \boldsymbol {W} _ {2} \boldsymbol {a} ^ {(i)} \| _ {2} ^ {2} \right] + \beta^ {2} \sum_ {k} \hat {a} _ {k} \| \boldsymbol {w} _ {k} \| _ {2} ^ {2}. \tag {5}
$$
The proof can be found in §C.1. Theorem 1 reveals that modulation with Gaussian multiplicative noise in the regression task implicitly regularizes the internal weight norm, with the regularization strength determined by the variance of the noise  $\beta^2$  and the magnitude of the features.
Despite analysis in a simplified two-layer system, the implicit weight regularization by the noise modulation applies to multi-layer and convolutional neural networks, which can be expressed as a combination of two-layer nets, and convolutional layers are a type of linear layer [37, 60, 3]. Assuming latent features in deep neural networks follow a Gaussian distribution [34], we can equate the Bernoulli noise modulation (Dropout) with  $Z \sim \mathcal{B}(\beta)$ , whose values are set to be  $1 / (1 - \beta)$  with probability  $1 - \beta$  and 0 otherwise, to the Gaussian multiplicative noise modulation in subsequent layers, making the regularization effects from Theorem 1 applicable to the Bernoulli noise as well.
Theorem 1 illustrates that one can modulate the latent features in the discriminator by the multiplicative noise to adaptively regularize the norm of its weights, leading to the reduced Rademacher complexity in Lemma 2 and improved GANs generalization in Lemma 1.
3.3 Consistency regularization
Although the noise modulation can reduce the Rademacher complexity, incorporating the noise increases the gradient norms of latent features and the inputs to the discriminator, making training of GAN unstable [56] and difficult to converge [36]. To understand the gradient-related challenges in the noise-modulated discriminator, we adopt the Taylor expansion with a focus on the pivotal first- and second-order terms, following standard practice [1, 12, 66].
Proposition 1 Define  $f \coloneqq f_{t} \circ \ldots \circ f_{2}$  as the feature extractor of the discriminator from the second layer onward. Let  $\tilde{\pmb{a}} = f_{1}(\tilde{\pmb{x}})$  and  $\pmb{a} = f_{1}(\pmb{x})$  be the latent features of the first layer for fake and real images, respectively. The discriminator is defined as  $h(\cdot) = \phi(\varphi(f(\cdot))$ , with  $\varphi$  as the prediction head and  $\phi$  as the loss function. Introduce a multiplicative noise  $z \sim \mathcal{N}(1, \beta^{2}\pmb{I}^{d^{\prime}})$  modulating solely the first layer. Let  $H_{kk}^{(h)}(\pmb{a})$  be the  $k$ -th diagonal entry of the Hessian matrix of  $h$  at  $\pmb{a}$  and  $a_{k}$  be the  $k$ -th element of  $\pmb{a}$ . Applying Taylor expansion to  $h(\cdot)$ , the GAN loss can be expressed as follows:
$$
\begin{array}{l} \min  _ {\boldsymbol {\theta} _ {d}} L _ {D} ^ {A N} := \mathbb {E} _ {\tilde {\boldsymbol {a}}} \mathbb {E} _ {\boldsymbol {z}} \left[ h (\boldsymbol {z} \odot \tilde {\boldsymbol {a}}) \right] - \mathbb {E} _ {\boldsymbol {a}} \mathbb {E} _ {\boldsymbol {z}} \left[ h (\boldsymbol {z} \odot \boldsymbol {a}) \right] \\ \approx \mathbb {E} _ {\tilde {\boldsymbol {a}}} [ h (\tilde {\boldsymbol {a}}) ] - \mathbb {E} _ {\boldsymbol {a}} [ h (\boldsymbol {a}) ] + \frac {\beta^ {2}}{2} \left(\mathbb {E} _ {\tilde {\boldsymbol {a}}} \left[ \sum_ {k} \tilde {a} _ {k} ^ {2} H _ {k k} ^ {(h)} (\tilde {\boldsymbol {a}}) \right] - \mathbb {E} _ {\boldsymbol {a}} \left[ \sum_ {k} a _ {k} ^ {2} H _ {k k} ^ {(h)} (\boldsymbol {a}) \right]\right), \tag {6} \\ \end{array}
$$
$$
\min  _ {\boldsymbol {\theta} _ {g}} L _ {G} ^ {A N} := - \mathbb {E} _ {\boldsymbol {z}} \mathbb {E} _ {\tilde {\boldsymbol {a}}} [ h (\boldsymbol {z} \odot \tilde {\boldsymbol {a}}) ] \approx - \mathbb {E} _ {\tilde {\boldsymbol {a}}} [ h (\tilde {\boldsymbol {a}}) ] - \frac {\beta^ {2}}{2} \mathbb {E} _ {\tilde {\boldsymbol {a}}} [ \sum_ {k} \tilde {a} _ {k} ^ {2} H _ {k k} ^ {(h)} (\tilde {\boldsymbol {a}}) ]. \tag {7}
$$
Prop. 1 (the proof is in §C.2) shows that introducing the noise modulation during the training of the discriminator amplifies the value of diagonal entry of Hessian matrix at real images, causing abrupt changes in their gradient and potentially an increase of the gradient norm of real images. Eq. 7 further shows that as the generator is optimized to produce images resembling real images, the gradient norm of the fake images also experiences amplification. Consequently, gradient norms for both real and fake images increase. Empirically, we observe an increase in the gradient norm of latent features before the prediction head, resulting in a larger gradient norm of samples.
To mitigate this problem, we propose a consistency regularization for  $f$ , promoting invariance of  $f$  to the same input under varying noise modulations. For simplicity, we take  $f: \mathbb{R}^{d'} \to \mathbb{R}$ , noting that this analysis readily extends to the consistency loss for  $f$  with a multidimensional output.