Improving the Training of the GANs with Limited Data via Dual Adaptive Noise Injection

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-08 12:15:27

Abstract: Recently, many studies have highlighted that training Generative Adversarial Networks (GANs) with limited data suffers from the overfitting of the discriminator ($D$). Existing studies mitigate the overfitting of $D$ by employing data augmentation, model regularization, or pre-trained models. Despite the success of existing methods in training GANs with limited data, noise injection is another plausible, complementary, yet not well-explored approach to alleviate the overfitting of $D$ issue. In this paper, we propose a simple yet effective method called Dual Adaptive Noise Injection (DANI), to further improve the training of GANs with limited data. Specifically, DANI consists of two adaptive strategies: adaptive injection probability and adaptive noise strength. For the adaptive injection probability, Gaussian noise is injected into both real and fake images for generator ($G$) and $D$ with a probability $p$, respectively, where the probability $p$ is controlled by the overfitting degree of $D$. For the adaptive noise strength, the Gaussian noise is produced by applying the adaptive forward diffusion process to both real and fake images, respectively. As a result, DANI can effectively increase the overlap between the distributions of real and fake data during training, thus alleviating the overfitting of $D$ issue. Extensive experiments on several commonly-used datasets with both StyleGAN2 and FastGAN backbones demonstrate that DANI can further improve the training of GANs with limited data and achieve state-of-the-art results compared with other methods.

Main paper: 
 introduction: 
 In recent years, Generative Adversarial Networks (GANs) [14] have achieved great success in generating contents, e.g., images [24, 27, 28, 59] , videos [8, 17, 41] , text [18, 53] and audio [29] , for social media. These generated contents can be applied in various multimedia applications, such as talking face [55, 60] . However, the success of GANs relies on the availability of a large amount of data. Collecting and cleaning these large datasets can be expensive, time-consuming, and even infeasible. Consequently, training GANs with limited data has received great attention.Recently, several approaches [7, 26, 65] have demonstrated that training GANs with limited data suffers from the overfitting of the discriminator (𝐷), i.e., 𝐷 becomes overly confident in distinguishing between real and fake data. To address this, existing methods employ various strategies such as data augmentation [7, 9, 23, 26, 65] , model regularization [12, 33, 51] , or pre-trained models [30, 45] . Despite the success of existing methods, noise injection [1, 4] is another plausible, complementary, yet not well-explored approach to alleviate the overfitting of 𝐷. Recent noise injection methods in GANs [2, 13, 22, 43, 47] have already shown their effectiveness in improving the training of GANs with a large amount of data. However, as stated in ADA [26] , directly applying noise injection to GANs with limited data suffers from the leaking problem [26, 63, 64, 66] , i.e., "noise augmentation leads to noisy results, even if there is none in the dataset", which can highly influence the performance of GANs training under limited data settings.In this paper, we propose a novel noise injection method for GANs with limited data, called Dual Adaptive Noise Injection (DANI). Specifically, DANI consists of two adaptive strategies, i.e., adaptive injection probability and adaptive noise strength. For the adaptive injection probability, we inject Gaussian noise into both real and fake images for 𝐺 and 𝐷 with a probability 𝑝, where the 𝑝 is controlled by the overfitting degree of 𝐷 adaptively. For the adaptive noise strength, we apply the adaptive forward diffusion process [20] to both real and fake images, respectively, to produce the Gaussian noise. Consequently, both adaptive strategies in DANI can effectively prevent the leaking problem caused by the noise injection. Furthermore, DANI can effectively alleviate the overfitting of 𝐷 problem, i.e., 𝐷 becomes overly confident in distinguishing the real and fake data. Specifically, we prove that DANI effectively increases the overlap between the supports of real and fake data distributions in GANs during training (see Theorem 3 in §3.2), as illustrated in Figure 1 . Then, based on the conclusion in ADA [26] , the increased overlap between the supports of real and fake data distributions provided by DANI strongly indicates that DANI mitigates 𝐷 becoming overly confident in distinguishing real from fake data. This demonstrates that DANI alleviates the overfitting of 𝐷 problem. As the overfitting of 𝐷 increases, 𝐷 becomes increasingly confident in distinguishing between real and fake data, leading to a decrease in the overlap between the supports of real and fake data distributions observed by 𝐷. Finally, when 𝐷 becomes overly confident, i.e., when 𝐷 reaches optimality as pointed out in [2, 14] , the overlapping parts disappear or can be ignored. Bottom: The real and fake data distributions with DANI observed by 𝐷 during training. Applying DANI for both real and fake samples can effectively provide more overlapping parts between the supports of real and fake distributions during training, thus resulting in better performance [2, 54] .To sum up, the main contributions of this paper are as follows:(1) We propose a novel method called Dual Adaptive Noise Injection (DANI) for training GANs with limited data. DANI can effectively alleviate the overfitting of 𝐷 and avoid the leaking problem. (2) We provide the theoretical analysis of DANI for training GANs with limited data, proving its convergence and rationality on both StyleGAN2 [28] and FastGAN [33] backbones. Furthermore, we prove that applying DANI to GANs can provide more overlapping parts between the supports of the real and fake data distributions. (3) Extensive experiments on several commonly used datasets demonstrate that the proposed DANI can further improve the training of GANs with limited data and achieve state-of-theart performance compared with other methods. Additionally, DANI achieves these benefits with only a negligible increase in computational cost. 
preliminaries 2.1 generative adversarial networks (gans): 
 Generative adversarial networks (GANs) [14] is a form of generative models [40, 52] in which a game is played between two players: A generator (𝐺) and a discriminator (𝐷). Specifically, 𝐺 aims to produce realistic-looking samples with some given noise 𝑧 to deceive 𝐷, while 𝐷 aims to distinguish whether the input sample is from the generator's output or real data. The objective function of GANs can be formulated as follows:EQUATIONThe parameters of 𝐺 and 𝐷 are updated iteratively with gradient descent methods. GANs are known to suffer from training instability, yielding poor quality and low diversity of generated images. To stabilize GANs training as well as improve the quality and diversity of generated images, various approaches have been proposed, focusing on more sophisticated network architectures [6, 38, 39, 42, 58, 61] , more stable objective functions [3, 15, 16, 36, 44] , and better training strategies [10, 24, 31, 34, 35, 56, 62] to achieve photorealistic results.Training GANs with limited data using data augmentation. Recently, data augmentation (DA) has played an important role in improving performance when training GANs with limited data. Many studies [7, 26, 50, 65] apply DA to both real and fake images for 𝐷 and 𝐺 to guide the discriminator avoiding overfitting. The most popular methods are Diff-Augment [65] , ADA [26] , and Diffusion-GAN [54] . Diff-Augment applies the DAs to both real and fake images for the 𝐷 and the 𝐺 without manipulating the target distribution. ADA is similar to Diff-Augment, while it further devises an adaptive approach that controls the strength of data augmentations. Diffusion-GAN applies the adaptive forward diffusion process [20] as DA to both real and fake images. Training GANs with limited data using model regularization. In recent years, several approaches have proposed novel model regularization methods to alleviate the overfitting of 𝐷, thus improving the training of GANs with limited data. The most significant methods include 𝑅 𝐿𝐶 [51] and DigGAN [12] . Specifically, 𝑅 𝐿𝐶 applies a regularized objective function for the discriminator to improve the training of GANs with limited data. DigGAN applies discriminator gap regularization to alleviate the overfitting of 𝐷. Training GANs with limited data using a pre-trained network. Recently, employing pre-trained network has been shown to For the adaptive injection probability strategy, the Gaussian noise is adaptively injected into both the real and fake images for 𝐺 and 𝐷 with a probability 𝑝, where 𝑝 is controlled by the overfitting degree 𝜂. For the adaptive noise strength strategy, the Gaussian noise is produced by applying an adaptive forward diffusion process to both the real and fake images for 𝐷 and 𝐺, controlled by Eq.( 7). In DANI, the parameters of 𝐷 are optimized using noise-injected real and fake samples (all paths), and the parameters of 𝐺 are optimized using noise-injected fake samples (red paths).significantly improve GANs training when using limited data. Many approaches [30, 45] apply a pre-trained network to the discriminator to extract the features from both the real and fake images to improve the training of GANs with limited data. One of the pioneering methods is Projected GAN. Specifically, Projected GAN applies a pre-trained network, i.e., EfficientNet-lite1 [48] , to both the real and fake images to extract projected features in image space. Then, Projected GAN applies a multi-scale discriminator architecture which can better utilize deeper layers of the pre-trained network. As a result, Projected GAN can effectively utilize the obtained projected features to benefit the training of GANs with limited data. The loss function of the Projected GAN can be formulated as:EQUATIONwhere {𝑃 𝑙 } is the set of feature projectors which map real and generated images to the discriminator's input space, {𝐷 𝑙 } is a set of independent discriminators operating on different feature projections and L = {1, ..., 𝑛} is index set for the different projectors.In recent years, Diffusion-based generative models [11, 20, 25] have shown their superiority in image-generation tasks. Diffusion-based generative models assume that 𝑝 𝜃 (𝑥 0 ) := ∫ 𝑝 𝜃 (𝑥 0:𝑇 )𝑑𝑥 1:𝑇 , where 𝑥 1 , ..., 𝑥 𝑇 are latent variables of the same dimensionality as the data 𝑥 0 ∼ 𝑝 (𝑥 0 ). There is a forward diffusion chain that gradually adds noise to the data 𝑥 0 ∼ 𝑞(𝑥 0 ) with pre-defined variance schedule 𝛽 𝑡 and variance 𝜎 2 in 𝑇 steps:EQUATIONA notable property is that 𝑥 𝑡 at an arbitrary time-step 𝑡 can be sampled in closed form as:EQUATIONwhere 𝛼 𝑡 := 1 -𝛽 𝑡 , ᾱ𝑡 := 𝑡 1 𝛼 𝑠 . A variational lower bound [5] is then used to optimize the reverse diffusion chain as:𝑝 𝜃 (𝑥 0:𝑇 ) := 𝑁 (𝑥 𝑇 ; 0, 𝜎 2 𝐼 ) 𝑇 𝑡 =1 𝑝 𝜃 (𝑥 𝑡 -1 |𝑥 𝑡 ).In recent years, several studies have injected noise [2, 22, 43, 47] into the discriminator's input to improve the training of GANs. Specifically, they add random Gaussian noise to the discriminator's input in GANs to achieve better performance. The technique can be formulated as follows:EQUATION)where 𝜆 is the parameter to scale the noise, 𝑐 represents the original data, and 𝑐 ′ represents the data obtained after noise injection. 
methodology 3.1 dual adaptive noise injection (dani): 
 Despite the success of existing methods, noise injection is another plausible, complementary, yet not well-explored approach to alleviate the overfitting of 𝐷 issue. To effectively utilize noise injection to improve GANs training with limited data, in this paper, we propose a novel noise injection method for training GANs with limited data, called Dual Adaptive Noise Injection (DANI). The overview of applying Dual Adaptive Noise Injection (DANI) to the GANs with limited data is shown in Figure 2 . DANI consists of two adaptive strategies, i.e., adaptive injection probability and adaptive noise strength.For the adaptive injection probability, we perform the noise injection based on a form of probability, 𝑝 ∈ [0, 1) for each real and fake sample. It is intuitive to let 𝑝 be adjusted adaptively based on the degree of overfitting without manual tuning regardless of data scales and properties. To achieve this goal, following ADA [26] and APA [23] , we apply an overfitting heuristic 𝜂 that quantifies the degree of 𝐷's overfitting as follows:EQUATIONwhere sign() indicates the sign function that returns +1 for a nonnegative input; -1, otherwise; 𝐷 𝑟𝑒𝑎𝑙 is the discriminator output on the real training data. Then, we set a threshold value 𝑑 target and follow the same step as in ADA [26] and APA [23] for using 𝜂 to adjust 𝑝. Specifically, we initialize 𝑝 to zero and adjust its value once every four minibatches based on the chosen overfitting heuristic 𝜂. If 𝜂 signifies too much/little overfitting regarding 𝑑 target (i.e., larger/smaller than 𝑑 target ), 𝑝 will be increased/decreased by one fixed-step. Using this step size, 𝑝 can increase from zero to one in 500k images shown to 𝐷. We adjust 𝑝 once every four iterations and clamp 𝑝 from below to zero after each adjustment. In this way, the noise injection probability can be adaptively controlled based on the degree of overfitting.For the adaptive noise strength, we follow Diffusion-GAN, aiming for the strength of the injected noise to be effectively controlled by the forward diffusion process. To this end, we use the same 𝜂 and 𝑑 target from the adaptive injection probability strategy to control the time-step 𝑡 in the forward diffusion process as follows:EQUATIONwhere 𝐶 is a constant. We update 𝑡 every four minibatches based on the chosen overfitting heuristic 𝜂. In this case, the strength of the noise in DANI can be controlled with an adaptive forward diffusion process.To sum up, DANI can be formulated as:EQUATIONwhere 𝐴 𝑡 is the adaptive forward diffusion process, 𝑥 represents the original data, and x represents the data obtained after noise injection. Following Diffusion-GAN [54] , we select the priority distribution for the forward diffusion process and the time step 𝑡 is controlled adaptively following Eq.( 7). According to [54] , the forward diffusion process variances 𝛽 𝑡 in Eq.( 3) are increased linearly from 𝛽 1 = 10 -4 to 𝛽 𝑇 = 0.02.Let 𝑃 𝑅 be the distribution of real data and 𝑃 𝐺 be the distribution of generated data. For the sample 𝑥, 𝐷 (𝑥) represents the estimated probability of sample 𝑥 being real or fake. To examine the rationality of DANI, we analyze it in a non-parametric setting, where a model is represented with infinite capacity by exploring its convergence in the space of probability density functions. Ideally, the estimated probability distribution 𝑃 𝐺 should perfectly model the distribution 𝑃 𝑅 without bias if given enough capability and training time.Since the probability 𝑝 is adaptively adjusted, to facilitate this theoretical analysis, we assume that 𝛼 is the expected strength, which approximates the effect of dynamic adjustment of distribution during the entire training procedure. Since 𝑝 ∈ [0, 1), we have 0 ≤ 𝛼 < 𝑝 𝑚𝑎𝑥 < 1, where 𝑝 𝑚𝑎𝑥 is the maximum probability throughout training. Based on Eq.( 1), the saturating loss function of DANI in GANs is shown as follows: minEQUATIONwhere 𝐴 𝑡 is the adaptive forward diffusion process, where the forward diffusion process is based on Eq.( 3) and the time step 𝑡 is controlled adaptively following Eq.( 7). Ideally, for ∀𝑥, 𝐴 𝑡 (𝑥) ∼ 𝑁 0, 𝜎 2 , it demonstrates that 𝐴 𝑡 (𝑥) can be directly regarded as Gaussian noise with different strengths during training. As a result, DANI in the GANs can be theoretically regarded as adding a set of Gaussian noise during training. In this case, we follow [22] to analyze the convergence of Eq. ( 9). For ∀𝑥, let 𝜀 = 𝛼𝐴 𝑡 (𝑥) represent the random Gaussian noise. Then, Eq.( 9) can be formulated as:EQUATIONaccording to [22] , we introduce a set 𝑆 of probability density functions in Eq. (10) , where 𝜀 ∼ 𝑃 𝜀 ∈ 𝑆. Then, based on [22, 45] , the optimal discriminator in Eq.( 10) can be formulated as:EQUATIONThen, according to [22] , the optimization of 𝐺 under optimal discriminator 𝐷 * (𝑥) for Eq.( 10) can be formulated as:EQUATIONwhere JSD is the Jensen-Shannon divergence. Following Theorem 1 in [22] , we can simplify Eq.( 12) as:EQUATIONwhere 𝑃 𝜀 is a zero-mean Gaussian function produced by DANI with a positive definite covariance Σ. Then, based on the proofs of Theorem 1 in [22] , we can conclude that the optimization of Eq.( 13) is equal to the optimization between the distributions 𝑃 𝐺 and 𝑃 𝑅 , which is the same as in the original GAN [14] , indicating that the proposed DANI does not influence the convergence. Next, we theoretically analyze the widely-used GANs with limited data backbones, i.e., StyleGAN2 and FastGAN with non-saturating loss and hinge loss, respectively.StyleGAN2. For the StyleGAN2, the non-saturating loss function can be formulated as:EQUATIONBased on the theoretical analysis of the saturating loss above, the non-saturating loss with DANI can be formulated as: Then, according to the original GAN and Theorem 2.5 as in [2] , optimizing the non-saturating loss in GANs is equivalent to minimizing the KL-2JS divergence item (under optimal 𝐷 * ). In this case, we follow [22] to consider images of 𝑚 × 𝑛 pixels and with values in a compact domain Ω ⊂ R 𝑚×𝑛 , then 𝑃 𝑅 can be represented as 𝐿 2 (Ω). In this case, the optimization of 𝐺 under Eq.( 15) can be formulated as Theorem 1 as follows. Theorem 1. Let us choose 𝑆 such that the optimization of 𝐺 in Eq.( 15) can be written asEQUATIONEQUATIONwhere 𝑃 𝜀 is a zero-mean Gaussian function produced by DANI with a positive definite covariance Σ. Let us also assume that the domain of 𝑃 𝐺 is restricted to Ω (and thus 𝑃 𝐺 ∈ 𝐿 2 (Ω)). Then, the global optimum of Eq.( 15) is 𝑃 𝐺 (𝑥) = 𝑃 𝑅 (𝑥), ∀𝑥 ∈ Ω.Proof. See supplementary materials.For the FastGAN, the hinge loss function can be formulated as:EQUATIONBased on the theoretical analysis of the saturating loss above, the hinge loss with DANI can be formulated as:EQUATIONAccording to [32, 38, 49] , optimizing the hinge loss is equivalent to minimizing the so-called reverse KL divergence item (under optimal 𝐷 * ). Therefore, following the definition in §3.2.1, the optimization of 𝐺 under Eq.( 18) can be formulated as Theorem 2 as follows. Theorem 2. Let us choose 𝑆 such that the optimization of 𝐺 in Eq.( 18) can be written asEQUATIONwhere 𝑃 𝜀 is a zero-mean Gaussian function produced by DANI with a positive definite covariance Σ. Let us also assume that the domain of 𝑃 𝐺 is restricted to Ω (and thus 𝑃 𝐺 ∈ 𝐿 2 (Ω)). Then, the global optimum of Eq.( 18) is 𝑃 𝐺 (𝑥) = 𝑃 𝑅 (𝑥), ∀𝑥 ∈ Ω.Proof. See supplementary materials.and Fast-GAN. Both StyleGAN2 and FastGAN apply regularization to 𝐷 in their loss function to enhance the training of GANs. To demonstrate the rigour and reasonableness of our theory above, we point out that regularization in both StyleGAN2 and FastGAN aims to avoid 𝐷 becoming overly confident. Therefore, regularization does not influence the soundness of our theoretical insights under optimal 𝐷 * . Additionally, based on the theory in [37] , applying regularization in GANs can still be convergent when initialized sufficiently close to the equilibrium point, which demonstrates that our theory analysis is reasonable. Next, we prove the proposed DANI can increase the overlapping parts between the supports of 𝑃 𝐺 and 𝑃 𝑅 , as shown in Theorem 3. Theorem 3. For two supports of the distributions 𝑃 and 𝑄 consisting of overlapping parts, if there exists one sample𝑥 ∈ 𝑃 ∩ 𝑄, ∀ function 𝐴 𝑡 , [𝑥 + 𝑝𝐴 𝑡 (𝑥)] ∈ [𝑃 + 𝑝𝐴 𝑡 (𝑃)] ∩ [𝑄 + 𝑝𝐴(𝑄)]. Proof. [𝑥 ∈ 𝑃 ∩𝑄] ⇒ [𝑥 ∈ 𝑃 and 𝑥 ∈ 𝑄] ⇒ [𝑝𝐴 𝑡 (𝑥) ∈ 𝑝𝐴 𝑡 (𝑃) and 𝑝𝐴 𝑡 (𝑥) ∈ 𝑝𝐴 𝑡 (𝑄)] ⇒ [𝑥 +𝑝𝐴 𝑡 (𝑥)] ∈ [𝑃 +𝑝𝐴 𝑡 (𝑃)]∩[𝑄 +𝑝𝐴 𝑡 (𝑄)]].The 𝑃 and 𝑄 in Theorem 3 can be replaced as 𝑃 𝐺 and 𝑃 𝑅 for the GANs, and the function 𝐴 𝑡 is the same as the adaptive forward diffusion function 𝐴 𝑡 in DANI. Theorem 3 shows that with the function 𝐴 𝑡 , if the sample 𝑥 is in the overlapping parts between the supports of 𝑃 𝐺 and 𝑃 𝑅 , 𝑥 + 𝑝𝐴 𝑡 (𝑥) can still be in the overlapping parts between the supports of 𝑃 𝐺 + 𝑝𝐴 𝑡 (𝑃 𝐺 ) and 𝑃 𝑅 + 𝑝𝐴 𝑡 (𝑃 𝑅 ). Because the probability 𝑝 and the function 𝐴 𝑡 in DANI are both controlled by the overfitting degree of 𝐷 adaptively, this means that 𝑝 and 𝐴 𝑡 will vary adaptively during training. In this case, one sample 𝑥 can produce a set of samples {𝑥 + 𝑝𝐴 𝑡 (𝑥)} during training. This demonstrates that the overlapping parts between the DANI distribution 𝑃 𝐺 + 𝑝𝐴 𝑡 (𝑃 𝐺 ) and 𝑃 𝑅 + 𝑝𝐴 𝑡 (𝑃 𝑅 ) consist of more samples, thus providing more overlapping parts, leading to better performance.Although the adaptive forward diffusion process has been already applied to the Diffusion-GAN, the two main differences between DANI and Diffusion-GAN are shown as follows. First, same to the data augmentation methods in GANs with limited data [26, 50, 54, 65] , the noise injection formulation in Diffusion-GAN is to transform both real and fake samples by an adaptive forward diffusion process. This design can cause the original real and fake data not to be visible to 𝐷 during training, which decreases the training of GANs under limited data settings, leading to sub-optimal performance. In contrast, the noise injection form of the proposed DANI allows the original real and fake data to be visible to 𝐷 during training (under adaptive injection probability 𝑝 = 0), which can improve the training of GANs under limited data settings, resulting in better performance. Second, according to [64] , the leaking problem still exists in Diffusion-GAN. Compared with the Diffusion-GAN, which only employs the adaptive noise strength to alleviate the leaking problem, the proposed DANI has an additional adaptive injection probability to further alleviate the leaking problem, therefore leading to better performance. More importantly, according to Eq.( 2), the DANI in Projected GAN is applied to the real and fake projected features rather than real and fake images. The commonly used Fréchet Inception Distance (FID) [19] is applied as the evaluation metric; the full dataset is used as the reference distribution for FID calculation, following prior work [26, 31, 65] . For the adaptive forward diffusion process 𝐴 𝑡 in DANI, based on Diffusion-GAN [54] , the initial value of 𝑡 is set as 5. Then, we set 𝑡 𝑚𝑖𝑛 = 5 and the 𝑡 𝑚𝑎𝑥 = 1000 for InsGen, and set the 𝑡 𝑚𝑖𝑛 = 5 and the 𝑡 𝑚𝑎𝑥 = 500 for Projected GAN. The forward diffusion process variances are set to constants increasing linearly from 𝛽 1 = 10 -4 to 𝛽 𝑇 = 0.02. More details of experiments, i.e., experimental results with the other evaluation metric Inception Score (IS) [44] and the link of the pre-trained model with test code, can be found in supplementary materials. 
results on low-shot datasets: 
 The results on low-shot datasets with StyleGAN2 and FastGAN are shown in Table 1 . Applying DANI to both InsGen and Projected GAN can achieve better performance. Projected GAN + DANI achieves the lowest FID compared with all of the other methods. More importantly, Projected GAN has already applied DA, regularizations and pre-trained models to achieve state-of-the-art performance on the low-shot datasets; adding DANI can further reduce the FID score by about 5% to 10% and achieve novel state-of-the-art performance. The images generated by Projected GAN + DANI on low-shot datasets are shown in Figure 3 .The results on the FFHQ datasets are shown in Table 2 . For the FFHQ dataset, following the FakeCLR [31] , we perform experiments on the subset of 100, 1K, 2K and 5K. Applying DANI to both InsGen and Projected GAN can achieve better performance and Projected GAN + DANI achieves state-of-the-art performance compared with other methods. The results on the LSUN-CAT datasets with the state-of-the-art method, i.e., Projected GAN, are shown in Table 3 .For the LSUN-CAT dataset, based on Diff-Augment [65] , we conduct experiments on the subset of 1K, 5K, 10K and 30K. Applying DANI to Projected GAN also achieves better performance. . We follow the setting as in [65] . MA means Massive Augmentation, i.e., xflipping, which has the same meaning as in [9] . comparing the proposed DANI with Diffusion-GAN with the Projected GAN backbone, and the results are shown in Table 5 . Projected GAN + DANI can achieve better performance compared with Diffusion-Projected GAN on low-shot datasets. More comparison results on the FFHQ dataset can be found in supplementary materials. 
conclusion: 
 In this paper, to improve GANs training with limited data, we propose a novel noise injection method called Dual Adaptive Noise Table 6 : FID score (lower is better) on the 100-shot-Obama dataset (256 × 256). Massive Augmentation [9] is applied to all of the methods. For a fair comparison, the FIDs are averaged over three runs; all standard deviations are less than 1% relatively.Injection (DANI), with a negligible computational cost increase. Specifically, DANI consists of two adaptive strategies, i.e., the adaptive injection probability and adaptive noise strength. For the adaptive injection probability, we inject Gaussian noise into both real and fake images for 𝐺 and 𝐷 through a probability 𝑝, where 𝑝 is controlled adaptively by the overfitting degree of 𝐷.  
 
 Appendix:  


Decision: The paper proposes a method to improve GANs with limited data. The idea is simple and effective. 

Most reviewers agree that the paper should be accepted. However, I strongly encourage the authors to add user study and quantitative evaluation on FFHQ 1k following the suggestions from Reviewer KTU8.

I did not consider the comment from Reviewer WQ8A because the limitation in the review is too general without any specific information/words related to this paper.

Reviews: review: paper_summary: paper_summary: To alleviate the overfitting of GANs with limited data, DANI introduces two adaptive strategies (adaptive injection probability and adaptive noise strength) to inject Gaussian noise into both real and fake data. The overall methodological framework is simple. Specifically, based on the adaptive forward diffusion process of Diffusion-GAN, DANI integrates the adaptive probability initially proposed by ADA to achieve adaptive noise injection into both real and generated images (or projected features).
strength_weakness: strength_weakness: 1. DANI provides a theoretical analysis of two classic loss functions (the non-saturating loss and the hinge loss) in two classic generative adversarial networks (i.e., StyleGAN and FastGAN).

2. According to the FID metric, DANI achieves state-of-the-art performance across a wide range of datasets with different scales under limited data.
limitations: limitations: 1. **Noise leaking problem.** Although DANI uses the adaptive probability proposed by ADA to alleviate the noise leaking problem, this issue still persists because the training is done on real data of noise injection. Furthermore, in my opinion, the harm caused by noise leaking is much greater than that of data augmentation leaking in ADA, because data-augmented images are still real images, whereas noise-injected data might just be random noise.

2. **Inconsistent quantitative results.** The FID results of InsGen reported in this paper are not consistent with the FID results from the original papers. For example, on FFHQ-1K, the original FID result of InsGen is 19.58, but 18.21 is reported in this paper. However, in the table captions, the paper only claims that the results of the Projected GAN are run based on the official codes. Please clearly specify in the paper which baseline results are cited from the original papers and which are reproduced using the official codes with the average taken over 5 runs.

3. **Insufficient quantitative results.** The paper reports the quantitative comparison of FID for all methods and the comparison of IS with Projected GAN in the appendix. Both FID and IS are calculated based on features extracted from classifiers pre-trained on ImageNet. However, as reported in previous work [1], aligning the histograms of Top-N ImageNet classifications between sets of generated and real images can substantially reduce metrics (e.g., FID) relying on the feature space of the ImageNet classifier without actually improving the quality of results. Therefore, I strongly recommend that the paper at least include non-ImageNet trained Fréchet distance metrics, such as CLIP FID, to validate the experimental results.
    
     **[1] The Role of ImageNet Classes in Fréchet Inception Distance. ICLR 2023.**

4. **Insufficient qualitative results.** The main text and appendix only provide quantitative comparisons with Projected GAN using four images. The number of images and the number of baseline methods compared are too few. Additionally, the paper does not include a user study, which limits the validation of the quantitative metrics' effectiveness.
paper_summary: paper_summary: Summary:
This paper studies adversarial training with a limited amount of data by mitigating the overfitting issue. Specifically, the authors propose a noise injection approach that employs diffusion process to adaptively inject noise into adversarial training. In order to prevent the overfitting problem, which is basically because the discriminator is overconfident in distinguishing real data and fake data, the proposed DANI approach controls the degree of noise injection via an adaptive probability p. By mapping the real data into noise, and meanwhile mapping the generated fake data into real data through a dual adaptive diffusion process, the discriminator can be trained more properly thus tackling the overfitting issue.
strength_weakness: strength_weakness: Strengths:
- This paper is well-written and easy to follow.
- The motivation is clear and reasonable, and the proposed solution is intuitive and effective.
- The empirical results are promising and the experiments are sufficient.
limitations: limitations: Weaknesses:
- The major concern is the computational efficiency of this method. It is known that generative models are quite computationally expensive, but this approach contains a diffusion model within a generative adversarial network. The authors provided computational cost analysis which is good, but it is confusing to me why the computational increase is so small given that a diffusion process is employed.
- Moreover, the intuition of employing the diffusion process into the framework is to add noise so that the training of GAN would not overfit. However, why is it necessary to use diffusion models but not using other noise injective models like VAEs or GANs? Could the authors provide further explanation to justify their motivation?
paper_summary: paper_summary: This paper proposes Dual Adaptive Noise Injection (DANI) to improve the training of GANs with limited data. DANI addresses discriminator overfitting by injecting Gaussian noise into real and fake images based on the overfitting degree of the discriminator and applying an adaptive forward diffusion process to determine the noise strength. By increasing the overlap between real and fake data distributions, DANI effectively alleviates overfitting. Experiments demonstrate DANI's superiority in improving GANs training with limited data compared to existing methods.
strength_weakness: strength_weakness: Proposes a novel method called Dual Adaptive Noise Injection (DANI) for training GANs with limited data, effectively alleviating discriminator overfitting and avoiding the leaking problem.
Provides theoretical analysis of DANI, proving its convergence and rationality on both StyleGAN2 and FastGAN backbones, and demonstrating increased overlap between real and fake data distributions.
Achieves state-of-the-art performance on several commonly used datasets with only a negligible increase in computational cost.
Open-source code available, ensuring good reproducibility of the proposed method.
limitations: limitations: It would be helpful to include the processing time of other methods or vary the number of images processed per iteration to more thoroughly demonstrate the minimal increase in computational time. This would provide a more comprehensive comparison and showcase the efficiency of the proposed method.
Conducting experiments on larger and more diverse datasets with a wider range of styles would further validate the effectiveness and generalizability of the proposed approach. This could strengthen the claims made in the paper and show the method's robustness across different data distributions.
The analysis in the experimental section could be more in-depth. Providing a more detailed discussion of the results, including insights into the behavior of the proposed method, its limitations, and potential areas for improvement, would enhance the overall quality and impact of the paper.
Adding the publication year when referring to the comparison methods would improve readability and provide better context for the reader. This would help the reader understand the chronological development of the field and the relative novelty of the proposed approach.