DigGAN: Discriminator gradIent Gap Regularization for GAN Training with Limited Data
Authors: Tiantian Fang
Ruoyu Sun
Alex Schwing
Venue: NeurIPS 2022 Accept (2022)
Keywords: Generative model
limited data
regularization

Abstract: Generative adversarial nets (GANs) have been remarkably successful at learning to sample from distributions specified by a given dataset, particularly if the given dataset is reasonably large compared to its dimensionality. However, given limited data, classical GANs have struggled, and strategies like output-regularization, data-augmentation, use of pre-trained models and pruning have been shown to lead to improvements. Notably, the applicability of these strategies is often constrained to particular settings, e.g., availability of a pretrained GAN, or increases training time, e.g., when using pruning. In contrast, we propose a  Discriminator gradIent Gap regularized GAN (DigGAN) formulation which can be added to any existing GAN. DigGAN augments existing GANs by encouraging to narrow the gap between the norm of the gradient of a discriminator's prediction w.r.t. real images and w.r.t. the generated samples. We observe this formulation to avoid bad attractors within the GAN loss landscape, and we find DigGAN to significantly improve the results of GAN training when limited data is available.

DigGAN: Discriminator gradlent Gap Regularization for GAN Training with Limited Data
Tiantian Fang, Ruoyu Sun, Alex Schwing University of Illinois Urbana-Champaign (tf6, ruoyus, aschwing)@illinois edu
Abstract
Generative adversarial nets (GANs) have been remarkably successful at learning to sample from distributions specified by given dataset, particularly if the given dataset is reasonably large compared to its dimensionality. However, given limited data, classical GANs have struggled, and strategies like output-regularizatoo, data-augmentation, use of pre-trained models and pruning have been shown lead to improvements. Notably, the applicability of these strategies is 1) often constrained particular settings, e.g., availability of a pretrained GAN; or increases training time, e.g., when using pruning. In contrast, we propose a Discriminator gradlent Gap regularized GAN (DigGAN) formulation which can be added to any existing GAN. DigGAN augments existing GANs by encouraging to narrow the gap between the norm of the gradient of discriminator's prediction w.r.t. real images and w.r.t. the generated samples. We observe this formulation to avoid bad attractors within the GAN loss landscape, and we find DigGAN to significantly improve the results of GAN training when limited data is available. Code available at ftthhtt:://// /gttbbb.. com AilsaF /DigGAN
1 Introduction
Generative Adversarial Nets (GANs) [3] have been remarkably successful at learning to sample from distributions specified by given dataset. In practice, this success has garnered lot of interest in GANs for a wide range of applications, from data augmentation [3]66]]
61] and domain adaptation
48
to image-to-image translation 63 18
24
and photo editing [4] 64).
This success of GANs strongly relies on the availability of large dataset. Unsurprisingly, in real- life circumstances, particularly when the dimensionality of the samples in the dataset is high, the available samples to train a GAN can be insufficient. Insufficient data may significantly reduce the performance of standard GANs. For instance, when we train a GAN on CIFAR- using just 10% of the data, BigGAN performance deteriorates from 13.54 FID score to 73.01 FID score, and the GAN generates images of single pattern (Fig. 1. To address this deteriorating performance of GANs trained with limited data, various strategies have been proposed recently, including use of
19]. However, despite improving results, each of these strategies also impose restrictions. The use of pretrained models works best if data domains remain similar. Pruning requires many rounds of training to increase the sparsity of the neural architecture, which raises the training cost. Data augmentation enhance the results, but the benefit is limited with insufficient data (Tab. 3). Regularization is cheap and potentially effective approach, and recent work by Tseng et al. [51 adopted this approach, controlling the distance between the diseriminator's prediction on the real image and the generated image. However, with limited data, this regularization doesn't 't show significant improvements (Tab, 3
pretrained model 660 42 28), pruning 77 and data augmentation 23
61
59
In this paper, we study new regularization to enhance the training of GANs with limited data. Instead of constraining the discriminator's ss output as done in prior work [5]] we propose the Discriminator
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

100% data
20% data
10% data
230
150
Baseline
Realta
Beration
10
Ours
02
12000 62530 106:00 156030 204030
Figure 1: CIFAR-100 100 generated samples by BigGAN (baseline) and BigGAN+DigGAN (ours), both trained with 100%, 20% and 10% data at the iteration with the best FID (left). We observe: 1) when training with 10% data, quantitative scores of the baseline deteriorate considerably (top right) and the gradient norm gap increases (bottom right). The gradient norm gap size and FID are inversely proportional the amount of training data. 2) The proposed DigGAN regularization stabilizes both quantitative scores and the gradient norm gap.
gradlent Gap GAN (DigGAN) regularization. Though multiple gradient-based regularizers have been proposed for GAN training 15 37 26). none of these target training of GANs with limited data. Instead, they focus on training stability of GANs with standard data.
In contrast to prior formulations, the DigGAN regularizer encourages to narrow the gap between I) the norm of the gradient of a discriminator's prediction w.r.t. real images, and 2) the norm of the gradient of a discriminator's prediction w.r.t. generated data. For readability, (Discriminator gradlent Gap). Our regularizer is motivated by empirical studies, during which we found that: 1) When training GANs with limited data, the DIG is large. 2) By reducing the DIG, the effects of bad attractors in the training dynamics of GANs can be reduced. We found this to improve training by preventing GANs from being pushed to bad attractors.
call this gap "DIG"
We conduct comprehensive experiments to show the effectiveness and consistency of the proposed regularizer. First, we use synthetic example to demonstrate the stabilizing effect of the proposed regularizer on the training, avoiding bad attractors that the original GAN is attached to. Second, using recent architectures that achieve state-of-the-ar results, including SNGAN, BigGAN, and StyleGAN2, we show that the regularizer improves results when only limited data is available.
2 Related Work
In the following we briefly discuss GAN variants, regularization for GANs, and prior work to address training of GANs with limited data.
Generative Adversarial Networks. Many GAN variants have been proposed to stabilize the training and improve the perceptual quality of the generated results. Among them, many variants study
different loss objectives 14. 35 41 B
43 15
54 39
40 20, T 10, 32 The design of
new architectures is another popular direction to obtain different GAN variants [4
12
63 5
22
21. Moreover, a variety of normalization techniques have also been proposed 46,
58
38 17 2].
Finally, techniques have also been devised to produce more diverse samples 33 55 and to improve
convergence [3 57 34 47
Regularization for GANs. Among all the aforementioned GANs, regularization techniques are widely used to stabilize the training. Gulrajani et [15) regularize the discriminator to be 1-Lipschitz by penalizing the norm of the gradient of the discriminator with respect to its input data points sampled by interpolating between real data and generated data. Kodali al. (26 propose to penalize only the Gaussian manifold around the real data instead. Roth et al. [45 encourage the gradient norm of the

---

discriminator on real data and generated data to be zero. Mescheder et al. [37] provide more detailed insights on the gradient penalty, Besides the gradient norm, constraining the discriminator is another common mechanism [59 62 and the weight penalty is also a common regularization method for GANs 4 [9] Different from these methods, we focus on improving results in the case of limited training data.
Training of GANS with limited data. Training GANs with limited data has attracted lot of attention
recently. Data scarcity causes classical GAN training to become more challenging 22 53 49 A
few methods have been proposed improve the performance of GANs trained with limited data.
One popular idea is data augmentation [62 59 23
61 19 6]. Jiang et al. [19] use generated data as an "augmentation" for the real data, while others do the augmentation on real instances. Chen et al. 77] leverage pruned neural networks to improve the performance. A few works [60] 42 28
used pre-trained GANs, which were learned on a similar data domain using sufficient data before being transferred to the target domain. Yang et al. [56] require the discriminator to do instance-level classification and distinguish every individual real and fake image instance as an independent category. Most closely related to method is work by Tseng al. [51], who regularize the distance between the discriminator evaluated on the real data and the discriminator evaluated on the generated data to be small. Our approach differs from these methods in that we propose new regularizer which encourages the Discriminator gradlent Gap of GAN to be small. This regularizer is orthogonal to other solutions (except (51)) and may be used concurrently.
3 Method
3.1 Review of Generative Adversarial Networks
Generative adversarial networks (GANs) consist of a generator G and a discriminator D, which are
pitted against each other. The generator G(2; 0). parameterized by learns map sample Z
drawn from simple, possibly low-dimensional distribution p(z) (e.g.. Gaussian) over domain 2,
to the complex, possibly high-dimensional data distribution domain X. The discriminator D(x) trained to distinguish between real data FR ~ X' and synthetically generated data 2p G(2; 0) obtained from the generator. More formally, the game between the generator and the discriminator D is described by the following two loss functions, which are often also referred to as the non- saturating GAN losses:
LG = -
CD (1 Dac.a//ooot.tiill-oooooaaeaaaaaa
Multiple loss functions have been used for LG and fp. For instance, Jenson-Shannon-GGA 14 uses fa(t) (t( log(1 exp(t)), and hingeGAN 9
uses (t(t) and (t) max(0, 1 t).
3.2 Discriminator Gradlent Gap (DigGAN)
Worse Performance of GANs with Limited Data. Karras et al. 23 and Tseng et al. [51] observed that fewer training samples can cause GANs to perform much worse. In empirical studies (c.g.. Fig. observe this too. Concretely, we train BigGAN with 100%, 20% and 10% CIFAR-100 data. The FID score (lower is better) increases dramatically when 10% or 20% data are used (top right part of Fig. T.. Subsequently the generator produces images of worse quality (see the images shown in the first row of Fig. I..
large gap of gradient norms. In comprehensive experiments across multiple datasets we observe that the gap between 1) the norm the gradient of discriminator's 's prediction w.r.t. real images, and 2) the norm of the gradient of a discriminator's 's prediction w.r.t generated data, i.., the squared distance
Observation:
(1
0D
aD
!)
(.D,
FR. xF) =
(2)
ZoR
Ozf
increases when the GAN trained with fewer data. Here, HR and oF G(2; 0) denote the training data and the generated data respectively. For readability, we refer to this gap via "DIG" (Discriminator gradlent Gap). This phenomenon can be observed in Fig. (see the bottom right part) and Fig. (see the right part).
3

---

Limneeliiing fass
making eggege
Image 2g
D
8D
Xx. 5.
D
aDVak;
(Ap)
(pplal
accersted
inSag
Figure 2: The DigGAN computation graph for the regularizer R(D, xR) given in Eq.
Motivated by the observation, we propose a possible explanation for the training failure of GANs when limited data is available:
Few data large "Discriminator gradlent Gap" unstable training of GANs.
In this chain, (a) is empirically observed in Fig. and Fig. and (b) is due to the explanation below.
We suspect that a large gap in discriminator gradient norms (DIG) is one reason for the bad perfor- mance of GANs trained with limited data. We provide an intuition here. We can treat the gradient as the rate with which the discriminator's output changes when the input image changes, and the norm as the magnitude of this rate. large gap between two norms can then interpreted as significant imbalance between 1) the discriminator's rate of change w.r.t. a real image change and 2) the discriminator's ' rate change w.r.t. generated image change. Consequently, the discriminator learns faster either real images or generated images resulting in an imbalance.
To improve the training GANs with limited data, natural reduce the DIG. We propose to use Eq. as regularizer so as to control the DIG during training. In turn, this aids to balance the discriminator's learning speed.
More specifically, we train the generator G using the objective La given in Eq. [ but train the discriminator / with the extra regularizer stated in Eq. (2), modified to account for the possible issue that data within different mini-batches can be extremely different, causing the regularizer to vary lot. To reduce the variance, we use a moving average for the gradient norm term in the regularization, with a smoothness factor a. Formally, when training the discriminator D we use the following regularized objective at iteration
ED = CD + AR(D, TR,
with R(D, DD, 2F), = ((gR): - (gF)))? where
0D
if = 1,
(g9);
(3)
0D
(1 a) (ga 1- +
ift 1.
Ozg
Here, À is a non-negative, scalar hyper-parameter a E (0, 1] is a scalar
lying between 0 and (inclusive), and B E (R. F) is used to refer real generated data. This process to compute the proposed regularizer is summarized Fig. 2
3.3 Small-Scale Case Study: Intuition for DigGAN Regularizer and Attractors
In this section, we analyze the advantage of the proposed regularization R from the perspective of attractors. For readability, we refer the local attractors unregularized GANs "unregularized- local-ation hocal-attractors" Prior work [50] argued that GAN training has sub-optimal unregularized- local- attractors, and this issue becomes less severe when there are more samples. fact, as number of samples goes to infinity, the empirical loss converges to population loss which is convex [4 50
Consequently as number of samples decreases, we expect the issue to become more severe. We will show that the proposed DigGAN can alleviate this issue at least simple example.

---

(b)
2.0
data position
gradient gap
Baseline
gener data
4
X
1.0
5000 iteration
10000
0
5000
10000
data position
iteration
true data
data position
gradient gap
gener ated
4
0.2
Ours
0
0.0
5000
10000
0
10000 iteration
5000
data position
iteration
Comparing GAN without (top row) and with (bottom row) the proposed DigGAN regular- ization in a two-point experiment. (a) Data position after the training. Baseline covers one true data point, while the proposed DigGAN regularization enables a GAN to cover all true data. (b) generated data (blue curves) movement for the whole training. (c) Gradient gap R (Eq. 2
Figure 3:
First, we empirically show that a vanilla GAN can get stuck in local attractors (Experiment I). Second, show that these local attractors are stable for vanilla GAN and cannot be escaped from (Experiment 2). Third, we show that DigGAN can improve the training by avoiding unregularized- local-attractors (Experiment 3). Forth, to understand why DigGAN can avoid them, empirically show that DigGAN can escape even starting from puregularized-lloal-attaaa
(Experiment demonstrates: the unregularized-
4). The experiment of "escaping mmregularized-locllatttaa local-attractors are no longer attractors for DigGAN. Thus DigGAN training doesn't 't get stuck at unregularized-Ioor
wwuriied-localltttee local-attractor which provides an explanation for Experiment 3. We summarize the logic for clarity:
In DigGAN, aaapallarizze--callatrraacee can be escaped (Experiment 4). They are no longer local attractors of DigGAN training (by definition of local attractors). DigGAN training will not get trapped in unregularized-locat attractors (Experiment 3) which trap a vanilla GAN (Experiment and 2).
Note that the four experiments serve different roles. The "avoiding experiment" (Experiment 3) is what naturally happens in practice, but not that clear how a regularizer can help avoid attractors. The "escaping experiment" (Experiment 4) is carefully designed and rarely happens in practice with random initialization (since DigGAN would not get trapped starting from a random initialization). But the experiment has two advantages: (i) results easily demonstrate the power of the regularizer; (ii) results illustrate the underlying mechanism of the "avoiding experiment".
For the four experiments, we consider the task of generating two real data points 0 and 4. The generator a 4-layer fully connected network (FCN) with 5 neurons per layer and tanh activation. The discriminator is a 4-layer FCN with 10 neurons per layer and sigmoid activation. Both the discriminator D and the generator G are updated with momentum coefficient 0.9 (25) and learning rate 0.01. We use A 1 and = 1 in Eq. (3). We use 10, 000 iterations to complete the training and study each of the four experiments:
Experiment 1: vanilla GAN can get stuck in unregularized-localaae
We train a vanilla
GAN from scratch with a random initialization D, and Gg. We get D, and G2 at the end of training. In Fig. top row, we plot (a) the final training snippet, (b) the 1-dimensional data movement, and (c) the gradient norm gap given in Eq. (2) throughout the entire training process. In plot (b), the y-axis represents the data position, while the x-axis represents the iteration. The movement of the generated data represented via the blue curves, and all true data represented by two straight red lines the y-axis positions 0 and 4. We observe that the vanilla GAN (top row in Fig. 3 gets stuck generating data from single-mode = (top row (a)) starting with roughly iteration 3000 (top row (b)). The baseline's 's gradient gap (top row (c)) increases significantly right before GAN training is trapped.
5

---

data position
data position
data position
data position
2000 4000
0
2000 4000
2000 4000
2000
4000 iteration
iteration
iteration
iteration
Figure 4: Discriminator perturbation experiments. Adding noise with variance (a) 0.1. (b) 1, (c)
and (d) 10 the vanilla discriminator parameters. Data generated during training shows that the attractor is reasonably strong (a, b).
(a)
(b)
(c)
(d)
gradient norm
10
data position
data position
genersted
X
0
0 2
4
5000 iteration
10000
5000 iteration
10000
data pasition
data pasition
Figure 5: Escaping an attractor. We initialize the GAN at local attractor (a) and illustrate the gradient norm in (b). The baselines get trapped at the initial point for the subsequent training (c), while the GAN with DigGAN regularizer can escape (d).
In this experiment, we verify that the end state Di, Gi a stable local attractor for a vanilla GAN, by testing if adding noise to D, can aid GAN training escape and converge to a better result. We experiment with four levels of noise variance, 0.1, 1, and 10 respectively. We visualize the data movement in Fig.
Experiment 2: vanilla GAN cannot escape saregularized-locaaattarer
We observe that 1) upon adding noise (0, 0.1) the GAN's initial state differs from the local attractor, but GAN training eventually converges to the same attractor; 2) the GAN's initial state differs more from the local attractor after (0,, 1) noise is added, but GAN training eventually still converges to the same result; 3) Noise (0, 5) has enough strength to help the GAN dynamics escape from the local attractor completely. However, GAN training is caught by another bad attactor, where both generated data points overlap
4; 4) Noise NO, 10) is too strong to make the dynamics converge. These experiments show that unregularized- wowedddooollaaaaeea attractosss can affect vanilla GAN training.
Experiment 3: DigGAN regularization helps to avoid that the comparison is fair, we train a DigGAN from the same starting point Do and Go as Experiment 1. We visualize the comparison with vanilla GAN in Fig. 3 bottom row. We observe that 1) DigGAN enables the GAN to avoid local attractors and cover two modes under the same settings (bottom row (a,b)). 2) Compared to the vanilla GAN whose gradient gap increases significantly before getting trapped, the proposed regularization R reduces the risk of getting trapped by encouraging the gradient norm gap to remain at low level (bottom row (c)). Note the scale difference of the y-axis.
prregularized-locl-atteeee
To ensure
We demonstrate that the DigGAN regularization can help the dynamics escape unregularized-loca. attractors. observe this, we initialize the GAN at state where trapped a local attractor, which is denoted as D2 and G2. Starting from this initialization, GAN without regularization remains trapped (see Fig. 44. In contrast, a GAN with DIG regularizer R is able to escape eventually.
Experiment DigGAN regularization helps escape pareguulrrizee-lcal-attrree
We provide more details regarding this experiment, At the initial state, the generated data covers one mode, and 10DD/axll final generator obtained when training the baseline in Experiment 2, which generates two points that overlap at O. To get Dz, we fix G2 and train the discriminator D to optimality using the objective given in Eq. B. This case is difficult because the proposed regularizer has little influence. However, because the discriminator's activation is non-linear, the discriminator aims to output a greater value for the missing point in subsequent training steps. This change in discriminator outputs causes gradient gap, which disrupts the system's balance. Eventually, DigGAN able to escape. We can detect growing perturbation for the generated data around iteration 2000 (bottom row (d)).
|aD/ /0 < le (Fig. 5 bottom row). We initialize G2 to be the

---

100% data
20% data
10% data
IS 1
FID 4
IS 1
FID
IS
FID 4
SNGAN
8.42
19.29
7.24
30.69
6.39
47.08
SNGAN DigGAN (ours)
8.54
15.11
7.78
22.27
7.36
29.43
BigGAN BigGAN + DigGAN (ours)
9.04
10.53
8.26
21.38
7.62
36.35
9.09
9.74
8.40
17.11
7.89
23.75
BigGAN + DiffAug
9.04
9.88
8.66
14.51
8.19
23.65
BigGAN + DiffAug + DigGAN (ours)
9.28
8.49
8.89
13.01
8.32
17.87
Table 1: Inception score (IS) (higher is better) and Fréchet Inception distance (FID) (lower is better) for BigGAN trained with 100%, 20%, 10% CIFAR-10 data, respectively.
100% data
20% data
10% data
IS f
FID
IS
FID +
IS f
FID +
BigGAN
10.58
13.54
8.66
33.64
5.35
73.01
BigGAN RLC 55
11.18
11.88
9.08
25.51
7.76
49.63
BigGAN + DigGAN (ours)
10.92
12.93
9.21
21.79
9.06
27.61
BigGAN + DiffAug
11.87
12.00
9.41
22.14
8.63
33.70
BigGAN + DiffAug + REe S1
10.77
11.84
9.52
21.78
8.89
26.91
BigGAN DiffAug + DigGAN (ours)
11.45
11.63
9.54
19.79
8.98
24.59
Table 2: Inception score (IS) (higher is better) and Fréchet Inception distance (FID) (lower is better) for BigGAN trained with 100%, 20%. 10% CIFAR- -EFAR-10 data, respectively.
4 Experiments
4.1 Experimental settings
To validate the effectiveness of the proposed DigGAN regularization. we conduct comprehensive experiments CIFAR-10 (27]. CIFAR-100, Tiny-ImageNet (30, CUB-200 (52) and multiple low- shot [6]] data using both class-conditional BigGAN, the unconditional SNGAN and StyleGAN2. We use two standard evaluation metries, i.e., the Fréchet Inception Distance (FID) [16] and the Inception Score (IS) [46). A smaller FID and a larger IS generally imply better GAN models. Moreover, we further improve our empirical results by combining data-augmentation trick. For the regularization, we set 1000/ (data availability percentage) no data augmentation is used; e.g., À 5000 if we train with 20% data. We set X 10/(data availability percentage)? if data augmentation is used. We also use a = 0.5 in all real data experiments. For both conditional and unconditional GANs, draw real samples and fake samples randomly in pairs (i.e., we pair samples irrespective of their class). All GANs are trained with NVIDIA Geforee RTX 2080 Ti GPUs.
regularization with DiffAug [6]] popular
4.2 CIFAR-10 10 and CIFAR-100
We test our regularization on CIFAR-10 and CIFAR- 100 data using two benchmark frameworks: BigGAN and SNGAN, following the implementation by Zhao et al. [61] and Miyato et al. [38] Both CIFAR-10 10 and CIFAR-100 datasets have 50,000 training images with an image size of 32 x 32. CIFAR- 100 is more challenging, since it includes 100 categories with less images per class, while CIFAR-10 has 10 categories. We conduct the experiments on 100%, 20%, 10% data, following the settings used in prior work [61 77.
We report the results in Tab. and Tab.
Four observations are notable. First, DigGAN can improve GAN performance generally, regardless of the amount of available data. Use of the regularization hence doesn't 't seem to harm GAN training. Second, our regularization can yield significant advantages when dealing with scarce data. Note that our regularization improves baseline FIDs by 11.85 score (33.64 vs. 21.79) and 45.4 score (73.01 vs. 27.61) for 20% and 10% CIFAR-100 data respectively. Third, with 10% CIFAR- 100 data, the regularization can even outperform BigGAN that uses DiffAug by 6.09 FID (33.70 vs. 27.61). Fourth,
7

---

1.50
200
150
11.00
100
Don.t 200
0.50
50
0.00
12460
60000
108600
156006
206300
12006
60000
10R00C
156000
204300
Figure 6: Comparison between BigGAN+DiffAus (baseline) and BigGAN-DiffAugtDiiCAA regularization (ours) CIFAR-I00, 100, trained with 100%, 20% and 10% training data. We show FID (left) and gradient norm gap (right) during training.
Figure 7: Generated images and their nearest neighbors in the training data, with 10% CIFAR-I0 (left) and 10% CIFAR-100 (right) training data.
DigGAN improves upon the regularization proposed by Tseng et al. [51] by 3.72 FID (25.51 vs. 21.79) and by 22.02 FID (49.63 vs. 27.61) in cases of 20% and 10% CIFAR- data respectively.
Data augmentation, a widely used trick for GAN training, is orthogonal to the studied regularization approach. demonstrate orthogonality to data augmentation, we follow the experimental settings of DiffAug [61], differentiable and data-level augmentation method. We report the results in Tab. and Tab. 2 as well. Our technique can consistently enhance performance and improve upon baselines with the augmentation, regardless of data availability. The FID scores improve from 23.65 to 17.87 and from 33.70 to 24.59 for CIFAR-10 10 and CIFAR-100 with our regularization.
Besides Fig. we also present the FIDs, gaps between the gradient of the discriminator's predictions on real images and generated images for CIFAR-100 with DiffAug in Fig. 6 We can draw similar conclusions: 1) For the baseline, the gaps get larger with increasingly scarcer data. 2) Our DigGAN regularization can control the discriminator gradient gaps at low level. These observations align with our analysis in Sec.
possible concern relates memorization of the scarce training data by the generator. To validate the generalization ability, we show the generated images and their 5 nearest neighbors from the training data, after training generator with only 10% training data (Fig. 7 We observe the generated data to not appear in the training data. This shows that the model doesn' 't just memorize.
4.3 Higher resolution generation with BigGAN: Tiny-ImageNet, CUB-200
Generating nigher-resolutie images when training with limited data is even more challenging. In this section, we report our results on ImageNet and CUB-200, We adopt the implementation contains 100,000 images of 200 classes
of the Pytorch StudioGAN codebase Tinn--Imagela a resolution 64 x 64. We test our proposed approach using 10%, 50%, and 100% of the data.
CUB-200 data contains 5,994 training images of 200 classes 128 x 128 resolution, and we 50% and 100% of the data.
We compare with two baselines: BigGAN and RiC [51] Tab. lists all the quantitative results. note that the conclusions are consistent with the CIFAR experiments. Specifically, 1) DigGAN outperforms the baseline when using all available data, regardless of the presence of the data augmentation; 2) DigGAN gains increase with fewer data. For instance, we improve RLC by 1.01

---

100% Tiny ImageNet
50% Tiny ImageNet
10% Tiny ImageNet
100%
50%
CUB-200
CUB-200
BigGAN
31.92
43,45
130.77
20.15
48.67
51 BigGAN + DigGAN (ours)
BigGAN+ RIC
28.
36.11
121.16
40.37
98.38
17.76
24.63
84.27
14.45
23.20
BigGAN + DiffAug
16.33
24.50
95,40
13.49
24.35
BigGAN+RLc Diff
16.30
23.67
83.76
12.81
23,49
BigGAN DiffAAg+ DigGAN (ours)
14.84
22.66
51.18
11.58
21.12
Table 3: Fréchet Inception distance (FID) for BigGAN with Tiny-ImageNet and CUB-200.
100-shot Obama
100-shot grumpy cat
AnimalFace Dog
AnimalFace Cat
StyleGAN+ADA StyLGGAN-ADA-iiGAAN
49.78
27.34
66.25
41.40
41.34
26.75
59.00
37.61
Table Fréchet Inception distance (FID) for StyleGAN2 with ADA on low-shot datasets.
Figure 8: Low-shot generation with DigGAN on Obama, Grumpy Cat, and AnimalFace datasets.
FID (23.67 vs. 22.66) with 50% Tiny-ImageNe data, and increase the improvements to 32.58 FID (83.76 vs. 51.18) with 10% Tiny-ImageNet data.
4.4 Low-shot generation with StyleGAN2
We also our DigGAN on low-shot generation tasks. We conduct new low-shot generation experiments with StyleGAN2+ADA (23). We run experiments on the 100-shot Obama, 100-shot Grumpy Cat, and AnimalFace dataset (160 cats and 389 dogs) provided by [6]. All datasets are at 256 x 256 resolution. We use the maximum training length of 600k images for all experiments.
We set the regularization weight as 100. We provide the results in Tab. and show that DigGAN achieves consistent gains on all datasets. Generated images are in Fig. 8
4.5 Comparison with other gradient- based regularization methods
Several gradient-based regularization methods have been proposed for GANs. Among them, GP- 1 [[5]. RI, R2 [45] and DraGAN (37] are the most popular ones. However, GP-O is usually used for an SNGAN structure [29], and R1 is usually applied in StyleGAN2 framework (22). R2 and DraGAN are not widely tested in popular model architectures. We provide a comparison on two settings: StyleGAN+CIFARI 10 and BigGAN+CIFARI
From Tab. 5 observe: GP-1 is not compatible with StyleGAN2, as it prevents StyleGAN2 from training properly. Further, cannot be applied on BigGAN, since the classes are not available for the interpolated data. 2) DigGAN, R1, R2 and DraGAN can be applied to StyleGAN2 and BigGAN. With the full dataset available, DigGAN does necessarily improve over other regularization methods. However, when limited training data are available, DigGAN is able to improve upon other gradient- based regularization techniques. Compared to the best baseline, DigGAN improves FID by 5.16 and 6.74 with 10% data availability.
4.6 Ablation study
Regularization strength. We conduct an ablation study regarding the regularization strength using 10% CIFAR-100 data and the BigGAN framework. We sweep the regularization strength À (1, 10, 100, 1k: 10k, 100k} and test the models' sensitivity. We report the results Fig. 9 We observe that the performance improves with increasing strength at first, better and better addressing overfitting. However, as expected, the performance deteriorates when the regularization is too strong. To provide more comprehensive comparison, we also sweep RI regularization [45 over the

---

|StylGGAN2 CIFAR-I0 BigGAN CIFAR-100
100%
10%
100%
10%
100
Ours
Baseline
13.54
73.01
+RI
11.07
36.02
15.63
40.62
+R2
9.14
34.93
14.61
34.35
+GP-1
200.51
228.39
+DRAGAN
9.56
39.21
13.09
52.79
100 1k 10k 100k regularization weight
+DigGAN
10.12
29.75
12.00
27.61
Table 5: Comparison with popular gradient-based regular- ization methods. We conduct experiments with two model architectures using two datasets, and report FID.
Figure 9: An ablation study of our regularization and R1 45 over reg- ularization strength.
100% CIAFR 10
10% CIAFR-10
100% CIAFR 100
10% CIAFR-10
Random pairing (Ours) Same-class pairing Gradient magnitude pairing
9.74
23.75
12,93
27.61
10.12
21.53
13.14
28.08
9,09
27.64
12.86
47.75
Table 6: Ablation study on real and fake samples pairing mechanism. All experiments are conducted with BigGAN structure.
strength set and report the results in Fig. 9 The behavior of R1 is similar to the proposed DigGAN regularization, however, its best quantitative results tend to be lower.
Pairing mechanism. We draw real samples and fake samples randomly in pairs and irrespective of their class for our experiments. To study the effect of the pairing method on the performance, we compare random pairing with two other methods: pairing within the same class and pairing based on gradient magnitude order. In this ablation study, we conduct the comparison using CIFAR data and BigGAN structure. We use the default regularization weight. We report results in Tab.
For the same-class pairing mechanism, each pair of the real sample and fake sample is drawn from the same class. Results are shown in Tab. 6 row 2: we do not observe significant difference to random pairing. We think that the use of exponential moving average (EMA) ensures that the trained model is not significantly influenced by one pair. For the gradient magnitude pairing mechanism, we pair real samples and fake samples by sorting norms ||D/ 00 @rrnll nll2 and /0/0arll ||2 within each batch to show how permutations of the real and fake samples affect the results. Results are in Tab. 6 row We observe that gradient magnitude pairing works better with 100% data availability for both datasets. However, DigGAN works much better than gradient magnitude pairing with 10% data availability. We think random pairing is important.
Conclusion and Broader Impact
We introduce DigGAN, gradient- based regularization for the discriminator which improves GAN training with limited data. DigGAN encourages small difference between the gradient norm of the discriminator w.r.t. the real data and w.r.t. the generated data. Empirically, we observe this difference to be large when GAN training fails. We also demonstrate that DigGAN regularization reduces the effect of local attractors. On a variety of datasets and architectures we show consistent results.
Limitation. We tried to analyze DigGAN from theoretical perspective. But a theoretical analysis of the studied loss extremely challenging because of the way batching performed. We could not find a compelling theoretical result that is meaningful, and we did not want to include derivations that don't add insights.
Societal implications. This work can have both positive and negative effects. On the positive side, training GANs with limited real-world data is important for fields where is expensive to collect large datasets. For instance, our research can make AI methods more powerful in rare disease diagnosis, antique authentication, etc. On the negative side, improved generative methods can be used to generate fake data and spread misinformation via DeepFakes.
Acknowledgements. Work supported in part by NSF under Grants 1718221, 2008387, 2045586. 2 106855, MRI 1725729, and NIFA award 2020-67021-32799, We thank NVIDIA for a GPU.
10

---

References
[1] M. Arjovsky, S. Chintala, and L. Bottou. Wasserstein gan. In JCML, 2017.
[2] J. L. Ba, J. R. Kiros, and G. E. Hinton. Layer normalization. arXiv preprint arXiv: 160.006607 2016.
[3] D. Berthelot, Schumm, and L. Metz. Began: Boundary equilibrium generative adversarial networks. arXiv preprint arXiv: 10717, 2017.
[4] A. Brock, T. Lim, J. M. Ritchie, and N. Weston. Neural photo editing with introspective adversarial networks. In Proc. ICLR, 2017.
[5] A. Brock, J. Donahue, and K. Simonyan. Large scale gan training for high fidelity natural image synthesis. arXiv preprint arXiv: 11096, 2018.
[6] Cao, Hou, M.-H. Yang, R. He, and Sun. ReMix: Towards Image-to- to-aaaee Translation with Limited Data. In CVPR, 2021.
[7] T. Chen, Y. Cheng, Z. Gan, J. Liu, and Z. Wang. Data-efficient gan training beyond (just) augmentations: A lottery ticket perspective. In Advances in Neural Information Processing Systems, 2021.
[8] Choi, T. Kim, and C. Kim. Self-ensembling with gan-based data augmentation for domain adaptation in semantic segmentation. In ICCV, 2019.
[9] R. W. A. Cully, H. J. Chang, and Y. Demiris. Magan: Margin adaptation for generative adversarial networks. arXiv preprint arXiv: 03811, 2017.
[10] I. Deshpande, Z. Zhang, and A. G. Schwing. Generative Modeling using the Sliced Wasserstein Distance. In Proc. CVPR, 2018.
[11] I. Deshpande, Y.-T. Hu, R. Sun, Pyrros, N. Siddiqui, Koyejo. Z. Zhao, Forsyth, and A. G. Schwing. Max-Slicoo Wasserstein Distance and its use for GANs. In CVPR, 2019.
[12] J. Donahue, P. Krähenbühl, and T. Darrell. Adversarial feature learning. arXiv preprint 1605.06660007789 2016.
[13] I. Goodfellow. Nips 2016 tutorial: Generative adversarial networks. arXiv preprint arXiv:1001.01160, 2016.
[14] I. Goodfellow, J. Pouget-Abadie, M. Mirza, Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. Generative adversarial nets. In NeurlPS, 2014.
[15] I. Gulrajani, F. Ahmed, M. Arjovsky, V. Dumoulin, and A. Courville. Improved training of wasserstein gans. In NeurIPS, 2017.
[16] M. Heusel, H. Ramsauer, Т. Unterthiner, B. Nessler, and S. Hochreiter. Gans trained by a two time-scale update nule converge a local nash equilibrium. In NeurlPS, 2017.
[17] S. loffe and Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167 2015.
[18] P. Isola, J.-Y. Zhu, T. Zhou, and A. A. Efros. Image-to-image translation with conditional adversarial networks. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
[19] L. Jiang, B. Dai, W. Wu, and C. Change Loy. Deceive d: Adaptive pseudo augmentation for gan training with limited data. In Advances in Neural Information Processing Systems, 2021.
[20] A. Jolicoeur-Martinen The relativistic discriminator: a key element missing from standard gan. In ICLR, 2018.
[21] Karras, T. Aila, Laine, and Lehtinen. Progressive growing of gans for improved quality, stability, and variation. In ICLR, 2018.

---

[22] T. Karras, S. Laine, and T. Aila. A style-based generator architecture for generative adversarial networks. Proceedings of the IEEE conference on computer vision and pattern recognition. pages 4401-4410, 2019.
[23] T. Karras, M. Aittala, J. Hellsten, S. Laine, J. Lehtinen, and T. Aila. Regularizing generative adversarial networks under limited data. In Advances in Neural Information Processing Systems, 2020.
[24] J. Kim, M. Kim, H. Kang, and K. H. Lee. U-gat-it: Unsupervised generative attentional networks with adaptive layer-instance normalization for image-to-image translation. In International Conference on Learning Representations, 2020.
[25] D. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: 2014.
[26] N. Kodali, Abernethy, Hays, and Z. Kira. How to train your dragan. arXiv preprint arXiv:1705.0721: 2017.
[27] A. Krizhevsky. Learning multiple layers of features from tiny images. https: // www. cs. toronto. edu/ "ta////r//// -ew///laannnneeeeeeeeee
2009 TR. pdf 2009.
[28] N. Kumari, R. Zhang, E. Shechtman, and J.-Y. Zhu. Ensembling off-the-shelf models for gan training. In CVPR, June 2022.
[29] K. Kurach, M. Lucic, X. Zhai, M. Michalski, and S. Gelly, A large-scale study on regularization and normalization in gans. In ICML, 2019.
[30] Y. Le and X. S. Yang. Tiny imagenet visual recognition challenge. In http: // vision. stanford. edu/ teaching/ cstpin/ httpl///weepr/////ee 2015/ 2015/pdfs// yle_ project. pdf 2015.
[31] C.-L. Li, W.-C. Chang, Y. Cheng, Y. Yang, and B. Póczos. Mmd gan: Towards deeper understanding of moment matching network. In NeurIPS, 2017.
[32] Y. Li, A. G. Schwing, K.-C. Wang, and R. Zemel. Dualing GANs. In NeurIPS, 2017.
[33] Z. Lin, A. Khetan, G. Fanti, and S. Oh. Pacgan: The power of two samples in generative adversarial networks. In NeurlPS, 2018.
[34] S. Liu, O. Bousquet, and K. Chaudhuri. Approximation and convergence properties of generative adversarial learning. In NeurlPS, 2017.
[35] X. Mao, Q. Li, H. Xie, R. Y. K. Lau, Z. Wang, and S. P. Smolley. Least Squares Generative Adversarial Networks. arXiv e-prints, 2016.
[36] E. V. Mazumdar, M. I. Jordan, and S. S. Sastry. On finding local nash equilibria (and only local nash equilibria) in zero-sum games. arXiv preprint /01000190, 2019.
[37] L. Mescheder, A. Geiger, and S. Nowozin. Which training methods for gans do actually converge? In ICMI,, 2018.
[38] T. Miyato, T. Kataoka, M. Koyama, and Y. Yoshida. Spectral normalization for generative adversarial networks. In ICLR, 2018.
[39] Y. Mroueh and T. Sercu. Fisher gan. In NeurIPS, 2017.
[40] Y. Mrouch, T. Sercu, and V. Goel. Mcgan: Mean and covariance feature matching gan, arXiv preprint 1702.01772. 2017.
[41] S. Nowozin, B. Cseke, and R. Tomioka. f-gan: Training generative neural samplers using variational divergence minimization. In NewrIPS, 2016.
[42] U. Ojha, Y. Li, J. Lu, A. A. Efros, Y. J. Lee, E. Shechtman, and R. Zhang. Few-shot Image Generation via Cross-domain Correspondence. In CVPR, 2021.
[43] B. Poole, A. A. Alemi, Sohl-Dickstein, and A. Angelova. Improved generator objectives for gans. arXiv preprint arXiv: 1612.0061 2016.
12

---

[44] A. Radford, L. Metz, and S. Chintala. Unsupervised representation learning with deep convolu- tional generative adversarial networks. In ICLR, 2016.
[45] K. Roth, A. Lucchi, S. Nowozin, and T. Hofmann. Stabilizing Training of Generative Adversarial Networks through Regularization. In Conference on Neural Information Processing Systems, 2017.
[46] T. Salimans, I. Goodfellow, W. Zaremba, V. Cheung, A. Radford, X. Chen, and X. Chen. Improved techniques for training gans. In NeurIPS, 2016.
[47] M. Sanjabi, J. Ba, M. Razaviyayn, and J. D. Lee. On the convergence and robustness of training gans with regularized optimal transport. In NeurlPS, 2018
[48] S. Sankaranarayanan, Y. Balaji, C. D. Castillo, and R. Chellappa. Generate to adapt: Aligning domains using generative adversarial networks. In CVPR, 2018.
[49] M. Shahbazi, M. Danelljan, D. Pani Paudel, and L. Van Gool. Collapse by Conditioning: Training Class-conditional GANs with Limited Data. In ICLR, 2022.
[50] R. Sun, T. Fang, and A. Schwing. Towards Better Global Loss Landscape of GANs. In Conference on Neural Information Processing Systems, 2020.
[51] H.-Y. Tseng, L. Jiang. C. Liu, M.-H. Yang. and W. Yang. Regularizing generative adversarial networks under limited data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021.
[52] C. Wah, S. Branson, P. Welinder, P. Perona, and S. J. Belongie. The caltech-ucsd birds- 200-2011 I dataset. In https: // authors. library. caltech. edu/ 27452/ 1/ CUB_ 200_ 2011. pdf] 2011.
[53] R. Webster, J. Rabin, L. Simon, and F. Jurie. Detecting Overfitting of Deep Generative Networks via Latent Recovery. In CVPR, 2019.
[54] J. Wu, Z. Huang, W. Li, J. Thoma, and L. Van Gool. Sliced wasserstein generative models. In CVPR, 2019.
[55] C. Xiao, P. Zhong, and C. Zheng. Bourgan: Generative networks with metric embeddings. In NeurlPS, 2018
[56] C. Yang, Y. Shen, Y. Xu, and B. Zhou. Data-Efficient Instance Generation from Instance Discrimination. In NeurlPS, 2021.
[57] Y. Yazıeı, C.-S. Foo, S. Winkler, K.-H. Yap, G. Piliouras, and V. Chandrasekhar. The unusual effectiveness of averaging in gan training. In ICLR, 2019.
[58] H. Zhang, I. Goodfellow, D. Metaxas, and A. Odena. Self-attention generative adversarial networks. In /CML., 2018.
[59] H. Zhang, Z. Zhang, A. Odena, and H. Lee. Consistency regularization for generative adversarial networks. International Conference on Learning Representations, 2020.
[60] M. Zhao, Y. Cong, and L. Carin. On leveraging pretrained gans for limited-data generation. In ICML, 2020.
[61] S. Zhao, 7. Liu, J. Lin, J.-Y. Zhu, and S. Han. Differentiable augmentation for data-efficient gan training. In Advances in Neural Information Processing Systems, 2020.
[62] Z. Zhao, S. Singh, H. Lee, Z. Zhang, A. Odena, and H. Zhang. Improved consistency regular- ization for gans. In Association for the Advancement of Artificial Intelligence, 2021.
[63] J.-Y. Zhu, T. Park, P. Isola, and A. A. Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In Computer Vision (ICCV), 2017 IEEE International Conference on, 2017.
[64] Zhuang, O. Koyejo, and A. G. Schwing. Enjoy Your Editing: Controllable GANs for Image Editing via Latent Space Navigation. In Proc. ICLR, 2021.
13

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either the supplemental material or as URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits. hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g.. with respect the random seed after running experi- ments multiple times)? [No]
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you'le roe're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
14


Reviews:
[{"review_id": "ykSCEjDybf9", "reviewer": "Reviewer_Mjjy", "summary": "This paper proposes a new method for improving gan training under limited data. The key idea is to regularize the gradient norm of discriminator between real and fake samples. Authors have done empirical studies showing that this regularization can help avoid bad attractors within the GAN loss landscape. Authors have applied the proposed regularization on BigGAN, leading to improvement across multiple datasets.", "questions": "Overall I think the idea proposed in this manuscript looks interesting and effective to some extent. But I think the experiments are insufficient to validate the effectiveness of the proposed method. Detailed questions are covered in paper weaknesses.", "limitations": "Authors didn't include the discussion of limitations in the main manuscript. I think authors can briefly discuss in what cases the proposed method may fail or lead to marginal improvement.\n", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n+ The manuscript is easy to follow\n+ The empirical studies provided by the authors are important to understand the effectiveness of proposed regularization\n\nWeaknesses:\n- Missing related work, and incorrect interpretation of related work. Specifically, [b] ([17] in manuscript) is not doing data augmentation in my understanding. And [a] is also related but is missed in the manuscript. Since both [a] and [b] are not doing data augmentation, I believe they should be compared as well.\n\n[a] Data-Efficient Instance Generation from Instance Discrimination\n[b] Deceive d: Adaptive pseudo augmentation for gan training with limited data.\n\n- while at L30 authors claim \"data augmentation can enhance the results, but the benefit is limited with insufficient data (Tab. 3).\" In table 3 authors didn't evaluate sufficient data augmentation methods to support this claim.\n\n- in the experiments, authors only compared to DiffAug and R_LC, which is insufficient. Authors should also include ADA, etc.\n\n- while previous works targeting gan training with limited data mainly evaluate their methods on StyleGAN and unconditional image synthesis, authors mainly evaluate the proposed method on BigGAN and conditional image synthesis. It's better to stay consistent with previous methods to ensure a fair comparison. Or at least expand table 4.\n\n-  It's great that authors provide empirical studies in Sec.3.3. It's better if authors include a theoretical analysis. \n\n\n\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "KI0Y-gbshqL", "reviewer": "Reviewer_ipzv", "summary": "The authors propose a novel approach to training GANs with limited data, one that is motivated by the gap in the norms of the gradients between the target and generated images. They demonstrate applications of the proposed regularizer on various BigGAN architectures with and without data augmentation. ", "questions": " - **Loss Formulation:** I have a fundamental concern with the form of the loss $\\left(  \\left\\| \\frac{\\partial D}{\\partial x_R} \\right\\|  - \\left\\| \\frac{\\partial D}{\\partial x_F} \\right\\|\\right)$. It is unclear how the samples are drawn. If $x_R$ and $x_F$ are drawn randomly in pair, then, the particular paining of the real and fake images will affect how this loss performs. While this issue might not be visible in the synthetic 1D experiment present, on the image scale, this might have effect. For example, are we making sure the pairs are drawn from the same class. In class-conditional GANs, it might be possible, bit, in unconditional variants, this cannot be guaranteed, and there is a strong possibility that the manifold nature of each class might itself affect how these terms interact. \n - To address the above, maybe the authors could consider some ablation study where, given the choice of the sample batch, how permutations of the real and fake samples affects the penalty?\n - Alternatively, have the authors considered batch-level sample averages of the gradient norms? This would essentially bring the terms closed to the R1 and R2 penalties. ", "limitations": "—", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Originality and Significance:** Limited data (LD) training with GANs has been gaining popularity in the recent years, and the proposed regularization approach to training shows potential. The fact that it can be used in parallel with other schemes for LD training such as DiffAug is an advantage. The gradient gap in the discriminator is generally well analyzed and adequate  experimental validations are provided. \n\n**Presentation and Clarity:** The paper is clear to read and the flow is consistent. \n - The synthetic experiments with Vanilla GAN consider a simplistic scenario — a similar experiment in 2D might have been more insightful, as the effect of different norm on the gradient vector (L1, L2), is clearer at least in 2D. Nevertheless, the given experiments help to give a clear intuition on the proposed method. \n - The notation is confusing at times. For example, The additional parentheses when time indexing time $t$ seems unnecessary, but this is just a personal opinion.\n - Some proof read would help, as there are a few typos here and there. L233 L146 Forth -> Fourth. L262 improvement to 32.56 -> improvement **by** 32.58, just to name a few. \n\n**Literature Survey:** Most of the relevant literature has been cited and the paper includes relevant discussions on related works. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "DuqSHFBsYBc", "reviewer": "Reviewer_Z5nH", "summary": "This paper introduces a new data-efficient training technique for GAN. The technique named Discriminator gradIent Gap (DIG) regularization aims to equalize the changes of discriminator’s judgments w.r.t its inputs (real and fake data) so that the discriminator can effectively learn representations for adversarial training without an imbalance issue. Experimental results demonstrate that GAN with the proposed DIG regularization exhibits better image generation results than the previous techniques (LeCam regularization and DiffAug).", "questions": "[Q1] Which gradient (gradient of a discriminator's prediction w.r.t real or fake sample) makes the DIG value large? I guess the norm of the gradient w.r.t real sample might be very large, as the discriminator tends to memorize training samples and predicts samples other than the training samples to be fake regardless of the realism of samples.", "limitations": "I cannot find any section that explains the limitations of the paper. Also, the described negative societal impact is not sufficient. I hope the authors will add more portions to tell the pros and cons of the paper and its societal impact.", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\n[S1] I think the proposed method is new to the GAN community. \n\n[S2] The empirical finding that the gap between the norms of the gradients of discriminator’s predictions increases when the GAN is trained with fewer data, seems to be very useful for future development of data-efficient training methods. \n\n[S3] The experiments in the paper show that the proposed DIG regularization is helpful for GAN training in data-hungry situations. To demonstrate the effectiveness of DIG regularization, the authors utilize various datasets (CIFAR10, CIFAR100, Tiny ImageNet, and CUB200), which means the proposed method can be used for real-world applications. \n\n[S4] The paper is well written and easy to follow. \n\nWeaknesses\n\n[W1] I think the theoretical explanation of the relation between DIG and the training failure is needed. Although experimental results show that applying DIG regularization can resolve the problem of getting stuck in local attractors, I am not sure why this is possible by imposing DIG regularization.\n\n[W2] FID scores in this paper do not match the results of previous papers [1, 2]. The FIDs of BigGAN on CIFAR10 in the papers [1, 2] are 8.57 and 8.08, respectively, while the FID of BigGAN in this paper is 10.53 (Table 1). Performance discrepancies occur for experiments across the paper, which significantly lowers the credibility of the paper.\n\n[W3] Experiments using high-resolution images are missing. Verifying the usefulness of DIG regularization using MetFaces, AFHQv2, or FFHQ will make the paper more convincing.\n\n[W4] Comparision with Adaptive Discriminator Augmentation (ADA) is missing. Since ADA is a standard method to train GAN with limited data, I think comparing DIG regularization with ADA is necessary. \n\n[1] Chen, T., Cheng, Y., Gan, Z., Liu, J., & Wang, Z. (2021). Ultra-Data-Efficient GAN Training: Drawing A Lottery Ticket First, Then Training It Toughly. ArXiv, abs/2103.00397.\nhttps://openreview.net/forum?id=BBVcs78PEDA\n\n[2] Kang, M., Shim, W., Cho, M., & Park, J. (2021). Rebooting ACGAN: Auxiliary Classifier GANs with Stable Training. NeurIPS.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}]

Author rebuttal:

Decision: Accept