Self-Diagnosing GAN: Diagnosing Underrepresented Samples in Generative Adversarial Networks
Authors: Jinhee Lee
Haeri Kim
Youngkyu Hong
Hye Won Chung
Venue: NeurIPS 2021 Poster (2021)
Keywords: Generative Adversarial Networks (GAN)
diversity in sample generation
fair generative model
minor attributes
training dynamics
statistics of log-density ratio
weighted sampling for SGD.

Abstract: Despite remarkable performance in producing realistic samples, Generative Adversarial Networks (GANs) often produce low-quality samples near low-density regions of the data manifold, e.g., samples of minor groups. Many techniques have been developed to improve the quality of generated samples, either by post-processing generated samples or by pre-processing the empirical data distribution, but at the cost of reduced diversity. To promote diversity in sample generation without degrading the overall quality, we propose a simple yet effective method to diagnose and emphasize underrepresented samples during training of a GAN. The main idea is to use the statistics of the discrepancy between the data distribution and the model distribution at each data instance. Based on the observation that the underrepresented samples have a high average discrepancy or high variability in discrepancy, we propose a method to emphasize those samples during training of a GAN. Our experimental results demonstrate that the proposed method improves GAN performance on various datasets, and it is especially effective in improving the quality and diversity of sample generation for minor groups.

Self-Diagnosing GAN: Diagnosing Underrepresented Samples in Generative Adversarial Networks
Jinhee Lee" School of Electrical Engineering KAIST
Haeri Kim"t Samsung Research
Samsung Electronics
jin. wee@kaist.ac...m...oooo.eeeommee
haeri www.ksassungg...... com
Youngkyu Hong*
Hye Won Chung
NAVER AI Lab NAVER vounkky.boogghavercorp..oo
School of Electrical Engineering KAIST huchung@kaist. kcr
Abstract
Despite remarkable performance in producing realistic samples, Generative Ad- versarial Networks (GANs) often produce low-quality samples near low-density regions of the data manifold, e.g., samples of minor groups. Many techniques have been developed to improve the quality of generated samples, either by post- processing generated samples or by pre-processing the empirical data distribution, but at the cost of reduced diversity. To promote diversity in sample generation without degrading the overall quality, we propose a simple yet effective method to diagnose and emphasize underrepresented samples during training of GAN. The main idea is to use the statistics the discrepancy between the data distribution and the model distribution each data instance. Based on the observation that the underrepresented samples have a high average discrepancy or high variability discrepancy, propose method emphasize those samples during training of GAN. Our experimental results demonstrate that the proposed method improves GAN performance various datasets, and it is especially effective in improving the quality and diversity of sample generation for minor groups
Introduction
Generative Adversarial Networks (GANs) have achieved remarkable performance in producing realistic samples for complex generation tasks, including image/video synthesis [5. 22], style trans- fer [45, 14], and data augmentation [29]. However, GANs often fail to cover sparse regions of data manifold [16, 9], leading to the underrepressenaatin of minor groups in the dataset [43]. In partieular, GANs generate samples of minor groups with low fidelity or even fail to generate such samples, exhibiting the mode collapse [43].
Many of previous techniques have focused on improving the overall sample quality of GANs, either by pre-processing the training dataset or by post- processccssiigg generated samples. The pre-processing aims to remove instances that cannot be well-represented by GANs even before the training starts and gains fidelity the focused samples [9]. A similar idea has been used to truncate the latent space by resampling or moving samples that fall outside of some acceptable range during training [16, 6].
"Equal contribution.
This work done as student at KAIST.
Corresponding author.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Post-processing on the other hand, is technique that can be applied after the training to remove low-quality generated samples by rejection sampling [3, 37]. All these approaches are effective in increasing the overall fidelity of samples from GANs, but reducing the diversity as trade-off, and may exacerbate biases against the minor groups in sample generation.
In this work, we aim to improve diversity in sample generation without degrading the overall quality, with special focus on coverage and quality improvement for minor groups. Toward this, we design methods to detect and emphasize underrepresented samples in training of GANs. Due to the lack of explicit labels available, detecting minor- muioor- samples is especially challenging for unsupervised learning. Therefore, we first develop two new metrics, which can be easily calculated from discriminator output of GANs, to detect underrepresented samples. The main idea is to measure the statistics (mean and variance) of the estimated discrepancy between the data distribution and model distribution at each data instance over multiple epochs of the training. The mean discrepancy indicates how close the data distribution is to the model distribution at each data over the training, while the variance in discrepancy measures how such discrepancy fluctuates across the training. We provide theoretical and empirical evidence that the mean discrepancy can effectively detect underrepresenteed samples, especially near collapsed modes, while the variance in discrepancy can detect minor data instances, which GANs suffer from modeling.
Based on these observations, we propose a novel method emphasize underrepresented samples during the training of GANs by score-based weighted sampling, where the score is defined as a weighted sum of the two metries we devised. We validate our method with thorough experiments over controlled and real datasets and demonstrate the efficacy of the proposed sampling method in improving not only the overall quality (both fidelity and diversity combined) of sample generation but also the coverage and quality for semantic features of minor subgroups. Our contributions can be summarized as follows.
We propose two new metrics, which can be simply computed from the discriminator, to diagnose GAN training and to detect underrepresented samples. By theoretical analysis and controlled experiments, we demonstrate that the proposed metrics are effective in detecting underrepresented minor samples.
We propose an algorithm that can effectively emphasize underrepresented data by score- based weighted sampling during the training of GANs. Our experiments controlled and real datasets show that our method improves diverse performance metrics on several GAN variants and enhances the coverage and quality of minor group generation.
Our code is publicly available at https: //githhb// cttee/ccoo//aooooe htt//rreyhonn/lllltigggnssinn gan.
2 Related Work
Promoting data coverage in GANs Due to the unstable nature of the min-max game between a generator and discriminator, GANs often suffer from mode collapse and produce samples with poor diversity. Several approaches have been proposed to promote better data coverage by modifying architectures [20, 23], loss functions [2, 1] adding regularizations [8, 4, 35]. While effective in promoting overall data coverage, these approaches do not provide special care on minor modes and often fail to recover them when the minority ratio for certain feature is extremely low. We provide method to promote data coverage for minor features even when the minority ratio is significantly low.
There exists another line of works to improve data coverage by designing hybrid generative models [32, 27, 43], which combines the idea of reconstructive models (e.g. variational autoencoder) GANs, to take advantages of the reconstructive models in recovering diverse modes. This hybrid method, however, requires relatively high computational overhead to guarantee data coverage for all (or partial) real modes by optimizing reconstruction error in feature domain. Our method directly detects and emphasizes underrepresented samples so that the computational overhead is much lower.
Improving GAN performance by diagnosing samples There have been promising attempts to improve GAN training by using the discriminator outputs to estimate the discrepancy between the data distribution and implicit model distribution. DRS [3] proposes the density ratio estimate based on the discriminator output to apply rejection sampling to filter generated samples. GOLD [26] uses the similar estimate to re-weight fake samples to emphasize underrepresented fake samples. In [10]

---

and [12], on the other hand, an external classifier is used to improve the density ratio estimates. There also exist some approaches to use discriminator outputs to select or weight "useful" fake samples during training. Top-k training [30] updates the generator by using only top-k fake samples with the largest discriminator outputs. In [31] and [40], discriminator-ban
importance re-weighting schemes for fake samples are developed, and in [39], latent samples are optimized to improve the fidelity.
Our method uses the discrepancy estimate proposed in [3], but its empirical mean and variance over multiple epochs, to extract more reliable and useful information to detect underrepresented minor group samples. We provide theoretical evidence of why not only the mean but also the variance of discrepancy estimate is effective in detecting underrepresented samples. Our method detects and emphasizes underrepresented real samples, not the fake samples. This difference is significant in promoting the data coverage of minor groups, since when fake samples already fail to cover minor modes, emphasizing a subset of fake samples cannot improve the data coverage for missed modes.
3
Two Metrics to Detect Underrepresented Samples During GAN Training
3.1 Measuring the discrepancy of GANs
GAN training aims to train a generator with an implicit model distribution (2((x) that closely matches
the data distribution Pdata (I). The discrepancy between Pdata (z)) and (z) can be measured by the log density ratio log (Paata (x) (x)pp(((a) (z)). but cannot be directly calculated in GANs, since Paata (z) is
unknown and Py (2) is implicit. Instead, the analysis in the original GAN paper [11] can be used to
define an estimate on the density ratio by using the discriminator output as explained in [3].
The original GAN solves the min-max optimization ming max V(D, G) for the loss V(D,G)
Eg-paa [log (xx)] + Egpp, [log(1 - (=)()))) For any fixed generator G, the optimal discriminator
(a)) and this allows us to define the Bog--Denityy-atii estimate (LDR) by yields D (xx = PMaes (c)
(1) When LDR(x) > 0, the data point x is underrepresented in the model, i.e., (d) > (2()), while when LDR(x) a < O, the data is overrepresented, i.e., Pdata (z)) < pg(r). Thus, we can leverage the value of
LDR(x)= = log(D(x))(11 D(x)))
When D(x)= (x)), the LDR(x) is equal the log density ratio (x))p
(2) of each instance x to give feedback to improve the generator if the estimation is valid.
Some prior works have used the LDR estimate to improve GAN training. As an example, GOLD [26] uses LDR(z) to evaluate the quality of the fake samples and re-weights the underrepresented fake samples when training the generator for conditional GANs. However, we later show that re- weighting fake samples is less effective than re- re-weighting real samples in improving diversity in sample generation. We also empirically show that LDR(x) is unstable metric to use. More detailed arguments are available in the Appendix $A.
As remedy, we propose to use statistics of LDR(ar), which are much more stable and informative metrics, to detect underrepresented data regions during the training. The main intuition to use training dynamics-the behavior of a model as training progresses-to diagnose the learning behavior of each sample. In supervised learning, training dynamics have been widely studied to detect "hard-to- learn" samples [7, 33, 38). However, in learning generative models, the metrics to diagnose training dynamics are not clear since there is no explicit reference to measure the accuracy of the model. Here we define metries that estimate the mean and variance of the discrepancy of GANs, LDRM (LDR Mean) and LDRV (LDR Variance), at each sample x across the training steps T = (ter..., te):
LDRM a;; 7) = [T]
Σ (D)( LDRV(z; T) = T KET
Σ [LDR(r), - LDRM xc; 7) KET
(2)
where LDR(x)& is the recorded LDR estimate (1) in the k-th training step. LDRM(x)) measures how close Pdata (α) is to Pg(xr) over the training at sample point æ, while LDRV(z) measures how such discrepancy fluctuates across training.
Intuitively, samples that have been well-learned and generalized will have consistently small LDR(a) since (zx) R 1/2 Paata
Paata (x: Rs p. (2)). thus will exhibit low LDRM and LDRV, while underrep- resented "hard-to-learn" samples will show high LDRM or LDRV values. the rest of this section, we thoroughly study the characteristics of data instances with high LDRM or high LDRV.
3

---

7 5 8 2 o 6
(Mine)) (Minnr))
(Miner)
Ganssian Colorrdinas -MNIST FMRST (Major)
1.0
(a) Major
(b) Minor
Payta Docal
075
1 1:59 9 3 7 4
0.5
030
(c) Major
(d) Minor
Minority
(e) Partial Recall of major/minor groups vs. minority level
Figure 1: Analysis on generated samples of GAN trained with (1) Single-mode Gaussian, (2) A mixture of MNIST (major) and FMNIST (minor), and (3) Colored MNIST with Red (major) and Green (minor) samples. (a) ~ (d) show the examples of generated samples with major/minor features, and (e) shows the Partial Recall of major (dotted)/minor (solid) samples in each dataset various minority levels. Both the sample quality and partial recall rate are higher for major groups.
3.2 LDRV is effective in detecting samples from minor groups
GANs have poor modeling for minor samples GANs are known to struggle with modeling minor samples [16]. To scrutinize this phenomenon, we use following toy datasets each of which includes major and minor group: (1) Single-mode Gaussian with distance from the origin as a factor dividing two groups. (2) A mixture of MNIST (major) and FMNIST (minor), and (3) Colored MNIST with Red (major) and Green (minor) digits. We vary the size of the minor group and define a minority level to represent the scarcity of the minor group, i.e., a higher level indicates the scarcer minor group. Details of each dataset are available in the Appendix $F. Figure I shows the poor quality of generated samples with minor features, relative to major features. To quantify the level of underrepresentedness, we examine the coverage of modes for major vs. minor groups with the Partial Recall [18], which is the portion of the subset of real samples that reside in the manifold of the fake samples. As shown in Figure le, major and minor groups have large recall gap and the gap gets worsen as the minority level gets severe. This observation indicates that the minor group suffers not only the poor quality problem but also the low coverage problem, and it gives strong motivation to detect the minor samples and emphasize them.
LDRV and minor samples We next provide heuristic arguments that LDRV can be used to detect samples with minor features, i.e., features of minor groups. In particular, we show that minor samples tend to have higher LDRV values. First, we view the discriminator as the logistic regression model: for each input 2, the discriminator takes the inner product between the feature vector Φ = F(x,) and the weight vector 0 of the last layer to produce the reality score (the probability that the sample I; is real (y,- = 1)), i.c.,
Da 0) 1/(1 + 00do- p(u: = 1 16:;; 0).
From a Bayesian perspective, assuming that prior distribution of 0 is p(0) = N0 so.). the posterior distribution over 0 is given by (0), 3t) x p(0)p(s? jo 0). To obtain a Gaussian approximation to the posterior distribution, we first find the maximum a posteriori estimate MAP that maximizes log p(0 (Ø,, 1)) which defines the mean of Gaussian. The covariance is then given by the inverse of the matrix of second derivatives of the negative log likelihood, which takes the form
-1
Sn Σ Daz; 0) D(z; 0))o:o7
(4)
50
Lastly, approximating D(2;; B) in (3) by the Taylor expansion at 0 = 0MAP. LDRV can be expressed as
LDRV(x;:) ~ var (log(D(z; 0)/(1 D(a;; 0))) ~ oJ S.0:
Details of the analysis is available in the Appendix $B.
This analysis shows an important aspect regarding LDRV and minor features. First, (5) shows that as the feature vector di becomes more correlated with the principal components of S, (eigenvectors with largest cigenvalues), its LDRV gets larger. Since each eigenvalue of S, is the reciprocal of that

---

420
LoW
400
220
mod:
(a) LDRM and generated samples
(b) LDRM distribution and the number
of high-quality samples over modes
Figure 2: (a) Training dynamics of 25 Gaussians. The index of cach mode is equal to 5x + y where the coordinates (z. 3) E {0, 1, 3, 4)2. LDRM of training samples is recorded for each mode with window [T] 50. Modes with high LDRM do not appear in the generated samples. (b) The empirical distribution of LDRM (box plot) for each mode from training samples, compared with the number of high-quality generated samples from each mode (blue). LDRM can effectively detect dropped modes.
of 1 we consider the characteristies of the eigenvector of ST with the least eigenvalue, which is the minimizer of
ST) Σ
(y) 0.)² const.
(6)
Eq. (6) shows if does not align with (or orthogonal to) majority of feature vectors {sh having
D(1 - D) > 0, then it tends to have smaller eigenvalue. Since minor feature vector Φ, may have a small component on the eigenspace formed by the majority of {Φr} having D(1 - D) O, when plug in Ø into (6), the summation becomes small. This shows that the minor feature vector Φj is correlated with the least eigenvector of S, and thus it will have higher LDRV.
In Table 1, we show that minor group indeed has higher LDRV. Thus, both theoretical and empirical evidence shows that we can detect minor samples by investigating LDRV of training samples.
3.3 LDRM is effective in detecting missing modes
Mixture of 25 Gaussians From the definition of LDRM (2), high LDRM samples x tend to have smaller Pg (2:) than (1) over the training, thus are underrepresented. We next investigate the ability of LDRM to detect the regions of data manifold not yet covered by the model distribution (x)) We consider mixture of 25 2D isotropic Gaussian distributions [20, 42, 37, 3]. During training, we record LDR(x) of the training samples and calculate LDRM values with window size [T|= =1]50. 50. We inspect LDRM values averaged over samples of each mode during the training. As shown in Fig. 2a, we observe that samples from underrepresented modes have higher mean LDRM values. This implies that we can detect the mode recovery by inspecting the mean of LDRM values.
To further examine the mode recovery in generated samples, we assign each generated sample to its closest mode and consider it as a "high-quality" sample if it is within four standard deviations from its assigned mode [42, 3]. We then count the number of high-quality samples of each mode among 10,000 generated samples and analyze the correlation between the high-quality sample counts and the distribution of LDRRM. As shown in Fig. 2b, modes with only a few high-quality samples tend to have higher LDRM. This indicates that LDRM of the data instances can be used to detect the regions of data manifold not yet covered by the model, even without looking at the generated samples.
Table 1: Averaged LDRV of major/minor groups on various datasets with majority rate 90%.
Group
Gaussian (a=3.0)
Colored MNIST
MNIST-FMNIST
Major
0.001
0.077
0.082
Minor
0.098
0.186
0.115
5

---

Lavest Dac. Scare Samplr
Cenerated Samples
(a) Images with lowest disc. score
3000
2000
Plant
(b) Images with highest disc. score
Intensitty
(c) Generated samples
(d) Histogram of pixel count over intensity level
Figure 3: Training images with (a) lowest, (b) highest discrepancy scores, and (c) generated samples. Generated samples resemble training images with lowest score. (d) A smoothed histogram of the intensities for 100 samples per group. The intensity levels of RGB channels are concatenated, resulting in total 768 256 x 3 levels. Images with the lowest scores (blue) and generated samples (green) have similar distribution, while images with the highest scores (red) show a high discrepancy.
4 Algorithm to Emphasize Underrepresented Samples
4.1 Proposed method: Stochastic Gradient Descent (SGD) sampled by discrepancy
We propose a simple modification to the GAN training procedures by using score-based weighted sampling for mini-batch SGD to emphasize underrepresented samples. Let D {2;} be the training
dataset. The mini- mini-batch of size B for the training dataset is formed by D {x(11 z(6
I; where ~ (i(i) for = =,..... B}. i.c., each sample σ, E D is sampled with certain probability (1). Our objective
design the sampling frequency P.(i) that can emphasize underrepresented
samples. Based on the observations Section 3, we first devise the discrepancy score s(x;; T) that
reflects the underrepresentedness of each sample follows:
s(r;; T) LDRM(
LDRRV(12 (T;
where is the set of steps used to calculate the discrepancy scores and & is the hyperparameter to modulate the contribution of each statistic. The score (7) can be interpreted as upper limit of the confidence interval of LDR estimate, or weighted sum LDRM and the square root of LDRV with weight controlled by k. To ensure every data is sampled with least some chance, clip the minimum value of s(x;) to be e = 0.01 (min_clipp) and clip the maximum value to have //min s(x) s<x)/min
max min ratio of 50, i.c., max
50 (max_clip). For the clipped score s' s((2;;
<(nTT) max_ mex_ccip(min,. [bx.cciipminncccipp((:::::: T)), our final weighted sampling frequency is (1) Σ, s'(y)TT)
4.2 Sample analysis of the discrepancy score
To check whether our discrepancy score indeed captures the underrepresented samples, we analyze the samples with lowest/highest discrepancy scores. We train SNGAN [25] CIFAR-10 [17] for 40k steps and measure the discrepancy score of each sample. We first present the images with lowest (Fig. 3a)/highest (Fig. 3b) discrepancy scores among training images, and compare them with generated samples (Fig, 3c). High-scoring images have properties that are distinct from the generated samples (e.g., unusual background or shape), while low-scoring images contain features that are also available in generated samples. Comparing the pixel intensity histogram (Fig. 3d) reveals the difference more clearly in sample properties. Images with lowest discrepancy scores exhibit similar intensity distribution with generated samples, while images with highest scores appear to show an extremely different tendency. We also analyze the Partial FID (FID [13] measured with a subset of training samples) of lowest/highest-scoe groups. The highest-score group has Partial FID of 94.64 while the lowest-score group has 22.43. The large gap between the two groups states that the generator fails to generate samples similar to high-scor group. These results imply that our discrepancy score

---

successfully identifies underrepresented data that may need emphasis in further training. For more examples of images with scores both for CIFAR-10 and CelebA, see the Appendix $C.
4.3 Post-processing by discriminator rejection sampling with auxiliary discriminator
Our weighted sampling gives bias toward underrepresented samples during training. Though effective improving diversity, this results in modified data distribution Paata()) (x) = = [(x)) (x) where f()) is the normalized sampling frequency. Thus, the trained model distribution Pg may be different from the original data distribution Pdata- To solve this, we utilize the Discriminator Rejection Sampling (DRS) [3] to correct the bias after training. The rejection sampling accepts generated sample with probability (2r for some constant M O. To conduct rejection sampling. DRS method needs an Mp, (x)
estimate for Pdata ( (2)r) calculated based on the discriminator outputs. Since our discriminator is trained with biased Paata (a), we add an auxiliary diseriminator and train it with uniform sampling (i.e., without applying our sampling technique) during the weighted sampling procedure to obtain the LDR estimate (1) for DRS, and use this measure for the rejection sampling of generated samples.
4.4 Self-Diagnosing GAN (Dia-GAN)
The overall algorithm (with details in the Appendix (D) can be summarized as below:
Phase 1 - Train and Diagnose: Train GAN and evaluate the discrepancy score for each data instance. Phase 2 Score-Based Weighted Sampling: Encourage GAN to learn underrepresented regions of data manifold through score- score-based weighted sampling (Section 4.1). Phase 3 DRS: After GAN training, correct the model distribution Pg by rejection sampling.
Experiments
5.1 Evaluation metrics and baselines
Evaluation metrics To evaluate the effect of our method on learned model distribution, we use various performance metrics including (1) Fréchet Inception Distance (FID) [13]. (2) Inception Score (IS) [28]. and (3) Precision and Recall (P&R) [18]. In addition to these global evaluation metrics, consider (4) Reconstruction Error (RE). RE score is calculated by first training convolutional autoencoder (CAE) with generated samples, and then calculating Euclidean distance between each training data and its reconstruction. RE can assess whether Py (z) covers Pdata (z) since CAE is known to have high RE for out-of-distribution samples [41, 44]. For more details, see the Appendix $E.
Baselines We compare the effect of our method with other methods that use the discriminator output for improving GAN training: 1) DRS [3], 2) Gap of log-densities (GOLD) [26], and 3) Top-k training [30]. GOLD* uses the LDR estimate on generated samples re-weight underrepresented samples (having high LDR) during training of GANs. Top-k training uses only top-k fake samples with the largest discriminator outputs, i.e., the samples believed to be the "most realistic", during the training of the generator. As our algorithm uses DRS after the training, we also analyze each method's performance with post-processing by DRS to measure the exact gain from our sampling method.
5.2 GAN performance enhancement real datasets
Experiments on CIFAR-10 and CelebA We first assess our method on two widely-studied GAN benchmark datasets, CIFAR-IO [17] and CelebA [21]. We evaluate our method on state-of-the-ard GANs; SNGAN [25] and SSGAN [36] with non-saturating variant of the original loss. We train our model for 50k (75k) steps for CIFAR-10 (CelebA), where for our method and GOLD, the phase 1 takes 40k (60k) steps, and the phase 2 takes the remaining. We record LDR every 100 steps and use the last 50 records for calculating the discrepancy score. For the discrepancy score (7), we use k = 0.3 (5.0) for CIFAR-IO (CelebA). Detailed configurations and hyperparameter search procedure are available in the Appendix $F.
4 As the original GOLD estimator is designed for conditional GANs [24], we consider the unconditional version by removing the conditional discrepancy term.
7

---

Table 2: Comparison of diverse sampling/weiihhingg methods for CIFAR-IO/CelebA image generation.
Dataset
CIFAR-10
CelebA
SNGAN
SSGAN
SNGAN
SSGAN
Methods
FID 4
IS 1
FID 4
IS 1
FID 4
P1
R
FID
P1
R 1
Vanilla
26.90
7.36
22.01
7.65
7.12
0.68
0.44
7.19
0.68
0.44
DRS [3]
24.54
7.57
20.51
7.77
7.04
0.68
0.44
7.08
0.68
0.45
GOLD [26]
28.86
7.21
21.90
7.57
7.31
0.69
0.44
7.46
0.68
0.43
GOLD + DRS
24.65
7.53
19.36
7.79
6.97
0.68
0.44
7.15
0.67
0.45
Top-k [30]
24.45
7.60
20.01
7.78
7.35
0.67
0.44
7.23
0.67
0.45
Top-k DRS
23.92
7.70
20.09
7.88
7.35
0.68
0.44
7.16
0.68
0.45
Dia-GAN
19.66
7.95
16.31
8.14
6.70
0.64
0.48
6.88
0.66
0.46
Table 3: StyleGAN2 on FFHQ 256x256.
Table 4: HingeGAN CIFAR-10 and CelebA.
FID
P
R f
CIFAR-10
CelebA
StyleGAN2 GOLD Dia-StyleGAN2
14.07
0.72
0.27
FID +
IS f
FID +
15.53
0.69
0.29
HingeGAN
21.99
7.67
6.66
11.89
0.69
0.30
Dia-HingeGAN
18.74
8.02
5.98
In Table first compare FID and over various methods on the CIFAR-10 dataset. Our proposed Dia-GAN achieves the best FID and IS with a great margin among all baseline methods in every GAN variant. This result demonstrates the wide applicability and effectiveness of our method in improving the overall quality (fidelity and diversity combined) of generated samples. Moreover, the comparison between DRS and our method assures that most of the gain indeed comes from our resampling method. Also, we compare FID and P&R over the methods on CelebA. Our method consistently improves FID over baseline GANs. Precision & Recall analysis shows more detailed reasons for the improvement of FID. Our method consistently improves recall (diversity) but with slight drop in precision (fidelity). As the increase diversity dominant, FID, which measures the combined effect of fidelity and diversity, is consistently improved with our method compared to the baselines. Examples generated samples from our Dia-GAN are also available in the Appendix $G.
Experiments on StyleGAN2 We further evaluate the scalability of our method with Style- GAN2 [16] FFHQ 256x256 [15] dataset. We train the model for 250k steps in total where phase takes 200k steps and the phase takes the remaining steps. We set the hyperparameter k == 3.0. Our method improves the FID of StyleGAN2 from 14.07 to 11.89 and recall of StyleGAN2 from 0.27 to 0.30 as shown in Table 3. This indicates that our method successfully scales to large state-of-the of-tthe-rtttttt GANs and high- nigh-resolution images.
Extension to hinge loss We further conduct experiments to show the applicability of our method to other GAN losses. Here, we focus on a commonly used loss, the hinge loss (HingeGAN) [19,34], Our method is not directly applicable the hinge loss since the output of the optimal discriminator D₁ (x)
is not (1, anymore. Instead, D (x) if (2) > (x(x) and if Pdata (z) < (x)).
(a)
One possible workaround is attaching an auxiliary layer to the discriminator and training it with the original GAN loss. However, we instead present empirical evidence showing that D₁ (x) itself still contains useful information about the degree of learning for the input We consider the variant of our method, -ingeiig by calculating the mean and variance of D (x) and using the same scoring rule of (7). In Table 4, we compare the performance of HingeGAN and Dia- Miaa-iineeGAN with the same configuration of the previous experiment. Interestingly, our method shows significant improvement in both CIFAR-10 10 and CelebA. This implies that despite the optimal form of the discriminator is different, the statisties of its output still provide meaningful information about the underrepresented features. We leave the thcoretical analysis of this variant method future work.

---

Table Reconstruction Error (RE) score of green (minor) training samples in Colored MNIST and
FMNIST (minor) samples in a mixture of MNIST and FMNIST on different majority rate p.
Dataset
Colored MNIST
MNIST-FMNIST
Majority rate p
99%
95%
90%
99%
95%
90%
Vanilla
0.838
0.236
0.218
0.290
0.227
0.215
GOLD [26]
0.813
0.297
0.200
0.296
0.241
0.218
Top-k [30]
0.831
0.210
0.223
0.281
0.232
0.221
PacGAN [20]
0.810
0.244
0.233
0.313
0.251
0.225
Inclusive GAN [43]
0.812
0.274
0.216
0.283
0.230
0.220
Dia-GAN
0.224
0.204
0.197
0.264
0.219
0.206
Table 6: CelebA minor attribute analysis. Averaged LDRV and averaged discrepancy score of CelebA samples with (W/) or without (W/O) minor attributes. stands for the occurrence of minor attributes among the generated samples in percentage (%) and stands for the Partial Recall.
Score
Method
LDRV
Discrepancy
Vanilla
Dia-GAN
W/
W/O
W/
W/O
f
R t
0f
R f
Bald (2.244%)
0.271
0.184
2.938
2.221
0.678
0.353
0.836
0.393
Double Chin (4.669%
0.219
0.184
2.525
2.224
0.440
0.411
0.522
0.461
Eyeglasses (6.512%)
0.254
0.181
2.783
2.200
3.300
0.400
4.053
0.449
Gray Hair (4.195%)
0.211
0.185
2.450
2.228
2.273
0.402
2.369
0.436
Mustache (4. 155%)
0.242
0.183
2.699
2.218
0.157
0.391
0.228
0,433
Pale Skin (4.295%)
0.190
0.186
2.240
2.238
0.346
0.380
0.453
0.427
Wearing Hat (4.846%)
0.357
0.177
3.651
2.164
2.307
0.380
3.595
0.408
5.3 Minor feature generation
Controlled experiments As method emphasizes underrepreeeenee samples in GAN training, we evaluate how much our method helps the generation of minor samples. To control the level of minority, design Colored MNIST dataset with red (major) and green (minor) samples, and MNIST-FNIST dataset with MNIST (major) FMNIST (minor) samples, with the majority rates p E (90, 95, 99)% %. We compare our method with the same of baseline methods in Section 5.1. Additionally, we compare our method with PacGAN [20], the approach to handle the mode collapse problem, and with Inclusive GAN [43], which also improves the data coverage over the minor groups by using a hybrid generative model.
Table 5 shows the results of each method in various majority rates. Here, focus the reconstruc- tion error (RE) score of minor training samples (green samples for Colored MNIST, and FMINST samples for MNIST-FMNIS method shows a significant improvement in RE score as only our method succeeds in generating minor samples while others fail. When the majority rate p decreases to 95% and 90%, vanilla model starts to generate minor samples but in low quality. For these rates, our method also shows improvement on the quality of generated samples with minor features, resulting in better RE scores. This result implies the efficacy of our method in improving the quality of generated samples with underrepresented features. Examples of samples with minor features for each method are available in the Appendix I1, and detailed configuration of the experiment is available in the Appendix $F.
dataset). For the training dataset with the majority rate of 99%,
For the weal-world example, we analyze how our method changes the generation of minor attributes of the CelebA [21] dataset, using the mnta-information available in dataset. Specifically, we focus on how much our method improves the occurrence rate of minor attributes, as they usually appear in much lower rate than its actual ratio. We train a binary classifier for each attribute to have train and test accuracy above 95%. We also evaluate the Partial Recall of the minor attributes, since minor samples suffer low-recall problem as explained in Section 3.2.
CelebA minor attribute analysis

---

Vanilla
Dia GAN
Figure 4: Generated samples vanilla SNGAN and Dia-SNGAN trained on CelebA
As shown in Table method improves both the occurrence (O) and Partial Recall (R) rates of ious minor attributes. Moreover, as explained Section 3.2, minor samples do have higher LDRV (2) and the discrepancy score (7). This indicates that our method indeed captures the underrepresented minor features and successfully promotes the generation of such features during training of GANs. Figure shows examples of generated samples with minor feature appeared by our Dia-GAN, Note that as we use the majority of the training time for Phase I (80% of total steps), the generator partially converges after Phase and thus the same latent vector 2 turns out give similar image with details changed after Phase 2 (e.g., wearing sunglasses, a hat).
The ability of method in capturing semantic features and improving the generation of minor samples also applies to high-resolution datasets. To demonstrate this, we conduct similar experiments for the high-resolution FFHQ dataset and present the results in the Appendix SH.
6 Discussion
We proposed two new metrics, LDRV and LDRM, that can detect underrepresented samples and devised simple approach emphasize detected underrepresented samples. Our method successfully improves overall quality of the generated samples in terms of FID and IS, and promotes generation of minor samples. However, we still find the trade-off relationship between the precision and recall of generated samples (Table 2). We leave the investigation of other approaches to the knowledge of detected underrepresented samples for further improvement GAN training as future work.
Societal impact We propose discrepancy score that can detect underrepressnteed minor samples training of GANs. On the good side, this results in enhanced generation of minor samples in GANs. The ability of our score could be further expanded and used for utilizing skewed datasets to train models representing more balanced datasets, by adding hyperparameter that can tune the level of emphasis for underrepresented samples. On the other hand, an abuser might instead be able to remove such minor subgroup samples and deteriorate the bias in sample generation.
Acknowledgement
This research was supported by the National Research Foundation of Korea under Grant 2017R IEI A A01076340 and 002RIICIC11005333 and by the Ministry of Science and ICT, Korea, under the ITP (Institute for Information and Communications Technology Panning and Evaluation) grant (No. N6..2020--000626)
10

---

References
[1] Jonas Adler and Sebastian Lunz. Banach wasserstein GAN. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa-Bianchi, and Roman Garnett, editors, Advances Neural Information Processing Systems, pages 6755-6764, 2018.
[2] Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. In International Conference Machine Learning, pages 214-223, PMLR, 2017.
[3] Samanch Azadi, Catherine Olsson, Trevor Darrell, Ian Goodfellow, and Augustus Odena. Discriminator rejection sampling. In International Conference on Learning Representations. 2019.
[4] David Berthelot, Thomas Schumm, and Luke Metz. Began: Boundary equilibrium generative adversarial networks. arXiv preprint arXiv: 10717, 2017.
[5] Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. arXiv preprint arXiv: 1096, 2018.
[6] Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale GAN training for high fidelity natural image synthesis. In 7th International Conference on Learning Representations, ICLR, 2019.
[7] Haw-Shiuan Chang, Erik Learned-Miller, and Andrew McCallum. Active bias: Training more accurate neural networks by emphasizing high variance samples. In Advances in Neural Information Processing Systems, pages 1002-1012, 2017.
[8] Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. Infogan: Interpretable representation learning by information maximizing generative ad adversarial nets, In Daniel D. Lee, Masashi Sugiyama, Ulrike von Luxburg, Isabelle Guyon, and Roman Garnett, editors, Advances in Neural Information Processing Systems, pages 2172-2180, 2016.
[9] Terrance Deiri, Michal Drozdzal, and Graham W. Taylor. Instance selection for gans. In Advances Neural Information Processing Systems, 2020.
[10] Xin Ding, Z Jane Wang, and William Welch. Subsampling generative adversarial networks: Density ratio estimation in feature space with softplus loss. IEEE Transactions on Signal Processing, 68:1910-1 2020.
[11] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in Neural Information Processing Systems, pages 2672-2680, 2014.
[12] Aditya Grover, Jiaming Song. Ashish Kapoor, Kenneth Tran, Alekh Agarwal, Eric Horvitz, and Stefano Ermon. Bias correction of learned generative models using likelihood-free importance weighting. In Advances in Neural Information Processing Systems, pages 11056-1 1066, 2019.
[13] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge a local nash equilibrium. In Advances in newral information processing systems, pages 6626-6637, 2017.
[14] Phillip Isola, Jun-Yan Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-image translation with conditional adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1125-1134, 2017.
[15] Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In CVPR, pages 4401-4410. Computer Vision Foundation / IEEE, 2019.
[16] Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of stylegan. In 2020 JEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8107-8116 IEEE, 2020.
[17] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[18] Tuomas Kynkäänniemi, Tero Karras, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Improved precision and recall metric for assessing generative models. Advances in Neural Information Processing Systems, pages 3929-3938, 2019.
[19] Jae Hyun Lim and Jong Chul Ye. Geometric gan. arXiv preprint arXiv: 170.1000502.4...

---

[20] Zinan Lin, Ashish Khetan, Giulia Fanti, and Sewoong Oh. PacGAN: The power of two samples in generative adversarial networks. IEEE Journal on Selected Areas in Information Theory, 1(1):324-30) -35, 2020.
[21] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of the IEEE international conference on computer vision, pages 3730-3738, 2015.
[22] Michael Mathieu, Camille Couprie, and Yann LeCun. Deep multi-scale video prediction beyond mean square error. arXiv preprint arXiv: 1.05440, 2015.
[23] Luke Metz, Ben Poole, David Pfau, and Jascha Sohl-Dickstein. Unrolled generative adversarial networks. In ICLR (Poster). OpenReview.net, 2017.
[24] Mehdi Mirza and Simon Osindero. Conditional generative adversarial nets. arXiv preprint arX.v.1411.1784, 2014.
[25] Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normal- ization for generative adversarial networks. In 6th International Conference on Learning Representations, ICLR. OpenReview.net, 2018.
[26] Sangwoo Mo, Chiheon Kim, Sungwoong Kim, Minsu Cho, and Jinwoo Shin. Mining gold samples for conditional gans. In Advances in Neural Information Processing Systems, pages 1170-6181, 2019.
[27] Mihaela Rosca, Balaji Lakshminarayanan, David Warde-Farley, and Shakir Mohamed. Vari- ational approaches for auto-encoding generative adversarial networks. arXiv preprint 17X6::1700.099777 2017.
[28] Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. In Advances in Neural Information Processing Systems, pages 2234-2242, 2016.
[29] Abhinav Shrivastava, Abhinav Gupta, and Ross Girshick. Training region- region-based object detectors with online hard example mining. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 761-769, 2016.
[30] Samarth Sinha, Zhengli Zhao, Anirudh Goyal ALIAS PARTH GOYAL, Colin A Raffel, and Augustus Odena. Top-k training of gans: Improving gan performance by throwing away bad samples. Advances in Neural Information Processing Systems. 33, 2020.
[31] Jiaming Song and Stefano Ermon. Bridging the gap between f-gans and wasserstein gans. In Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 9078-9087 PMLR, 2020.
[32] Akash Srivastava, Lazar Valkov, Chris Russell, Michael U. Gutmann, and Charles Sutton. VEEGAN: reducing mode collapse in gans using implicit variational learning, In NIPS, pages 3308-3318, 2017.
[33] Swabha Swayamdipta, Roy Schwartz, Nicholas Lourie, Yizhong Wang, Hannaneh Hajishirzi, Noah A. Smith, and Yejin Choi. Dataset cartography: Mapping and diagnosing datasets with training dynamics. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 9275-9293, 2020.
[34] Dustin Tran, Rajesh Ranganath, and David M. Blei. Hierarchical implicit models and likelihood- free variational inference. In Advances in Neural Information Processing Systems, pages 5523-5533, 2017.
[35] Ngoc-Trung Tran, Tuan-Anh Bui, and Ngai-Man Cheung. Dist-gan: An improved GAN using distance constraints. In Vittorio Ferrari, Martial Hebert, Cristian Sminchisescu, and Yair Weiss, editors, Computer Vision ECCV 2018- 15th European Conference, volume 11218 of Lecture Notes in Computer Science, pages 387-401. Springer, 2018.
[36] Ngoc-Trung Tran, Viet-Hung Tran, Ngoc-Bao Nguyen, Linxiao Yang, and Ngai-Man Cheung. Self-supervised GAN: analysis and improvement with multi-class minimax game. In Advances in Neural Information Processing Systems, pages 13232- 132433, 2019.
[37] Ryan Turner, Jane Hung, Eric Frank, Yunus Saatchi, and Jason Yosinski. Netropolis-hasting: generative adversarial networks. In International Conference on Machine Learning, pages 6345-6353. PMLR, 2019.
12

---

[38] Xiaoxia Wu, Ethan Dyer, and Behnam Neyshabur. When do curricula work? arXiv preprint arXiv:2012.03107. 2020.
[39] Yan Wu, Jeff Donahue, David Balduzzi, Karen Simonyan, and Timothy Lillicrap. Logan: Latent optimisation for generative adversarial networks. arXiv preprint arXiv: 2019.
[40] Yue Wu, Pan Zhou, Andrew Gordon Wilson, Eric Xing, and Zhiting Hu. Improving GAN training with probability ratio clipping and sample reweighting, In Advances in Neural Informa- lion Processing Systems, 2020.
[41] Yan Xia, Xudong Cao, Fang Wen, Gang Hua, and Jian Sun. Learning discriminative reconstruc- tions for unsupervised outlier removal. In 2015 IEEE International Conference on Computer Vision, ICCV, pages 1511-1519, IEEE Computer Society, 2015.
[42] Jiahui Yu, Zhe Lin, Jimei Yang, Xiaohui Shen, Xin Lu, and Thomas S Huang. Generative image inpainting with contextual attention. Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5505-5514, 2015-.
[43] Ning Yu, Ke Li, Peng Zhou, Jitendra Malik, Larry Davis, and Mario Fritz. Inclusive gan: Improving data and minority coverage in generative models. In European Conference on Computer Vision, pages 377-393. Springer, 2020.
[44] Chong Zhou and Randy C. Paffenroth. Anomaly detection with robust deep autoencoders. In Proceedings of the 23rd ACM SIGKDD International Conference Knowledge Discovery and Data Mining, pages 665-674. ACM, 2017.
[45] Mun-Yan& Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In Proceedings of the IEEE international conference on computer vision, pages 2223-2232, 2017.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Section 6.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See
Section 6.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] See Section
3.2 and Appendix $B.
(b) Did you include complete proofs of all theoretical results? [Yes] See Section 3.2 and Appendix $B.
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed reproduce the main ex- perimental results (either in the supplemental material or as a URL)? [Yes] They are contained in the supplemental material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Appendix $F.
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes] See Appendix $J.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Appendix $D.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
13

---

(d) Did you discuss whether and how consent was obtained from people whose data you' re using/curating? [N/A]
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "OhNlAwGLziQ", "reviewer": "Reviewer_FXfS", "summary": "This submission addresses the problem of limited diversity in GANs due to not modeling underrepresented samples. The authors propose two metrics based on the log-density ratio (LDR), namely, its mean and variance over time. This allows the model to monitor underrepresented and underrepresented samples. Then, the authors propose a training method where data is sampled according to an heuristic based on those two metrics. The results outperform other related methods used for the same purpose, across different architectures and datasets.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality:\nThe method is built on top of previous GAN frameworks and the LDR statistic, which are described clearly in the submission. The two metrics proposed are straightforward yet effective. Score-based sampling is also not novel, although the authors propose also a simple yet effective method. \n\nQuality:\nThe submission is sound, and the claims are supported by experiments with different GAN frameworks in both synthetic and realistic datasets.\n\nClarity:\nThe organization and writing are clear. The toy examples with synthetic datasets are very illustrative.\n\nSignificance:\nThe idea is sound, yet borrowing heavily from previous works, and achieves a moderate gain in most experiments. Other practicioners can benefit from this approach. Similarly, the paper also includes useful insights.  ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors mention the in the conclusion. I think the authors should also include some words on the misuse of GANs.", "review_text": "", "extra_scores": {"needs_ethics_review": "Yes", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "D81nlqMlMRk", "reviewer": "Reviewer_bY4D", "summary": "This paper proposes a sample reweighing scheme for GANs that emphasizes underrepresented samples in the training data. Log-density ratio statistics from the discriminator are used to reweigh the sample probability of data during training, ultimately resulting in more diverse generations.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- As far as I can tell, this is the first work to propose reweighing real samples for unconditional GAN training. Previous work has reweighed fake samples or performed instance selection on real samples (which could be seen as a binary form of reweighing), but none have used the discriminator to inform sampling probabilities of real data.\n- Problem is motivated both empirically and theoretically.\n- Includes proof-of-concept experiments on synthetic datasets, as well as experiments on real world datasets, all demonstrating improvement from the proposed technique.\n- Nice visual examples demonstrating high and low discrepancy scores (Figure 3a-c).\n- Well written and organized.\n- Includes many extra details in appendix, including sufficient details for reproducing experiments.\n- Work addresses an important limitation of GANs. GANs are notorious for having poor diversity, and recent work has demonstrated many cases of bias in generative models. Being able to improve diversity while still retaining decent image quality is a useful contribution.\n\nWeaknesses:\n- Proposed method introduces additional hyperparameters that needs to be tuned (lengths of stage 1 and 2), k value. Not a huge problem since they can be optimized with grid search, but does add an extra level of complexity (and compute/training time) to the solution.\n- Missing ablation study of Dia-GAN, specifically Dia-GAN without DRS (i.e., weighted sampling only).\n- Missing Precision and Recall metrics for CIFAR-10 and FFHQ 256x256 experiments. Since the goal of this work is to improve diversity, I think it is important to include these metrics for all experiments. Inception Score could be moved to appendix or removed completely as it isn't very informative in this setting.\n\nQuestions:\n- In Algorithm 1 it appears that the sampling probabilities are set a single time, at the end of stage 1. Have you considered adjusting the sampling probabilities multiple times throughout training, and would this be expected to yield a better final result?\n- How would vanilla GAN + DRS perform if combined with the truncation trick [1,2] with truncation values >1? That is, multiply the latent noise by some value >1, like 1.5 or 2 during inference. This method improves sample diversity in exchange for image quality, but I am curious whether combination with DRS would avoid the quality degradation issue. This would be a good simple baseline to compare to.\n- In Figure 4, how are the samples so similar? This is unusual for unconditional models. Is the vanilla GAN from stage 1 of Dia-GAN training? If so this should be explained in the caption.\n- Are all GANs trained to convergence, or are they stopped early? From what I can tell, the number of iterations for training the GANs in this paper is much less than normal (e.g. ~50k iterations vs 500k iterations in [3] for CIFAR10, 4000k images vs 10000k/25000k images in [4] for FFHQ). This is important because standard GANs generally learn to represent minority samples later in training, so early stopping could produce misleading results.\n\nOther comments/minor concerns:\n- In Figure 1e, what does \"minority level\" mean? Is it the percentage of the dataset composed of minority samples? If so, it would be nice if the graph reflected the actual percentages on the x axis.\n- In Figure 1, it would be nice if 1e were on the left side of the image, since single-mode Gaussian is the first one described in the caption (or swap the order in the caption).\n- Figure 1, claims that DCGAN was used to train single-mode Gaussian, but appendix says that non-convolutional GAN was used.\n- In Figure 3d, the concatenation of RGB channels in the histogram feels weird to me, although I'm not sure if there are any better ways to show this data. Perhaps you could try a separate subplot for each colour channel stacked vertically? Your choice.\n\nOverall, I think this is a decent paper. The proposed technique is reasonable and seems effective, if a bit clunky (requires staged training, additional hyperparameter tuning, and DRS to fix distribution shift). Currently I recommend acceptance, but very weakly. I would be much reassured if the authors would be able to address my questions above, particularly with regards to early stopping. \n\n[1] Brock, Andrew, Jeff Donahue, and Karen Simonyan. \"Large scale GAN training for high fidelity natural image synthesis.\" arXiv preprint arXiv:1809.11096 (2018).  \n[2] Karras, Tero, Samuli Laine, and Timo Aila. \"A style-based generator architecture for generative adversarial networks.\" arXiv preprint arXiv:1812.04948 (2018).  \n[3] https://github.com/ajbrock/BigGAN-PyTorch/blob/master/scripts/launch_cifar_ema.sh  \n[4] https://github.com/NVlabs/stylegan2", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Authors mention positive impact of making sure that machine learning models include minorities.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4.5"}, "extra_text": {}}, {"review_id": "2Dsp4UVFEU", "reviewer": "Reviewer_n1Zt", "summary": "The paper proposes a scoring to measure to what extend an image in a dataset is characterized by an under-represented visual appearance. GANs fail to accurately model these under-represented modes of the data distribution or drop them completely. Here, the score is based on the discriminator predictions on real images, and is used as a basis for score-weighted sampling of real data during training, to show under-represented real images more often. The method is shown to improve mode coverage and to result in better image quality/diversity than the considered baselines.\n", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n\n-\tThe paper studies an important and challenging problem. The ability to better learn underrepresented modes has importance for fairness and is not well studied in the GAN literature. \n\n-\tThe proposed method falls into the category of sampling and reweighting methods for image generation. It is used to boost underrepresented modes of the data. The main novelty is that this method is applied to real instead of fake samples, the advantage of which is explained on lines 86-89.\n\n-\tThe theoretical support is provided for the hypothesis that the proposed scoring detects samples from underrepresented modes. \n\n- The method also shows a notable improvement in image quality for a state-of-the-art GAN (StyleGANv2) in Table 3.\n\nWeaknesses:\n\n-\tPerforming the comparison in Table 2 also for StyleGAN2 and on a more complex dataset with a higher resolution (e.g. ImageNet or FFHQ) would increase the value of the paper. Since state-of-the-art models are of the more interest to the community, and for more complex, higher resolution datasets it is more challenging to capture underrepresented samples. The discriminator would have a much harder job to capture the differences between samples, e.g. semantic classes of ImageNet. \n\n-  The discriminator predictions are used to estimate the discrepancy score, which is used for weighted sampling during training. Thus it would be interesting to see how discriminator network architecture, its design and capacity, influense the performance of the proposed approach. As the score is highly dependent on the discriminator's ability to learn the features which are best describe the data.\n\n-\tI’m still doubtful how effective the proposed approach and scores would be on more realistic cases, when some semantic classes are underrepresented. Based on Fig.3 it seems that the high discrepancy score captures more variance in terms of colors and not high-level features. It would be beneficial to see the analysis in Table 1 and Fig.3 on e.g. ImageNet, where number of images per class is quite imbalanced. \n\n-\tMost of the experiments are performed using datasets with low-resolution images, exploiting “outdated” GAN models, such as DCGAN, SNGAN, SSGAN. \n\n\n\n- The writing and clarity of the paper can be improved:\n\nLine 15 (abstract): It is not very clear what is meant with “minor features” when reading the abstract for the first time.\n\nLine 43/44: What is the difference between minor data instances and under-represented samples? Especially at this point of the text that is not clear.\n\nSection 3.2 and Fig. 1(e): It should be explained in the main text what the “minority levels” stand for, not in the appendix.\n\nThe paragraph from line 151-160 is hard to follow and should be described in more detail.\n\nSection 4.4: How does the number of steps in phase 1 affect the outcome? Likewise, is there an optimal number of steps for phase 2?\n\nSection 5.2.: Please specify the resolution of the CIFAR10 and CelebA experiments.\n\nLine 313: “Figure 4 shows examples of generated samples with minor feature appeared by our Dia-GAN.” This sentence should be rephrased.\n\nHow was it possible to create almost the same images for SNGAN and Dia-SNGAN in Fig. 4? Even when the latent codes are the same and all seeds are fixed, that is very unlikely to happen. Was the same checkpoint of SNGAN copied after phase 1 and then one copy continued to be trained as SNGAN and the other as Dia-SNGAN in phase 2?\n\nMinor comments:\n\n- Line 26: There is a misplaced dot on this line (“.”)\n- Typo in Fig. 3(d): “Historgram”\n\n\n---\n## Post-rebuttal feedback \n\nThanks to the authors for their feedback and clarifications. Additional experiments and ablations requested by reviewers have been provided, showing the advantages of the proposed approach. I still think it would be beneficial for the paper if the method is also tested on the datasets containing multiple semantic classes, where some of the classes are severely underrepresented, such as ImageNet. Cifar or the faces datasets, such as Celeba or FFHQ, might be too simplistic for this setting. Nevertheless, I think the proposed approach has a potential to be adopted by practitioners and may encourage further research in this direction. Thus I'm more inclined towards acceptance and raise my score to 6.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations and positive societal impact are discussed in section 6, with one sentence each. A negative societal impact is not mentioned, thus it would be good to add the discussion about it to the paper.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4.5h"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper addresses the problem of mode-dropping of underrepresented areas of the data manifold in the context of GANs. The main result relies on a reweighing scheme for the real data, which is novel according to the reviewers.

Two reviewers think the paper is clear and well written while the third one believes the clarity could be improved.

One recurring comment is that the experimental section could be stronger by using more recent models and larger datasets.

In general, the reviewers tend to agree that this is an incremental progress, based on other papers, but all agree that it is important enough to be published.

Ethics: The reviewers only highlight general possible ethics issues which are common to most GAN (and generative models). The authors did acknowledge some of these issues in the paper and their answers to the reviewers, along to the planned editing, are satisfactory.

Given all that, and the general consensus for acceptance, I recommend this paper to be published as a poster.
Decision: Accept (Poster)