Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: This submission addresses the problem of limited diversity in GANs due to not modeling underrepresented samples. The authors propose two metrics based on the log-density ratio (LDR), namely, its mean and variance over time. This allows the model to monitor underrepresented and underrepresented samples. Then, the authors propose a training method where data is sampled according to an heuristic based on those two metrics. The results outperform other related methods used for the same purpose, across different architectures and datasets.
main_review: Originality:
The method is built on top of previous GAN frameworks and the LDR statistic, which are described clearly in the submission. The two metrics proposed are straightforward yet effective. Score-based sampling is also not novel, although the authors propose also a simple yet effective method. 

Quality:
The submission is sound, and the claims are supported by experiments with different GAN frameworks in both synthetic and realistic datasets.

Clarity:
The organization and writing are clear. The toy examples with synthetic datasets are very illustrative.

Significance:
The idea is sound, yet borrowing heavily from previous works, and achieves a moderate gain in most experiments. Other practicioners can benefit from this approach. Similarly, the paper also includes useful insights.  
limitations_and_societal_impact: The authors mention the in the conclusion. I think the authors should also include some words on the misuse of GANs.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper proposes a scoring to measure to what extend an image in a dataset is characterized by an under-represented visual appearance. GANs fail to accurately model these under-represented modes of the data distribution or drop them completely. Here, the score is based on the discriminator predictions on real images, and is used as a basis for score-weighted sampling of real data during training, to show under-represented real images more often. The method is shown to improve mode coverage and to result in better image quality/diversity than the considered baselines.

main_review: Strengths

-	The paper studies an important and challenging problem. The ability to better learn underrepresented modes has importance for fairness and is not well studied in the GAN literature. 

-	The proposed method falls into the category of sampling and reweighting methods for image generation. It is used to boost underrepresented modes of the data. The main novelty is that this method is applied to real instead of fake samples, the advantage of which is explained on lines 86-89.

-	The theoretical support is provided for the hypothesis that the proposed scoring detects samples from underrepresented modes. 

- The method also shows a notable improvement in image quality for a state-of-the-art GAN (StyleGANv2) in Table 3.

Weaknesses:

-	Performing the comparison in Table 2 also for StyleGAN2 and on a more complex dataset with a higher resolution (e.g. ImageNet or FFHQ) would increase the value of the paper. Since state-of-the-art models are of the more interest to the community, and for more complex, higher resolution datasets it is more challenging to capture underrepresented samples. The discriminator would have a much harder job to capture the differences between samples, e.g. semantic classes of ImageNet. 

-  The discriminator predictions are used to estimate the discrepancy score, which is used for weighted sampling during training. Thus it would be interesting to see how discriminator network architecture, its design and capacity, influense the performance of the proposed approach. As the score is highly dependent on the discriminator's ability to learn the features which are best describe the data.

-	I’m still doubtful how effective the proposed approach and scores would be on more realistic cases, when some semantic classes are underrepresented. Based on Fig.3 it seems that the high discrepancy score captures more variance in terms of colors and not high-level features. It would be beneficial to see the analysis in Table 1 and Fig.3 on e.g. ImageNet, where number of images per class is quite imbalanced. 

-	Most of the experiments are performed using datasets with low-resolution images, exploiting “outdated” GAN models, such as DCGAN, SNGAN, SSGAN. 



- The writing and clarity of the paper can be improved:

Line 15 (abstract): It is not very clear what is meant with “minor features” when reading the abstract for the first time.

Line 43/44: What is the difference between minor data instances and under-represented samples? Especially at this point of the text that is not clear.

Section 3.2 and Fig. 1(e): It should be explained in the main text what the “minority levels” stand for, not in the appendix.

The paragraph from line 151-160 is hard to follow and should be described in more detail.

Section 4.4: How does the number of steps in phase 1 affect the outcome? Likewise, is there an optimal number of steps for phase 2?

Section 5.2.: Please specify the resolution of the CIFAR10 and CelebA experiments.

Line 313: “Figure 4 shows examples of generated samples with minor feature appeared by our Dia-GAN.” This sentence should be rephrased.

How was it possible to create almost the same images for SNGAN and Dia-SNGAN in Fig. 4? Even when the latent codes are the same and all seeds are fixed, that is very unlikely to happen. Was the same checkpoint of SNGAN copied after phase 1 and then one copy continued to be trained as SNGAN and the other as Dia-SNGAN in phase 2?

Minor comments:

- Line 26: There is a misplaced dot on this line (“.”)
- Typo in Fig. 3(d): “Historgram”


---
## Post-rebuttal feedback 

Thanks to the authors for their feedback and clarifications. Additional experiments and ablations requested by reviewers have been provided, showing the advantages of the proposed approach. I still think it would be beneficial for the paper if the method is also tested on the datasets containing multiple semantic classes, where some of the classes are severely underrepresented, such as ImageNet. Cifar or the faces datasets, such as Celeba or FFHQ, might be too simplistic for this setting. Nevertheless, I think the proposed approach has a potential to be adopted by practitioners and may encourage further research in this direction. Thus I'm more inclined towards acceptance and raise my score to 6.
limitations_and_societal_impact: Limitations and positive societal impact are discussed in section 6, with one sentence each. A negative societal impact is not mentioned, thus it would be good to add the discussion about it to the paper.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes a sample reweighing scheme for GANs that emphasizes underrepresented samples in the training data. Log-density ratio statistics from the discriminator are used to reweigh the sample probability of data during training, ultimately resulting in more diverse generations.
main_review: Strengths:
- As far as I can tell, this is the first work to propose reweighing real samples for unconditional GAN training. Previous work has reweighed fake samples or performed instance selection on real samples (which could be seen as a binary form of reweighing), but none have used the discriminator to inform sampling probabilities of real data.
- Problem is motivated both empirically and theoretically.
- Includes proof-of-concept experiments on synthetic datasets, as well as experiments on real world datasets, all demonstrating improvement from the proposed technique.
- Nice visual examples demonstrating high and low discrepancy scores (Figure 3a-c).
- Well written and organized.
- Includes many extra details in appendix, including sufficient details for reproducing experiments.
- Work addresses an important limitation of GANs. GANs are notorious for having poor diversity, and recent work has demonstrated many cases of bias in generative models. Being able to improve diversity while still retaining decent image quality is a useful contribution.

Weaknesses:
- Proposed method introduces additional hyperparameters that needs to be tuned (lengths of stage 1 and 2), k value. Not a huge problem since they can be optimized with grid search, but does add an extra level of complexity (and compute/training time) to the solution.
- Missing ablation study of Dia-GAN, specifically Dia-GAN without DRS (i.e., weighted sampling only).
- Missing Precision and Recall metrics for CIFAR-10 and FFHQ 256x256 experiments. Since the goal of this work is to improve diversity, I think it is important to include these metrics for all experiments. Inception Score could be moved to appendix or removed completely as it isn't very informative in this setting.

Questions:
- In Algorithm 1 it appears that the sampling probabilities are set a single time, at the end of stage 1. Have you considered adjusting the sampling probabilities multiple times throughout training, and would this be expected to yield a better final result?
- How would vanilla GAN + DRS perform if combined with the truncation trick [1,2] with truncation values >1? That is, multiply the latent noise by some value >1, like 1.5 or 2 during inference. This method improves sample diversity in exchange for image quality, but I am curious whether combination with DRS would avoid the quality degradation issue. This would be a good simple baseline to compare to.
- In Figure 4, how are the samples so similar? This is unusual for unconditional models. Is the vanilla GAN from stage 1 of Dia-GAN training? If so this should be explained in the caption.
- Are all GANs trained to convergence, or are they stopped early? From what I can tell, the number of iterations for training the GANs in this paper is much less than normal (e.g. ~50k iterations vs 500k iterations in [3] for CIFAR10, 4000k images vs 10000k/25000k images in [4] for FFHQ). This is important because standard GANs generally learn to represent minority samples later in training, so early stopping could produce misleading results.

Other comments/minor concerns:
- In Figure 1e, what does "minority level" mean? Is it the percentage of the dataset composed of minority samples? If so, it would be nice if the graph reflected the actual percentages on the x axis.
- In Figure 1, it would be nice if 1e were on the left side of the image, since single-mode Gaussian is the first one described in the caption (or swap the order in the caption).
- Figure 1, claims that DCGAN was used to train single-mode Gaussian, but appendix says that non-convolutional GAN was used.
- In Figure 3d, the concatenation of RGB channels in the histogram feels weird to me, although I'm not sure if there are any better ways to show this data. Perhaps you could try a separate subplot for each colour channel stacked vertically? Your choice.

Overall, I think this is a decent paper. The proposed technique is reasonable and seems effective, if a bit clunky (requires staged training, additional hyperparameter tuning, and DRS to fix distribution shift). Currently I recommend acceptance, but very weakly. I would be much reassured if the authors would be able to address my questions above, particularly with regards to early stopping. 

[1] Brock, Andrew, Jeff Donahue, and Karen Simonyan. "Large scale GAN training for high fidelity natural image synthesis." arXiv preprint arXiv:1809.11096 (2018).  
[2] Karras, Tero, Samuli Laine, and Timo Aila. "A style-based generator architecture for generative adversarial networks." arXiv preprint arXiv:1812.04948 (2018).  
[3] https://github.com/ajbrock/BigGAN-PyTorch/blob/master/scripts/launch_cifar_ema.sh  
[4] https://github.com/NVlabs/stylegan2
limitations_and_societal_impact: Authors mention positive impact of making sure that machine learning models include minorities.
review_title: null

Meta-review: The paper addresses the problem of mode-dropping of underrepresented areas of the data manifold in the context of GANs. The main result relies on a reweighing scheme for the real data, which is novel according to the reviewers.

Two reviewers think the paper is clear and well written while the third one believes the clarity could be improved.

One recurring comment is that the experimental section could be stronger by using more recent models and larger datasets.

In general, the reviewers tend to agree that this is an incremental progress, based on other papers, but all agree that it is important enough to be published.

Ethics: The reviewers only highlight general possible ethics issues which are common to most GAN (and generative models). The authors did acknowledge some of these issues in the paper and their answers to the reviewers, along to the planned editing, are satisfactory.

Given all that, and the general consensus for acceptance, I recommend this paper to be published as a poster.
Decision: Accept (Poster)