Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper propses MaskedGAN to help GANs learn from the limited data by introducing two image masking strategies. One is the shifted spatial masking and the other is the balanced spectral masking. Those two masking strategies complement each other which together encourage GANs learning effectively and robustely on the limited data.
strengths_and_weaknesses: Strengths:

The motivation of this paper is clear, and the authors did lots of experiments to demonstrate the effectiveness of their method. But the experiment setting part needs to be improved as listed in the weaknesses or question part.


Weaknesses:

1. More data augmentation methods are supposed to be involved in the comparison of CIFAR and Imagenet. For example, StyleGAN2-ADA is required rather than the original StyleGAN2, especially with limited training data. Overall, some tables seem unfair since many baselines are without data augmentations, while the proposed method could be regarded as one novel data augmentation. 

2. Some important baselines are missing. For instance, InsGen and APA (both are NeurIPS 2021 work) achieved strong performances with limited training data on FFHQ, and AFHQ.

InsGen: Data-Efficient Instance Generation from Instance Discrimination, Yang. et al, NeurIPS 2021.
APA: Deceive D: Adaptive Pseudo Augmentation for GAN Training with Limited Data, Jiang, et al. NeurIPS 2021.
questions: 1. Differences from data augmentation? As it lacks the reconstruction part of MAE, the masking strategy can be regarded as another type of data augmentation. From this perspective, what if we have sufficient data, is improvement consistent? Besides, the relation to data augmentation is required to be discussed in related work.

2. As claimed in the contribution summary, the holistic understanding of images is encouraged. Is there any experimental support? 
limitations: The authors do not adequately addressed the limitations and potential negative societal impact of their work.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes two novel masked-based data augmentation methods for image generation task with GAN, namely shifted spatial masking and balanced spectral masking. Extensive experiments are conducted to demonstrate the effectiveness and generalizability of the proposed methods.
strengths_and_weaknesses: Pro:
1. The paper is well-written and easy to follow.
2. The proposed method is technically sound and intuitive. The paper is well-motivated. 
3. Extensive experiments are conducted to support the author's points.
4. Competitive performances are achieved by using the proposed method. The proposed method is generalizable enough to a lot of GAN architectures.

Cons:
1. Missing augmentation comparison: in table 3, I think a proper comparison would be 1. biggan, 2. biggan + diffaug[1], 3. biggan + [2], 4. biggan + masked-aug (MaskedGAN). Same issue also exists in table. 5~7.

[1] Zhao, S., Liu, Z., Lin, J., Zhu, J. Y., & Han, S. (2020). Differentiable augmentation for data-efficient gan training. Advances in Neural Information Processing Systems, 33, 7559-7570.

[2] Karras, T., Aittala, M., Hellsten, J., Laine, S., Lehtinen, J., & Aila, T. (2020). Training generative adversarial networks with limited data. Advances in Neural Information Processing Systems, 33, 12104-12114.
questions: 1. Is the proposed method orthogonal to existing augmentation methods like [1][2]? The author might consider to add an experiment that using both the proposed mask-based augmentation with either [1] or [2] to see whether this would give an extra performance boost.

[1] Zhao, S., Liu, Z., Lin, J., Zhu, J. Y., & Han, S. (2020). Differentiable augmentation for data-efficient gan training. Advances in Neural Information Processing Systems, 33, 7559-7570.

[2] Karras, T., Aittala, M., Hellsten, J., Laine, S., Lehtinen, J., & Aila, T. (2020). Training generative adversarial networks with limited data. Advances in Neural Information Processing Systems, 33, 12104-12114.
limitations: The author may consider to put more visualization results in the paper.
review_title: null

final_score: rating: 3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
review_content: summary: This work suggests two strategies for masking in GANS and it claims that their Mask Generative Adversarial Networks (MaskedGAN) are robust with limited training data.
The two strategies are:

1) Shifted spatial masking (random shifts in the spatial domain)
2) Balanced spectral masking (multiple bands with self-adaptive probabilities)

Hypothesis: masking helps the model to learn hard-to-discriminate band and create a challenging scenario
Support for the hypothesis: training on multiple architectures (not covering all datasets)
Limited dataset setup:
10%(5K), 20%(10K), and 100%(50K) of data on CIFAR10, 100 
2.5%(25K), 5%(50K), and 10%(100K) of data on ImageNet


strengths_and_weaknesses: Strengths:
- The paper introduces an interesting subject for the community.
- There are multiple experiments.
- Multiple models are used for the experiments.

Weaknesses:
- The writing is somewhat incoherent and unclear: in the abstract and introduction, problem statements lack motivation and clarity.
- The literature review section does not provide coherent statements for each cited work, and it bulk references some papers that might not be directly related. e.g. for Masked Autoencoders, it references basic papers of autoencoders(45, 39, 8, 17, 2, 20, 48), which needs to be discussed in the Autoencoder section.
- The mentioned shifting and Fourier domain manipulation (masking) are not clearly discussed.
- It is not clear how the sampling happens in the limited data scenario and if it is heterogeneous or homogenous.
- The number of training rounds is not specified.
- It seems it works as an augmentation, and with multiple random masking, we actually increase the number of data samples. I think after mentioning the number of iterations, it would be appropriate to compare with SOTA augmentation techniques in GANs.

questions: - What do you mean by Masking? Are you zeroing some pixels? In that case, are you using Gated or Partial Convolutions? If yes, then the models are different, and if no, how do you handle the normalization of the output to adjust for the fraction of missing data?
- Is the sampled data heterogeneous or homogeneous? How have you handled it?
- What are the learning rate, batch-size, and epochs of training for each of the experiments?
- In Fig2, I don't understand how the masked image is used in the model? Specifically, can you make it clear if the model receives the right images directly? If it receives the lower image (Masked Image in the spectral domain), it seems a completely different image! Do some of the pixels retain the original pixel values?

limitations: No. They mentioned the limited data training, which is the main setting of the paper, but as far as I understand, listing the limitation means to discuss where the methodology does not work.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes MaskedGAN to deal with the problem of training GAN with limited data. MaskedGAN uses two masking strategies, shifted spatial masking (masking spatial patches) and balanced spectral masking (masking spectral bands), on both real images and generated images during the training of GAN. Experimental results shows the effectiveness of the proposed method over the original GAN, as well as the superiority over other techniques for training GAN with limited data, across different architectures and datasets.
strengths_and_weaknesses: Pros:
1. The idea of developing masking strategies to enhance the training of GAN with limited data is great. It shows the power of masking in the generative field. 

2. The results are strong. This paper tests the proposed method with different architectures and datasets, and shows consistent and remarkable improvements over baselines, especially when the ratio of available data is small. 

Cons:

In the experiments, the proposed method compares with other techniques for training GAN with limited data (e.g., DA, GenCo, etc) only on very small datasets (e.g., CIFAR, 100-shot) but not on large datasets (e.g., ImageNet, FFHQ). On large datasets, it only compares with the baseline.
questions: 1. My major concern is why the comparison results with other techniques for training GAN with limited data on AFHQ, FFHQ, and ImageNet are missing. The experimental results are not convincing enough without them.

2. Masking, including spatial masking and spectral masking, can also be viewed as a kind of data augmentation strategy. What makes masking, or the specially designed masking in this paper, superior to other data augmentation strategies?

3. How much does the computation overhead increase after adding the masking strategy during GAN training?
limitations: The authors did not state the limitation of the proposed method clearly in the paper. They only claimed possible future applications, e.g., applying MaskedGAN to multi-modality generation, but those are not the limitation of the method or result itself.
review_title: null

Meta-review: metareview: This paper proposes two masking strategies to improve GANs with limited data. The idea is novel and these two strategies can nicely complement each other. The experiment results are promising. The reviewers unanimously raised questions on missing comparison, which seem to be well addressed after author-reviewer discussion. Two reviewers end up raising their scores. There are still some claims in the paper that may need better experimental evidence support, and also more visual cues would be helpful in paper presentation. Also, all reviewers pointed out that the discussion of limitations and broader impact seem not adequate. I would recommend weak acceptance however strongly encourage the authors to address the above-mentioned concerns in their next version. 
confidence: Less certain
Decision: Accept