Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: The paper studies the potential causes of  discrepancy in high frequency statistics between real and GAN-generated images. Specifically, it studies the upsampling choices in the generator, impact of downsampling in the discriminator and the corresponding frequency bias exhibited by such choices. The paper introduces clever experimental conditions to test various hypothesis related to the above study. Unsurprisingly, the authors claim that adding spectral matching loss or spectral discriminator helps match the spectral statistics. They don’t talk about the introduction of other artifacts in such conditions.  Authors conclude by saying that fully eliminating the discrepancy in the spectral statistics between real and generated distributions of images is still and open open problem. One of the key learnings from the paper is that the discriminators can provide good signals for learning high frequencies, but struggle to model frequencies with low magnitudes.

main_review: Overall, I found the paper to be illuminating about how specific layer choices in generator and discriminator biases the learning of the distribution of generated images differently. The experiment design was quite clever and sufficient to substantiate various claims made in the paper. The paper is a good step towards understanding and solving the identified problems.

Some questions for the authors:
1. Authors choose 10 images for studying the choices made in discriminators and generators. What is the need of using such a small number of images? Will using more images forces the models to show their biases more clearly?
2. While correctly pointed out that there is a discrepancy in the generated and fake images, the goal of image synthesis is not just to match spectral statistics. What are the other side-effects introduced by spectral discriminator or spectral matching loss?


The paper is very easy to read and understand. The experiment design is very sound and convincing. 
limitations_and_societal_impact: The authors don't talk much about the limitations of the work.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper provides insights on GANs via various experiments:
* Upsampling in G leads to different spectral biases.
  * Both bilinear and nearest neighbor upsampling bias the generator towards predicting little high-frequency content.
  * Zero insertion introduces checkerboard artifacts, but they can be compensated by a loss function sensitive to them.
* D struggles with frequencies of low magnitude, not with detecting high frequencies.
* Downsampling in D can impair training signals, but not if high-frequency has high magnitude.
* D benefits from additional input in frequency domains.

main_review: Originality
* (+) The authors investigate G and D on isolated environments.
  * (+) G is given reconstruction task with L2 loss on RGB and spectrum to penalize overall spectrum.
  * (+) Signals from D is evaluated by directly optimizing the pixel values.

Quality
* (+) The proposed experiments and implications are technically sound.
* (+) The paper provides ablation study from the isolated setting to the practical setting.

Clarity
* (+) Writing and organization are clear for understanding.

Significance
* (+) Many empirical evaluations are insightful for future research.
* (-) Most experiments are done with ProgressiveGAN and only the last part works with StyleGAN2.

Misc.
* L115 investigatesnor
limitations_and_societal_impact: Limitation: n/a
Impact: Yes.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The authors explore spectral properties of models in GAN training, isolating the generator and discriminator in turn, to address questions about inability for current frameworks to model certain data.

main_review: -- The conclusion about the generator's ability to match spectral properties under the right setting is critical. I think the current assumption places much of the burden on high-frequency checkerboard artifacts on the generator's upsampling. To show the biases may exist, but that under some loss regimes the generator can learn to correct for them, is a very important observation.

-- The shift of focus both to the discriminator and to the magnitude of frequencies rather than if they are high or low is interesting, and motivates the adoption of alternative different discriminator architectures than otherwise would be explored.

-- A minor quibble: the authors claim that the MSE-based pixel-level supervision baseline of studying the generator are neutral and don't impart any biases of its own, e.g. with the claim 'otherwise, even with a perfect discriminator it would be impossible to generate a correct image'. While pixel-level supervision has the perfect minimum, it is possible that taking gradient descent steps toward it along a MSE-minimizing path will not reach that minimum while some other loss could. This is still a good baseline to use and I don't object to any of the conclusions drawn with it, making my disagreement very minor indeed.

-- One more question I would have liked to see explored is the impact on the distribution-level modeling of the data distribution. While obviously the point of this study is analyzing an individual point/image for its fidelity to a real point/image, I am curious about these statistics at a distribution level across the entire dataset. For example, if generator image i has particular artifacts, are those artifacts sensitive to what other points are in the training set (e.g. how diverse it is)? Or is that truly fundamental to the particular image i? To be clear, I understand this is a large question that need not be answered here, but it would have been interesting to see explored.

I think this work overall makes many important points that will positively impact the direction of future research. It is well-written and well-supported, and both raises and then answers interesting questions.

limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: Generative models, such as GANs, are known to not reproduce the spectral statistics. Various methods in the generator and discriminator have been proposed in papers. This paper systematically indexes and studies these tools and characterizes their effect on the spectral statistics of real vs generated images. The paper aims to use these insights to improve GAN training, with mixed results.
main_review: This paper performs a systematic study of tools for both the generator and discriminator. The paper uses a "toyset", which consists of synthetically generated images with two dominant frequencies, as well as low-resolution faces (64x64). The paper first tests on small datasets (10 images). The paper focuses on the generation in Section 3 and the discriminator in Section 4. Section 5 attempts to draw upon these lessons to improve GANs. The paper evaluates on how well the generations reproduce the spectral properties, as well as image generation metrics like FID. The paper also presents on "detectability" from a classifier trained on spectral properties. The paper's presents a clear story and focus and systematically studies different factors.

I did have several concerns regarding the paper's presentation.
- I am confused regarding the colorbar in Figures 2, 4, 5. First, I recommend explaining the heatmap in the caption, rather than buried in the text. In L145, the text mentions the colorbar is clipped between -1 and +1. I understand how a frequency can be overrepresented by +100%, but how can a frequency can be underpresented by more than 100%? In the worst case, if a frequency is not represented, it should be at -100%? Furthermore, most of the plot is either clipped at solid blue or red. This seems quite surprising that almost none of the frequencies are even close.
- Figure 3 shows error bars, which adds to the value of the plot. However, the size of the error bars seem quite large. It is also quite difficult to tell the error bars apart, as they are heavily overlapping. Is it possible that standard deviation, rather than standard error, was plotted?

Minor
- Table 1. "39.81" has an extra significant figure
- L115 "investigatesnor"
- Equation 2: x used for H and y used for W is nonintuitive (to me)

While the paper systematically sudies different issues, the conclusions in this paper are not too impactful. As described in L170-175 and L229, many of the findings are in agreement with previous literature. This is good corroboration, but ideally this paper would have lead to new insights that could improve GAN training. In Section 5, the attempt at improving GANs are mixed. Lastly, while the paper focuses on quantitative results, there are not many qualitative results. The most impactful potential result would be clear, qualitative examples, where a more accurate spectral result provides more compelling generations as well.

In conclusion, I appreciate the systematic, focused study of this aspect of image generation. As summarized above, I have some concerns regarding the presentation; the paper does not show improvement in GAN training, and it's unclear if the paper's will lead to future improvements. I am currently borderline on this paper.
limitations_and_societal_impact: Yes
review_title: null

Meta-review: The reviewers agreed that the paper was a strong systematic study of frequency bias of GANs. While the overall impact of the paper is somewhat limited, as previous work show similar results, it does provide sufficient new insight that it will be useful to the community. 
Decision: Accept (Poster)