Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper notes that the current GAN architectures do not synthesize images in a natural hierarchical manner. Finer details in the images are dependent on pixel coordinates, which is undesirable. This is evident from the internal representations learned by the network. The authors attribute this problem to aliasing and propose changes to get rid of it. Specifically, they propose changes to upsampling, downsampling, and non-linearity to make the features rotation and translation invariant. Finally, the authors apply those changes to StyleGAN2 architecture and show comparable FID scores and improved translation and rotation invariance.
main_review: Overall, the paper is well motivated and does a good job of explaining the problem. Authors introduce "non-critical sampling" and the "flexible layers" which clearly show to increase desirable invariances for GANs and increase FID scores. The paper is moderately well written, clarifications.

Here are some questions that need discussion:
Why would the handling of boundaries and upsampling hurt FID? Why would filtered non-linearities hurt FID?  Does the change from D to E keep the filter size to be the same? Wondering if the increase in translation invariance with E is causes just by using a larger filter size! Why does adding noise is contrary to the goal of this paper? Can they be treated as inputs, and hence, they should be translated (or rotated) to compute translation (or rotation) invariance metrics?

The biggest contributor to the performance seems to come from “non-critical sampling” and the “flexible layers”. It will be valuable for readers to know how does the performance look with just these two changes.


limitations_and_societal_impact: Even if the proposed changes apply to any convolutional architecture, the tests have been done only on StyleGAN2, which limits the empirical soundness of the results. The analysis is limited to GANs without attention layers. Either this should be specified in the limitations or experiments related to attention layers should be added.
review_title: null

final_score: rating: 10: Top 5% of accepted NeurIPS papers, seminal paper
review_content: summary: Starting from the observation that current generative models in fact do not synthesize images in a natural hierarchical manner (which they call “texture sticking”), this paper defines the problem systematically, analyze the root of it (unintentional information leaking and aliasing), and proposes a set of simple architectural solutions that are motivated from the signal processing tools. 
The key component of the approach are: 
- pointing out the presence of “texture sticking” phenomenon that much of the fine details of synthesized images appear to be fixed in the absolute pixel coordinates, not the relative position of the underlying objects being drawn. 
- analyzing signals that flow through a network in a continuous domain and interpreting the actual, discrete features as merely a convenient sampled encoding of it
- fixing the operations (up/down sampling and nonlinearity) that are the root of the aliasing by applying a proper low-pass filtering. 
After applying these fixes, the generator becomes translation- and/or rotation- invariant and provides authentic 2D images as if there is actual 3D model and it is capturing pictures in different views. 

main_review: This is a very nice work with impressive results, a great progress achievement in the field of image generation. Very well written.
In particular, I am very impressed with how deeply the authors investigate the deficiencies of the current state-of-the-art, which could have been easily overlooked. The way that the authors analyze the problem is very convincing, and not only do the theoretical analyses provide deep insights but the empirical results shows big advances by fixing the issues at a fundamental level. I believe these insights will be very useful for the future developments of the field. 
Except a few minor things for clarification, I recommend the paper to be accepted as is. 

1) If the ReLU-type of nonlinearity is the problem, would using other smooth type of nonliearity (like swish) function give better result? 
2) In config R, every 3x3 convolution is now replaced to 1x1. Then, is it correct to understand that the network is now entirely relying on the interpolation function (upsamlpling part) for learning the relationship between the neighboring pixels? 

2) Line 135, “convolving it with a discretized downsampling filter …” I think this should be changed to low-pass filter. 
3) Line 34, “faint after-images of the pixel grid …”: what does this mean? It was not clear for me even after reading the entire paper. Could you please clarify this a bit more in the paper?
4) Line 93, “In the remainder of this paper we shall interpret …” In the remainder of this paper, we shall interpret … (comma is missing)
5) Line 153, “This discrete operation cannot be realized without temporarily entering the continuous
representation.” This was not obvious to me. Isn’t the downsampling operation has the same issue (since it needs to be convolved with low-pass filter in continuous domain)? Can you elaborate a bit more? 
6) Line 158 to Line 161, Does this mean that if we limit the range of novel frequencies very strictly, the generator won't be able to add any fine details? 

Overall, as stated above, the paper is well written. However, considering the impact of the paper to the field, it would nice if the authors elaborate the details (that I pointed here) a bit more for the ones who are not familiar with signal processing context. 

---

update after the rebuttal: I am satisfied with the authors' response and I keep my score. 

Some questions I actually forgot to ask: 
1) "Recall that sampling rate s equals the size of the canvas in pixels, given our definitions in..." Here, what is the (physical) size of the canvas then? How do you set it? 
2) "but as a separable filter it compromises rotation equivariance in particular.": why? 
3) "they contain strong overlays.": what does this "overlays" mean? 

---
limitations_and_societal_impact: No comment.
review_title: null

final_score: rating: 9: Top 15% of accepted NeurIPS papers, strong accept
review_content: summary: This paper explores in detail the intrinsic properties of StyleGAN generators that lead to a variety of aliasing artifacts. The source of these artifacts is traced to several distinct phenomena, which are pointwise addressed via a lens of analysis in the continuous signal space, relying heavily on fundamental signal processing techniques. The resulting solutions are applied to a StyleGAN generator, resulting in an “alias-free” model that is better able to handle effects like pose variation while still maintaining the sample quality of the baseline.
main_review: This is an excellent paper based around a solid premise, with strong empirical results against relevant and up-to-date baselines. The analysis is easy to follow and the proposed methods are sensible. This paper is a clear accept without need for revision; I feel that (barring actual mistakes, which I am not able to spot after reading through in detail) seeking out minor points to trip the authors up on would be a waste of time for all involved.
limitations_and_societal_impact: Yes. It's important to note that StyleGAN models in particular have been used to fuel bot armies and disinformation campaigns through the use of generated faces, and the authors acknowledge this.
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: This paper analyzed the aliasing and equivariance issues of standard GANs components through the lens of signal processing and proposed several corresponding techniques to modify the StyleGAN2 generator to make it better suited to video and animation tasks. The visual results are impressive and will benefit a variety of following works.
main_review: Overall, this is a high-quality paper with solid contributions to be published at NeurIPS.

**Originality**: This paper makes the first step in solving aliasing and equivariance issues in GANs using a series of newly proposed methods. Both the tasks and methods are new to the GAN community.

**Quality**: This paper is of high quality. Both the theoretical reasoning and the experimental results are convincing. However, it could be better to include experiments or some discussions on the following points:
1. The method is built on the assumption of limited bandwidth signals, whose valid bandwidth is restricted by the sampling rate or image resolution. So how does the proposed method work on low-resolution images where the sampling rate is more challenging?
2. What exactly happens when the training dataset contains very high-frequency signals? Will it simply fail (as indicated by the authors) or will the generated images look like the ones that have been passed through a low-pass filter? This may help better understand the limitation of the proposed method.

**Clarity**: This paper is well-organized but some parts of it need more explanation:
1. There seems to be a gap in motivation. The “texture sticking” phenomenon, positional references and equivariance are easy to understand, but how are they related to aliasing? I do not see strong evidence showing that “texture sticking” is caused by aliasing. So why is the paper named “Aliasing-free GANs” but not something like “Equivariant GANs”? Equivariance seems to be a more straightforward motivation.
2. How do the modulation and demodulation influence the equivariance?
3. The visualization in Figure 6 looks impressive. However, the explanation of how it is generated and how to interpret it is not enough (e.g. why the proposed method yield the grid/wave-like patterns?).

**Significance**: The results are impressive and the contributions are solid. I believe it will attract many follow-up works in GAN-based video and animation synthesis and processing.



limitations_and_societal_impact: The authors adequately addressed the limitations and potential negative societal impact of their work.
review_title: null

Meta-review: The paper identifies the "texture sticking" problem in image synthesis GANs, argues that this is caused by aliasing in the convolutions, proposes architectural modifications to fix the aliasing, and demonstrates strong improvements in equivariance / smooth transitions. All reviewers agreed that the results were impressive. Separately and equally importantly, all reviewers agreed that the problem was clearly motivated and multiple reviewers (gQig, 1gcw) also agreed that the analysis was insightful. Papers which succeed on both of these fronts are rare; I recommend acceptance.
Decision: Accept (Oral)