On the Frequency Bias of Generative Models
Authors: Katja Schwarz
Yiyi Liao
Andreas Geiger
Venue: NeurIPS 2021 Poster (2021)
Keywords: GAN
Generative Models
Generative Adversarial Networks
Deepfakes
Frequency Bias
High-Frequency Artifacts
Spectrum Discrepancies
Spectrum
Frequency

Abstract: The key objective of Generative Adversarial Networks (GANs) is to generate new data with the same statistics as the provided training data. However, multiple recent works show that state-of-the-art architectures yet struggle to achieve this goal. In particular, they report an elevated amount of high frequencies in the spectral statistics which makes it straightforward to distinguish real and generated images. Explanations for this phenomenon are controversial: While most works attribute the artifacts to the generator, other works point to the discriminator.  We take a sober look at those explanations and provide insights on what makes proposed measures against high-frequency artifacts effective. To achieve this, we first independently assess the architectures of both the generator and discriminator and investigate if they exhibit a frequency bias that makes learning the distribution of high-frequency content particularly problematic. Based on these experiments, we make the following four observations: 1) Different upsampling operations bias the generator towards different spectral properties. 2) Checkerboard artifacts introduced by upsampling cannot explain the spectral discrepancies alone as the generator is able to compensate for these artifacts. 3) The discriminator does not struggle with detecting high frequencies per se but rather struggles with frequencies of low magnitude. 4) The downsampling operations in the discriminator can impair the quality of the training signal it provides.
In light of these findings, we analyze proposed measures against high-frequency artifacts in state-of-the-art GAN training but find that none of the existing approaches can fully resolve spectral artifacts yet. Our results suggest that there is great potential in improving the discriminator and that this could be key to match the distribution of the training data more closely.

On the Frequency Bias of Generative Models
Katja Schwarz
Yiyi Liao
Andreas Geiger
Autonomous Vision Group
University of Tübingen and MPI for Intelligent Systems
(firstrame lastname @uni buni-tuabingen.co
Abstract
The key objective of Generative Adversarial Networks (GANs) is to generate new data with the same statistics as the provided training data. However, multiple recent works show that state-of-te thetttrrrtt architectures yet struggle achieve this goal. In particular, they report an elevated amount of high frequencies the spectral statistics which makes straightforward to distinguish real and generated images. Explanations for this phenomenon are controversial: While most works attribute the artifacts to the generator, other works point to the discriminator. We take sober look at those explanations and provide insights on what makes proposed measures against figh-ffreueecy artifacts effective. To achieve this, first independently assess the architectures of both the generator and discriminator and investigate if they exhibit frequency bias that makes learning the distribution of freeufrry content particularly problematic. Based on these experiments, we make the following four observations: 1) Different upsampling operations bias the generator towards different spectral properties. 2) Checkerboard artifacts introduced by upsampling cannot explain the spectral discrepancies alone the generator is able to compensate for these artifacts. 3) The discriminator does not struggle with detecting high frequencies per se but rather struggles with frequencies of low magnitude. 4) The downsampling operations in the discriminator can impair the quality of the training signal it provides. light of these findings, we analyze proposed measures against figh-frreuency artifacts in state-of-the-arr GAN training but find that none of the existing approaches can fully resolve spectral artifacts yet. Our results suggest that there is great potential in improving the discriminator and that this could be key to match the distribution of the training data more closely.
Introduction
In recent years, unconditional Generative Adversarial Networks (GANs) have achieved impressive photo-realism for image synthesis tasks. While this has hampered the identification of generated images based on visual cues, multiple recent works show that it is straightforward to distinguish real and generated images based on their high-frequency content [7-9, 14, 22, 34, 38]. This has aroused considerable interest because it reveals a fundamental problem in state-of-the-ar GANs: Existing approaches evidently struggle learn the correct data distribution. While GAN training is notoriously hard, learning the distribution of high-frreqency content is particularly problematic [7- 9, 14, 22, 34, 34. 38]. This indicates a systematic problem existing approaches that could make training suboptimal. For example, if generating high-fie figh-frequencies is difficult for the generator but detecting them straightforward for the discriminator this imbalance could impair the stability of the training. Conversely, if the discriminator struggles to detect high frequencies generating fine details also becomes more difficult which could impede convergence. Therefore, we argue that important to better understand the expressivity of both the generator and the discriminator. Indeed, numerous works suggest that the architecture the generator can hamper the generation of high
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

perkitic
30
1//m
1//mm
grrend peevlertiess
greral
pertictire
30 D01
10
D0
(a) Ground truth
(b) Reconstruction of generator with bilin- ear upsampling using pixel-level supervision
(c) Reconstruction guided by a discriminator with BlurPool downsampling
Figure I: Spectral Properties of Generator and Discriminator. (a) We show natural image and a high-pass filtered version (amplified for clarity) to illustrate the networks' behavior for both low and high magnitudes at high frequencies. (b) Our experiments reveal that bilinear upsampling biases the generator towards generating data with little high-frest tigh-frequency content, regardless of the magnitude. (c) The discriminator shows such bias but provides better guidance for frequencies with high magnitude. The results further reveal artifacts due to downsampling that impair the training signal. We include the training details for both experiments in the supplementary.
frequencies [4,7,9, 10, 14. However, another line of works question the quality of the training signal provided by the discriminator [5, 10, 15]. But there indeed a frequency bias that prevents learning high frequencies in existing GAN models?
This is non-trivial question as GAN training involves two players where architectures for both the generator and discriminator, loss functions, as well as the dataset statistics can all affect the generated images. To narrow down potential factors, we first develop isolated testbeds for both the generator and discriminator: Then, we extend our findings to state-of-the-art GANs on large-scale datasets.
The majority of works attribute the high-frequener artifacts the upsampling operations in the generator [4, 7,9,25,38] Upsampling typically follows pre-defined scheme, e.g. bilinear or nearest neighbor interpolation, or zero insertion between pixels. While the latter introduces too many high frequencies, interpolation has been shown to reduce artifacts [4, 7]. But is interpolation always good scheme for upsampling? Our experiments indicate that both bilinear and nearest neighbor upsampling bias the generator towards predicting little high-frequen frequency content, see Fig. 1b. While zero insertion can be more flexible, it prone to introducing checkerboard artifacts, i.e. too many high frequencies [,,7, But why do the learnable filters in subsequent layers not learn to compensate for these artifacts? Depending on the training objective, checkerboard artifacts might be penalized only slightly. Our experiments evidence that when the loss function is sensitive to these artifacts the generator is indeed capable of compensating for them.
Generator:
Discriminator: This leads us to question the quality of the training signal: Can the discriminator detect high frequencies and provide the necessary supervision? Chen et al. [5] argue that the discriminator cannot detect high- freeeeryyy information due to the downsampling operations. Our experiments corroborate that downsampling can introduce artifacts in the training signal. However, we also observe that the discriminator can indeed provide a meaningful training signal for the spectral statistics at high frequencies when their magnitude is large enough, e.g., Fig. 1c Other works propose to train both the generator and discriminator in wavelet space [ [10] or add a discriminator on the spectrum of the images to reduce high-frequeene artifacts [15]. Motivated by these approaches we ask: Is it enough consider only the spatial domain? ? In agreement with these works, our results suggest that the discriminator can benefit from (additional) input in frequency- domains. Further, our testbed yields insights which of these measures is most effective and reveals that the training signal from the discriminator might remain problematic.
We take a sober look at explanations for high-freque frequency artifacts in generated images and unify the efforts that have been done so far. In particular, we develop isolated testbeds for both the generator and the discriminator which allows us to analyze what makes existing efforts effective. The conclusions drawn by this paper shed new light on limitations of common design choices for both generator and discriminator: i) Bilinear and nearest neighbor upsampling bias the generator towards
Contributions:

---

predicting little high-frequee fegh-rreuency content. ii) Zero insertion is prone to producing checkerboard artifacts the generated images. However, with suitable loss function, the learnable filters of the generator can compensate for the artifacts. This indicates that the upsampling in the generator alone cannot explain the spectral discrepancies. iii) In general, the discriminator able to detect high frequencies and provide supervision to learn the correct spectral statistics. However, while the exponential decay of the spectrum for natural images creates the impression of the discriminator being insensitive to high frequencies, it actually struggles with low magnitudes. iv) We find that all commonly used downsampling operations the discriminator can impair the quality of the training signal.
Lastly, we demonstrate that these findings extend to full GAN training. In agreement with [15], we find that spectral discriminators can bring us one step closer to matching the spectral statistics. Nonetheless, generated images remain straightforward to classify based on their spectral statistics only. While recent works on GANs largely focus on improving the generator, e.g.. [16, 18, 19], our findings suggest that the design of the discriminator plays an equally important role and deserves more attention in future work. We believe that our testbeds for both the generator and discriminator can be useful tool for future investigations. We release our code and dataset at https://ithhub.com/ autonomouuviiiioffreqqunnyyyy bias.
Preliminaries
GAN Training: A generative adversarial network consists of generator and discriminator that are trained jointly in a 2-player game. Given latent variables z ~ Pz, the generator synthesizes images while the diseriminator tries distinguish the synthesized images from real images I ~ pD- sampled from data distribution pp. Let Ge and De denote a generator G and discriminator D with parameters & and 4 respectively. The parameters of the models updated in alternating steps using non-saturating GAN objective [11] which is often combined with RI RI-regularization stabilize training [24]
V 4 Ex-p, ((D+(GG(z))) + EI-PD [((-D /// x
(I)
where f(t) = - log(1 + exp(-t)) and À controls the strength of the regularizer.
Image Processing the Frequency Domain: The discrete 2D Fourier transform maps gray scale image I E #W to the frequency domain:
W-1
i[k
1
Σ 0=0
Σ
exp
2mi * exp
2mi t/ I[x,,
(2)
HW
y=0
for k: = ,....,,... H - and / = O,..., magnitudes of the Fourier components Sik, I = jik, 2 Similar to [7,8] we consider the reduced spectrum Š, i.e. the azimuthal average over the spectrum in normalized polar coordinates E [0. 1}, 0 E 2x)
, W - 1. The power spectral density is estimated by the squared
1
2n S(r, () with r=
22 + 12 [HH + W2)
5(r)
and atan2(k:
(3)
Since images are discretized in space, the maximum frequency is determined by the Nyquist fre- quency. For square image, H = W, is given by Snyg / Vi2 1 H/v2, i.e. for 1. Spectral Classifier: To detect generated images, Dzanic et al. [8] propose to classify real and gen- erated images based on their reduced spectrum. More specifically, they fit a power-law function to the tail of each reduced spectrum for frequencies above given threshold re 0.75 and train a binary classifier on the fit parameters of each spectrum.
3 Are high frequencies more difficult to generate?
In this section, we investigate there a frequency bias in the generator that impedes the generation of high frequencies. Therefore, we first consider the generator in an isolated setting assuming pixel- level supervision otherwise, even with a perfect discriminator it would be impossible to generate a correct image. We extend our findings to the full GAN setting in Section 5.

---

There are two lines of argumentation in existing works: The first investigates artifacts that arise from the upsampling operations in the generator 7,7.. These works analyze the spectral properties of the generator predictions at convergence. We extend this analysis from the perspective of frequency bias to see if low frequencies are learned earlier during training. The second line of works argues for a frequency bias of the learnable filters [14, 20]. Motivated by their analysis we investigate if learnable filters are at all able to compensate for the artifacts introduced by upsampling.
Experimental Setting: To isolate the generator from the discriminator we consider a conditional re- construction task. In particular, we take 10 images from a dataset and pair them with 10 latent codes drawn from normal distribution. Given latent code, the generator is optimized to reconstruct the corresponding image with a pixel-wise 1.2-loss
H-1W-
1 HW
L1
Σ
Σ |[G yl 1z, xill2 Y=
(4)
We choose the generator from PGAN [16] because of its simple architecture comprising only con- volutions and upsampling operations. Further, when combined with RI -e-reaularization it trains stably in GAN setting [24]. allowing us to use the same generator in Section We reduce the number of channels for faster training because our primary focus is on the spectral properties. As upsampling operations, we investigate bilinear and nearest neighbor interpolation, zero insertion, and reshaping in the channel dimension [29]. We ensure that all networks have similar number of parameters. More details are provided in the supplementary.
Datasets: In natural images, the spectral density follows an exponential decay. To isolate the ef- fects of frequency range and magnitute, we create a Toyset of images with two Gaussian peaks N( (1), N 02) of equal magnitude in the spectrum, see Fig. 2a, 2b. Based on the Nyquist frequency fngg = H/V2, we create samples using 01 = 02 = 1/V2fngg and draw Yr ~ U40005 0.15fngu) and μl2 ~ U((0755/mm 0.85f Together with a uniformly distributed phase, we apply the inverse Fourier transform to create images. We further test our setting on natural images with a downsampled version of CelebA [21]. Both datasets have a resolution of 642 pixels. Evaluation Metries:
On the spatial domain, we report PSNR on the RGB values and on the fre- quency domain we evaluate the reduced spectrum, see Eq. (3). To analyze frequency- deneeddt convergence we further visualize the evolution of the spectrum similar to [26]. Here, the x-axis denotes the (frequency) radius r in normalized polar coordinates and the y-axis denotes the training iterations. The color corresponds to the relative error of the average predicted reduced spectrum wrt. the ground truth, where positive and negative values indicate too many and too few predicted frequencies, respectively. We clip the colorbar at 1, i.e., when the relative eтror exceeds 100%.
Do generators exhibit a frequency bias? The spectral evolution in Fig. 2 shows different behavior for bilinear and nearest neighbor upsampling compared to zero insertion and reshaping. Particularly on the Toyset, a generator with bilinear or nearest neighbor upsampling learns the lower frequencies earlier in training than the high frequencies, see Fig. 2c and 2d. While the generator with bilinear upsampling struggles with high frequencies throughout training, for nearest neighbor upsampling eventually fits the high-frequency peak, suggesting that its bias towards little high-frequency content is not as strong. In contrast, a generator with zero insertion or reshaping learns both peaks approx- imately at equal speed but is prone to generating checkerboard artifacts as indicated by the large error at the highest frequency in Fig. 2c and 2f. Between the peaks of the toyset, where frequencies have small magnitudes, all methods struggle to match the statistics of the training data because the L2-loss penalizes errors at frequencies with low magnitude only slightly. This explains why for CelebA there an overall trend towards learning lower frequencies earlier during training. The PSNR values in Table I support these findings in the spatial domain. The lack of high frequencies for bilinear and nearest neighbor upsampling manifests in low PSNR, particularly for the Toyset which contains many high frequencies by construction.
Can the generator learn to compensate for the artifacts? Since the L2-loss is less sensitive to frequencies with low magnitudes (see supplementary for a formal derivation), we now add an L2-loss on the logarithm of the reduced spectrum:
H/ H/V2-1
Ls = H
Σ
log (5 (5(G)(2)) [k] - log (Š(1) [+]]
(5)
k=
The logarithm penalizes errors at low magnitudes more strongly and can therefore reduce the checker- board artifacts introduced by zero insertion and reshaping, see Fig. 3. Hence, given suitable

---

Bilinear
NN
Zeros
Reshape
AvgPool
BlurPool
Stride
MLP
LB
Le
+ La
Ls
+ SD
SD
SD
+SD
Toyset 21.1 20.9 30.7
30.7 38.6 38.0 38.6
36.5
Toyset 16.3 19.2 23.3 15.6
23.3
24.1
46.5 28.1 33.7
26.2
CelebA 34.9 37.0 40.9
39.8
42.4
43.6
42.3 40.9
CelebA 25.4 27.0 24.2
24.4
25.5
25.6
Table 1: PSNR for different upsampling opera- tions in the generator at resolution 64 pixels.
Table 2: PSNR for different downsampling opera-
tions in the discriminator at resolution 64 pixels.
Tharne hoatn H1200
Toyset
0.5 F1lmm
10
1/5m
fifm
1/4m
Tounzz treake In/uus
100
101
CelebA
Sperted
10
R/S fi/m
0.5
1.0 0.0
0.
0.5
1.0 010
0.5
0.0
2/2mm
Fi5m
(a) GT Sample
(b) GT Spectrum
(c) Bilinear
(d) NN
Zeros
(1) Reshape
Figure 2: Spectrum Error Evolution for the Generator. We show one sample from the dataset in the first column and the mean and standard deviation of the reduced spectra for all 10 samples in the second column for reference. For the Toyset the dashed lines mark the mean of the Gaussian peaks at 0.1f and 0.8 fng. Upsampling with bilinear or nearest neighbor interpolation biases the generator towards predicting little high-frequency content. Conversely, zero insertion and reshaping are prone introducing checkerboard artifacts, indicated by the large errors the highest frequency. The color corresponds to the relative error of the average predicted reduced spectrum wrt. the ground truth and clipped at 1, i.e. when the relative error exceeds 100%.
gronnd trath
Ei
film
Bilinear
(b) NN
(c) Zeros
(d) Reshape
Figure 3: Reduced Spectrum for the Generator on the Toyset. We plot the mean and standard deviation of the reduced spectrum above 0.75 The peak at the highest frequencies from upsampling with zero insertion or reshaping is removed by an additional loss on the spectrum.
objective, learnable filters can indeed compensate for high-frrquency artifacts introduced by zero insertion and reshaping. Interestingly, bilinear upsampling does not benefit as much from the spectral loss. This suggests that strong bias towards little high-frequency content can be more difficult to compensate for. As expected, the additional loss does not alter the evaluation in the spatial domain significantly and yields similar PSNR values in Table 1.
Implications: Overall, these results lead us to conclude that different upsampling operations bias the generator towards different spectral properties. Nearest neighbor and particularly bilinear up- sampling introduce a bias towards fitting functions with little high-frequency content. In Section 5, we will see that this can also be beneficial when working with natural images as their spectral density follows an exponential decay. On the other hand, zero insertion and reshaping introduce bias towards checkerboard artifacts. However, with a suitable loss function, network filters can learn to compensate for these artifacts. Therefore, the upsampling in the generator alone cannot explain the spectral discrepancies. This suggests that the training signal provided by the discriminator might be suboptimal in the first place.
5

---

Toyset
Toy
17/m
Film
M0m
17/m
The www. A110
The bwxc Xsta
CelebA
ColebA
Film
115
125
(a) AvgPool (b) BlurPool (c) Stride
(d) MLP
(a) Stride
(b) +SD (c) +F-Mninng (d) +Wavelet
Figure 4: Spectrum Error Evolution for Dis- criminators with Different Downsampling Operations. The downsampling operations do not significantly bias the discriminator to- wards any frequency range. Instead, it gener- ally struggles with frequencies of low magni- tude. The color corresponds to the relative error of the average predicted reduced spectrum wrt. the ground truth and is clipped at 1, i.e. when the relative error exceeds 100%.
Figure 5: Spectrum Error Evolution for Discriminators on Different Input Domains. The spectral discriminator greatly improves the spectral statistics on both datasets while hard example mining in the frequency domain (F- mining) and wavelets alter results only slightly. The color corresponds to the relative error of the average predicted reduced spectrum wrt. the ground truth and is clipped at 1, ie. when the relative error exceeds 100%.
4 Can the discriminator provide a good training signal?
In this section, we investigate how good the training signal is that the discriminator can provide. Only a few existing works consider the discriminator as a cause for the spectral discrepancies. Chen al. [5] attribute the high-frequency artifacts to information loss in the downsampling operations. In particular, they analyze how high frequencies affect the output of the discriminator at convergence. Instead, we consider how downsampling affects the training signal by assessing the input to the dis- criminator. Further, we analyze how the spectral statistics evolve during training to see if downsam- pling introduces a bias towards correcting low frequencies earlier than high frequencies. To reduce spectral discrepancies, Chen et al. [5] propose regularizer based on the reduced spectrum to per- form hard example mining in the frequency domain, cf. Eq. (3). Similarly, Jung et al. [15] define an additional discriminator on the reduced spectrum. While spectral discrepancies are not the main fo- cus of their work. Gal et [10] also argue for "unfavorable loss functions" and propose to train both the generator and discriminator in wavelet space. In the second part of this section, we therefore aim to understand what makes (additional) inputs from other domains valuable and which of the proposed measures are most effective to correct the high-frequene artifacts.
Experimental Setting: Similar to the generator experiments Section 3, we propose conditional reconstruction task to assess the quality of the training signal independently of the generator architec- ture. More specifically, we train a class-conditional GAN with single sample per class. To minimize the impact of the generator, we directly optimize the pixel values of the fake images. In practice, we pair 10 images with 10 labels training data and optimize 10 learnable tensors conditioned on the la- bels. We optimize the learnable tensors and discriminator weights in an alternating fashion using the GAN two-player game setting. Similar to Section 3, we use the PGAN discriminator [16] with a re- duced number of channels because our primary focus is on the spectral properties. As downsampling operations, we investigate strided convolution, average pooling, and blurring with subsequent average pooling [37]. We further train MLP the flattened input image baseline without any downsam- pling operations. When adding a spectrum discriminator, we weigh both discriminator losses equally as in [15]. Note, that training GAN such few images is non-trivial task that requires careful tuning to train stably. We ensured that this is the case for our results, see supplementary for details.
Datasets and Metries: We use the same datasets and metrics as in Section 3 and evaluate them on the reconstructions guided by the gradients from the discriminator. This allows us to assess the quality of the training signal.
How does downsampling affect the training signal? Fig. 4 shows the evolution the average spec- trum of the reconstructed images. On the Toyset, both the low- and high-frequent peaks are learned approximately at an equal pace. This suggests that the discriminator can indeed detect and correct the spectral statistics at high frequencies, regardless of the choice of downsampling operation. However,

---

(a) GT
(b) MLP
(c) AvgPool (d) BlurPool
(e) Stride
(1) Stride
(g) Stride
(h) Stride
SD
F-mining
+ Wavelet
Figure 6: Reconstruction Guided by the Discriminator. All discriminators with downsampling lead to significant deviations from the ground truth. While F-mining and wavelets slightly improve the reconstruction, the spectral discriminator decreases image fidelity.
all approaches struggle correct frequencies of low magnitude. For natural images, the power spectrum decays exponentially which creates the impression that the discriminator struggles with high frequencies, while it actually struggles with the low magnitude of the high- high-frequency content. In image space, amongst the downsampling operations strided convolution achieves the best PSNR. However, it is much lower than the values for the generator in Table I which reflects the harder task due to instance- instead pixel-level supervision. Fig. 6 shows the learned tensors at the last training iteration. While the reconstruction from the MLP is reasonably good, the downsampling operations introduce artifacts in the training signal provided by the discriminator.
Is it enough to train in the spatial domain? In Fig. 5 compare the spectrum evolution for addi- tional spectral discriminator (SD) [15], hard example mining in the frequency domain (F-Mining) [5] and training in wavelet space (Wavelet) [10]. Since the spectral discriminator is directly applied to the reduced spectrum, it can guide the generator on the spectral statistics of the dataset and greatly reduces the error in Fig. 5b. Instead, F-mining and wavelets only slightly reduce the spectral discrepancies, e.g.. on CelebA for frequencies between 0.5 - 0.755 myg- While F-mining increases the weight of samples with poor spectral realness, it does not make the discriminator
sensitive slight changes in im- age space. This is in agreement with the findings in [5]. Wavelets separate the input wrt. the frequency but not wrt. the magnitude which could explain why low magnitude artifacts remain difficult to detect. In the spatial domain, however, the reconstructions with the spectral discriminator in Fig, 6e and 6f reveal a caveat: While the spectrum is better aligned, the reconstruction with the additional spec- tral discriminator qualitatively becomes worse. As the spectrum computation and the azimuthal integration discard information, images with the same reduced spectra can look very different. Consequently, penalizing the reduced spectrum might not be sufficient for improving the training signal alone. This also reflects in the PSNR in Table which, except for the MIP, remains largely unaffected by the spectral discriminator. In the supplementary, we verify that replacing the reduced spectrum with the full Fourier transform indeed improves the image fidelity. However, as the spectral discriminator is an MLP, this does not trivially scale to real-world settings.
Implications: In contrast to existing hypotheses, our findings evidence that the discriminator gener- ally struggles with frequencies that have a low magnitude but that high frequencies are not per se more difficult to detect. An additional discriminator on the reduced spectrum can greatly facilitate learning the spectral statistics of the data but might not improve the image fidelity. Even in our simple testhed, none of the convolutional architectures with downsampling is able to provide artifact-free supervision. This indicates that the discriminator might play more important role in reducing high-frequent facts than currently anticipated in the field. While a simple spectral discriminator provides a good first step, we conclude that more work in this area is required to solve the problem. We believe that one key is to reduce the downsampling artifacts, e.g.. by exploring alternative downsampling operations or by considering approaches that allow for pixel-level supervision of the discriminator in [30] or [28].
arti-
5 Improving GANs
We now analyze the full GAN training, i.e., when generator and discriminator are two competing neural networks. First, we extend the discriminator analysis from Section 4 and verify if these results are also valid for full GAN training. Next, we investigate if the most effective discriminator is able to resolve high-frequency artifacts of the different upsampling strategies discussed in Section 3. Lastly, we analyze its effect on StyleGAN2 [19], state-of-the-art GAN model, which shows the characteristic peak at the highest frequencies in the reduced spectrum.
7

---

ground tmuth
original
original
1..5
100
6/5
110
0.5
Bb6 film
005
11:00
1000
fifoe
175mm
(a) Bilinear
(b) NN
(c) Zeros
(d) Reshape
Figure 7: Reduced Spectrum for PGAN on FFHQ64. We plot the mean and standard deviation of
the reduced spectrum above f.75 The spectral discriminator prevents the high peak for zero insertion and reshaping but cannot fully correct the spectral discrepancies. Similar to Fig. 3, bilinear upsampling struggles to match the spectral statistics the dataset.
Original Acc
Wavelet Acc
F-Mining Ace
SD
Bilinear Acc
NN
Zeros
Reshape Acc
FID
FID
FTD
Acc
FID
FID Acc
FID
Ace
FID
FID
FFHQ64
72 31.0
73
39.8
73
33.7
62
31.0
FFHQ64 Cats
57 36.3
62
31.0
66
34.9
67
35.2 70
Cass128
88
106.1
74
122.5
77
119.3
62
114.6
62
138.2
62
114.6
77
128.8
129.7
Table 3: Spectral Classification Accuracy and FID for PGAN with discriminators on different input domains.
Table 4: Spectral Classification Accuracy and FID for PGAN using different upsampling strate- gies with spectral discriminator.
Experimental Setting: We start by combining the architectures from Section 3 and Section to obtain PGAN [16] with reduced number of channels, which train from scratch with R1- regularization and without progressive growing [19,26 24] until the discriminator has seen 15M images. Next, extend our analysis to StyleGAN2 on resolutions up to 1024² pixels. The StyleGAN2 generator uses bilinear upsampling which, according to our previous findings, should not cause the el- evated amount of high frequencies. Therefore, finetune pre-trained models with the most effective discriminator following the training protocol from [17] until the discriminator has seen 2.5M images.
We consider large-scale real- world datasets in this section. We train version of PGAN on a downsampled version of FFHQ [18] at resolution 642 pixels and a downsampled version of 200k images from LSUN Cats [36] resolution 1 128 pixels. We finetune StyleGAN2 on LSUN Cats (256² pixels), AFHQ Dog [6] (512² pixels) and FFHQ (10242 pixels). For AFHQ Dog, we use adaptive discriminator augmentation due to the small size of the dataset [17].
Datasets:
Evaluation Metrics: In the spatial domain, we report FID [12] on the full dataset and 50k gen- erated images. To measure spectral discrepancies we deploy the spectral classifier described in Section 2. We follow the proposal in [4] and replace the KNN classifier with an SVM to ob- tain stronger classifier. We use 1000 real and fake images each and use 90% and 10% for training and evaluation, respectively. Here, a classification accuracy of around 50% is ideal as this indicates a perfect generator because the classifier is forced make a random guess.
How do the isolated settings transfer to full GAN training? We ablate PGAN with discriminators on different input domains in Table 3. In agreement with the findings from Section 4, the wavelet discriminator (Wavelet) and hard frequency mining (F-Mining) improve the spectral statistics only slightly. Hence, generated images can still be classified with high accuracy. The most effective method learn the spectral statistics remains the additional spectral discriminator (SD) as indicated by the lower accuracy of the spectral classifier on all datasets. Con- sistent with our observation on the testbed, the image quality in the spatial domain remains largely unaffected.
Original Acc FID
+SD
Acc
FID
Cats256
92
6.5
80
8.4
AFHQ Dog
92
7.4
66
12.1
FFHQ
99
2.8
94
3.1
Table 5: Finetuning StyleGAN2 with an additional disciminator on the reduced spectrum.
Recalling the implication of Section 3, that the generator can learn to compensate for feeeufreqeescs artifacts given a suited training objective, we now investigate whether the spectral discriminator satis- fies such a requirement for different upsampling operations in the generator. Consistent with our obser- vation on the generator testbed, Fig. shows that the spectrum discriminator also able to significantly reduce the peak at the highest frequency for both zero insertion and reshaping upsampling. However, the magnitude at the highest frequencies remains slightly elevated because the generator only receives supervision through the discriminator (real vs. fake) instead of full ground truth spectra considered in
8

---

the testbed. On the other hand, the bias towards little high-freeuenee content for bilinear and nearest neighbor upsampling aligns well with the spectral statistics of the datasets. This also reflects in Table where images generated with zero insertion and reshaping are still detected with higher accuracy than images generated with bilinear and nearest neighbor upsampling. Considering both the spatial statis- tics and image fidelity, we observe that upsampling with nearest neighbor yields the best performance.
Lastly, we evaluate the effect of the spectral discriminator when applied to StyleGAN2 on datasets with higher resolution, namely LSUN Cats (256*), AFHQ Dog (512²) and FFHQ (1024²). The results in Table 5 show mixed results: On AFHQ Dog the spectral discriminator results in a strong improvement on the spectrum but also significantly increases the FID. In contrast, on FFHQ the FID stays similar but the spectral discriminator also only slightly improves the spectral statisties and not able to correct the peak at the highest frequency'. This suggests that in existing architectures there is a tradeoff between perceptual image quality and matching the spectral statistics of the data.
Our experiments show that both of our testbeds on the generator and the discriminator provide consistent observations which are aligned with those for full GAN training. Our findings suggest that the spectral discriminator can reduce the spectral discrepancies but the misalignment in the high frequencies is not fully addressed. We further observe that nearest neighbor upsam- pling together with an additional spectral discriminator is a good combination for natural images where freeeeree content is low. However, when applying existing measures to StyleGAN2, we observe that none of them can solve the spectral discrepancies completely. This suggests that the high-frequency artifacts in GANs are still an open problem that require further research. Our work takes an important step towards understanding the underlying mechanisms and sheds light on the effectiveness and limitations of existing approaches.
Implications:
6 Other Related Work
the last years, great progress on unconditional Generative Adversarial Net- works enabled photo-realistic image synthesis with deep neural networks [1,11, 18,1,,,9,,,4,,,,,, 19,24,311 With the rapid development in image synthesis, fake image detection becomes equally important [23,27,34]. Albeit very high photorealism, multiple works show that generated images can still be easily dis- tinguished from real images based on their high-frequency content [7-9, 14, 22,34, 38]. Consistent behavior across various architectures suggests a systematic problem in sfate-of-the-ar GANs. Thus, it is important to understand if there is a frequency bias in convolutional generators or discriminators.
Deepfake Detection:
Spectral Bias: Fully connected ReLU networks are known to fit low-frequency modes of target function faster than its high-frequency modes [2,3, 13, 26, 25J. This behavior, referred to as spectral bias, has been shown to greatly impact the practical expressiveness of such models due to frequency dependent learning rates (2,3, 13,26,35] For CNNs, spectral bias is often mentioned to explain the good generalization of largely overparameterized networks [32,33,35] 35] but is theoretically less well understood. Inspired by Rahaman et al. [26] who study the spectral bias of fully connected ReLU networks, we analyze if similar bias exists in convolutional generators and discriminators. Note that Rahaman et al. [26] consider one-dimensional regression tasks in which the spectrum is evaluated over the output samples. Instead, calculate the spectrum over all pixels of one image. To highlight this difference, we use the term term frequency bias in our work instead.
7 Conclusion
In this work, we take a thorough look at existing explanations for systematic artifacts in the spectral statistics of generated images and unify the efforts that have been done so far. While our experiments suggest that upsampling introduces a frequency bias in the generator this alone does not explain the spectral discrepancies. In agreement with [5, 15], we find that advancing the discriminator brings us one step closer to learning the correct data distribution. However, none of the existing measures can faithfully recover the spectral statistics yet, such that generated images are still straightforward to classify solely based on their spectra. While state-of-the-art GANs benefit from highly engineered generator architectures, e.g., [16, 18, 19], our findings suggest that the design of the discriminator
Jung et al. [15] conduct a similar experiment but evaluate the average over the logarithmic reduced spectra, i.e. 1 N ΣN log(S.) while we consider N log (I>, 5.). We find that the peak at the highest frequencies is only significant in the latter computation and confirmed this difference with the authors.

---

plays an equally important role and deserves more attention in future work. We encourage future work to explore alternative downsampling operations and to consider approaches that enable pixel- level supervision for the discriminator. Interesting starting points could for example be [30] or [28]. Another interesting direction might be to explore discriminator augmentation [17, 39] wrt. high- frequency artifacts. We believe that our testbeds for both the generator and discriminator may serve as useful tools for future investigations and will make our code publicly available upon acceptance. Lastly, we remark that working on generative models always bears the risk of manipulation and the creation of misleading content. While the insights from our work could be used to make Deepfakes harder to detect, we believe that better understanding of existing architectures and their limitations is key to advancing the field, such that the benefits of our work outweigh the societal risks.
Acknowledgments
We acknowledge the financial support by the BMWi in the project KI Delta Learning (project number 19A190130) and the support from the BMBF through the Tuebingen AI Center (FKZ: O1IS 18039A). We thank the International Max Planck Research School for Intelligent Systems (IMPRS-IS) for supporting Katja Schwarz. This work was supported by NVIDIA research gift.
References
[1] Anokhin, K. Demochkin, T. Khakhulin, Sterkin, Lempitsky, and D. Korzhenkov. Image generators with conditionally-nndeeedee
pixel synthesis. Proc. IEEE Conf. on Computer Vision and Pattern
Recognition (CVPR), 2021. 9
[2] S. Arora, Du, W. Hu. Li, and R. Wang. Fine-grained analysis optimization and generalization for overparameterized two-layer neural networks. In Proc. of the International Conf. on Machine learning (ICML), 2019, 9
[3] R. Basri, D. W. Jacobs, Kasten, and S. Kritchman. The convergence rate of neural networks for learned functions of different frequencies. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
[4] K. Chandrasegaran, N. Tran, and Cheung. A closer look at fourier spectrum discrepancies for cnn- generated images detection. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR),
2021. 2, 4,8 8
[5] Y. Chen, G. Li, C. Jin, S. Liu, and T. Li. SSD-GAN: measuring the realness in the spatial and spectral domains. In Proc. of the Conf. on Artificial Intelligence (AAAI), 2021. 2, 6, 7,
[6] Y. Choi, Uh, J. Yoo, and J.WW. Ha. Stargan v2: Diverse image synthesis for multiple domains, In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2020. 8
[7] R. Durall, M. Keuper, and J. Keuper. Watch your up-convolution: CNN based generative deep neural networks are failing to reproduce spectral distributions. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2020. 1, 2, 3, 4,
[8] T. Dzanic, Shah, and F. D. Witherden. Fourier spectrum discrepancies in deep network generated
images. In Advances in Neural Information Processing Systems (NeurIPS), 2020. 1, 1,3,9 9
[9] J. Frank, T. Eisenhofer, L. Schönhem, A. Fischer. D. Kolossa, and T. Holz. Leveraging frequency analysis deep fake immaee recognition. In Proc. of the International Conf. on Machine learning (ICML), 2020. 1,
2, 4, 9
[10] R. Gal, Cohen,
Bermano, and D. Cohen-Or. SWAGAN: A style- based wavelet-driven generative model. CORR, 2102.06108, 2021, 2, 6,
[11] I. J. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. C. Courville. and
Y. Bengio. Generative adversarial nets. In Advances Neural Information Processing Systems (NeurIPS),
2014. 3,9
[12] M. Heusel, H. Ramsauer. T. Unterthiner, B. Nessler, and S. Hochreiter. Gans trained by time-scale update rule converge to local nash equilibrium. Advances Nearal Information Processing Systems (NeurIPS), 2017.
[13] A. Jacot, Hongler, and F. Gabriel. Neural tangent kernel: Convergence and generalization in neural networks. Advances in Neural Information Processing Systems (NeurIPS), 2018.
[14] L. Jiang, B. Dai, W. Wu, and C. C. Loy. Focal frequency loss for generative models. arXiv.org, 2012.12821.
2020. 1, 4, 9
[15] S. Jung and M. Keuper. Spectral distribution aware image generation. In Proc. of the Conf. on Artificial Intelligence (AAAI), 2021. 2, 3, 6, 7,9
[16] T. Karras, T. Aila. S. Laine, and J. Lehtinen. Progressive growing of GANs for improved quality, stability.
and variation. In Proc. of the International Conf. Learning Representations (ICLR), 2018. 3, 4, 6. 8,9
10

---

[17] T. Karras, M. Aittala, J. Hellsten, S. Laine, J. Lehtinen, and T. Aila. Training generative adversarial
networks with limited data. In Advances in Neural Information Processing Systems (NeurIPS), 2020. 8, 10 [18] T. Karras, S. Laine, and T. Aila. A style-based generator architecture for generative adversarial networks.
In Proc. IEEE Conf. Computer Vision and Pattern Recognition (CVPR), 2019. 3, 8, 9
[19] T. Karras, S. Laine. M. Aittala, Hellsten, J. Lehtinen, and T. Aila. Analyzing and improving the image quality of StyleGAN. In Proc. IEEE Conf. on Computer Vision and Puttern Recognition (CVPR), 2020. 3,
7, 8, 9
[20] M. Khayatkhoei and A. Elgammal. Spatial frequency bias convolutional generative adversarial networks. arXiv.org, 2010.01473, 2020. 2,
[21] Z. Liu, P. Luo. X. Wang. and X. Tang. Deep learning face attributes in the wild. In Proc. of the IEEE
International Conf. on Computer Vixion (ICCV), 2015. 4
[22] Z. Liu, X. Qi, and P. H. S. Tomr. Global texture enhancement for fake face detection in the wild. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2020. 1.9
[23] F. Marra, D. Gragnaniello, D. Cozzolino, and I.. Verdoliva. Detection of gan-generated fake images over social networks. In Proc. IEEE Conf. on Multimedia Information Processing and Retrieval (MIPR), 2018. 9
[24] L.. Mescheder, A. Geiger, and S. Nowozin. Which training methods for gans do actually converge? In Proc. of the International Conf. on Machine learning (ICML), 2018. 3, 4, 8,
[25] A. Odena, V. Dumoulin, and C. Olah. Deconvolution and checkerboard artifacts. Distill, 2016.
[26] N. Rahaman, A. Baratin, D. Arpit, E. Draxler, M. Lin, F. A. Hamprecht, Y. Bengio, and A. C. Courville. On the spectral bias of neural networks. In Proc. of the International Conf. on Machine learning (ICML.),
2019. 4,9 9
[27] A. Rössler, D. Cozzolino, L. Verdoliva, C. Riess, J. Thies, and M. Nießner. Faceforensics**: Learning to detect manipulated facial images. In Proc. of the IEEE International Conf. Computer Vision (ICCV),
2019. 9
[28] F. Schönfeld, B. Schiele, and A. Khoreva. A u-net based discriminator for generative adversarial networks. In Proc. IEEE Conf. Computer Vision and Pattern Recognition (CVPR). 2020. 7, 10
[29] W. Shi, J. Caballero, F. Huszar, J. Totz, A. P. Aitken, R. Bishop. D. Rueckert, and 7.. Wang. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In Proc.
IEEE Conf. Computer Vision and Pattern Recognition (CVPR). 2016. 4
[30] A. Shocher, B. Feinstein, N. Haim, and M. Trani. From discrete to continuous convolution layers, arXiv.org.
2020. 10
[31] I. Skorokhodov, S. Ignatyev, and M. Elhoseiny. Adversarial generation of continuous images. Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2021.
[32] D. Ulyanov, A. Vedaldi, and V. S. Lempitsky. Deep image prior. In Proc. IEEE Conf. on Computer Vision
and Pattern Recognition (CVPR), 2018. 9
[33] H. Wang, X. Wu, Z. Huang, and E. P. Xing. High-frequency component helps explain the generalization of convolutional neural networks. In Proc. IEFE Conf. Computer Vision and Pattern Recognition (CVPR),
2020. 9
[34] S. Wang, O. Wang, R. Zhang, A. Owens, and A. A. Efros. Cnn-generated images are surprisingly easy to spot... for now. In Proc. IEFE Conf. on Computer Vision and Pattern Recognition (CVPR), 2020. 1,9
[35] Z. J. Xu, Y. Zhang, Luo, Y. Xiao, and Z. Ma. Frequency principle: Fourier analysis sheds light on deep
neural networks. arXiv.org, 0.652,,,,, 2019. 9
[36] F. Yu, Y. Zhang, S. Song, A. Seff, and J. Xiao. Lsun: Construction of large-scale image dataset using
deep learning with humans in the loop. arXiv.org, 1506.03365, 2015. 8
[37] R. Zhang. Making convolutional networks shift-invariant again. In Proc. of the International Conf. on Machine learning (ICML), 2019. 6
[38] X. Zhang, S. Karaman, and S. Chang. Detecting and simulating artifacts in GAN fake images. In IEEE
International Workshop Information Forensics and Security, 2019. 1, 2,9
[39] S. Zhao, Z. Liu, Lin, J. Zhu, and S. Han. Differentiable augmentation for data-efficient GAN training. In Advances in Neural Information Processing Systems (NeurIPS), 2020. 10
11


Reviews:
[{"review_id": "y2VzV6UY55h", "reviewer": "Reviewer_hXku", "summary": "The paper studies the potential causes of  discrepancy in high frequency statistics between real and GAN-generated images. Specifically, it studies the upsampling choices in the generator, impact of downsampling in the discriminator and the corresponding frequency bias exhibited by such choices. The paper introduces clever experimental conditions to test various hypothesis related to the above study. Unsurprisingly, the authors claim that adding spectral matching loss or spectral discriminator helps match the spectral statistics. They don’t talk about the introduction of other artifacts in such conditions.  Authors conclude by saying that fully eliminating the discrepancy in the spectral statistics between real and generated distributions of images is still and open open problem. One of the key learnings from the paper is that the discriminators can provide good signals for learning high frequencies, but struggle to model frequencies with low magnitudes.\n", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, I found the paper to be illuminating about how specific layer choices in generator and discriminator biases the learning of the distribution of generated images differently. The experiment design was quite clever and sufficient to substantiate various claims made in the paper. The paper is a good step towards understanding and solving the identified problems.\n\nSome questions for the authors:\n1. Authors choose 10 images for studying the choices made in discriminators and generators. What is the need of using such a small number of images? Will using more images forces the models to show their biases more clearly?\n2. While correctly pointed out that there is a discrepancy in the generated and fake images, the goal of image synthesis is not just to match spectral statistics. What are the other side-effects introduced by spectral discriminator or spectral matching loss?\n\n\nThe paper is very easy to read and understand. The experiment design is very sound and convincing. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors don't talk much about the limitations of the work.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "gMC5A6hZnPx", "reviewer": "Reviewer_Us7c", "summary": "This paper provides insights on GANs via various experiments:\n* Upsampling in G leads to different spectral biases.\n  * Both bilinear and nearest neighbor upsampling bias the generator towards predicting little high-frequency content.\n  * Zero insertion introduces checkerboard artifacts, but they can be compensated by a loss function sensitive to them.\n* D struggles with frequencies of low magnitude, not with detecting high frequencies.\n* Downsampling in D can impair training signals, but not if high-frequency has high magnitude.\n* D benefits from additional input in frequency domains.\n", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality\n* (+) The authors investigate G and D on isolated environments.\n  * (+) G is given reconstruction task with L2 loss on RGB and spectrum to penalize overall spectrum.\n  * (+) Signals from D is evaluated by directly optimizing the pixel values.\n\nQuality\n* (+) The proposed experiments and implications are technically sound.\n* (+) The paper provides ablation study from the isolated setting to the practical setting.\n\nClarity\n* (+) Writing and organization are clear for understanding.\n\nSignificance\n* (+) Many empirical evaluations are insightful for future research.\n* (-) Most experiments are done with ProgressiveGAN and only the last part works with StyleGAN2.\n\nMisc.\n* L115 investigatesnor", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitation: n/a\nImpact: Yes.", "review_text": "", "extra_scores": {"ethical_concerns": "Fine.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1.5"}, "extra_text": {}}, {"review_id": "WHiRvjqoxNI", "reviewer": "Reviewer_xGhA", "summary": "The authors explore spectral properties of models in GAN training, isolating the generator and discriminator in turn, to address questions about inability for current frameworks to model certain data.\n", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "-- The conclusion about the generator's ability to match spectral properties under the right setting is critical. I think the current assumption places much of the burden on high-frequency checkerboard artifacts on the generator's upsampling. To show the biases may exist, but that under some loss regimes the generator can learn to correct for them, is a very important observation.\n\n-- The shift of focus both to the discriminator and to the magnitude of frequencies rather than if they are high or low is interesting, and motivates the adoption of alternative different discriminator architectures than otherwise would be explored.\n\n-- A minor quibble: the authors claim that the MSE-based pixel-level supervision baseline of studying the generator are neutral and don't impart any biases of its own, e.g. with the claim 'otherwise, even with a perfect discriminator it would be impossible to generate a correct image'. While pixel-level supervision has the perfect minimum, it is possible that taking gradient descent steps toward it along a MSE-minimizing path will not reach that minimum while some other loss could. This is still a good baseline to use and I don't object to any of the conclusions drawn with it, making my disagreement very minor indeed.\n\n-- One more question I would have liked to see explored is the impact on the distribution-level modeling of the data distribution. While obviously the point of this study is analyzing an individual point/image for its fidelity to a real point/image, I am curious about these statistics at a distribution level across the entire dataset. For example, if generator image i has particular artifacts, are those artifacts sensitive to what other points are in the training set (e.g. how diverse it is)? Or is that truly fundamental to the particular image i? To be clear, I understand this is a large question that need not be answered here, but it would have been interesting to see explored.\n\nI think this work overall makes many important points that will positively impact the direction of future research. It is well-written and well-supported, and both raises and then answers interesting questions.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1"}, "extra_text": {}}, {"review_id": "W539toaSWS", "reviewer": "Reviewer_GSWk", "summary": "Generative models, such as GANs, are known to not reproduce the spectral statistics. Various methods in the generator and discriminator have been proposed in papers. This paper systematically indexes and studies these tools and characterizes their effect on the spectral statistics of real vs generated images. The paper aims to use these insights to improve GAN training, with mixed results.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper performs a systematic study of tools for both the generator and discriminator. The paper uses a \"toyset\", which consists of synthetically generated images with two dominant frequencies, as well as low-resolution faces (64x64). The paper first tests on small datasets (10 images). The paper focuses on the generation in Section 3 and the discriminator in Section 4. Section 5 attempts to draw upon these lessons to improve GANs. The paper evaluates on how well the generations reproduce the spectral properties, as well as image generation metrics like FID. The paper also presents on \"detectability\" from a classifier trained on spectral properties. The paper's presents a clear story and focus and systematically studies different factors.\n\nI did have several concerns regarding the paper's presentation.\n- I am confused regarding the colorbar in Figures 2, 4, 5. First, I recommend explaining the heatmap in the caption, rather than buried in the text. In L145, the text mentions the colorbar is clipped between -1 and +1. I understand how a frequency can be overrepresented by +100%, but how can a frequency can be underpresented by more than 100%? In the worst case, if a frequency is not represented, it should be at -100%? Furthermore, most of the plot is either clipped at solid blue or red. This seems quite surprising that almost none of the frequencies are even close.\n- Figure 3 shows error bars, which adds to the value of the plot. However, the size of the error bars seem quite large. It is also quite difficult to tell the error bars apart, as they are heavily overlapping. Is it possible that standard deviation, rather than standard error, was plotted?\n\nMinor\n- Table 1. \"39.81\" has an extra significant figure\n- L115 \"investigatesnor\"\n- Equation 2: x used for H and y used for W is nonintuitive (to me)\n\nWhile the paper systematically sudies different issues, the conclusions in this paper are not too impactful. As described in L170-175 and L229, many of the findings are in agreement with previous literature. This is good corroboration, but ideally this paper would have lead to new insights that could improve GAN training. In Section 5, the attempt at improving GANs are mixed. Lastly, while the paper focuses on quantitative results, there are not many qualitative results. The most impactful potential result would be clear, qualitative examples, where a more accurate spectral result provides more compelling generations as well.\n\nIn conclusion, I appreciate the systematic, focused study of this aspect of image generation. As summarized above, I have some concerns regarding the presentation; the paper does not show improvement in GAN training, and it's unclear if the paper's will lead to future improvements. I am currently borderline on this paper.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers agreed that the paper was a strong systematic study of frequency bias of GANs. While the overall impact of the paper is somewhat limited, as previous work show similar results, it does provide sufficient new insight that it will be useful to the community. 
Decision: Accept (Poster)