On the Frequency Bias of Generative Models

1 Introduction . In recent years , unconditional Generative Adversarial Networks ( GANs ) have achieved impressive photo-realism for image synthesis tasks . While this has hampered the identification of generated images based on visual cues , multiple recent works show that it is straightforward to distinguish real and generated images based on their high-frequency content [ 7–9 , 14 , 22 , 34 , 38 ] . This has aroused considerable interest because it reveals a fundamental problem in state-of-the-art GANs : Existing approaches evidently struggle to learn the correct data distribution . While GAN training is notoriously hard , learning the distribution of high-frequency content is particularly problematic [ 7– 9 , 14 , 22 , 34 , 38 ] . This indicates a systematic problem in existing approaches that could make training suboptimal . For example , if generating high-frequencies is difficult for the generator but detecting them is straightforward for the discriminator this imbalance could impair the stability of the training . Conversely , if the discriminator struggles to detect high frequencies generating fine details also becomes more difficult which could impede convergence . Therefore , we argue that it is important to better understand the expressivity of both the generator and the discriminator . Indeed , numerous works suggest that the architecture of the generator can hamper the generation of high 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . ( a ) Ground truth ( b ) Reconstruction of a generator with bilinear upsampling using pixel-level supervision ( c ) Reconstruction guided by a discriminator with BlurPool downsampling Figure 1 : Spectral Properties of Generator and Discriminator . ( a ) We show a natural image and a high-pass filtered version ( amplified for clarity ) to illustrate the networks ’ behavior for both low and high magnitudes at high frequencies . ( b ) Our experiments reveal that bilinear upsampling biases the generator towards generating data with little high-frequency content , regardless of the magnitude . ( c ) The discriminator shows no such bias but provides better guidance for frequencies with high magnitude . The results further reveal artifacts due to downsampling that impair the training signal . We include the training details for both experiments in the supplementary . frequencies [ 4 , 7 , 9 , 10 , 14 , 20 ] . However , another line of works question the quality of the training signal provided by the discriminator [ 5 , 10 , 15 ] . But is there indeed a frequency bias that prevents learning high frequencies in existing GAN models ? This is a non-trivial question as GAN training involves two players where architectures for both the generator and discriminator , loss functions , as well as the dataset statistics can all affect the generated images . To narrow down potential factors , we first develop isolated testbeds for both the generator and discriminator . Then , we extend our findings to state-of-the-art GANs on large-scale datasets . Generator : The majority of works attribute the high-frequency artifacts to the upsampling operations in the generator [ 4 , 7 , 9 , 25 , 38 ] . Upsampling typically follows a pre-defined scheme , e.g . bilinear or nearest neighbor interpolation , or zero insertion between pixels . While the latter introduces too many high frequencies , interpolation has been shown to reduce artifacts [ 4 , 7 ] . But is interpolation always a good scheme for upsampling ? Our experiments indicate that both bilinear and nearest neighbor upsampling bias the generator towards predicting little high-frequency content , see Fig . 1b . While zero insertion can be more flexible , it is prone to introducing checkerboard artifacts , i.e . too many high frequencies [ 4 , 7 , 25 ] . But why do the learnable filters in subsequent layers not learn to compensate for these artifacts ? Depending on the training objective , checkerboard artifacts might be penalized only slightly . Our experiments evidence that when the loss function is sensitive to these artifacts the generator is indeed capable of compensating for them . Discriminator : This leads us to question the quality of the training signal : Can the discriminator detect high frequencies and provide the necessary supervision ? Chen et al . [ 5 ] argue that the discriminator can not detect high-frequency information due to the downsampling operations . Our experiments corroborate that downsampling can introduce artifacts in the training signal . However , we also observe that the discriminator can indeed provide a meaningful training signal for the spectral statistics at high frequencies when their magnitude is large enough , e.g. , Fig . 1c . Other works propose to train both the generator and discriminator in wavelet space [ 10 ] or add a discriminator on the spectrum of the images to reduce high-frequency artifacts [ 15 ] . Motivated by these approaches we ask : Is it enough to consider only the spatial domain ? In agreement with these works , our results suggest that the discriminator can benefit from ( additional ) input in frequency-based domains . Further , our testbed yields insights on which of these measures is most effective and reveals that the training signal from the discriminator might remain problematic . Contributions : We take a sober look at explanations for high-frequency artifacts in generated images and unify the efforts that have been done so far . In particular , we develop isolated testbeds for both the generator and the discriminator which allows us to analyze what makes existing efforts effective . The conclusions drawn by this paper shed new light on limitations of common design choices for both generator and discriminator : i ) Bilinear and nearest neighbor upsampling bias the generator towards predicting little high-frequency content . ii ) Zero insertion is prone to producing checkerboard artifacts in the generated images . However , with a suitable loss function , the learnable filters of the generator can compensate for the artifacts . This indicates that the upsampling in the generator alone can not explain the spectral discrepancies . iii ) In general , the discriminator is able to detect high frequencies and provide supervision to learn the correct spectral statistics . However , while the exponential decay of the spectrum for natural images creates the impression of the discriminator being insensitive to high frequencies , it actually struggles with low magnitudes . iv ) We find that all commonly used downsampling operations in the discriminator can impair the quality of the training signal . Lastly , we demonstrate that these findings extend to full GAN training . In agreement with [ 15 ] , we find that spectral discriminators can bring us one step closer to matching the spectral statistics . Nonetheless , generated images remain straightforward to classify based on their spectral statistics only . While recent works on GANs largely focus on improving the generator , e.g. , [ 16 , 18 , 19 ] , our findings suggest that the design of the discriminator plays an equally important role and deserves more attention in future work . We believe that our testbeds for both the generator and discriminator can be a useful tool for future investigations . We release our code and dataset at https : //github.com/ autonomousvision/frequency_bias . 2 Preliminaries . GAN Training : A generative adversarial network consists of a generator and a discriminator that are trained jointly in a 2-player game . Given latent variables z ∼ pz , the generator synthesizes images while the discriminator tries to distinguish the synthesized images from real images I ∼ pD , sampled from data distribution pD . Let GΘ and DΦ denote a generator G and a discriminator D with parameters Θ and Φ , respectively . The parameters of the models are updated in alternating steps using a non-saturating GAN objective [ 11 ] which is often combined with R1-regularization to stabilize training [ 24 ] V ( Θ , Φ ) = Ez∼pz [ f ( DΦ ( GΘ ( z ) ) ) ] + EI∼pD [ f ( −DΦ ( I ) ) − λ‖∇DΦ ( I ) ‖2 ] ( 1 ) where f ( t ) = − log ( 1 + exp ( −t ) ) and λ controls the strength of the regularizer . Image Processing in the Frequency Domain : The discrete 2D Fourier transform maps a gray scale image I ∈ RH×W to the frequency domain : Î [ k , l ] = 1 HW H−1∑ x=0 W−1∑ y=0 exp−2πi x·k H exp−2πi y·l W ·I [ x , y ] ( 2 ) for k = 0 , . . . , H − 1 and l = 0 , . . . , W − 1 . The power spectral density is estimated by the squared magnitudes of the Fourier components S [ k , l ] = |Î [ k , l ] |2 . Similar to [ 7 , 8 ] we consider the reduced spectrum S̃ , i.e . the azimuthal average over the spectrum in normalized polar coordinates r ∈ [ 0 , 1 ] , θ ∈ [ 0 , 2π ) S̃ ( r ) = 1 2π ∫ 2π 0 S ( r , θ ) dθ with r = √ k2 + l2 1 4 ( H 2 +W 2 ) and θ = atan2 ( k , l ) ( 3 ) Since images are discretized in space , the maximum frequency is determined by the Nyquist frequency . For a square image , H = W , it is given by fnyq = √ k2 + l2 = H/ √ 2 , i.e . for r = 1 . Spectral Classifier : To detect generated images , Dzanic et al . [ 8 ] propose to classify real and generated images based on their reduced spectrum . More specifically , they fit a power-law function to the tail of each reduced spectrum for frequencies above a given threshold rc = 0.75 and train a binary classifier on the fit parameters of each spectrum . 3 Are high frequencies more difficult to generate ? . In this section , we investigate if there is a frequency bias in the generator that impedes the generation of high frequencies . Therefore , we first consider the generator in an isolated setting assuming pixellevel supervision – otherwise , even with a perfect discriminator it would be impossible to generate a correct image . We extend our findings to the full GAN setting in Section 5 . There are two lines of argumentation in existing works : The first investigates artifacts that arise from the upsampling operations in the generator [ 4 , 7 , 9 ] . These works analyze the spectral properties of the generator predictions at convergence . We extend this analysis from the perspective of a frequency bias to see if low frequencies are learned earlier during training . The second line of works argues for a frequency bias of the learnable filters [ 14 , 20 ] . Motivated by their analysis we investigate if learnable filters are at all able to compensate for the artifacts introduced by upsampling . Experimental Setting : To isolate the generator from the discriminator we consider a conditional reconstruction task . In particular , we take 10 images from a dataset and pair them with 10 latent codes drawn from a normal distribution . Given a latent code , the generator is optimized to reconstruct the corresponding image with a pixel-wise L2-loss LI = 1 HW H−1∑ x=0 W−1∑ y=0 ‖GΘ ( z ) [ x , y ] − I [ x , y ] ‖22 ( 4 ) We choose the generator from PGAN [ 16 ] because of its simple architecture comprising only convolutions and upsampling operations . Further , when combined with R1-regularization it trains stably in a GAN setting [ 24 ] , allowing us to use the same generator in Section 5 . We reduce the number of channels for faster training because our primary focus is on the spectral properties . As upsampling operations , we investigate bilinear and nearest neighbor interpolation , zero insertion , and reshaping in the channel dimension [ 29 ] . We ensure that all networks have a similar number of parameters . More details are provided in the supplementary . Datasets : In natural images , the spectral density follows an exponential decay . To isolate the effects of frequency range and magnitute , we create a Toyset of images with two Gaussian peaks N ( µ1 , σ1 ) , N ( µ2 , σ2 ) of equal magnitude in the spectrum , see Fig . 2a , 2b . Based on the Nyquist frequency fnyq = H/ √ 2 , we create samples using σ1 = σ2 = 1/ √ 2fnyq and draw µ1 ∼ U ( 0.05fnyq , 0.15fnyq ) and µ2 ∼ U ( 0.75fnyq , 0.85fnyq ) . Together with a uniformly distributed phase , we apply the inverse Fourier transform to create images . We further test our setting on natural images with a downsampled version of CelebA [ 21 ] . Both datasets have a resolution of 642 pixels . Evaluation Metrics : On the spatial domain , we report PSNR on the RGB values and on the frequency domain we evaluate the reduced spectrum , see Eq . ( 3 ) . To analyze frequency-dependent convergence we further visualize the evolution of the spectrum similar to [ 26 ] . Here , the x-axis denotes the ( frequency ) radius r in normalized polar coordinates and the y-axis denotes the training iterations . The color corresponds to the relative error of the average predicted reduced spectrum wrt . the ground truth , where positive and negative values indicate too many and too few predicted frequencies , respectively . We clip the colorbar at 1 , i.e. , when the relative error exceeds 100 % . Do generators exhibit a frequency bias ? The spectral evolution in Fig . 2 shows different behavior for bilinear and nearest neighbor upsampling compared to zero insertion and reshaping . Particularly on the Toyset , a generator with bilinear or nearest neighbor upsampling learns the lower frequencies earlier in training than the high frequencies , see Fig . 2c and 2d . While the generator with bilinear upsampling struggles with high frequencies throughout training , for nearest neighbor upsampling it eventually fits the high-frequency peak , suggesting that its bias towards little high-frequency content is not as strong . In contrast , a generator with zero insertion or reshaping learns both peaks approximately at equal speed but is prone to generating checkerboard artifacts as indicated by the large error at the highest frequency in Fig . 2e and 2f . Between the peaks of the toyset , where frequencies have small magnitudes , all methods struggle to match the statistics of the training data because the L2-loss penalizes errors at frequencies with low magnitude only slightly . This explains why for CelebA there is an overall trend towards learning lower frequencies earlier during training . The PSNR values in Table 1 support these findings in the spatial domain . The lack of high frequencies for bilinear and nearest neighbor upsampling manifests in a low PSNR , particularly for the Toyset which contains many high frequencies by construction . Can the generator learn to compensate for the artifacts ? Since the L2-loss is less sensitive to frequencies with low magnitudes ( see supplementary for a formal derivation ) , we now add an L2-loss on the logarithm of the reduced spectrum : LS = 1 H/ √ 2 H/ √ 2−1∑ k=0 ∥∥∥log ( S̃ ( GΘ ( z ) ) ) [ k ] − log ( S̃ ( I ) ) [ k ] ∥∥∥2 2 ( 5 ) The logarithm penalizes errors at low magnitudes more strongly and can therefore reduce the checkerboard artifacts introduced by zero insertion and reshaping , see Fig . 3 . Hence , given a suitable objective , learnable filters can indeed compensate for high-frequency artifacts introduced by zero insertion and reshaping . Interestingly , bilinear upsampling does not benefit as much from the spectral loss . This suggests that a strong bias towards little high-frequency content can be more difficult to compensate for . As expected , the additional loss does not alter the evaluation in the spatial domain significantly and yields similar PSNR values in Table 1 . Implications : Overall , these results lead us to conclude that different upsampling operations bias the generator towards different spectral properties . Nearest neighbor and particularly bilinear upsampling introduce a bias towards fitting functions with little high-frequency content . In Section 5 , we will see that this can also be beneficial when working with natural images as their spectral density follows an exponential decay . On the other hand , zero insertion and reshaping introduce a bias towards checkerboard artifacts . However , with a suitable loss function , the network filters can learn to compensate for these artifacts . Therefore , the upsampling in the generator alone can not explain the spectral discrepancies . This suggests that the training signal provided by the discriminator might be suboptimal in the first place .

Summary: The paper studies the potential causes of  discrepancy in high frequency statistics between real and GAN-generated images. Specifically, it studies the upsampling choices in the generator, impact of downsampling in the discriminator and the corresponding frequency bias exhibited by such choices. The paper introduces clever experimental conditions to test various hypothesis related to the above study. Unsurprisingly, the authors claim that adding spectral matching loss or spectral discriminator helps match the spectral statistics. They don’t talk about the introduction of other artifacts in such conditions.  Authors conclude by saying that fully eliminating the discrepancy in the spectral statistics between real and generated distributions of images is still and open open problem. One of the key learnings from the paper is that the discriminators can provide good signals for learning high frequencies, but struggle to model frequencies with low magnitudes.