FreGAN: Exploiting Frequency Components for Training GANs under Limited Data
Authors: Mengping Yang
Zhe Wang
Ziqiu Chi
Yanbing Zhang
Venue: NeurIPS 2022 Accept (2022)
Keywords: Generative adversarial networks
Limited data
Frequency analysis
Image generation

Abstract: Training GANs under limited data often leads to discriminator overfitting and memorization issues, causing divergent training. Existing approaches mitigate the overfitting by employing data augmentations, model regularization, or attention mechanisms. However, they ignore the frequency bias of GANs and take poor consideration towards frequency information, especially high-frequency signals that contain rich details. To fully utilize the frequency information of limited data, this paper proposes FreGAN, which raises the model's frequency awareness and draws more attention to synthesising high-frequency signals, facilitating high-quality generation. In addition to exploiting both real and generated images' frequency information, we also involve the frequency signals of real images as a self-supervised constraint, which alleviates the GAN disequilibrium and encourages the generator to synthesis adequate rather than arbitrary frequency signals. Extensive results demonstrate the superiority and effectiveness of our FreGAN in ameliorating generation quality in the low-data regime (especially when training data is less than 100). Besides, FreGAN can be seamlessly applied to existing regularization and attention mechanism models to further boost the performance.

FreGAN: Exploiting Frequency Components for Training GANs under Limited Data
Mengping Yang" Zhe Wang Ziqiu Chit
Yanbing Zhang
'Deparmment of Computer Science and Engineering, ECUST, China
Key Laboratory of Smart Manufacturing in Energy Chemical Process, ECUST, China
wangzhedecust. edu. en mengpingyang@maill www.aatti..eeee....oooooommmo..e
Abstract
Training GANs under limited data often leads discriminator overfitting and memorization issues, causing divergent training. Existing approaches mitigate the overfitting by employing data augmentations, model regularization, or attention mechanisms. However, they ignore the frequency bias of GANs and take poor consideration towards frequency information, especially high-frequency signals that contain rich details. To fully utilize the frequency information of limited data, this paper proposes FreGAN, which raises the model's frequency awareness and draws more attention to producing high-frequency signals, facilitating high-quality generation. addition to exploiting both real and generated images' frequency in- formation, we also involve the frequency signals of real images self-supervised constraint, which alleviates the GAN disequilibrium and encourages the generator to synthesize adequate rather than arbitrary frequency signals. Extensive results demonstrate the superiority and effectiveness of our FreGAN ameliorating gen- eration quality in the low-data regime (especially when training data is less than 100). Besides, FreGAN can be seamlessly applied to existing regularization and attention mechanism models to further boost the performance.
1 Introduction
Generative adversarial networks (GANs) [10] have shown impressive achievements in synthesising plausible and photorealistic visual objects, such as image [18] [7 and video 33 generation, image inpainting 225. image translation [35] and so on. However, prerequisite of such success is sufficient training data, which impedes applications of GANs in areas where only dozens of data are available or where it is challenging to collect massive data due geographical, spatial, temporal, privacy reasons. Thus developing data-efficient GANs that can generate plausible images under limited data, without compromising the quality, is necessary and meaningful.
Training GANs under limited data often leads to overfitting and instability issues 16
Specifical- ly, when the discriminator (D) overfits to the limited training data, it simply remembers the input real images and classifies others as fake images, thus providing meaningless feedback the generator (G), leading to divergent training and poor-quality generation. Ameliorating the synthesize quality under limited data still an unexplored problem. Recent approaches for this problem include enlarging the
training set with different data augmentations [52 36 51
16 14 regularizing the output of D with an additional constraint [37], and devising new network architectures [24]. However, existing methods are mainly developed from the perspective of data scale and model capacity, and they ignore
Conrespon author
Our codes are available at http://///taaaooooo////////////r
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

FastGAN
FreGAN (Ours)
Grumpy cat Flat-colored
(100 images) (36 images
ArtPainting (1000 images)
Figure 1: Images generated by FastGAN [24) and our FreGAN given limited training data. The details of FastGAN deteriorate while our FreGAN effectively ameliorates the synthesize quality by raising the model's frequency awareness, producing plausible images with better fine details.
a critical property of the data itself, i.e., frequency signals. GANS have been demonstrated to have spectral bias in fitting frequency signals [29] [34]. They preferentially fit ow-frequency signals and tend to ignore high-frequency signals 45 which encode fine details like vertical and horizontal edges
Missing them may lead to unrealistic image synthesize with unsatisfactory artifacts model, termed as FreGAN, to raise the frequency awareness of G and D. By encouraging to generate more reasonable and adequate high-frequency signals, our FreGAN ameliorates the synthesize quality under limited data, as shown in Fig.
(47 9.
(see Fig. 1 This paper proposes a frequency-awan
To fully exploit the frequency information of limited training data, we first decompose images into different frequency components via Haar wavelet transformation [6]. Unlike traditional wavelet transformation that employed at the image level, we perform it the intermediate features of both D and G. We then employ figh-freeuency discriminator (HFD) and frequency skip connection (FSC) raise the frequency awareness of and D, respectively. However, G still has no explicit clue about what high-free frequrryy signals it should synthesize, and D is overconfident in making real/fake decisions after seeing real and fake images. Such an unbalanced competition motivates us to perform high-frequency alignment (HFA) to alleviate the information asymmetry between G and Innovatively, we explicitly exploit the frequency signals of real images induced from as a self-supervised constraint to guide G to leverage the frequency knowledge properly. Besides, HFD and HFA are applied on multi-scale features to thoroughly excavate the frequency signals of limited data, mitigating frequency bias and loss of high-frequency information.
The primary contributions of this paper are three-fold: 1) we propose FreGAN raise the mod- el's frequency awareness, which successfully mines and exploits frequency information of limited data, and as a byproduct, FreGAN alleviates the unhealthy competition between G and D; 2) we demonstrate the compatibility of our model by combining our method with other techniques like regularization (37 and attention mechanism [22]: 3) we perform extensive experiments on various datasets with limited data, and our FreGAN achieves state-of-ther the-tar performance on these dataset- s, indicating the effectiveness and superiority of our method for ameliorating synthesize quality, especially when training data is extremely limited.
2 Related Work
Generative Adversarial Networks. Generative adversarial networks (GANs) 44 [15
[T] which
target at generating plausible and realistic images, have made massive progress since the pioneering work [10]. The capability of GANs enables various visual applications like image [8
[9 and
video generation [39 [38 image inpainting 48 (25). image manupulation 9 33 and super-
resolution [42) etc. However, GANs are notoriously difficult to train as several issues like mode collapse and instability happen easily. Numerous techniques have been proposed to stabilize training and improve the synthesize quality by designing new optimization objectives or network architec- tures [[3]. WGAN 2 and /-GAN (7 minimize the Wasserstein distance and the f-divergence of real and generated distribution instead of minimizing JS divergence in [10]. BigGAN 7 and

---

17 have made breakthrough progress in producing realistic images. However, the performance of these models deteriorates when given limited data.
StyleGAN series [1] 18
19 16
Training GANs under limited data. Improving the synthesize quality under limited data remains an underexplored problem, which has drawn extensive attention recently. Insufficient training data leads to discriminator overfitting, thus degrading the quality of generated images. One straightforward way to address such data scarcity is to expand the training set with various augmentations. In addition to employing conventional augmentation techniques [3) [22] (c.g.. fli, crop), ADA [I9] and DiffAug [51] propose adaptive and differentiable augmentation to enlarge the training data, respectively. APA 14 deceives based on the degree of overfitting with an adaptive pseudo augmentation. InsGen [46] involves instance discrimination an auxiliary task to encourage D to distinguish every individual image, which improves the discriminative power of the discriminator. Lecam [37 regularizes the output of the discriminator throughout the training process. FastGAN [24] employs a skip-layer channel-wise excitation module and a sulf--upervised discriminator to stabilize and accelerate the training. The most recently MoCA [2 improves few-shot image generation quality with a prototype memory with an attention mechanism. Benefit from the significant progress of large-scale pre-trained visual recognition models, Vision-aided GAN (20 uses available off- the-shelf models to help the GAN training and ProjectedGAN [1 improve GANs by projecting generated and real images into pre-trained feature spaces. Another category of methods transfer and reuse knowledge from models that are pre-trained on large-scale data, i.e. few-shot GAN 28 In this paper, ameliorate the synthesize quality under limited data from the frequency domain perspective. By raising the frequency awareness of GANs and providing more fine details G, facilitate photorealistie image generation. Our work is complementary to previous model regularization and attention mechanism approaches, and our method promotes equilibrium between G and D.
adaptation 41
23 40
Wavelet Transformation in GANs. Schwarz al. (32 prove that GANs exhibit frequency bias and resolving frequency artifacts is necessary for photorealistic image generation. Consequently, GANs tend to ignore high-frequency signals as they are hard to generate, compromising the generation quality. Wavelet transformation [6], which decomposes images into frequency components with different bands, has been wildly used in various applications of GANs, such as style transfer [4] [47),
image inpainting [48), image editing [9], etc. HiFA [9] alleviates the generator's ' pressure of producing high-fre frequueuyyncy signals by directly feeding high-frequency components to the generator. WaveFill [48] disentangles different frequency signals and explicitly fills the missing regions in each frequency band, achieving superior image inpainting. Zhang al. (49) propose wavelet knowledge distillation towards efficient maage-to-image translation without a performance drop. SWAGAN [8] incorporates wavelet with the hierarchical training of StyleGAN2 19 and performs wavelets the image level. Our FreGAN is more flexible by directly decomposing intermediate features of the generator and the discriminator into the wavelet domain, and no additional down/up sampling are required to convert images to higher/lower resolution as in SWAGAN, which makes our method more efficient. Unlike existing methods that are performed on ample data, this paper addresses the more challenging few-shot generation problem. In addition to raising the frequency awareness of the model, we also mitigate the unhealthy competition by lessening the frequency gap between and
3 Methodology
The overall framework our FreGAN is illustrated in Fig. To formulate our method, we explicitly utilize wavelet transformation to decompose features into different frequency components. We then employ high-frequency discriminator (HFD) and frequency skip connection (FSC) to raise the frequency awareness of G and D, respectively. Moreover, we perform fighffrequency alignment (HFA) to further guide to synthesize adequate frequency signals.
3.1 Wavelet Transformation
To decompose images into different frequency components, we adopt a simple but effective wavelet transformation, iee., Haar wavelet. Haar wavelet consists of two mirror operations: wavelet pooling and wavelet unpooling. The former converts images into the hlwavelet domain, and the latter inversely reconstructs frequency components into the spatial domain. There four kernels in wavelet pooling
3

---

Frequency Skip Connection
Wavelet Transformation
HFD
2-)(0,1)
Input
Wavelet Transformation
Wavelet Transformation
Images
Freh--Freuency Alignment (HFA)
Frequency SRIp Connection (FSC)
Hlghh Frequency Diseriminater (HFD)
Forward
Teasor Addition
Fi-s
Frequency Componcnts
TerrStention
Uxpeoling
Transfomation
F
Frequency Alignment
Features
Figure 2: The overall framework our proposed FreGAN, Composed of three key ingredients, ie., frequency skip connection (FSC), high-frequency discriminator (HFD), and high- high-frequency alignment (HFA), our FreGAN raises the model's frequency awareness, facilitating high-quality image synthesize under limited data.
Reals
LH-HL-HH
Figure 3: Illustration of frequency components obtained from Haar wavelet transformation.
The low (L) pass filter captures images' overall textures and outlines, and the high (H) pass filter concentrates on details such as the background and edges.
operation: LLT LHT HLT,, HHT, where L = 2 11, 1]. HT 11,, L and H denotes the low and high pass filters, respectively. The low (L) pass filter captures the outline and surface of images, while the high (H) pass filter focuses on detailed information like the edges and delicate textures. Fig. 3 illustrates the obtained frequency components of given images via Haar wavelet. We can observe that low-frequency component LL contains the overall surface of images, while components that decomposed by high pass filters, i.e., LH, HL, H contain more fine details. Further, by summing up the three gegh-rrequecy components, we approximately obtain all details information of images, e.g., the eyes of the cat and the teeth of Obama.
3.2 High-Frequency Discriminator
To raise the frequency awareness of D. devise high- high-frequency discriminator (HFD). HFD is responsible for distinguishing the real images from the generated images from the perspective of the frequency domain. Formally, for the i-th layer in the discriminator, on the intermediate features and obtain LLD. LH HDD H H then combine the three high- frequency components by tensor addition, i.e., HF), LH', + HLD + HH which contains sufficient details of features. By applying traditional convolution and downsampling operations following the original discriminator, define the ad loss of our HFD as:
adopt wavelet pooling
CHF = -EHr HF [mi D DHHHHHF (H rea))] E1 [min (0, -1 DDs (H rake)] (1)
CHF EHFi D (D(HHFFa) (F
(2)

---

where Fral and Fike are the high-frequeney information of real and fake images, respectively. Dy the high-frequency discriminator. Since the high-frequency information may be eschewed by D as the network goes deeper, we perform scale HFD on multi-layers of the discriminator. The multi-scale operation ensures fully mine and exploit the frequency information of limited data, which further improves D's frequency awareness. Notably, being guided by the HFD with Eq. 2 G is also optimized to produce rich high-frequency details.
3.3 Frequency Skip Connection
The generator is capable of producing plausible frequency signals after employing HFD (see Tab. 5) However, as GANs fit frequency signals from low high and the high-frequency signals may be ignored as the network goes deeper. To prevent the loss of high-freegg gegh-fre information and further en- courage the generator to produce rich details, we propose frequency skip connection (FSC). Concrete- ly, utilize wavelet unpooling operation of the frequency components LLG; LH HLg. H obtained from wavelet transformation on the features of G's i-th layer, which reconstructs the high- frequency representation the original features. Then we explicitly feed the reconstructed frequency representations to the next layer of G. Formally,
F = Fi+1 Unpooling(LLLs LHT: HTG HH()
where F; denotes the features of the i-th layer and Unpooling is the wavelet unpooling operation. F(+1 is the obtained features after FSC, which will be fed into the subsequent layer for further operation. Such skip connection prevents loss of high-frequency information and maintains high- frequency details.
3.4 High-Frequency Alignment
Adding HFD and FSC explicitly raises the frequency awareness of G, but G can only synthesize arbitrary frequency signals. How G can utilize the frequency signals is still ambiguous, and still dominates the competition since it learns discriminative knowledge from both real and generated images. To balance the unhealthy competition between G and D, we propose high- ligh-freuency align- ment (HFA), which involves high-free freg-frreeency signals of real images induced from D as a regularizer to guide G, promoting G to synthesize more reasonable and realistic fine details. Specifically, we extract the frequency representations of intermediate features of G at different layers. For the i-th layer of G, we obtain frequency components LLG, LH HLG, H We ignore LL' and combine the three figh-frequency components, i.e., H = LH + HL + H Then we use the fighfrreuency components of the discriminator H Fb as a self-supervision constraint. In addition to fool D, G is expected to minimize the distance of high-frequency information between the generated and real images. The alignment loss is defined as:
Calign |HED - H
(4)
where || * || denotes the L -norm. Such alignment encourages G to synthesis frequency signals that approach real frequency signals, mitigating the unhealthy competition and facilitating generation quality. To take full advantage of frequency signals of real images from D, we perform HFA on multi- scale features like HFD as shown in Fig. 2 The ablative experiment results in Sec. 2.2. demonstratt the rationality and effectiveness of employing HFA and HFD on multi-scale features.
3.5 Optimization
Following
24
we adopt the hinge version of adversarial loss to train our model.
LD Eezira [min(0, -1 + (xx)) Es [min(0,
(min -1
D())
(5)
Lg - E |D((<())))
(6)
We also use the reconstruction loss 24 to encourage the discriminator extract more representative features.
Lrecons -Ee-Deuns Im) [I) (x)
where f is the intermediate features of D. 9 and T denote the processing on the features f and the input images In sum, discriminator is optimized by LD. Lrecnns and C#F Our generator is optimized by Lc. LHF and Caligu the coefficient of each loss is set to 1.
5

---

FasIGAN
FreGNN (Ours)
FastGAN
FreGAN (Ours)
256: Obama (100 images), Animnall AnimalFFcc--au (100 images)
512"512: Anime-face (120 images), Art painting (1000 images)
FastGAN
FreGAN (Ours)
FauGAN
FreGAN (Ours)
1024-1024: Shells (64 images), Skulls (97 images)
512"512: AFIIQ-CACCC Cat (515) images), AFIIQ-Dup (47.39 images)
Qualitative comparison results of our FreGAN and baseline FastGAN. The images from left to right are generated images, low-frequency, and high-frequency components, respectively. Our FreGAN improves the overall quality of generated images and raises the model's frequency awareness, encouraging the generator to produce precise high- high-frequency signals with fine details.
Figure 4:
4
Experiments
Datasets. We test the effectiveness of our method on low-shot datasets from various domains with different resolutions. On 256 256 resolution, as 100-shot-Panda, Obama, and Grumpy_car cat [31]. On 512 x 512 resolution, we use Anime-Face,
Animal Face Dogs and Cat (33), well Art Paintings, Moongate, Flat-colored, and Fauvism-still- use Pokemon, Skulls, Shells, MetFace [16] and BrecaHAD [1 These datasets contain a limited number of samples (mostly less than 1,000) and cover art paintings, realistic photos, human faces, etc. For datasets that are not strictly equal to the corresponding resolution, we resize them to the closest resolution implementation. Besides, we use AnimalFace HQ (AFHQ) datasets to evaluate the performance of our model when training with more data (~5k).
24 On 1024 1024 resolution,
Evaluation metries and baseline. We adopt two common metries evaluate the synthesize quality: Fréchet Inception Distance (FID) [12] and Kernel Inception Distance (KID) []. The lower FID and KID is, the better the generation quality is. FID quantifies the distance between the distribution of the generated and the real images. KID, which is designed unbiased, has been proven more descriptive for small datasets (16), note that all KID scores reported in our paper need x 10 following [16).
Following 224, we calculate FID and KID by measuring the distance between all available training
images and 5k generated images. We also provide the LPIPS (50] IS 30) Precision, Recall 21].
Density, Coverage (26 results in the appendix.
We compare model with: 1) the tfate-of-the-art generative model StyleGAN2 [19), and SWA- GAN [8] which incorporates wavelet into StyleGAN2; 2) data augmentation- approaches that is designed for training GANs with limited data, i.e., ADA (16). DiffAug (51), APA [14; ; 3) the
state-of -fte--of-the-art few-shot generative model FastGAN [24]. We reimplement all baselines with their released official code under consistent settings for fair comparison. Implementation details of baseline models are given in the appendix.
Implementation details. We choose the current state-of-the-art few-shot generative model Fast- GAN (44 as the backbone and implement our proposed techniques upon all other settings remain the same (24). We decompose the intermediate 8 x 8, 16 x 16, 32 32 features of G and D discriminator, and
into frequency components for our frequency skip connection, high-frequency alignment. More implement details are given in the appendix.
4.1 Main Results
Quantitative comparison on datasets with limited data amounts. The quantitative comparison results of our FreGAN and baseline methods different resolutions are given in Tab. [ Tab. and Tab. We save the best training snapshots of each method and generate 5k images to compute FID and KID. The whole training set is adopted the referenced distribution. We can observe from the

---

Table I: The FID (lower is better) and KID (lower is better) scores of our method compared to state-of-the- methods on 256 x 256 datasets with limited data amounts.
Animal Face
100-shot
Dog (389
imgs)
Cat (160 imgs)
Panda
Obama
Grumpy cat
Method
FID
KID
FID
KID
FID
KI1>
FII
KID
FID
KID
StyleGAN2
[9]
113.86
91.31
79.04
34.43
18.05
7.40
69.01
52.63
35.00
11.01
SWAGAN
8
82.47
80.46
59.71
19.35
27.55
11.92
71.05
55.82
38,44
17.03
ADA [66
55.48
18.42
37.95
6.43
14.17
6.53
43.17
13.23
43.80
45.01
APA 14]
81.16
26.42
42.60
7.97
19.21
10.80
42.97
15.71
28.10
5.53
DiffAug [51
61.34
24.51
41.84
12.27
11.52
3.57
48.85
23.31
26.89
9.42
FastGAN [24]
52.46
18.22
33.85
4.99
9.70
1.60
35.80
5.50
25.75
3.41
FreGAN (Ours)
47.85
13.49
31.05
2.44
8.97
0.91
33.39
3.76
24.93
3.89
Table 2: The FID (lower is better) and KID (lower is better) scores of our method compared to
state-of-the-ar methods 512
512 datasets with limited data amounts.
AnimeFace
ArtPainting
Moongate
Flat
Fauvism
120 imgs
1000
imgs
136 imgs
36 imgs
124 imgs
Method
FID
KID
FID
KID
FID
KID
FID
KID
FID
KID
StyleGAN2
[9]
183.44
242.83
100.35
113.75
288.25
93.14
285.61
214.47
299.15
220.14
SWAGAN
8
189.71
216.39
56.95
22.50
302.72
99.47
293.94
232.53
291.66
226.21
ADA [6
59.67
16.02
46.38
12.26
149.06
43.21
248.46
62.89
201.99
86.64
APA [4]
58.38
15.73
47.23
10.60
193.67
50.52
233.52
166.53
197.47
66.
DiffAug [1
135.85
148.51
49.25
18.42
136.12
48.04
340.14
247.41
223.58
117.10
FastGAN [24]
55.87
11.17
45.06
10.26
114,79
23.57
216.27
36.88
178.42
58.01
FreGAN (Ours)
50.19
4.58
43.13
9.71
107.13
15.58
178.10
18.35
171.95
49.81
Table 3: The FID (lower is better) and KID (lower is better) scores of our method compared to
ofate-of-the-art methods on 1024 x 1024 datasets with limited data amounts.
Shells
Skulls
Pokemon 833 imgs
BrecaHAD
MctFace 1336 imgs
64 imgs
97 imgs
162 imgs
Method
FID
KID
FID
KID
FID
KID
FID
KID
FID
KID
StyleGAN2
19
133.31
33.36
234.54
209.22
161.28
161.98
174.07
176.32
66.97
55.53
SWAGAN
8
185.96
85.25
203.49
178.96
80.94
68.02
162.53
119.64
31.56
13.96
ADA [16
133.22
29.12
97.05
12.33
66.41
76.67
21.38
24.74
10.23
APA 14]
136.52
58.77
99.46
12.74
51.05
59.29
75.89
25.08
26.03
5.58
DiffAug [51
151.94
54.73
124.23
38.12
62.73
50.68
93.71
31.62
27.45
11.55
FastGAN
24
141.71
37.00
101.94
12.10
44.96
17.31
59.80
7.24
26.80
7.08
FreGAN (Ours)
125.77
20.58
86.12
5.47
38.88
10.42
54.88
3.41
25.42
5.93
results that, although evaluated on various datasets that have different resolutions and data amounts, all these datasets. Our FreGAN consistently improves both FID and KID metrics on 14 of the 15 datasets, demonstrating the effectiveness and generalizability of our proposed techniques. Notably, for those datasets with extremely limited data (less than 100), i.e., Flat (Tab. 2). Shells and Skulls (Tab. 3). our method improves the performance more significantly, e.g., the FID from 216.27 178.10 on Flat and from 101.94 to 86.12 on Skulls, and the corresponding KID is improved doubled, further reflecting our model's potential for training GANs with extremely limited data. More quantitative results are presented in the appendix.
our proposed FreGAN achieves superior performance
Qualitative Comparison. The qualitative results of FastGAN and our FreGAN on various datasets illustrated in Fig. 4 For each dataset in Fig. from left right are generated images, the visualization of the low and high-frequency components of the generated images. The images generated by FastGAN contain unsatisfactory artifacts and some of them are incongruous, e.g., the generated images of cat and dog in the bottom right of Fig[4] the cat has artifacts around the head, and the dog's 's cars are distorted. Our FreGAN significantly facilitates image quality in coordination. rationality, and fine details. As can been seen from Fig. the human face Obama generated by our FreGAN more photorealistic, the details of the anime lace, such as eye color and hair texture, are more realistic, and the synthesized animal faces of cats and dogs are also more plausible. Besides, the frequency components of the images generated by our FreGAN contain wealthier details. For example, the generated image of AnimalFace HaimalFace-Car Cat has richer background, and the generated image of Skulls has more clear contours of the eye and nose. Such observation reflects that the proposed FreGAN: 1) ameliorates the quality of generated images under limited data; 2) raises frequency awareness of synthesizing high-frequency signals with richer fine details of images; and 3) takes full advantage of limited data's 's frequency information. More qualitative results are given in the appendix.
7

---

Table 4: The FID (lower is better) and KID (lower is better) scores of our method compared to the state-of-the-ar FastGAN on AFHQ [5) datasets with more training data (~5k).
AFHQ-Cat (5153 imgs)
AFHQQDog (4739 imgs)
AFHQ--wil (4738 imgs)
Method
FID
KID
FID
KID
FID
KID
FastGAN [24]
10.17
4,91
25.36
14.29
7.30
1.93
+Ours
6.62
1.95
20.75
11.45
6.37
1.31
Table Ablation studies on different components our FreGAN, We remove each component to evaluate the efficacy of the three ingredients of our method, i.e., HFD, HFA, and FSC. The "Full" represents the the full version combining three techniques used the main experiments.
100-shot- Obama (256 256)
Anime Face (512 x 512)
Pokemon (1024
1024)
Module
FID
KID
FID
KID
FID
KID
Baseline
35,80
5.50
55.87
11.17
44.96
17.31
w/o HFD
35.67
7.78
55.17
8.16
41.75
13.69
w/o HFA
34.28
4.60
54.40
10.70
40.27
12.53
w/o FSC
33.52
4.18
51.15
4.83
39.41
11.13
Full
33.39
3.76
50.19
4.58
38.88
10.42
Effectiveness under datasets with more data. To investigate the effectiveness of our FreGAN more comprehensively, evaluate the performance datasets with more training data, i.e., AnimalFace- HQ (AFHQ) [S]. which includes 3 sub-datasets with close to 5k images, the results are shown in Tab. Similarly, our method yields compelling improvements on both FID and KID metries when training with more data. Combined with the generated images in Fig. 4 the results further validate our FreGAN's contribution to the synthesize quality. Our method boosts the performance under different amounts of data, suggesting the generalization of our model.
4.2 Ablation Studies
Ablation studies on variants of FreGAN. There are three ingredients of our FreGAN, i.e., the high- frequency discriminator (HFD), high-frequency alignments (HFA), and frequency skip connection (FSC). We evaluate the efficacy of each component by removing each of them from the full version of our FreGAN. We choose one from each of the different resolution datasets, i.e., 100-shot-Obama Anime face and pokemon for 256, 512 and 1024 resolution, respectively. As shown in Tab. 5 removing any of the three techniques leads to a performance drop, reflecting the contribution of each component. Still, all these variants outperform baseline FastGAN on both FID and KID, which implies that the combination of different components of our method consistently boosts model performance. Moreover, the performance drops the most when removing the HFD module, which is reasonable because the HFD raises the frequency awareness of G and D, and the frequency awareness of D serves as self- cule-superviiion to guide G to synthesize adequate and reasonable frequency signals. Qualitative comparison results of ablation studies are given the appendix.
Ablation studies on different scale of features. We employ proposed HFD and HFA on multi- scale features of G and D, namely, 8, 16, and 32 scales of features. Here we provide the ablation studies on different scales in Tab. 6 It can be seen that performing HFD and HFA on multi-scale features boosts the model performance. Besides, when only performing HFD and HFAA on single-scale features, the obtained results still outperform the FastGAN baseline, suggesting the effectiveness of HFA and HFD. Notably, despite adding more scales features may bring further performance advancement, the required additional convolutional and downsampling layers increases for higher scales features(e.g) g., 128, 256), bringing non-neglinn neg-eeglii computational costs.
components are ob- tained from wavelet transformation on the features, i.e., LH, H and H Each of them encodes different details of features as shown in Fig. 3 we sum them to fuse all the detail information for further operation our main experiments. Here we conduct experiments on three components respectively to verify their contribution and the necessity of fusing them. As shown in Tab. 6 each high- figh-frequency component contributes to the model performance compared with the baseline, and fusing them can better promote the generation quality.
Ablation studies on different frequency components. Three

---

Table 6: Ablation studies on the different scales of HED and HIA on AnimalFace-Dog. dataset. The numbers after "Feat" indicate the scale of features. "LHI" and "HL" denote the high-frequency components. The "Full" represents the full version of our FreGAN used in the main experiments.
Module
Metric
Bascline
Feat8
Feat8 16
LH
LH HI.
Full
HFD
52.46
51.86
50.63
52.15
51.79
47.85
HID
KID
18.22
17.13
16.28
17.94
16.57
13.49
FID
52.46
52.23
51.93
51.60
49.71
47.85
IIA
KID
18.22
17.15
16.87
16.92
15.87
13.49
Table 7: The FID (lower is better) and KID (lower is better) scores of method combined with
model regularization and attention mechanism techniques on 256 x 256 datasets.
Animal Face
100-shot
Dog (389 imgs)
Cat (160 imgs)
Panda
Obama
Grumpy cat
Method
FID
KID
FID
KID
FID
KID
FID
KID
FID
KID
Lecam 37
54.88
34.18
10.16
33.16
24.93
+ Ours
48.29
14.16
31.77
2.22
8.87
1.06
32.69
4.99
24.39
2.36
MoCA [22]
54.04
19.25
38.04
8,40
11.24
4.00
42.26
17.03
25.59
4.20
Ours
50.96
16.06
35.47
4.92
9.05
1.13
34.13
5.53
24.78
3.11
4.3
Analysis on Compatibility and GAN Equilibrium
Compatibility of Our Model. Lecam 37 and MoCA 22 exploit regularization and attention mechanism for training GANs under limited data, respectively. We implement our proposed tech- niques them to test the compatibility of our method. We keep the original setting unchanged and the set the regularization weight to 0.1. The FID results are given in Tab. from which we can see that FreGAN can further boost the performance of MoCA and Lecam, demonstrating that our method is complementary to the model regularization and attention mechanism methods.
GAN Equilibrium is improved. Our HFA module aligns the frequency components of real and generated images, guiding G to synthesize precise instead of arbitrary high-frequency signals. Mean- while, a byproduct, the HFA mitigates the domain gap between G and D, alleviating the unhealthy competition. As shown in Fig. 5 (a), our discriminator converges to a better point, and our generator better fool the discriminator, while the discriminator of FastGAN surpass the generator, thus providing less informative guidelines and degrading the synthesize quality. Besides, we plot the FII) and KID curves throughout the training process in Fig. (b), from which we can observe that our FreGAN are consistently better. Moreover, we plot the multi-scale HFA loss curves in Fig. 5 (c), where each line denotes the loss of each scale. These curves indicate that the frequency signals are well aligned, lessening the domain gaps and promoting the GAN equilibrium.
Diseussion
Conclusions. In this paper, propose frequency-aware method for training GANs under limited data, i.e., FreGAN. The proposed FreGAN ameliorates the synthesize quality by raising the model's frequency awareness, encouraging the model to pay more attention to frequency signals, especially high-frequency signals, which encode fine details images. We conduct extensive experiments on various datasets with different amounts of data and different resolutions to demonstrate the efficacy of our proposed method. Qualitative results suggest that our model successfully makes the generator to generate precise high-frequency signals, facilitating high-quality image generation. Quantitative results indicate that our method 1) substantially boosts the performance, especially when data is
D-Score (Cuus)
G-Score (Ours)
D-Score (FastGAN)
FID (FAGAN)
loss
G-Score (FasGAA))
FID (Ours)
Fre loss Fre loss
KID (FasGAN)
KID (Ouns
Fre All loss
11
100
(2)
Figure 5: Equilibrium is improved with our proposed techniques. (a) Our generator can better deceive the discriminator. (b) Our FreGAN yields better performance throughout the training process. (c) The aligned HFA loss mitigates the asymmetrical information and facilitates the GAN equilibrium.

---

extremely limited (less than 100), and 2) is complementary to existing regularization and attention models. Moreover, the proposed model alleviates the disequilibrium of GANs by lessening the frequency information gap. In the future, we plan to implement our techniques on more backbones, e.g. StyleGANN2 [9] and apply our method to more applications.
Limitations. Despite achieving significant improvements on various low-data datasets, our FreGAN still struggles in generating photorealistic images when given datasets with limited data but various contents, e.g. only dozens of images, and their contents vary widely. When the low-data datasets are imbalanced [43 or even long-tailed, the proposed method may fail to generalize, which is limited by the intrinsic reasons of the data distribution. Developing more effective ways to train generative models with insufficient training data still requires more efforts.
Acknowledgment
This work is supported by Shanghai Science and Technology Program "Distributed and generative
few-shot algorithm and theory research" under Grant No. 2051 100600 and "Federated based cross-domain and cross-task incremental learning" under Grant No. 21511 100000, Natural Science Foundation of China under Grant No. 62076094, Chinese Defense Program of Science and Tech- nology under Grant No. N0.2201-ICIO-JJ00041 China Aerospace Science and Technology Corporation Codustt--UniverstyyRRSssss Cooperation Foundation of the Eighth Research Institute under Grant No.SAST2021-007.
References
[1] Alper Aksac, Douglas J. Demetrick, Tansel Ozyer, and Reda Alhajj. Brecahad: a dataset for breast cancer histopathological annotation and diagnosis. BMC Research Notes, 2019.
[2] Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. In International conference machine learning, pages 214-223, PMLR, 2017.
[3] Mikohaj Binkkwski, Danica Sutherland, Michael Arbel, and Arthur Gretton. Demystifying mmd gans. In International Conference on Learning Representations, 2018
[4] Tai- Tai-Yin Chiu and Danna Gurari. Photowet2: Compact autoencoder for photorealistic style transfer resulting from blockwise training and skip connections of high-frequency residuals. Proceedings of the IEEF/CVF Winter Conference on Applications of Computer Vision, pages 2868 2877, 2022.
[5] Yunjey Choi, Youngjung Uh. Jaejun Yoo, and Jung-Woo Woo Ha. Stargan v2: Diverse image synthesis for multiple domains. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 8188 189-8197,98 2020.
[6] Ingrid Daubechies. The wavelet transform, time-freyuency localization and signal analysis. IEEE transactions on information theory, 3665):911-1055 1990.
[7] Jeff Donahue and Karen Simonyan. Large scale adversarial representation learning. Advances Neural Information Procexsing Systems. 32, 2019.
[8] Rinon Gal, Dana Cohen Hochberg, Amit Bermano, and Daniel Cohen-Or. Swagan: A style-based wavelet-driven generative model. ACM Transactions on Graphics (TOG), 40(4): 11, 2021.
[9] Yue Gao, Fangyun Wei, Jianmin Bao, Shuyang Gu. Dong Chen, Fang Wen, and Zhouhui Lian. High-fidelity and arbitrary face editing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16115-16124, 2021.
[10] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.
[11] Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron C Courville. Improved training of wasserstein gans. Advances in neural information processing systems, 30, 2017.
[12] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two simel update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017.
10

---

[13] Abdul Jabbar, Xi Li. and Bourahla Omar. A survey generative adversarial networks: Variants, applica- tions, and training. ACM Computing Surveys, 54(8):1-49, 2021.
[14] Liming Jiang. Bo Dai, Wayne Wu, and Chen Change Loy. Deceive d: Adaptive pseudo augmentation for gan training with limited data. Advances in Neural Information Processing Systems, 34, 2021.
[15] Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for improved quality, stability, and variation. In International Conference Learning Representations, 2018.
[16] Tero Karras, Miika Aittala, Janne Hellsten, Samuli Laine, Jaakko Lehtinen. and Timo Aila. Training generative adversarial networks with limited data. Advances in Neural Information Processing Systems, 33, 2020.
[17] Tero Karras, Miika Aittala, Samuli Laine, Erik Härkönen, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Alias-fres generative adversarial networks. Advances in Neural Information Processing Systems, 34, 2021.
[18] Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1401-4410, 2019.
[19] Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of stylegan. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 81 8110-8119, 2020.
[20] Nupur Kumari, Richard Zhang. Eli Shechtman, and Jun-Yan Zhu, Ensembling off-the-shelf models for gan training. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pottern Recognition, pages 10651-10662, 2022.
[21] Tuomas Kynkäänniemi, Tero Karras, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Improved precision and recall metric for assessing generative models. Advances in Neural Information P Processing Systems, 32, 2019.
[22] Tianqin Li. Zijie Li, Andrew Luo, Harold Rockwell, Amir Barati Farimani, and Tai Sing Lee. Prototype memory and attention mechanisms for few shot image generation. In International Conference on Learning Representations, 2022.
[23] Yijun Li, Richard Zhang, Jingwan Cynthia Lu, and Eli Shechtman. Few-shot image generation with elastic weight consolidation. Advances in Neural Information Processing Systems, 33, 2020.
[24] Bingchen Liu, Yizhe Zhu, Kunpeng Song, and Ahmed Elgammal. Towards faster and stabilized gan training for high-fidelity few-shot image synthesis. In International Conference on Learning Representations, 2021.
[25] Hongyu Liu, Ziyu Wan, Wei Huang, Yibing Song, Xintong Han, and Jing Liao. Pd-gan: Probabilistic diverse gan for image inpainting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9371 2371-9381, 2021.
[26] Muhammad Ferjad Nacem, Scong Joon Oh, Youngjung Uh, Yunjey Choi, and Jacjun Yoo. Reliahle fidelity and diversity metries for generative models. International Conference on Machine Learning, pages 7176-7185. PMLR, 2020.
[27] Sebastian Nowozin, Botond Cseke, and Ryota Tomioka. f-gan: Training generative neural samplers using variational divergence minimization. Advances in neural information processing systems, 29, 2016.
[28] Utkarsh Ojha, Yijun Li, Jingwan Lu, Alexei A Efros, Yong Jae Lee. Eli Shechtman, and Richard Zhang. Few-shot image generation via cross-domain correspondence. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 10743-10752, 2021.
[29] Nasim Rahaman, Aristide Baratin, Devansh Arpit, Felix Draxler, Min Lin, Fred Hamprecht, Yoshua Bengio, and Aaron Courville. On the spectral bias of neural networks. In International Conference on Machine Learning, pages 5301 13100 PMLR, 2019.
[30] Tim Salimans, Ian Goodfellow, Wojeiech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. Advances in neural information processing systems, 29, 2016.
[31] Axel Sauer, Kashyap Chitta, Jens Müller, and Andreas Geiger. Projected gans converge faster. Advances in Neural Information Processing Systems, 00117788414977 2021.
[32] Katja Schwarz, Yiyi Liao, and Andreas Geiger. On the frequency bias of generative models. Advances in Neural Information Processing Systems, 34, 2021.

---

[33] Zhangzhang Si and Song-Chun Zhu. Learning hybrid image templates (hit) by information projection. IEEE Transactions on pattern analysis and machine intelligence, 34(7):1354-1367, 2011.
[34] Matthew Tancik, Pratul Srinivasan, Ben Mildenhall, Sara Fridovich-Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimensional domains. Advances in Neural Information Processing Systems, 33, 2020.
[35] Omer Tov, Yuval Alaluf, Yotam Nitzan, Or Patashnik, and Daniel Cohen-Or. Designing encoder for stylegan image manipulation. ACM Transactions on Graphics, 40(4))1-14, 2021.
[36] Ngoe-Trume TrungT Tran, Viet-Hung Tran, Ngoc-Bao Nguyen, Trung-Kien Nguyen, and Ngai-Man Cheung. On data augmentation for gan training. IEEE Transactions on Image Processing, 30:1 1882-189877 2021.
[37] Hung- Yu Tseng, Lu Jiang, Ce Liu, Ming-Hsuan Yang, and Weilong Yang. Regularizing generative adversarial networks under limited data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7921-7931, 2021.
[38] Sergey Tulyakov, Ming- Yu Liu, Xiaodong Yang, and Jan Kautz. Mocogan: Decomposing motion and content for video generation. In Pruceedings of the IEEE conference on computer vision and puttern recognition, pages 1526-1535, 2018.
[39] Lin Wang, Yo-Sung Ho, and Kuk-Jin Yoon. Event-based high dynamic range image and very high frame rate video generation using conditional generative adversarial networks. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 10081-10090. 2019.
[40] Vaxing Wang, Abel Gariia,, David Berga, Luis Herranz, Fahad Shahbaz Khan, and Joost van de Weijer. Minegan: effective knowledge transfer from gans to target domains with few images. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pages 9332-9341, 2020.
[41] Yaxing Wang, Chenshen Wu, Luis Herranz, Joost van de Weijer, Abel Gonzalez-Garcia, and Bogdan Raducanu. Transferring gans: generating images from limited data. In Proceedings of the European Conference on Computer Vision, pages 218-234, 2018.
[42] Yuntao Wang, Lin Zhao, Liman Liu, Huaifci Hu, and Wenbing Tao. Umet: A u-shaped residual network for lightweight image super-resolution. Remote Sensing, 13(19)33848, 2021.
[43] Zhe Wang, Chenjie Cao, and Yujin Zhu. Entropy and confidence-based undersampling boosting ran- dom forests for imbalanced problems. IEEE Transactions on Neural Networks and Learning Systems, 31(12):5178-519 2020.
[44] Zhengwei Wang, She, and Tomas F. Ward. Generative adversarial networks in computer vision: survey and taxonomy. ACM Computing Surveys (CSUR), 54(2):1-38, 2021.
[45] Zhi Qin John Xu, Yaoyu Zhang, Tao Luo, Yanyang Xiao, and Zheng Ma. Frequency principle: Fourier analysis sheds light on deep neural networks. arXiv preprint arXiv: /01. 2019.
[46] Ceyuan Yang, Yujun Shen, Yinghao Xu, and Bolci Zhou. Data-efficient instance generation from instance discrimination. Advances in Neural Information Processing Systems, 34:9378 9390, 2021.
[47] Jaejun Yoo, Youngjung Uh. Sanghyuk Chun, Byeongkyu Kang, and Jung- Woo Ha. Photorealistic style transfer via wavelet transforms. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9036 900669905,222 2019.
[48] Yingchen Yu, Fangneng Zhan, Shijian Lu, Jianxiong Pan, Feiying Ma. Xuansong Xie, and Chunyan Miao. Wavefill: A based generation network for image inpainting. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 14114-14123, 2021.
[49] Linfeng Zhang, Xin Chen, Xiaobing Tu, Pengfei Wan, Ning Xu, and Kaisheng Ma. Wavelet knowledge distillation: Towards efficient image-to- to-image translation. arXiv preprint urXiv:2203.06322 2022.
[50] Richard Zhang, Phillip Isola, Alexei Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 186-595,10 2018.
[51] Shengyu Zhao, Zhijian Liu, Lin, Yan Zhu, and Song Han. Differentiable augmentation for data- efficient gan training. Advances in Neural Information Processing Systems, 33, 2020.
[52] Zhengli Zhao, Zizhao Zhang, Ting Chen, Sameer Singh, and Han Zhang. Image augmentations for gan training, arXiv preprint 22252000000255, 2020.
12

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] Section 5
(c) Did you discuss any potential negative societal impacts of your work? [Yes] Section S
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you experiments..
(a) Did you include the code. data, and instructions needed reproduce the main ex- perimental results (either the supplemental material or as a URL)? [Yes] Code is provided in the supplemental material. We will also make our code and pre-trained models publicly available
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] Implement details Section
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [No]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] Implement details Section
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you eite the creators? [Yes] Section 4
(b) Did you mention the license of the assets? [No]
(c) Did you include any new assets either in the supplemental material or as URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you're 're
using/curating? [Yes] Datasets in Section 4
(e) Did you discuss whether the data you are using/curating contains personally identifiable information offensive content? [No] The data use contains no personally
identifiable information or offensive content, the description of datasets given in Section 4
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
"Our codes are available htt.tt:://gittbb.ooo////o http://ttt...cooooeeeeegeg/////a
13


Reviews:
[{"review_id": "k9p1ZDt7-bE", "reviewer": "Reviewer_Xdtv", "summary": "This paper aims to stabilize GAN training in the low-data regime by incorporating wavelet information into the generator and discriminator. Specifically, the paper proposes three measures to improve image synthesis in the low-data regime: i) frequency skip connections (FSC), ii) an additional high-frequency discriminator (HFD), and iii) high-frequency alignment (HFA). The proposed method improves over the baselines on various datasets and for various metrics.\n", "questions": "Please add the comparisons to SWAGAN and ProjectedGAN discussed under weaknesses or justify why they are not applicable.  \n\nWhat is wavelet unpooling? Is it NN upsampling/bilinear upsampling in the wavelet domain? Please define a mathematical operator for this to make it clear.\n\nFig.3: what is the color scheme? I am confused, shouldn’t the lowpass filter (L) lead to a blurry version of the original image instead of these weird color distortions?\n\nLastly, adding a spectral analysis of the generated images (see e.g. Schwarz et al) would be instructive to validate the claim that the proposed method ‘raises the frequency awareness’ of the generator.\n", "limitations": "The results in the appendix indicate that recall is very low for all methods and all datasets. While this limitation seems not specific to the proposed method, it should be discussed in more detail in the paper.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\nGenerally, the idea to exploit frequency information for image synthesis in the low data regime is interesting as it might provide additional guidance to the model.\nThe proposed method is evaluated on a multitude of datasets and metrics. Further, the proposed design choices are validated in an ablation study that demonstrates their effectiveness.\n\nWeaknesses:\n\nThe paper is missing any comparison to SWAGAN[1]. While [1] is not aiming at the low-data regime specifically, it features many similarities to the proposed method as it considers image synthesis in the wavelet domain. There needs to be a detailed analysis on how the proposed method differs from SWAGAN as well as to include results for [1] as a baseline. Without this, it is difficult to judge how novel the contributions of this work are.\nThe paper uses FastGAN as baseline. However, ProjectedGAN [2] improves over FastGAN, also for small datasets. The comparison to ProjectedGAN is missing from the related work and the experimental analysis and thereby lacks comparison to one important state-of-the-art method. \nIt would also be interesting if the proposed method can be combined with ProjectedGANs for even further improvements.\nThe language is often colloquial, see misc.\n \n[1] SWAGAN: A Style-based WAvelet-driven Generative Model, Gal et al, SIGGRAPH 21\n[2] Projected GANs Converge Faster, Sauer et al, NeurIPS 21\n \nMisc:\nL.43, 106 “employ *a* high-frequency discriminator (HFD) and frequency skip connection*s* (FSC)”\nL.45, 141, 151, etc  synthesis -> synthesize\nL.54,101, etc ‘unhealthy’ -> unbalanced\nL.71 However, their (StyleGAN/BigGAN) success is mainly from sufficient training data” - I disagree. The main success of StyleGAN/BigGAN is due to architectural choices, carefully tuned training strategy, and regularization. Nonetheless, it is correct that their performance degrades with limited data.\nL.90 prove -> demonstrate\nL.112 the Haar transform does not transform an image into the frequency domain, but rather to the wavelet domain.\nL.138 add a pointer to ablation where this is shown\nL.198 Implement -> Implementation\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "U5rIJREp0uH", "reviewer": "Reviewer_BaYJ", "summary": "The paper proposes to improve the training of GANs with limited data by means of more careful attention to spectral properties of real and fake images. The proposed model, FreGAN, utilizes wavelets to map the generator’s and discriminator’s intermediate features to the frequency domain. These features are used as input for an additional discriminator’s branch (HFD) to raise the awareness of the discriminator to the frequency domain of images, as well as for an additional loss (HFA) which is used to provide supervision in the frequency domain for the generator. FreGAN is evaluated on a range of limited-size datasets (64-5000 images) and exhibits better FID and KID scores compared to previous few-shot GAN models. \n", "questions": "[1] My major concern is that the design of FreGAN favors the memorization of the training images more than the FastGAN baseline. Given the limited dataset size, the additional discriminator HFD can memorize all the possible HF for real images, and can thus force the generated images to have exactly same HF, which forces to fully reproduce the original training images, and not to generate any new images (as they would have different HF). On the other hand, HFA is a rather restrictive L1 reconstruction loss that also makes the generated images look closer to the exact training examples across all G layers. Missing evaluation of diversity, as well as some visual examples (e.g., 2 memorized examples for “shells” in figure 6 in supplementary) second my concerns. Please comment.\n\n[2] Please also comment on the other concerns 2,3,4.\n", "limitations": "The authors have adequately addressed the limitations and potential negative societal impact of their work.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "--- Strengths ---\n\n[1 - Originality] The paper combines the task of training GANs under limited data and the task of addressing the spectral discrepancy between real and GAN-generated images. The experiments demonstrate that raising the frequency awareness helps to achieve both improved image quality (FID, KID) and better rendering of high frequencies (visually). The paper is the first to explore this synergy for this specific task.\n\n[2 - Results] The results in tables look strong. The gain in FID and KID in comparison to the baseline FastGAN is clearly visible and is consistent across different datasets, image resolutions, and the number of training images.\n\n\n--- Weaknesses ---\n\n[1 - Evaluation (major concern)] In few-shot image synthesis, image quality metrics (FID, KID) can be often improved at the cost of the synthesis diversity [1] (figure 4). For example, the model that perfectly reproduces the training set can achieve near-to-zero FID and KID. The diversity aspect is not evaluated in this work at all. This makes the whole evaluation of the paper unconvincing. Does FreGAN improve FID of FastGAN at the cost of diversity? Is FreGAN memorizing training images more than FastGAN? Are the interpolations in the latent space of FreGAN smooth? It is not possible to answer with the provided figures and tables.\n\nIn addition to FID or KID, I suggest adding a metric that explicitly measures the diversity of generated images, as done in some previous works. One can choose pairwise LPIPS as in [2] (tables G,H), or intra-cluster LPIPS as in [3] (table 2). Moreover, it would be good to measure the average LPIPS from generated images to the nearest example from the training set, which would help to assess the degree of memorization of the training set ([4], table 8). Lack of diversity is also partially supported by low recall across different datasets reported in the supplementary materials. On the qualitative side, it would be good to add examples of latent space interpolations as in [4] (figure 5) and the nearest real neighbors to some of generated images ([4], figure 12-13). Overall, without this analysis, given the size of the used datasets, only improving FID and KID is not a convincing result to me. \n\n\n[2 - Claim (major concern)] The paper claims that the proposed method alleviates the unhealthy competition between G and D (lines 11, 47, 54) in the low data regime. The caption of figure 5 says that the FreGAN generator can better deceive the discriminator. The justification for the claim is claimed in the loss curves plot in figure 5(a). \n\nIn fact, what is seen in figure 5(a) seems to say the opposite. The discriminator loss of FreGAN is lower than the one of FastGAN (even though it has an additional term L^HF). This indicates that the discriminator can dissect real and fake images more confidently. It would be interesting to see the curves of the discriminator outputs during training (e.g., as in [5], figure 1(b-c)), where this effect should be more visible. Consequently, based on the plot 5(a), the FreGAN generator struggles to fool the discriminator more than FastGAN. This is confirmed by the larger G loss for FreGAN. It is thus not clear how the better GAN equilibrium is quantified by the authors, so one of the main claims is not supported.\n\n\n\n[3.1 – Clarity (major concern)] Generally, it is not clear why the technical proposals of the paper are introduced as techniques for training GANs in low data regimes. The frequency bias for GANs exists not only when the training data size is limited [6]. In principle, any GAN model should benefit from generating images with better spectral properties (e.g., the motivation in lines 33-39 also fits to training GANs on large datasets). The explanation on mitigating unhealthy G and D competition is, in my opinion, unsupported (see Weaknesses-2). \n\nWhy are the introduced modifications tested only in low-data regimes? Do the proposed techniques help to improve the quality of images or spectral properties of GANs trained on large datasets?\n\n\n[3.2 – Related work (major concern)] The proposed method is by design aimed to improve the quality of images in the frequency domain. There are other works with similar motivation (e.g., [6] as an overview). These methods can be potentially placed instead of the techniques introduced in section 3.2-3.4. It is not clear whether the effect of the proposed techniques could not be achieved with already existing techniques mitigating spectral biases of GANs. A proper comparison to previous works is expected.\n\n[3.3 – Evaluation (major concern)] The proposed method is aimed to improve the quality of images in the frequency domain, but this aspect is not evaluated quantitatively. The only provided comparisons of high/low frequencies are visual, which makes it subjective, especially since ground truth spectrums to compare to are not available. Quantitative assessment of GAN images spectrums was approached before, for example by measuring the accuracy of a binary classifier trained on spectrum features of real and fake images [7]. A proper quantitative comparison of spectral properties of images for different models is expected.\n\n[4 – Clarity (moderate concern)] The motivation of the frequency alignment loss (lines 151-161) is not clear. Firstly, what is meant by “G can only synthesize arbitrary frequency signals”? The synthesis of G is guided with the discriminator loss, which provides supervision on the image realism, which includes the intensity of details of different frequencies. So they should not be arbitrary. Secondly, why is HDA called a “regularizer for D”, while it is included only to the objective of G? \n\nFinally, eq. (4) is a form of a reconstruction loss, but it is computed between D’s features of a real image and G’s features of a randomly sampled fake image. These images are not necessarily aligned spatially, so it is not clear why imposing a pixel-wise L1 reconstruction loss is meaningful. Please explain. \n\n\n--- References ---\n\n[1] Few-Shot Adaptation of Generative Adversarial Networks. Robb et al. Arxiv, 2020.\n\n[2] Generating Novel Scene Compositions from Single Images and Videos. Sushko et al. Arxiv, 2021.\n\n[3] Few-shot Image Generation via Cross-domain Correspondence. Ojha et al. CVPR 2021.\n\n[4] Towards Faster and Stabilized GAN Training for High-fidelity Few-shot Image Synthesis. Liu et al. ICLR 2021.\n\n[5] Training Generative Adversarial Networks with Limited Data. Karras et al. NeurIPS 2020.\n\n[6] On the Frequency Bias of Generative Models. Schwarz et al. NeurIPS 2021.\n\n[7] Watch your Up-Convolution: CNN Based Generative Deep Neural Networks are\nFailing to Reproduce Spectral Distributions. Durall et al. CVPR 2020.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "M28lIMZALcD", "reviewer": "Reviewer_MrWM", "summary": "The paper proposes an effective approach to the problem of few-shot GAN learning by leveraging the frequency information from the generated images. Specifically, the author designs a model, FreGAN, which integrates wavelet transformation, a high-frequency discriminator (HFD), frequency skip connections (FSC), and high-frequency alignment (HFA) together. Experiments are conducted on multiple datasets with different number of samples. The ablation of the experiment well explains the functionality for each component.", "questions": "See weakness.\n\nAs a side note, I am also interested to know how the author would integrate the approach into StyleGAN. Would the high-frequency information be extracted from the modulation signal or the modulated feature maps?", "limitations": "Yes.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. The method is novel. To improve high-frequency components for low-shot GAN learning is interesting.\n2. The comparisons are extensive. Both qualitative and quantitative comparisons show strong results.\n3. The ablation study well explains the effectiveness for each component.\n\nWeaknesses:\n\n1. Using some visual examples/figures to illustrate the effectiveness of each component could be even better. For example, how does HFA work? As HFD already provides a supervision to enforce G to synthesize high-frequency component similar to the real images, why should we add additional constraint on the intermediate feature maps? Does HFA stabilize the training?\n2. The clarity of the paper can be improved, for example, it is not clear WHAT $\\mathcal{G}$ and $\\mathcal{T}$ are in Eqn. (7). ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept