Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper proposes an effective approach to the problem of few-shot GAN learning by leveraging the frequency information from the generated images. Specifically, the author designs a model, FreGAN, which integrates wavelet transformation, a high-frequency discriminator (HFD), frequency skip connections (FSC), and high-frequency alignment (HFA) together. Experiments are conducted on multiple datasets with different number of samples. The ablation of the experiment well explains the functionality for each component.
strengths_and_weaknesses: Strengths:

1. The method is novel. To improve high-frequency components for low-shot GAN learning is interesting.
2. The comparisons are extensive. Both qualitative and quantitative comparisons show strong results.
3. The ablation study well explains the effectiveness for each component.

Weaknesses:

1. Using some visual examples/figures to illustrate the effectiveness of each component could be even better. For example, how does HFA work? As HFD already provides a supervision to enforce G to synthesize high-frequency component similar to the real images, why should we add additional constraint on the intermediate feature maps? Does HFA stabilize the training?
2. The clarity of the paper can be improved, for example, it is not clear WHAT $\mathcal{G}$ and $\mathcal{T}$ are in Eqn. (7). 
questions: See weakness.

As a side note, I am also interested to know how the author would integrate the approach into StyleGAN. Would the high-frequency information be extracted from the modulation signal or the modulated feature maps?
limitations: Yes.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper aims to stabilize GAN training in the low-data regime by incorporating wavelet information into the generator and discriminator. Specifically, the paper proposes three measures to improve image synthesis in the low-data regime: i) frequency skip connections (FSC), ii) an additional high-frequency discriminator (HFD), and iii) high-frequency alignment (HFA). The proposed method improves over the baselines on various datasets and for various metrics.

strengths_and_weaknesses: Strengths:

Generally, the idea to exploit frequency information for image synthesis in the low data regime is interesting as it might provide additional guidance to the model.
The proposed method is evaluated on a multitude of datasets and metrics. Further, the proposed design choices are validated in an ablation study that demonstrates their effectiveness.

Weaknesses:

The paper is missing any comparison to SWAGAN[1]. While [1] is not aiming at the low-data regime specifically, it features many similarities to the proposed method as it considers image synthesis in the wavelet domain. There needs to be a detailed analysis on how the proposed method differs from SWAGAN as well as to include results for [1] as a baseline. Without this, it is difficult to judge how novel the contributions of this work are.
The paper uses FastGAN as baseline. However, ProjectedGAN [2] improves over FastGAN, also for small datasets. The comparison to ProjectedGAN is missing from the related work and the experimental analysis and thereby lacks comparison to one important state-of-the-art method. 
It would also be interesting if the proposed method can be combined with ProjectedGANs for even further improvements.
The language is often colloquial, see misc.
 
[1] SWAGAN: A Style-based WAvelet-driven Generative Model, Gal et al, SIGGRAPH 21
[2] Projected GANs Converge Faster, Sauer et al, NeurIPS 21
 
Misc:
L.43, 106 “employ *a* high-frequency discriminator (HFD) and frequency skip connection*s* (FSC)”
L.45, 141, 151, etc  synthesis -> synthesize
L.54,101, etc ‘unhealthy’ -> unbalanced
L.71 However, their (StyleGAN/BigGAN) success is mainly from sufficient training data” - I disagree. The main success of StyleGAN/BigGAN is due to architectural choices, carefully tuned training strategy, and regularization. Nonetheless, it is correct that their performance degrades with limited data.
L.90 prove -> demonstrate
L.112 the Haar transform does not transform an image into the frequency domain, but rather to the wavelet domain.
L.138 add a pointer to ablation where this is shown
L.198 Implement -> Implementation

questions: Please add the comparisons to SWAGAN and ProjectedGAN discussed under weaknesses or justify why they are not applicable.  

What is wavelet unpooling? Is it NN upsampling/bilinear upsampling in the wavelet domain? Please define a mathematical operator for this to make it clear.

Fig.3: what is the color scheme? I am confused, shouldn’t the lowpass filter (L) lead to a blurry version of the original image instead of these weird color distortions?

Lastly, adding a spectral analysis of the generated images (see e.g. Schwarz et al) would be instructive to validate the claim that the proposed method ‘raises the frequency awareness’ of the generator.

limitations: The results in the appendix indicate that recall is very low for all methods and all datasets. While this limitation seems not specific to the proposed method, it should be discussed in more detail in the paper.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper proposes to improve the training of GANs with limited data by means of more careful attention to spectral properties of real and fake images. The proposed model, FreGAN, utilizes wavelets to map the generator’s and discriminator’s intermediate features to the frequency domain. These features are used as input for an additional discriminator’s branch (HFD) to raise the awareness of the discriminator to the frequency domain of images, as well as for an additional loss (HFA) which is used to provide supervision in the frequency domain for the generator. FreGAN is evaluated on a range of limited-size datasets (64-5000 images) and exhibits better FID and KID scores compared to previous few-shot GAN models. 

strengths_and_weaknesses: --- Strengths ---

[1 - Originality] The paper combines the task of training GANs under limited data and the task of addressing the spectral discrepancy between real and GAN-generated images. The experiments demonstrate that raising the frequency awareness helps to achieve both improved image quality (FID, KID) and better rendering of high frequencies (visually). The paper is the first to explore this synergy for this specific task.

[2 - Results] The results in tables look strong. The gain in FID and KID in comparison to the baseline FastGAN is clearly visible and is consistent across different datasets, image resolutions, and the number of training images.


--- Weaknesses ---

[1 - Evaluation (major concern)] In few-shot image synthesis, image quality metrics (FID, KID) can be often improved at the cost of the synthesis diversity [1] (figure 4). For example, the model that perfectly reproduces the training set can achieve near-to-zero FID and KID. The diversity aspect is not evaluated in this work at all. This makes the whole evaluation of the paper unconvincing. Does FreGAN improve FID of FastGAN at the cost of diversity? Is FreGAN memorizing training images more than FastGAN? Are the interpolations in the latent space of FreGAN smooth? It is not possible to answer with the provided figures and tables.

In addition to FID or KID, I suggest adding a metric that explicitly measures the diversity of generated images, as done in some previous works. One can choose pairwise LPIPS as in [2] (tables G,H), or intra-cluster LPIPS as in [3] (table 2). Moreover, it would be good to measure the average LPIPS from generated images to the nearest example from the training set, which would help to assess the degree of memorization of the training set ([4], table 8). Lack of diversity is also partially supported by low recall across different datasets reported in the supplementary materials. On the qualitative side, it would be good to add examples of latent space interpolations as in [4] (figure 5) and the nearest real neighbors to some of generated images ([4], figure 12-13). Overall, without this analysis, given the size of the used datasets, only improving FID and KID is not a convincing result to me. 


[2 - Claim (major concern)] The paper claims that the proposed method alleviates the unhealthy competition between G and D (lines 11, 47, 54) in the low data regime. The caption of figure 5 says that the FreGAN generator can better deceive the discriminator. The justification for the claim is claimed in the loss curves plot in figure 5(a). 

In fact, what is seen in figure 5(a) seems to say the opposite. The discriminator loss of FreGAN is lower than the one of FastGAN (even though it has an additional term L^HF). This indicates that the discriminator can dissect real and fake images more confidently. It would be interesting to see the curves of the discriminator outputs during training (e.g., as in [5], figure 1(b-c)), where this effect should be more visible. Consequently, based on the plot 5(a), the FreGAN generator struggles to fool the discriminator more than FastGAN. This is confirmed by the larger G loss for FreGAN. It is thus not clear how the better GAN equilibrium is quantified by the authors, so one of the main claims is not supported.



[3.1 – Clarity (major concern)] Generally, it is not clear why the technical proposals of the paper are introduced as techniques for training GANs in low data regimes. The frequency bias for GANs exists not only when the training data size is limited [6]. In principle, any GAN model should benefit from generating images with better spectral properties (e.g., the motivation in lines 33-39 also fits to training GANs on large datasets). The explanation on mitigating unhealthy G and D competition is, in my opinion, unsupported (see Weaknesses-2). 

Why are the introduced modifications tested only in low-data regimes? Do the proposed techniques help to improve the quality of images or spectral properties of GANs trained on large datasets?


[3.2 – Related work (major concern)] The proposed method is by design aimed to improve the quality of images in the frequency domain. There are other works with similar motivation (e.g., [6] as an overview). These methods can be potentially placed instead of the techniques introduced in section 3.2-3.4. It is not clear whether the effect of the proposed techniques could not be achieved with already existing techniques mitigating spectral biases of GANs. A proper comparison to previous works is expected.

[3.3 – Evaluation (major concern)] The proposed method is aimed to improve the quality of images in the frequency domain, but this aspect is not evaluated quantitatively. The only provided comparisons of high/low frequencies are visual, which makes it subjective, especially since ground truth spectrums to compare to are not available. Quantitative assessment of GAN images spectrums was approached before, for example by measuring the accuracy of a binary classifier trained on spectrum features of real and fake images [7]. A proper quantitative comparison of spectral properties of images for different models is expected.

[4 – Clarity (moderate concern)] The motivation of the frequency alignment loss (lines 151-161) is not clear. Firstly, what is meant by “G can only synthesize arbitrary frequency signals”? The synthesis of G is guided with the discriminator loss, which provides supervision on the image realism, which includes the intensity of details of different frequencies. So they should not be arbitrary. Secondly, why is HDA called a “regularizer for D”, while it is included only to the objective of G? 

Finally, eq. (4) is a form of a reconstruction loss, but it is computed between D’s features of a real image and G’s features of a randomly sampled fake image. These images are not necessarily aligned spatially, so it is not clear why imposing a pixel-wise L1 reconstruction loss is meaningful. Please explain. 


--- References ---

[1] Few-Shot Adaptation of Generative Adversarial Networks. Robb et al. Arxiv, 2020.

[2] Generating Novel Scene Compositions from Single Images and Videos. Sushko et al. Arxiv, 2021.

[3] Few-shot Image Generation via Cross-domain Correspondence. Ojha et al. CVPR 2021.

[4] Towards Faster and Stabilized GAN Training for High-fidelity Few-shot Image Synthesis. Liu et al. ICLR 2021.

[5] Training Generative Adversarial Networks with Limited Data. Karras et al. NeurIPS 2020.

[6] On the Frequency Bias of Generative Models. Schwarz et al. NeurIPS 2021.

[7] Watch your Up-Convolution: CNN Based Generative Deep Neural Networks are
Failing to Reproduce Spectral Distributions. Durall et al. CVPR 2020.

questions: [1] My major concern is that the design of FreGAN favors the memorization of the training images more than the FastGAN baseline. Given the limited dataset size, the additional discriminator HFD can memorize all the possible HF for real images, and can thus force the generated images to have exactly same HF, which forces to fully reproduce the original training images, and not to generate any new images (as they would have different HF). On the other hand, HFA is a rather restrictive L1 reconstruction loss that also makes the generated images look closer to the exact training examples across all G layers. Missing evaluation of diversity, as well as some visual examples (e.g., 2 memorized examples for “shells” in figure 6 in supplementary) second my concerns. Please comment.

[2] Please also comment on the other concerns 2,3,4.

limitations: The authors have adequately addressed the limitations and potential negative societal impact of their work.
review_title: null

Meta-review: metareview: The majority of reviewers voted for accept, and we had a 7 among the mix, which I think cancels out the 4. This paper proposes using a wavelet method to bias the GAN towards generating the proper frequency distribution, which helps especially in the low-data regime. Overall I think low-data data modeling is important, and although this approach doesn't seem extremely novel, it seems useful, and on balance the reviewers voted to accept. I agree.
confidence: Less certain
Decision: Accept