Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: Summary:
This paper proposes a lightweight GAN architecture which is tuned for learning generative models in the case where one has access to only a relatively small datasets, as well as a simple autoencoding modification for GAN discriminators to help prevent overfitting and mode collapse. Results are presented on a range of benchmark and new datasets, using standard metrics to compare performance against existing models. The new models compare favorably in the target regime, while unsurprisingly not being as strong as the baseline models in the large-data regime.

My take:

This is a decent paper, with reasonable (albeit not perfect) presentation clarity and passably significant results. The architectural modifications are not trivial (i.e. one could just try and make a StyleGAN less wide, but the authors give specific attention to the design of the higher resolution layers in G), and the changes to the training procedure are simple and effective, while being sufficiently different from previous autoencoder-based approaches to merit being called novel. The results show that the proposed models perform well (qualitatively and quantitatively) in the low-data regime against a full-size StyleGAN baseline, particularly when controlling for compute budget. I would rate this paper about a 6.5: I do not have any major concerns (I would evaluate the technical and methodological soundness of this paper as high) but I also would not expect this paper to have an especially high impact. As I tend to accept, I expect to reconsider my rating to a 7 after the discussion period unless major concerns are raised.

My main suggestion to the authors which I think could strengthen this paper would be to compare against a baseline StyleGAN with the width multipliers decreased. If I were a practitioner seeking to improve model performance in the low-data regime, this would be my first approach: to take an existing, working model, and make it smaller. As the authors’ architectural changes are not this simple, one would expect that, for an equivalent FLOP budget and training budget, the new architecture would outperform a “StyleGAN-slim,” but it would be good to have quantitative evidence on this front.

If the authors have the compute budget, it would also be good to see how this model “scales up,” in the case where it is made deeper or wider (simply changing the width multipliers) and tested against StyleGAN on the full FFHQ; a plot comparing FID over time for the two models (so that a practitioner could see how long one would have to train an equivalent StyleGAN on the full dataset to outperform this model, or vice versa) would be useful. Since the model has a different architecture I would expect it to have different scaling properties. However, this reviewer appreciates that this would be a compute intensive experiment that is likely not possible to run in the revision period, and does not wish to push the authors in this direction given that it is outside the target scope of low-data modeling.


“Note that we collect the last six datasets in the wild, which we do not have a license to re-distribute, yet one can freely collect them just like us.”

I appreciate that the authors have given some considered the legal implications of distributing potentially copyrighted images (and given that there’s not much established legal precedent that I’m aware of on whether doing research using copyrighted images of Pokemon constitutes fair us, this reviewer does not consider this cause for concern). For open-sourcing, the authors might want to consider releasing the URLs of the datasets to enable reproducibility (at least for as long as the images are up), which is an approach that has been used for other datasets. This would also allow the authors revise this sentence to be a bit more “academic,” e.g. “Note that we do not have a license to re-distribute the last six datasets, which we collect from the wild, but we provide the URLs in order to enable reproducibility.”

Edit: As mentioned in my comment below, I believe the authors have done sufficient work in their revision to address my concerns and am revising my score to an acceptance.

review_title: Tend to Accept

final_score: rating: 7: Good paper, accept
review_content: Summary:  
This paper introduces a new GAN architecture that targets high resolution generation for small datasets. Two techniques are introduced for this purpose: skip-layer channel-wise excitation (SLE) modules, and regularization of the discriminator via a self-supervised auxiliary task. The proposed architecture is shown to outperform current SOTA models on a variety of small datasets, while training in less time.  

Strengths:  
-Paper is well written and easy to understand.  
-SLE combines benefits of skip-connections, channel-attention, and style-modulation in a single operation.  
-Self-supervised discriminator appears to be very effective at preventing overfitting in low data regimes.  
-Strong generation results on a variety of datasets, including better image quality, and faster training time than the baseline models with similar numbers of parameters.  
-Ablation study demonstrates the usefulness of each of the proposed components.  

Weaknesses:  
-No significant weaknesses that I can think of.  

Recommendation and Justification:  
I quite like this paper and tend to vote for acceptance. It is refreshing to see a new architecture designed specifically for the low data, low compute regime, rather than simply reducing the capacity of existing architectures. I particularly like the idea of regularizing the discriminator with an auto-encoding task. Many other methods that attempt to combine auto-encoding and GANs seem to constrain the model too much due to the requirement of mapping all examples in the dataset into the latent space, but this method does not appear to share this constraint. It also has the added benefit of sharing discriminator capacity, rather than introducing an additional encoder which further increases computational cost.  

Clarifying Questions:  
-Why perform random cropping in the discriminator at 16x16 resolution? Why not perform reconstruction on the full image? Is this mainly for computational savings?  
-Is there any weighting on the reconstruction loss in Equation 3 or is the weighting effectively equal to 1 here?
-In Table 4, the full StyleGAN2 model outperforms the proposed model when more images are available. However, as is stated in the paper, the StyleGAN2 model has twice as many parameters. If the number of parameters in both models were equal (either by doubling the amount in the proposed model of halving the amount in the StyleGAN2), which would be expected to achieve better performance?  

review_title: Review

final_score: rating: 7: Good paper, accept
review_content: Paper summary

This work studies training GANs on small datasets (in a few-shot setting) for high-resolution image synthesis. To generate high-quality samples with minimum computation cost, and to alleviate overfitting and training instabilities, two techniques are proposed: 1) For the generator the Skip-Layer channel-wise Excitation (SLE) module is introduced, “skip-connecting” low-scale layers with high-scale ones, which facilitates the gradient flow and allows style-content mixing from different images. 2) The discriminator is trained with additional small decoders to reconstruct given images, which acts as self-supervision and helps to reduce overfitting. Experiments show that the proposed GAN model copes well with high-resolution image synthesis task (256x256 – 1024x1024) while being trained on small datasets (down to 60 – 100 images), providing a significant speed up for this setting compared to existing approaches.

Strengths

1) The paper proposes an approach for a very challenging and important task of training GANs with the small amount of training data. To my knowledge, this paper is the first to generate high-resolution realistic images from datasets of such a small scale. This is valuable, as it potentially extends the domain of possible GAN applications. It is also good to see that the paper compares to the recent advances in low-data GANs (Karras et al., 2020a).

2) The paper achieves good results. The performance gain in comparison to StyleGAN2 and the considered baseline is visible across multiple small-scale datasets, see Tables 2 and 3. The improvement in visual quality is also clearly seen from Figure 5, though the evaluation setting might be unfair to StyleGAN2 (see my comments below). 

3) SLE module seems interesting, as it is a novel way for designing a skip-connection between layers of different spatial resolutions in the generator. Besides facilitating the gradient flow, which helps the generator to learn quicker, it serves as a tool for style-content mixing by modulating high-scale features on low-scale encoding of another image.


Weaknesses

1) Limited technical novelty and missing comparisons to the closely related work. Each of the two proposed technical solutions have been proposed in similar forms in previous works, and the paper does not directly compare with them. 

- SLE:
The proposed SLE module is a combination of skip connection + plus channel attention mechanism. However, no clear comparison with the related work is provided. As skip connections, one could also use simply residual connections, or use MSG-GAN (Karnevar & Wang, 2019) or StyleGAN2 approaches to improve the gradient flow (see Fig. 7 in the StyleGAN2 paper). They are probably heavier in terms of training speed, but I think there has to be an ablation where the proposed SLE is fairly compared to other ways of using skip connections to improve the gradient flow (memory, speed, performance). 
From channel attention mechanism, the proposed technique resembles the Squeeze-and-Excitation module (SE) proposed by Hu et al. However, there were other follow up works that show superior results, such as Efficient Channel Attention (ECA) module proposed by Wang et al. CVPR’2020 or Convolutional Block Attention Module by Woo et al. ECCV’18. The paper doesn’t compare the proposed SLE to the above methods, thus it’s hard to judge how effective it is in comparison. 

- SS-discriminator:
The employed auto-encoding approach is a typical method for self-supervised learning. However, there has been a line of works which uses different self-supervision techniques (e.g. Auxiliary Rotation Loss in Chen et al. CVPR 2019) or regularizations on the discriminator side (e.g. Zhao et al. 2020) for the same purpose as the proposed self-supervision, and it would be beneficial to see the comparison of the proposed s-s to existing approaches. Table 6 provides only the comparison with the two SS techniques, which might be suboptimal for the task at hand.

Generally, the proposed model compares well to the considered baseline (DCGAN + extras), but the need for the proposed solutions is not totally justified. Other similar existing solutions, mentioned above, implemented on top of the baseline, potentially could lead to the same performance improvement. 

2) Incomplete evaluation and lack of the experimental support of some claims.

-	The comparison is done using only one metric - FID. This metric is known for not being able to detect overfitting, and, as was recently shown, is not a proper metric in low-data regimes (see Fig.4 in [*]). This raises the concern that on such small datasets the metric simply shows the degree of overfitting to the training set. With a limited training time used for the reported experiments this is naturally simpler to achieve for the proposed (lighter) model than for StyleGAN2, which might explain such a performance gap between two models. Overall, I disagree with the claim “We find it unnecessary to involve other metrics”, as the used FID metric could be misleading. It would be beneficial to employ also other metrics to measure the diversity of the generated samples. Thus, I find the evaluation presented in Tables 2,3 incomplete.

[*] Robb et al. FEW-SHOT ADAPTATION OF GENERATIVE ADVERSARIAL NETWORKS, ArXiv’2020. 

- Overfitting is certainly one of the main challenges in a few-shot synthesis setting. However, the paper pays relatively small attention on analyzing the issue. Also in its current state it’s not clear if the proposed solutions actually help to avoid overfitting. On small datasets, the generated images probably resemble the training examples. This is seen for “Skull” in Figure 5, where for each generated image one could find the similar training example, also noticeable for “Shell” in Figure 6, where the interpolations tend to resemble the image on the left or on the right.
I agree that Table 5 is valuable, and that it shows relative overfitting strength compared to the baseline. However, I would also expect the analysis on the absolute values, as well as the comparison to StyleGAN2. For example, reporting LPIPS to the nearest training example would be helpful, together with showing generated samples together with the closest training examples. 

-	Looking at the results in Table 4, as the number of training images become larger, the StyleGAN2 outperforms the proposed model. This also shows that even with a half of the parameters the StyleGAN2 model capacity is too large for less than 2k images. So I would expect for a low data regime setting StyleGAN2 with fewer parameters may have better results than the halved StyleGAN2. Given also that on larger datasets (5k-10k, Table 4) the proposed model underperforms, the problem might be the limited capacity of the proposed model, so increasing the number of parameters, e.g. number of channels, might help to improve the overall performance.  This trade-off between the model capacity and the size of the training set is not analysed in the paper, and from my point of view lead to unfair comparison in Tables 2,3,4.

-	For Figure 5, the images shown are from a different epoch than in Tables 3 and 4. Moreover, it might be unfair to clip StyleGAN2 at 10 hours of training, as its best epochs are coming later. The figure illustrates the speed-up from the proposed model, but the reader cannot match it to FID values in tables, also it is not possible to see the performance of StyleGAN2 in its best checkpoints on the studied settings.

-	It is unclear, what is meant by the “robustness” of the model in the paper. The model is claimed to be robust, but this claim is not really explained and supported experimentally. 


Minor

1) Why also not to employ the SLE module in the discriminator?
2) How does the style mixing via SLE compares to other approaches, e.g. StyleGAN2?

Overall, I give the paper a borderline rating. I note that the paper studies an important problem, achieves good results, and advances GANs extending their application areas. On the other hand, I find some incompleteness in the experimental evaluation and unsupported claims in the paper, and have concerns about the limited novelty of the proposed technical solutions. 

Post-rebuttal:
I believe the authors have done sufficient work in their revision to address my concerns. Thus I'm leaning towards acceptance and raising my score to a 7.
review_title: Borderline

final_score: rating: 7: Good paper, accept
review_content: In this paper, the authors introduce a new framework for unconditional image generation. The introduce a skip-layer excitation module (SLE) that allows gradient flow between activations of different spatial size. They also included a discriminator that is forced to autoencode the image. The authors claim that their framework is able to produce images of higher quality compared to SOTA with less resources.

1) Figures and tables are not great
Figure 3: The figure on right flows from top to bottom to top, making it a little difficult to follow. Arrows sometimes correspond to an operation (upsampling) or the workflow (the red arrows). Different fonts are used. 

Figure 4: Similarly, hard to follow. Arrows aren’t straight. What’s the resolution of the crop? 

Figure 5: StyleGAN2 and proposed method’s image sizes should be the same for easy qualitative comparison. Perhaps plot real images in a single row at the top. Current organization is difficult to follow. 

Table 2:organization can be better. What is stylegan2 ft?  

2) The choice of sigmoid sigmoid in SLE is not justified, especially since AdaIN[1], a very similar method, does not use sigmoid. 

3) No intuition is given for cropping the feature maps for decoder. Why does it help? 

4) Experiments in table 2 and table 3 are computed with different gpus which makes their results/training time incomparable.

5) Similarly, comparisons with StyleGAN2 have different training time and batch sizes. I understand the point the authors are trying to point out is with less resources, their model performs better. However, it is unclear if their model would mode collapse with longer training or would it perform worse on bigger batches. I would like to see a fair comparison and also a line plot of FID w.r.t training time.

6) Under 4.2, the authors claim that a well generalized G is the key for good inversions. This claim is not true since Image2StyleGAN[2] showed that even a network with randomly initialized weights can achieve good inversion. 

7) Under 4.2 the authors claim that “The smooth interpolations from the back-tracked latent vectors also suggest little overfitting and mode-collapse of our G.” I am aware that in StyleGAN[3], they claim smoothness of interpolation in latent space is correlated with how disentangled the latent space. But I am not aware of work that show interpolation is related to overfitting and mode collapse. Citation is needed.

8) How good are the decoders? A figure showing the reconstruction will be good. 

9) For comparisons with other self-supervision method, the experimental setup is missing.

10) In which layers are style mixing being performed on? In my opinion, the style mixing results show more of a color change rather than anything semantically meaningful. StyleGAN showed that styles from different layers correspond to different semantic concepts. This seems to be missing in this model.


General comments: Writing can be improved. Some sentences can be rewritten to flow better
1)	“The biggest challenge lies on the overfitting issue on D, thus leading to mode-collapse on G, given the sparse training samples and a low computing budget.”  
2)	“elevated model depth”
3)	“In contrast, our model maintains a good synthesis quality, even double the training time, thanks to the decoding regularization of D”
4)	“at most an eight hours’ training is needed”

Overall, I think the idea of SLE is interesting, but clear comparisons and ablations should be done to validate its usefulness. 

[1] Huang, Xun, and Serge Belongie. "Arbitrary style transfer in real-time with adaptive instance normalization." Proceedings of the IEEE International Conference on Computer Vision. 2017.

[2] Abdal, Rameen, Yipeng Qin, and Peter Wonka. "Image2StyleGAN: How to Embed Images Into the StyleGAN Latent Space?." (2019).

[3] Karras, Tero, Samuli Laine, and Timo Aila. "A style-based generator architecture for generative adversarial networks." Proceedings of the IEEE conference on computer vision and pattern recognition. 2019.


review_title: Interesting work but lacks polish and proper comparisons 

Meta-review: The paper proposes a method for training GANs in few-shot setting. Two key components of the method are: a skip-layer channel-wise excitation (SLE) module that encourages gradient flow across resolutions, and a self-supervised loss of autoencoding to regularize the discriminator. The results presented in the paper are indeed impressive in the few-shot setting. Reviewers had some concerns about training set memorization which have been addressed by the authors with additional evaluations using LPIPS metric. Overall, the paper tackles an important problem of few-shot learning of GANs and will be a good addition to the ICLR program. 
Decision: Accept (Poster)