Towards Faster and Stabilized GAN Training for High-fidelity Few-shot Image Synthesis
Authors: Bingchen Liu
Yizhe Zhu
Kunpeng Song
Ahmed Elgammal
Venue: ICLR 2021 Poster (2021)
Keywords: deep learning
generative model
image synthesis
few-shot learning
generative adversarial network
self-supervised learning
unsupervised learning

Abstract: Training Generative Adversarial Networks (GAN) on high-fidelity images usually requires large-scale GPU-clusters and a vast number of training images. In this paper, we study the few-shot image synthesis task for GAN with minimum computing cost. We propose a light-weight GAN structure that gains superior quality on 1024^2 resolution. Notably, the model converges from scratch with just a few hours of training on a single RTX-2080 GPU, and has a consistent performance, even with less than 100 training samples. Two technique designs constitute our work, a skip-layer channel-wise excitation module and a self-supervised discriminator trained as a feature-encoder. With thirteen datasets covering a wide variety of image domains (The datasets and code are available at https://github.com/odegeasslbc/FastGAN-pytorch), we show our model's superior performance compared to the state-of-the-art StyleGAN2, when data and computing budget are limited.

Published as a conference paper at ICLR 2021
TOWARDS FASTER AND STABILIZED GAN TRAINING FOR HIGH-FIDELITY FEW-SHOT IMAGE S SYNTHESIS
Bingchen Liu Yizhe Zhu?, Kunpeng Song" :2, Ahmed Elgammal"
'Playform Artrendex Inc., USA "Department of Computer Science, Rutgers University (bingchen. 1iu, yi rizhe. zhu, kunpeng song) edu e elgammal & Cartreereeed com
ABSTRACT
Training Generative Adversarial Networks (GAN) on high-fidelity images usually requires large-scale GPU-clusters and vast number of training images. In this paper, we study the few-shot image synthesis task for GAN with minimum com- puting cost. We propose a light-weight GAN structure that gains superior quality on 1024 x 1024 resolution. Notably, the model converges from scratch with just few hours of training on a single RTX-2080 GPU, and has consistent perfor- mance, even with less than 100 training samples. Two technique designs consti- tute our work, a skip-layer channel-wise excitation module and self-supervised discriminator trained as a feature-encoder. With thirteen datasets covering a wide variety of image domains [ to the state-of-the-ar StyleGAN2, when data and computing budget are limited.
we show our model's superior performance compared
INTRODUCTION
The fascinating ability to synthesize images using the state-of-the-ar (SOTA) Generative Adversar- ial Networks (GANs) (Goodfellow et al. 2014) display a great potential of GANs for many intrigu- ing real-life applications, such as image translation, photo editing, and artistic creation. However, expensive computing cost and the vast amount of required training data limit these SOTAs in real applications with only small image sets and low computing budgets.
In real-life scenarios, the available samples to train a GAN can be minimal, such as the medical images of a rare disease, a particular celebrity's portrait set, and specific artist's artworks. Transfer- learning with pre-trained model (Mo et al., 20200 Wang et al., (020) is one solution for the lack of training images. Nevertheless, there is no guarantee to find a compatible pre- pre-training dataset. Furthermore, if not, fine-tuning probably leads even worse performance (Zhao al. 2020).
Figure 1: Synthetic results on 1 resolution of our model, trained from scratch on single RTX 2080-Ti GPU, with only 1000 images. Left: 20 hours on Nature photos; Right: 10 hours on FFHQ.
In recent study, it was highlighted that in art creation applications, most artists prefers to train their models from scratch based on their own images to avoid biases from fine-tuned pre-trained model. Moreover, shown that most cases artists want train their models with datasets of less than 'The datasets and code available http://gttt...cooooggaaasssfaaas

---

Published as a conference paper at ICLR 2021
100 images [E] Elgammal al. 2020). Dynamic datt-augmenttttion (Karras al. 2020a; Zhao al.
2020 smooths the gap and stabilizes GAN training with fewer images. However, the computing cost from the SOTA models such as StyleGAN2 (Karras et al. 2020b) and BigGAN (Brock et al.
2019 remain to be high, especially when trained with the image resolution on 1024 x 1024.
In this paper, our goal is to learn unconditional GAN high-resolution images, with low compu- tational cost and few training samples. As summarized in Fig. [ these training conditions expose the model high risk of overfitting and mode-collapse (Arjovsky & Bottou 2017 Zhang & Khoreva 2018). To train a GAN given the demanding training conditions, we need a generator (G) that can learn fast, and discriminator (D) that can continuously provide useful signals to train G. To address these challenges, we summarize our contribution as:
We design the Skip-Layer channel-wise Excitation (SLE) module, which leverages low- scale activations to revise the channel responses on high-scale feature-maps. SLE allows more robust gradient flow throughout the model weights for faster training. It also leads to an automated learning of style/content disentanglement like StyleGAN2.
We propose self-supervised discriminator D trained as feature-encoder with extra decoder. We force D to learn a more descriptive feature-map map covering more regions from input image, thus yielding more comprehensive signals to train G. We test multiple self- supervision strategies for D, among which we show that auto-encoding works the best.
We build a computationall-fficiee GAN model based on the two proposed techniques, and show the model's robustmess on multiple high-fidelity datasets, as demonstrated Fig.
2 RELATED WORKS
Speed up the GAN training: Speeding up the
training of GAN has been approached from var-
Training settings
ious perspectives. Ngxande et al.) propose to re- duce the computing time with depth-wise con-
Requirements
Low VRAM
Risks
volutions. Zhhon et al.] adjust the GAN objec- tive into a min max- min-max max-min min problem for shorter optimization path. [Sinha et al.) suggest to pre- pare each batch of training samples via a core- set selection, leverage the better data prepara- tion for a faster convergence. However, these methods only bring limited improvement in training speed. Moreover, the synthesis quality is not advanced within the shortened training time.
image resolution
Small batch-size
Overfitting and mode collapse
Short training time
Small model size
Few training samples
The causes and challenges for training GAN our studied conditions.
Figure 2:
Train GAN high resolution: High-resolution training for GAN can be problematic. Firstly, the increased model parameters lead to a more rigid gradient flow to optimize G. Secondly, the target distribution formed by the images
1024 1024 resolution is super sparse, making GAN much harder to converge. Denton et al. (2015): Zhang et al. (2017);; Huang et al. (2017): Wang et al.) (2018); Karras et al. (2019); Karnewar & Wang 2020)) Karras al. (2020b); Liu et (202 develop the multi-scale GAN structures to alleviate the gradient flow issue, where G outputs images and receives feedback from several resolutions simultaneously. However, all these approaches further increase the computational cost, consuming even more GPU memory and training time.
Stabilize the GAN training:
Mode-collapse on G one of the big challenges when training GANs. And it becomes even more challenging given fewer training samples and lower computational budget (a smaller batch-size). As / more likely to overfitting on the datasets, thus unable to provide meaningful gradients to train G (Gulrajani et al. 2017).
Prior works tackle the overfitting issue by seeking a good regularization for D, including different objectives (Arjovsky et al. 2017 Lim & Ye 2017: Tran et al. 2017); regularizing the gradients (Gulrajani et al. 2017]] Mescheder et al.) 2018); normalizing the model weights (Miyato et al. 2018); and augmenting the training data (Karras et al. 2020al Zhao et al. 2020). However, the effects of these methods degrade fast when the training batch-size is limited, since appropriate batch statistics hardly be calculated for the regularization (normalization) over the training iterations.
2

---

Published as a conference paper at ICLR 2021
Meanwhile, self-superviison D has been shown to be effective method to stabilize the GAN training as studied in Tran et al. 2019); Chen et al. (2019). However, the auxiliary self-suuern tasks in prior works have limited using scenario and image domain. Moreover, prior works only studied on low resolution images (32² to 128²), and without a computing resource limitation.
3 METHOD
We adopt a minimalistic design for our model. In particular, we use a single conv-layer on each resolution in G, and apply only three (input and output) channels for the conv-layers on the high resolutions (2 512 x 512) in both and D. Fig. and Fig. Fillustaet the model structure for our G and D, with descriptions of the component layers and forward flow. These structure designs make our GAN much smaller than SOTA models and substantially faster to train. Meanwhile, our model remains robust on small datasets due its compact size with the two proposed techniques.
47
Newrest Upnampler DaschMere GLU
BIAC Beteee SLLL
Impuri uenter
Sklp-layer exciiation
Shipe Sayan evnlieden
Loutyytiic. Ul (B:3)
Image
Curr Jolk dils
restuttin
322
layse restutton
Taath
6
128
128
256
256
512
$12
1024
The forward flow of our Generator
Figure 3: The structure of the skip-layer excitation module and the Generator. Yellow boxes repre- sent feature-maps (we show the spatial size and omit the channel number), blue box and blue arrows represent the same up-sampling structure, red box contains the SLE module as illustrated on the left.
3.1 SKIP-LAYER CHANNEL-WEL WISE EXCITATION
For synthesizing higher resolution images, the generator G inevitably needs to become deeper, with more conv-layers, in concert with the upssamping needs. A deeper model with more convolution layers leads to longer training time of GAN, due to the increased number of model parameters and weaker gradient flow through G (Zhang et al., 2017 Karras et al., 2018 Karnewar & Wang, 2020). To better train deep model, He et al. design the Residual structure (ResBlock), which uses a skip-layer connection to strengthen the gradient signals between layers. However, while ResBlock has been widely used in GAN literature (Wang et al. 2018 Karras et al. 2020b). it also increases the computation cost.
We reformulate the skip-connection idea with two unique designs into the Skip-Layer Excitation module (SLE). First, ResBlock implements skip-connection as an element-wise addition between the activations from different conv-layers. It requires the spatial dimensions of the activations to be the same. Instead of addition, we apply channel-wise multiplications between the activations, eliminating the heavy computation of convolution (since side of the activations dimension of 12). Second, in prior GAN works, skip-connections are only used within the same resolution. In contrast, we perform skip-connection between resolutions with much longer range (e.g., 82 and 128², 162 and 256²), since an equal spatial-dimension is no longer required. The two designs make SLE inherits the advantages of ResBlock with a shorteut gradient flow, meanwhile without an extra computation burden.
has spatial
Formally, we define the Skip-Layer Excitation module as:
y F(xtour- (W.)) Xhigh
(1)
Here x and y are the input and output feature-man of the SLE module, the function F contains the
operations Ntows and W, indicates the module weights to be leamed. The left panel in Fig, shows an SLE module in practice, where Xtow and Xhigh are the feature-mapp at 8 x and 128 x 128
resolution respectively. An adaptive average-pooling layer in F first down-samples Xtow into 4 x
3

---

Published as a conference paper at ICLR 2021
along the pimiial-imenssoons then conv-layer further dawn-samples it into 1 x 1. A LeakyReLU is used to model the non- -ineariar, and another conv-layer projects Xlow to have the same channel size Xhigh- Finally, after gating operation via Sigmoid function, the output from F multiplies Xhigh along the channel dimension, yielding y with the same shape as Xhigh-
SLE partially resembles the Squeeze-ande aneeee-ndd--eeeeee and-Excitation module (SE) proposed by Hu et al., However, SE operates within one feature-map as a self- self-gating module. In comparison. SLE performs between feature-maps that are far away from each other. While SLE brings the benefit of channel-wise feature re-calibration just like SE, it also strengthens the whole model's gradient flow like ResBlock. The channel-wise multiplication in SLE also coincides with Instance Normalization (Ulyanov et al.j 2016 Huang & Belongne (017), which is widely used in style-transfer. Similarly. we show that SLE enables to automatically disentangle the content and style attributes, just like StyleGAN Karras et al. 2019). As SI.E. performs on high-resolution feature-maps, altering these feature- maps is shown to be more likely to change the style attributes of the generated image (Karras et al., 2019 Liu et al. 202I). By replacing Xtow in SLE from another synthesized sample, our G can generate an image with the content unchanged, but in the same style of the new replacing image.
3.2 SELF-SUPERVISED DISCRIMINATOR
Our approach to provide a strong regularization for D is surprisingly simple. We treat D as an encoder and train it with small decoders. Such auto-encoding training forces D to extract image features that the decoders D on a simple reconstruction loss, which is only trained on real samples:
give good reconstructions. The decoders are optimized together with
Lrecons Ee Deneeedd (1), a~ireat [f) -Tx))]]]
(2)
where f is the intermediate feature-maps from D, the function & contains the processing on f and the decoder, and the function T represents the processing on sample xr from real images Treat-
Down
LeakyfaLU Cone p3 eakkymocc
Simple Decosler
Real image 1o24
Nearest Upsarpple
Boh p.1
X 4
Feom RR
Frmr Riar
Cons 466, sil, più
Simple Decodler
Decoder
Cors axd, p.3
Cone 3xb, p.
Conu 363, 6:1, aid Cony 4n4, Ell, P13
128
64
32
16
Figure 4: The structure and the forward flow of the Discriminator. Blue box and arrows represent the same residual down-sampling structure, green boxes mean the same decoder structure.
Our self-supervised D is illustrated in Fig. where we employ two decoders for the feature-maps on two scales: on 162 and f2 on 82 The decoders only have four conv-layers produce images 128 x 128 resolution, causing little extra computations (much less than other regularization methods). We randomly crop fj with _ of its height and width, then crop the real image on the same portion to get part. We resize the real image to get /. The decoders produce 1 part from the cropped f,, and I' from fz. Finally, D and the decoders are trained together minimize the loss in eq. 2 by matching I part to part and I' to I.
Such reconstructive training makes sure that D extracts a more comprehensive representation from the inputs, covering both the overall compositions (from f2) and detailed textures (from fj). Note that the processing in G and are not limited to cropping; more operations remain to explored for better performance. The auto-encoding approach we employ is a typical method for self-supervised learning, which has been well recognized to improve the model robustness and generalization ability (He et al. 2020 Henddyycks et al. 2019: Jing & Tian 2020 Goyal et al., 2019). In the context of GAN, we find that regularized D via self-supervision training strategies significantly improves the synthesis quality G, among which auto-encoding brings the most performance boost.
Although our self-supervision strategy for D comes in the form of an auto-encoder (AE), this ap- proach is fundamentally different from works trying to combine GAN and AE (Larsen al. 2016
4

---

Published as a conference paper at ICLR 2021
Guo al. 2019 Zhao et al., 2016 Berthelot al. 2017). The latter works mostly train G decoder on a learned latent space from D, or treat the adversarial training with D as an supplemen- tary loss besides AE's training. In contrast,
model is a pure GAN with much simpler training schema. The auto-encoding training is only for regularizing D, where G is not involved.
In sum, employ the hinge version of the adversarial loss (Lim & Ye 20177; Tran et al. (017 iteratively train our and G. We find the different GAN losses make little performance difference, while hinge loss computes the fastest:
LD Eew [m -1+ D(=) - (())(())mmn(00 -1 -1- -D(3)] + Crecons LG B
(3)
(4)
4 EXPERIMENT
Datasets: We conduct experiments on multiple datasets with a wide range of content categories. On 256 256 resolution, we test Animal-Face Dog and Cat (Si & Zhu 2011), 100-Shot-Obama, Panda, and Grumpy-cat (Zhao et al., 2020). On 1024 x 1024 resolution, we test on Flickr-Face- HQ (FFHQ) (Karras al., 2019), Oxford-flowers (Nilsback & Zisserman 2006), art paintings from WikiArt (wikiart.org), photographs on natural landscape from Unsplash (unsplash.com) Pokemon (pokemon.com)) anime face, skull, and shell. These datasets are designed cover images with different characteristics; photo realistic, graphic-illussratioo and art-like images.
Metries: We use two metrics to measure the models' synthesis performance: Fréchet Inception Distance (FID) (Heusel et al. 2017) measures the overall semantic realism of the synthesized im- ages. For datasets with less than 1000 images (most only have 100 images), we let G' generate 5000 images and compute FID between the synthesized images and the whole training set. 2) Learned per- ceptual similarity (LPIPS) (Zhang et al., 2018) provides a perceptual distance between two images. We use LPIPS to report the reconstruction quality when we perform latent space back-tracking on G given real images, and measure the outo-encoding performance. We find it unnecessary to involve other metries, FID is unlikely to be inconsistent with the others, given the notable performance gap between our model and the compared ones. For all the testings, we train the models times with random seeds, and report the highest scores. The relative error less than five percent on average.
We compare our model with: 1) the state-of-oe oftie--f--hehan- the-art (SOTA) unconditional model, StyleGAN2, 2) baseline model ablated from our proposed one. Note that we adopt Style- GAN2 with recent studies from (Karras et al. 2020a Zhao et al. 2020), including the model con- figuration and differentiable data-augmenntttion, for the best training on few-sample datasets. Since StyleGAN2 requires much more computing-cov (cc) to train, we derive an extra baseline model. In sum, we compare our model with StyleGAN2 on the absolute image synthesis quality regardless of cc, and use the baseline model for the reference within comparable ec range.
Compared Models:
The baseline model is the strongest performer that we integrated from various GAN techniques based on DCGAN (Radford et al. (015)): 1) spectral norraaizioo (Miyato et al.) 2018). 2) exponential-
4) GLU (Dauphin et al. 2017) instead of ReLU in G. We build our model upon the baseline with the two proposed techniques: the skip-layer excitation module and the self-supervised discriminator.
moving-average (Yzıiı et al. 2018) optimization on G, 3)
Table 1: Computational cost comparison of the models.
StyleGAN2@0.25
StyleGAN2@0.5
StyleGAN2
Buseline
Qurs
Resolution: 2562
Training time (hour / 10k äter) Training vram (GB)
1.8
3,8
0,7
Batch-size:
Model parameters (million)
27.557
45.029
16
108.8
44,359
47,363
6,5
Resolution: 1024
Training time (hour äter)
3.6
1.7
Training vram (GB)
23
10
Batch-size:
Model parameters (million)
27.591
45.15
109,229
44,377
47,418
Table. D presents the normalized ec figures of the models on Nvidia's RTX 2080-Ti GPU, imple- mented using PyTorch (Paszke et al. 2017). Importantly, the slimed StyleGAN2 with parameters cannot converge on the tested datasets at 1024² resolution. We compare to the StyleGAN2 with y parameters (if not specifically mentioned) in the following experiments.
5

---

Published as a conference paper at ICLR 2021
4.1 IMAGE SYNTHESIS PERFORMANCE
Collecting large-scale image datasets are expensive, or even impossible, for certain character, genre, or topic. On those few-shot datasets, data-efficient model becomes especially valuable for the image generation task. In Table. and Table. 3 we show that our model not only achieves superior performance on the few-shot datasets, but also much more computational- efficient than the compared methods. We save the checkpoints every 10k iterations during training and report the best FID from the checkpoints (happens least after 15 hours of training for Style- GAN2 all datasets). Among the 12 datasets, our model performs the best on 10 of them.
Few-shot generation:
Please note that, due to the VRAM requirement for StyleGAN2 when trained on 1024² resolution, we have to train the models in Table. on a RTX TITAN GPU. In practice, 2080-TI and TITAN share a similar performance, and our model runs the same time on both GPUs.
Table 2: FID comparison at 256 resolution on few-sample datasets.
Animal Face Dog
Animal Face Cat
Obuma
Punda
Grampy-cal
Image number
389
160
100
100
100
StyleGAN2
20
StyleGAN2 finetume
61.03
46.07
35.73
14.5
29.34
Training time RTX 2080-Ti
Bascline
108. 19
150,3
62.74
15.4
42.13
Baseline+Skip
52.50
Aaselinesdecoole
56.25
36.74
44.34
10.12
29.38
Ours (B-SKip4dccode)
50.66
35.11
41.05
10.03
26.65
Training from scratch vs. fine-tuning: Fine-tuning from a pre-trained GAN (Mo et al. 2020
Noguchi & Harada 2019 Wang et al. 2020) has been the go-to method for the image generation task on datasets with few samples. However, its performance highly depends on the semantic consistency between the new dataset and the available pre-trained model. According Zhao al.) fine-tuning performs worse than training from seratch in most cases, when the content from the new dataset strays away from the original one. We confirm the limitation of current fine-tuning methods from Table. 2 and Table. 3 where we fine-tune StyleGAN2 trained on FFHQ use the Freeze-D method from Mo et al. Among all the tested datasets, only Obama and Skull favor the fine-tuning method, making sense since the two sets share the most similar contents to FFHQ
Module ablation study: We experiment with the two proposed modules in Table. 2 where both SLE (skip) and decoding-om, on (decode) can separately boost the model performance. It shows that the two modules are orthogonal to each other in improving the model performance, and the self-supervised D makes the biggest contribution. Importantly, the baseline model and StyleGAN2 diverge fast after the listed training time. contrast, our model is less likely to mode collapse among the tested datasets. Unlike the baseline model which usually model-collapse after trained for 10 hours, our model maintains a good synthesis quality and won't ' collapse even after trained for 20 hours. We argue that it is the decoding regularization D that prevents the model from divergence.
Table 3: FID comparison at 10242 resolution on few-sample datasets.
Art Paintings
FTH
Flower
Pokemon
Anime Face
Skull
Shell
Image number
1000
1000
1000
800
120
100
Training time an one RTX TITAN
24 hour
StyleGAN2 StyleGAN2 finetune
74.56
25.66
45.23
190.23
152.73
61.23
107.68
127.98
241.37
8 hour
Baseline
62.27
45.08
24.45
38.35
42.25
25.66
67.86
57.19
101.23
59.38
130.05
186.45
155.47
202.32
Table 4: FID comparison at 1024 resolution on datasets with more images.
Dataset
Art Paintings
FFHQ
Nature Photograph
Model
Image number
2k
Sk
10k
2k
5k
10k
70k
2k
5k
10k
StyleGAN2
70.02
48.36
41.23
18.38
10.45
7.86
4.4
67.12
41.47
39.05
Buseline
60.02
51.23
49.38
36.45
27.86
25.12
17.62
71.47
66.05
62.28
Ours
44.57
43.27
42.53
19.01
17.93
16.45
12.38
52.47
45.07
43.65

---

Published as a conference paper at ICLR 2021
Panda
256
Table 5: LPIPS of back-tracking with G
Obama
Cat
Dog
FFHQ
Resolution
256
1024
Baseline 20k
2,113
2,073
2.589
2916
Baseline @ 40k iter
2.513
2.171
2.583
2.812
FFHQ
Ours or 40kk iter
1821
1.918
2.425
2.624
1024
Ours 80k iter
1.897
1.986
2.342
2.601
Shell
Table FID of self-supervisions for D
Art 1o04
Art paintings
Nature photos
contrastive
47.14
57.04
b. predict aspect ratio
49.21
59, 22
Resl
Interpolation beween back tracked images
Reall
nato-encoding
42.53
43.65
intege
a+b aehic
46.02
54.23
44.21
47.65
Figure 6: Latent space back-tracking and interpolation.
Training with more images: For more thorough evaluation, also test our model on datasets with more sufficient training samples, as shown in Table. 4 We train the full StyleGAN2 for around five days on the Art and Photograph dataset with batch-size of 16 on two TITAN RTX GPUs, and use the latest official figures on FFHQ from Zhao et al.] Instead, we train our model for only 24 hours, with a batch-size of on a single 2080-Ti GPU. Specifically, for FFHQ with all 70000 images, we train our model with larger batch-size of 32, to reflect an optimal performance of our model.
In this test, we follow the common practice of computing FID by generating 50k images and use the whole training set as the reference distribution. Note that StyleGAN2 has more than double the parameters compared to our model, and trained with a much larger batch-size on FFHQ. These factors contribute to its better performances when given enough training samples and computing power. Meanwhile, our model keeps up well with StyleGAN2 across all testings with a considerably lower computing budget, showing a compelling performance even larger-scale datasets, and consistent performance boost over the baseline model.
The advantage of our model becomes isons in Fig. [5] Given the same batch-size and training time, StyleGAN2 either converges slower or suffers from mode collapse. contrast, our model consistently generates satisfactory images. Note that the best results from our model on Flower, Shell, and Pokemon only take three hours' training, and for the rest three datasets, the best performance is achieved at training for eight hours. For StyleGAN2 on "shell", "anime face", and "Pokemon", the images shown in Fig. 5 are already from the best epoch, which they match the scores Table. [[and Table. 3 For the rest of the datasets, the quality increase from StyleGAN2 is also limited given more training time.
Qualitative results:
clear from the qualitative compar-
4.2 MORE ANALYSIS AND APPLICATIONS
Testing mode collapse with back-tracking: From well trained GAN, one can take a real image and invert it back vector in the latent space of G, thus editing the image's content by altering the back-tracked vector. Despite the various back- weac-traciing methods (Zhu et al. 2016 Lipton & Tripathi] Zhu et al. 2020 Abdal et 2019). a well generalized G is arguably as important for the good inversions. To this end, we show that our model, although trained on limited image samples, still gets a desirable performance on real image back-tracking
2017
In Table 5 we split the images from each dataset with training/testing ratio of 9:1, and train G' on the training set. We compute reconstruction error between all the images from the testing set and their inversions from G. after the same update of 1000 iterations on the latent vectors (to prevent the vectors from being far off the normal distribution). The baseline model's 's performance is getting worse with training iterations, which reflects mode-collapse better reconstructions with consistent performance over more training iterations. Fig. 6 presents the back-tracked examples left-most and right-most samples in the middle panel) given the real images.
G. In contrast,
model gives
7

---

Published as a conference paper at ICLR 2021
Shell
Painting
Pokemon
Anime Face
Skull
Real data
Stylegan2
Ours
Figure 5: Qualitative comparison between our model and StyleGAN2 1024 resolution datasets. The left-most panel shows the training images, and the right two panels show the un- curated samples from StyleGAN2 and our model. Both models trained from scratch for 10 hours with batch-size of 8. The samples are generated from the checkpoint with the lowest FID.
The smooth interpolations from the back-tracken latent vectors also suggest little mode-collapse of our G (Radford al., 2015: Zhao et al. 2020 Robb et al. 2020).
In addition, we show qualitative comparisons in appendix D, where our model maintains a good generation while StyleGAN2 and baseline are model-collapsed.
The self-supervision methods and generalization ability on D: Apart from the auto-encoding training for D, we show that D with other common self-supervising strategies also boost GAN's performance in training settings. We test five self-supervision settings, as shown in Table 6 which all brings a substantial performance boost compared to the baseline model. Specifically, setting-a refers to contrastive learning which treat each real image unique class and let D classify them. For se we train to predict the real image's original aspect-ratio since they reshaped square when fed to D. Setting-e is the method employ in model, which
8

---

Published as a conference paper at ICLR 2021
Animelf ace Cel
AnimalFace Dog
Obama
Content
Condent
Art Painting
Shell
Pokemon
Figure Style-mixing results from our model trained for only hours on single GPU.
trains D encoder with decoder reconstruct real images. To better validate the benefit of self-supervision on D, all the testings are conducted on full training sets with 10000 images, with a batch-size of 8 to be consistent with Table 4 We also tried training with larger batch-size of 16, which the results are consistent to the batch-size of 8.
Interestingly, according Table 6 while setting-c performs the best, combining it with the rest two settings lead to a clear performance downgrade. The similar behavior can be found on some other self- self-supervision settings, e.g. when follow follow(Chen Chen al. (2019) with a "rotation- peatiion-preictinng task on art-paintings and FFHQ datasets, we observe a performance downgrade even compared to the baseline model. We hypothesis the reason being that the auto-encoding forces D to pay attention to more areas of the input image, thus extracts more comprehensive feature- map to describe the input image (for a good reconstruction). In contrast, a classification task does not guarantee to cover the whole image. Instead, the task drives D to only focus on small regions because the model can find class cues from small regions of the images. Focusing on limited regions (i.e., react to limited image patterns) is typical overfitting behavior, which is also widely happening for D in vanilla GANs. More discussion can be found in appendix B.
Style mixing like StyleGAN. With the channel-wise excitation module, our model gets the same functionality as StyleGAN: learns to disentangle the images' high-level semantic attributes (style and content) in an unsupervised way, from G's conv-layers at different scales. The style-mixing results are displayed in Fig. 7 where the top three datasets are 256 x 256 resolution, and the bot- tom three are 1024 1024 resolution. While StyleGAN2 suffers from converging on the bottom high-resolution datasets, our model successfully learns the style representations along the channel dimension on the "excited" layers (i.e., for feature-maps on 256 x 256, 512 512 resolution). Please refer to appendix A and C for more information on SLE and style-mixing.
5 CONCLUSION
We introduce two techniques that stabilize the GAN training with an improved synthesis quality, given sub-hundred high-fidelity images and a limited computing resource. On thirteen datasets with diverse content variation, show that skip-layer channel-wist excitation mechanism (SLE) and a self-supervised regularization on the discriminator significantly boost the synthesis performance of GAN. Both proposed techniques require minor changes to a vanilla GAN, enhancing GAN's practicality with a desirable blug-and-play property. We hope this work can benefit downstream tasks of GAN and provide new study perspectives for future research.

---

Published as a conference paper at ICLR 2021
REFERENCES
Rameen Abdal, Yipeng Qin, and Peter Wonka. Image2stylegan: How to embed images into the stylegan latent space? In Proceedings of the IEEE international conference on computer vision, pp. 4432 4441, 2019.
Martin Arjovsky and Léon Bottou. Towards principled methods for training generative adversarial networks. In International Conference on Learning Representations, 2017.
Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative ad adversarial networks. In International conference on machine learning, pp. 214-223. PMLR, 2017.
David Berthelot, Thomas Schumm, and Luke Metz. Began: Boundary equilibrium generative ad- versarial networks. arXiv preprint arXiv:1703.10711 2017.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale GAN training for high fidelity natural image synthesis. In International Conference on Learning Representations, 2019.
Ting Chen, Xiaohua Zhai, Marvin Ritter, Mario Lucic, and Neil Houlsby. Self-supervised gans via auxiliary rotation loss. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 12154-12163, 2019.
Yann N Dauphin, Angela Fan, Michael Auli, and David Grangier. Language modeling with gated convolutional networks. In International conference on machine learning, pp. 933-941, 2017.
Emily L Denton, Soumith Chintala, Rob Fergus, et al. Deep generative image models using laplacian pyramid of adversarial networks. In Advances neural information processing systems, pp. 1486-1494, 2015.
Ahmed Elgammal, Marian Mazzone, et al. Artists, artificial intelligence and machine-basee creativ- ity in playform. Artnodes, (26):1-8, 2020.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural infor- mation processing systems, pp. 2672-2680, 2014.
Priya Goyal, Dhruv Mahajan, Abhinav Gupta, and Ishan Misra. Scaling and benchmarking self- supervised visual representation learning. In Proceedings of the IEEE International Conference on Computer Vision, pp. 1390, 2019.
Ishaan Guirajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron C Courville. Im- proved training of wasserstein gans. Advances in neural information processing systems, pp- 5767-5777, 2017.
Yong Guo, Qi Chen, Jian Chen, Qingyao Wu, Qinfeng Shi, and Mingkui Tan. Auto-embedding generative adversarial networks for high resolution image synthesis. IEEE Transactions Mul- timedia, 21(11):27262273) 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 9729- 2020.
Dan Hendrycks, Mantas Mazeika, Sauray Kadavath, and Dawn Song. Using self-supervised learning can improve model robustness and uncertainty. In Advances in Neural Information Processing Systems, pp. 15663-15674, 2019.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium. In Advances in neural information processing systems, pp. 6626-6637, 2017.
10

---

Published as a conference paper at ICLR 2021
Jie Hu, Li Shen, and Gang Sun. Squeeze-and-exiitatee networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7132-7141, 2018.
Xun Huang and Serge Belongie. Arbitrary style transfer in real-time with adaptive instance nor- malization. In Proceedings of the IEEE International Conference on Computer Vision, pp. 1501- 1510, 2017.
Xun Huang, Yixuan Li, Omid Poursaeed, John Hopcroft, and Serge Belongie. Stacked generative adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5077-5086, 2017.
Longlong Jing and Yingli Tian. Supe-supe visual feature learning with deep neural networks: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2020.
Animesh Karnewar and Oliver Wang. Msg-gan: Multi-scale gradients for generative adversarial networks. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recog- nition, pp. 7799-7808, 2020.
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of GANs for im- proved quality, stability, and variation. In International Conference on Learning Representations, 2018.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE conference on computer vision and patlern recognition, pp. 4401-4410, 2019.
Tero Karras, Miika Aittala, Janne Hellsten, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Training generative adversarial networks with limited data. arXiv preprint arXiv:2006.0666 06676, 2020a.
Tero Karras, Samuli Laine, Mika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyz- ing and improving the image quality of stylegan. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 81 10 8119, 2020b.
Anders Boesen Lindbo Larsen, Søren Kaae Sønderby, Hugo Larochelle, and Ole Winther. Autoen- coding beyond pixels using a learned similarity metric. In International conference on machine learning, pp. 1558-1566 PMLR, 2016.
Jac Hyun Lim and Jong Chul Ye. Geometric gan. arXiv preprint arXiv:1705.028%0 2017
Zachary C. Lipton and Subarna Tripathi. Precise recovery of latent vectors from generative adver- sarial networks. ICLR workshop, 2017.
Bingchen Liu, Kunpeng Song, Yizhe Zhu, Gerard de Melo, and Ahmed Elgammal. Time: Text and image transslttonnlntionn adversarial networks. In Thirty- AAAI Conference on Artificial Intelligence, 2021.
Lars Mescheder, Andreas Geiger, and Sebastian Nowozin. Which training methods for gans do actually converge? In International conference on machine learning, pp. 3481-3490. PMLR, 2018.
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. In International Conference on Learning Representations, 2018.
Sangwoo Mo, Minsu Cho, and Jinwoo Shin. Freeze discriminator: simple baseline for fine-tuning gans. arXiv preprint arXiv:2002.10964, 2020.
Mkhuseli Ngxande,
Tapamo, and Michael Burke. Depthwisegans: Fast training generative adversarial networks for realistic image synthesis. In 2019 Southern African Universi- ties Power Engineering Conference/Robootic and Mechatronics/Paterr Recognition Association of South Africa (SAUPECRROMcch/PRAAAA)) pp. 111-116. IEEE, 2019.
Maria-Elena Nilsback and Andrew Zisserman. A visual vocabulary for flower classification. In IEEE Conference Computer Vision and Pattern Recognition, volume 2, pp. 1447-1454, 2006.

---

Published as a conference paper at ICLR 2021
Atsuhiro Noguchi and Tatsuya Harada. Image generation from small datasets via batch statistics adaptation. In Proceedings of the IEEE International Conference on Computer Vision, pp. 2750- 2758, 2019.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary De Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv: 2015.
Esther Robb, Wen-Sheng Chu, Abhishek Kumar, and Jia-Bin Huang. Few-shot adaptation of gener- ative adversarial networks. arXiv preprint arXiv:2010. 1943, 2020.
Zhangzhang Si and Song-Chun Zhu. Learning hybrid image templates (hit) by information projec- tion. IEEE Transactions on pattern analysis and machine intelligence, 34(():1354-1300 2011.
Samarth Sinha, Han Zhang, Anirudh Goyal, Yoshua Bengio, Hugo Larochelle, and Augustus Odena. Small-gan: Speeding up gan training using core-sets. arXiv preprint 1rXiv:1910.135400 2019.
Dustin Tran, Rajesh Ranganath, and David M Blei. Deep and hierarchical implicit models. arXiv preprint 1.V::0702.0886, 7(3):13, 2017.
Ngoc-Trung Tran, Viet-Hung Tran, Bao-Ngoc Nguyen, Linxiao Yang, and Ngai-Man Man Cheung- Self-super gan: Analysis and improvement with multi-class minimax game. Advances in Neural Information Processing Systems, 12:13253-13264 2019.
Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Instance normalization: The missing in- gredient for fast stylization. arXiv preprint arXiv:1607.08022 2016.
Ting- Ting-Chun Wang. Ming- Yu Liu, Jun-Yan Zhu, Andrew Tao, Jan Kautz, and Bryan Catanzaro. High- resolution image synthesis and semantic manipulation with conditional gans. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 8798 8798-8807, 2018.
Yaxing Wang, Abel Gonzalez-Garche David Berga, Luis Herranz, Fahad Shahbaz Khan, and Joost van de Weijer. Minegan: effective knowledge transfer from gans to target domains with few im- ages. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 9332-9341, 2020.
Yasin Yazıeı, Chuan- Chuan-Sheng Foo, Stefan Winkler, Kim-Hui Yap, Georgios Piliouras, and Vi- jay Chandrasekhar. The unusual effectiveness of averaging in gan training. arXiv preprint arXiv: 1806.04498 2018.
Dan Zhang and Anna Khoreva. Pa-gan: Improving gan training by progressive augmentation. 2018.
Han Zhang, Tao Xu, Hongsheng Li, Shaoting Zhang, Xiaogang Wang. Xiaolei Huang, and Dim- itris N Metaxas. Stackgan: Text to photo-realistic image synthesis with stacked generative ad- versarial networks. In Proceedings of the IEEE international conference on computer vision, pp- 5907-5915, 2017.
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 586-595, 2018.
Junbo Zhao, Michael Mathieu, and Yann LeCun. Energy-based generative adversarial network. arXiv preprini 160iv:1609: 000 /2,,,,,55 2016.
Shengyu Zhao, Zhijian Liu, Ji Lin, Jun-Yan Zhu, and Song Han. Differentiable augmentation for data-efficient gan training. arXiv preprint arXiv:2006. 107.38800.078. 2020.
Jiachen Zhong, Xuanqing Liu, and Cho-Jui Hsieh. Improving the speed and quality of gan by adversarial training. arXiv preprint arXiv:2008.00 03364, 2020.
Jiapeng Zhu, Yujun Shen, Deli Zhao, and Bolei Zhou. In-domain gan inversion for real image editing. arXiv preprint arXiv:2004. 00049, 2020.
12

---

Published as a conference paper at ICLR 2021
Jan Zhu, Philipp Krähenbühl, Eli Shechtman, and Alexei A Efros. Generative visual manipu- lation on the natural image manifold. In European conference on computer vision, pp. 597-613. Springer, 2016.
13


Reviews:
[{"review_id": "jjSVXXJQsM", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:  \nThis paper introduces a new GAN architecture that targets high resolution generation for small datasets. Two techniques are introduced for this purpose: skip-layer channel-wise excitation (SLE) modules, and regularization of the discriminator via a self-supervised auxiliary task. The proposed architecture is shown to outperform current SOTA models on a variety of small datasets, while training in less time.  \n\nStrengths:  \n-Paper is well written and easy to understand.  \n-SLE combines benefits of skip-connections, channel-attention, and style-modulation in a single operation.  \n-Self-supervised discriminator appears to be very effective at preventing overfitting in low data regimes.  \n-Strong generation results on a variety of datasets, including better image quality, and faster training time than the baseline models with similar numbers of parameters.  \n-Ablation study demonstrates the usefulness of each of the proposed components.  \n\nWeaknesses:  \n-No significant weaknesses that I can think of.  \n\nRecommendation and Justification:  \nI quite like this paper and tend to vote for acceptance. It is refreshing to see a new architecture designed specifically for the low data, low compute regime, rather than simply reducing the capacity of existing architectures. I particularly like the idea of regularizing the discriminator with an auto-encoding task. Many other methods that attempt to combine auto-encoding and GANs seem to constrain the model too much due to the requirement of mapping all examples in the dataset into the latent space, but this method does not appear to share this constraint. It also has the added benefit of sharing discriminator capacity, rather than introducing an additional encoder which further increases computational cost.  \n\nClarifying Questions:  \n-Why perform random cropping in the discriminator at 16x16 resolution? Why not perform reconstruction on the full image? Is this mainly for computational savings?  \n-Is there any weighting on the reconstruction loss in Equation 3 or is the weighting effectively equal to 1 here?\n-In Table 4, the full StyleGAN2 model outperforms the proposed model when more images are available. However, as is stated in the paper, the StyleGAN2 model has twice as many parameters. If the number of parameters in both models were equal (either by doubling the amount in the proposed model of halving the amount in the StyleGAN2), which would be expected to achieve better performance?  \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "Fp-PDcSGBdj", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThis paper proposes a lightweight GAN architecture which is tuned for learning generative models in the case where one has access to only a relatively small datasets, as well as a simple autoencoding modification for GAN discriminators to help prevent overfitting and mode collapse. Results are presented on a range of benchmark and new datasets, using standard metrics to compare performance against existing models. The new models compare favorably in the target regime, while unsurprisingly not being as strong as the baseline models in the large-data regime.\n\nMy take:\n\nThis is a decent paper, with reasonable (albeit not perfect) presentation clarity and passably significant results. The architectural modifications are not trivial (i.e. one could just try and make a StyleGAN less wide, but the authors give specific attention to the design of the higher resolution layers in G), and the changes to the training procedure are simple and effective, while being sufficiently different from previous autoencoder-based approaches to merit being called novel. The results show that the proposed models perform well (qualitatively and quantitatively) in the low-data regime against a full-size StyleGAN baseline, particularly when controlling for compute budget. I would rate this paper about a 6.5: I do not have any major concerns (I would evaluate the technical and methodological soundness of this paper as high) but I also would not expect this paper to have an especially high impact. As I tend to accept, I expect to reconsider my rating to a 7 after the discussion period unless major concerns are raised.\n\nMy main suggestion to the authors which I think could strengthen this paper would be to compare against a baseline StyleGAN with the width multipliers decreased. If I were a practitioner seeking to improve model performance in the low-data regime, this would be my first approach: to take an existing, working model, and make it smaller. As the authors’ architectural changes are not this simple, one would expect that, for an equivalent FLOP budget and training budget, the new architecture would outperform a “StyleGAN-slim,” but it would be good to have quantitative evidence on this front.\n\nIf the authors have the compute budget, it would also be good to see how this model “scales up,” in the case where it is made deeper or wider (simply changing the width multipliers) and tested against StyleGAN on the full FFHQ; a plot comparing FID over time for the two models (so that a practitioner could see how long one would have to train an equivalent StyleGAN on the full dataset to outperform this model, or vice versa) would be useful. Since the model has a different architecture I would expect it to have different scaling properties. However, this reviewer appreciates that this would be a compute intensive experiment that is likely not possible to run in the revision period, and does not wish to push the authors in this direction given that it is outside the target scope of low-data modeling.\n\n\n“Note that we collect the last six datasets in the wild, which we do not have a license to re-distribute, yet one can freely collect them just like us.”\n\nI appreciate that the authors have given some considered the legal implications of distributing potentially copyrighted images (and given that there’s not much established legal precedent that I’m aware of on whether doing research using copyrighted images of Pokemon constitutes fair us, this reviewer does not consider this cause for concern). For open-sourcing, the authors might want to consider releasing the URLs of the datasets to enable reproducibility (at least for as long as the images are up), which is an approach that has been used for other datasets. This would also allow the authors revise this sentence to be a bit more “academic,” e.g. “Note that we do not have a license to re-distribute the last six datasets, which we collect from the wild, but we provide the URLs in order to enable reproducibility.”\n\nEdit: As mentioned in my comment below, I believe the authors have done sufficient work in their revision to address my concerns and am revising my score to an acceptance.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "BuOeihtmW_G", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors introduce a new framework for unconditional image generation. The introduce a skip-layer excitation module (SLE) that allows gradient flow between activations of different spatial size. They also included a discriminator that is forced to autoencode the image. The authors claim that their framework is able to produce images of higher quality compared to SOTA with less resources.\n\n1) Figures and tables are not great\nFigure 3: The figure on right flows from top to bottom to top, making it a little difficult to follow. Arrows sometimes correspond to an operation (upsampling) or the workflow (the red arrows). Different fonts are used. \n\nFigure 4: Similarly, hard to follow. Arrows aren’t straight. What’s the resolution of the crop? \n\nFigure 5: StyleGAN2 and proposed method’s image sizes should be the same for easy qualitative comparison. Perhaps plot real images in a single row at the top. Current organization is difficult to follow. \n\nTable 2:organization can be better. What is stylegan2 ft?  \n\n2) The choice of sigmoid sigmoid in SLE is not justified, especially since AdaIN[1], a very similar method, does not use sigmoid. \n\n3) No intuition is given for cropping the feature maps for decoder. Why does it help? \n\n4) Experiments in table 2 and table 3 are computed with different gpus which makes their results/training time incomparable.\n\n5) Similarly, comparisons with StyleGAN2 have different training time and batch sizes. I understand the point the authors are trying to point out is with less resources, their model performs better. However, it is unclear if their model would mode collapse with longer training or would it perform worse on bigger batches. I would like to see a fair comparison and also a line plot of FID w.r.t training time.\n\n6) Under 4.2, the authors claim that a well generalized G is the key for good inversions. This claim is not true since Image2StyleGAN[2] showed that even a network with randomly initialized weights can achieve good inversion. \n\n7) Under 4.2 the authors claim that “The smooth interpolations from the back-tracked latent vectors also suggest little overfitting and mode-collapse of our G.” I am aware that in StyleGAN[3], they claim smoothness of interpolation in latent space is correlated with how disentangled the latent space. But I am not aware of work that show interpolation is related to overfitting and mode collapse. Citation is needed.\n\n8) How good are the decoders? A figure showing the reconstruction will be good. \n\n9) For comparisons with other self-supervision method, the experimental setup is missing.\n\n10) In which layers are style mixing being performed on? In my opinion, the style mixing results show more of a color change rather than anything semantically meaningful. StyleGAN showed that styles from different layers correspond to different semantic concepts. This seems to be missing in this model.\n\n\nGeneral comments: Writing can be improved. Some sentences can be rewritten to flow better\n1)\t“The biggest challenge lies on the overfitting issue on D, thus leading to mode-collapse on G, given the sparse training samples and a low computing budget.”  \n2)\t“elevated model depth”\n3)\t“In contrast, our model maintains a good synthesis quality, even double the training time, thanks to the decoding regularization of D”\n4)\t“at most an eight hours’ training is needed”\n\nOverall, I think the idea of SLE is interesting, but clear comparisons and ablations should be done to validate its usefulness. \n\n[1] Huang, Xun, and Serge Belongie. \"Arbitrary style transfer in real-time with adaptive instance normalization.\" Proceedings of the IEEE International Conference on Computer Vision. 2017.\n\n[2] Abdal, Rameen, Yipeng Qin, and Peter Wonka. \"Image2StyleGAN: How to Embed Images Into the StyleGAN Latent Space?.\" (2019).\n\n[3] Karras, Tero, Samuli Laine, and Timo Aila. \"A style-based generator architecture for generative adversarial networks.\" Proceedings of the IEEE conference on computer vision and pattern recognition. 2019.\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "4aFHNC1P53_", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Paper summary\n\nThis work studies training GANs on small datasets (in a few-shot setting) for high-resolution image synthesis. To generate high-quality samples with minimum computation cost, and to alleviate overfitting and training instabilities, two techniques are proposed: 1) For the generator the Skip-Layer channel-wise Excitation (SLE) module is introduced, “skip-connecting” low-scale layers with high-scale ones, which facilitates the gradient flow and allows style-content mixing from different images. 2) The discriminator is trained with additional small decoders to reconstruct given images, which acts as self-supervision and helps to reduce overfitting. Experiments show that the proposed GAN model copes well with high-resolution image synthesis task (256x256 – 1024x1024) while being trained on small datasets (down to 60 – 100 images), providing a significant speed up for this setting compared to existing approaches.\n\nStrengths\n\n1) The paper proposes an approach for a very challenging and important task of training GANs with the small amount of training data. To my knowledge, this paper is the first to generate high-resolution realistic images from datasets of such a small scale. This is valuable, as it potentially extends the domain of possible GAN applications. It is also good to see that the paper compares to the recent advances in low-data GANs (Karras et al., 2020a).\n\n2) The paper achieves good results. The performance gain in comparison to StyleGAN2 and the considered baseline is visible across multiple small-scale datasets, see Tables 2 and 3. The improvement in visual quality is also clearly seen from Figure 5, though the evaluation setting might be unfair to StyleGAN2 (see my comments below). \n\n3) SLE module seems interesting, as it is a novel way for designing a skip-connection between layers of different spatial resolutions in the generator. Besides facilitating the gradient flow, which helps the generator to learn quicker, it serves as a tool for style-content mixing by modulating high-scale features on low-scale encoding of another image.\n\n\nWeaknesses\n\n1) Limited technical novelty and missing comparisons to the closely related work. Each of the two proposed technical solutions have been proposed in similar forms in previous works, and the paper does not directly compare with them. \n\n- SLE:\nThe proposed SLE module is a combination of skip connection + plus channel attention mechanism. However, no clear comparison with the related work is provided. As skip connections, one could also use simply residual connections, or use MSG-GAN (Karnevar & Wang, 2019) or StyleGAN2 approaches to improve the gradient flow (see Fig. 7 in the StyleGAN2 paper). They are probably heavier in terms of training speed, but I think there has to be an ablation where the proposed SLE is fairly compared to other ways of using skip connections to improve the gradient flow (memory, speed, performance). \nFrom channel attention mechanism, the proposed technique resembles the Squeeze-and-Excitation module (SE) proposed by Hu et al. However, there were other follow up works that show superior results, such as Efficient Channel Attention (ECA) module proposed by Wang et al. CVPR’2020 or Convolutional Block Attention Module by Woo et al. ECCV’18. The paper doesn’t compare the proposed SLE to the above methods, thus it’s hard to judge how effective it is in comparison. \n\n- SS-discriminator:\nThe employed auto-encoding approach is a typical method for self-supervised learning. However, there has been a line of works which uses different self-supervision techniques (e.g. Auxiliary Rotation Loss in Chen et al. CVPR 2019) or regularizations on the discriminator side (e.g. Zhao et al. 2020) for the same purpose as the proposed self-supervision, and it would be beneficial to see the comparison of the proposed s-s to existing approaches. Table 6 provides only the comparison with the two SS techniques, which might be suboptimal for the task at hand.\n\nGenerally, the proposed model compares well to the considered baseline (DCGAN + extras), but the need for the proposed solutions is not totally justified. Other similar existing solutions, mentioned above, implemented on top of the baseline, potentially could lead to the same performance improvement. \n\n2) Incomplete evaluation and lack of the experimental support of some claims.\n\n-\tThe comparison is done using only one metric - FID. This metric is known for not being able to detect overfitting, and, as was recently shown, is not a proper metric in low-data regimes (see Fig.4 in [*]). This raises the concern that on such small datasets the metric simply shows the degree of overfitting to the training set. With a limited training time used for the reported experiments this is naturally simpler to achieve for the proposed (lighter) model than for StyleGAN2, which might explain such a performance gap between two models. Overall, I disagree with the claim “We find it unnecessary to involve other metrics”, as the used FID metric could be misleading. It would be beneficial to employ also other metrics to measure the diversity of the generated samples. Thus, I find the evaluation presented in Tables 2,3 incomplete.\n\n[*] Robb et al. FEW-SHOT ADAPTATION OF GENERATIVE ADVERSARIAL NETWORKS, ArXiv’2020. \n\n- Overfitting is certainly one of the main challenges in a few-shot synthesis setting. However, the paper pays relatively small attention on analyzing the issue. Also in its current state it’s not clear if the proposed solutions actually help to avoid overfitting. On small datasets, the generated images probably resemble the training examples. This is seen for “Skull” in Figure 5, where for each generated image one could find the similar training example, also noticeable for “Shell” in Figure 6, where the interpolations tend to resemble the image on the left or on the right.\nI agree that Table 5 is valuable, and that it shows relative overfitting strength compared to the baseline. However, I would also expect the analysis on the absolute values, as well as the comparison to StyleGAN2. For example, reporting LPIPS to the nearest training example would be helpful, together with showing generated samples together with the closest training examples. \n\n-\tLooking at the results in Table 4, as the number of training images become larger, the StyleGAN2 outperforms the proposed model. This also shows that even with a half of the parameters the StyleGAN2 model capacity is too large for less than 2k images. So I would expect for a low data regime setting StyleGAN2 with fewer parameters may have better results than the halved StyleGAN2. Given also that on larger datasets (5k-10k, Table 4) the proposed model underperforms, the problem might be the limited capacity of the proposed model, so increasing the number of parameters, e.g. number of channels, might help to improve the overall performance.  This trade-off between the model capacity and the size of the training set is not analysed in the paper, and from my point of view lead to unfair comparison in Tables 2,3,4.\n\n-\tFor Figure 5, the images shown are from a different epoch than in Tables 3 and 4. Moreover, it might be unfair to clip StyleGAN2 at 10 hours of training, as its best epochs are coming later. The figure illustrates the speed-up from the proposed model, but the reader cannot match it to FID values in tables, also it is not possible to see the performance of StyleGAN2 in its best checkpoints on the studied settings.\n\n-\tIt is unclear, what is meant by the “robustness” of the model in the paper. The model is claimed to be robust, but this claim is not really explained and supported experimentally. \n\n\nMinor\n\n1) Why also not to employ the SLE module in the discriminator?\n2) How does the style mixing via SLE compares to other approaches, e.g. StyleGAN2?\n\nOverall, I give the paper a borderline rating. I note that the paper studies an important problem, achieves good results, and advances GANs extending their application areas. On the other hand, I find some incompleteness in the experimental evaluation and unsupported claims in the paper, and have concerns about the limited novelty of the proposed technical solutions. \n\nPost-rebuttal:\nI believe the authors have done sufficient work in their revision to address my concerns. Thus I'm leaning towards acceptance and raising my score to a 7.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a method for training GANs in few-shot setting. Two key components of the method are: a skip-layer channel-wise excitation (SLE) module that encourages gradient flow across resolutions, and a self-supervised loss of autoencoding to regularize the discriminator. The results presented in the paper are indeed impressive in the few-shot setting. Reviewers had some concerns about training set memorization which have been addressed by the authors with additional evaluations using LPIPS metric. Overall, the paper tackles an important problem of few-shot learning of GANs and will be a good addition to the ICLR program. 
Decision: Accept (Poster)