Projected GANs Converge Faster
Authors: Axel Sauer
Kashyap Chitta
Jens Müller
Andreas Geiger
Venue: NeurIPS 2021 Poster (2021)
Keywords: GANs
Generative Models
Image Synthesis
Pretrained Models

Abstract: Generative Adversarial Networks (GANs) produce high-quality images but are challenging to train. They need careful regularization, vast amounts of compute, and expensive hyper-parameter sweeps. We make significant headway on these issues by projecting generated and real samples into a fixed, pretrained feature space. Motivated by the finding that the discriminator cannot fully exploit features from deeper layers of the pretrained model, we propose a more effective strategy that mixes features across channels and resolutions. Our Projected GAN improves image quality, sample efficiency, and convergence speed. It is further compatible with resolutions of up to one Megapixel and advances the state-of-the-art Fréchet Inception Distance (FID) on twenty-two benchmark datasets. Importantly, Projected GANs match the previously lowest FIDs up to 40 times faster, cutting the wall-clock time from 5 days to less than 3 hours given the same computational resources.

Projected GANs Converge Faster
Axel Sauer 1.2
Kashyap Chitta 1.2
Jens Müller
Andreas Geiger
Max Planck Institute for Intelligent Systems, Tübingen CComptter Vision and Learning Lab, University Heidelberg [firstname. lastname mpg.de "(firstname Lastnamej@iwr
University of Tübingen
Abstract
Generative Adversarial Networks (GANs) produce high-quality images but are challenging train. They need careful regularization, vast amounts of compute, and expensive syper-parameter sweeps. We make significant headway on these is- sues by projecting generated and real samples into a fixed, pretrained feature space. Motivated by the finding that the discriminator cannot fully exploit features from deeper layers of the pretrained model, we propose more effective strategy that mixes features across channels and resolutions. Our Projected GAN improves im- age quality, sample efficiency, and convergence speed. is further compatible with resolutions of up to one Megapixel and advances the state-of-the-aar Fréchet In- ception Distance (FID) on twenty-two benchmark datasets. Importantly, Projected GANs match the previously lowest FIDs up to 40 times faster, cutting the wall-clock time from days to less than hours given the same computational resources.
SYyAGGANN-AA
SyycGAN2-ADA
StyleGAN2- (ronweged))
200
Prejeted GAN
(bg sxobo)
100
Projected GAS (smоsиррa)
25
Projected GAN
17x
FiD
15
20
Toiting Time (hou))
Figure 1: Convergence with Projected GANs. Evolution of samples for fixed latent code during training on the AFHQ-Dog dataset [5]. We find that discriminating features in the projected feature space speeds up convergence and yields lower FIDs. This finding consistent across many datasets.
1 Introduction
A Generative Adversarial Network (GAN) consists of generator and a discriminator. For image synthesis, the generator's s task to generate an RGB image; the discriminator aims to distinguish real from fake samples. On closer inspection, the discriminator's 's task is two-fold: First, it projects the real and fake samples into a meaningful space, i.c., it learns a representation Second, it discriminates based on this representation. Unfortunately, training the discriminator jointly with the generator is notoriously hard task. While discriminator regularization techniques help to balance the adversarial game [31], standard regularization methods like gradient penalties [36]
the input space.
susceptible to hyperparameter choices [26] and can lead to substantial decrease in performance [4].
In this paper, we explore the utility of pretrained representations to improve and stabilize GAN training. Using pretrained representations has become ubiquitous in computer vision [29,30,4] 48]
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

and natural language processing 418,45,47]. While combining pretrained perceptual networks [58] with GANs for image-to-image translation has led to impressive results 114,49,59 64], this idea has not yet materialized for unconditional noise-to-image synthesis. Indeed, we confirm that a naive application of this idea does not lead to state-of-the-e results (Section 4) as strong pretrained fea- tures enable the discriminator to dominate the two-player game, resulting in vanishing gradients for the generator [2]. In this work, we demonstrate how these challenges can be overcome and identify two key components for exploiting the full potential of pretrained perceptual feature spaces for GAN training: feature pyramids to enable multi-scale feedback with multiple discriminators and random projections to better utilize deeper layers of the pretrained network.
We conduct extensive experiments on small and large datasets with resolution of up to 1024² pixels. Across all datasets, we demonstrate offteoof-the-arr image synthesis results at significantly reduced training time (Fig. 1). We also find that Projected GANs increase data efficiency and avoid the need for additional regularization, rendering expensive hyperparameter sweeps unnecessary. Code, models, and supplementary videos can be found on the project page https://sitess http://iiee.ooogllcooo/c/e/e///e projected- gan.
2 Related Work
We categorize related work into two main areas: pretraining for GANs and discriminator design.
Pretrained Models for GAN Training, Work on leveraging pretrained representations for GANs can be divided into two categories: First, transferring parts of a GAN to a new dataset [15,38, 65, 71] and, second, using pretrained models to control and improve GANs. The latter is advantageous as pretraining does not need to be adversarial. Our work falls into this second category. Pretrained models can be used as guiding mechanism to disentangle causal generative factors [54], for text-driven image manipulation [44], matching the generator activations to inverted classifiers [19, or to generate images via gradient ascent in the latent space of a generator [41]. The novesaveraa approach of [S3] learns generative models with moment matching in pretrained models; however, the results remain far from competitive standard GANS. An established method is the combination of adversarial and perceptual losses [21]. Commonly, the losses are combined additively [10, 14, 32,52, 64]. Additive combination, however, is only possible if a reconstruction target is available, e.g., in paired image-to- image translation settings [74]. Instead of providing the pretrained network with a reconstruction target, Sungatullina al. [59] propose to optimize an adversarial loss frozen VGG features [58]. They show that their approach improves CycleGAN [74] on image translation tasks. In a similar vein, [49] recently proposed a different perceptual discriminator. They utilize a pretrained VGG and connect its features with the prediction of a pretrained segmentation network. The combined features are fed into multiple discriminators at different scales. The two last approaches are specific to the image-to- image translation task. We demonstrate that these methods do not work well for the more challenging unconditional setting where the entire image content is synthesized from a random latent code.
Discriminator Design. Much work on GANs focuses on novel generator architectures [4, [4,26,27, 27, 69]. while the discriminator often remains close a vanilla convolutional neural network or mirrors the generator. Notable exceptions are [55,70] which utilize an encoder-decoder discriminator architecture. However, in contrast us, they neither use pretrained features random projections. A different line of work considers a setup with multiple discriminators, applied to either the generated RGB image [8, 13] or low-dimensional projections thereof [1. 40]. The use of several discriminators promises improved sample diversity, training speed, and training stability. However, these approaches are not utilized in current state-of-the-arr systems because of diminishing returns compared to the increased computational effort. Providing multi-scale feedback with one or multiple discriminators has been helpful for both image synthesis [23, and image-to-imee to-imae translation (43,64) While these works interpolate the RGB image at different resolutions, our findings indicate the importance of multi-scale feature maps, showing parallels to the success of pyramid networks for object detection [34]. Lastly, to prevent overfitting of the discriminator, differentiable augmentation methods have recently been proposed 635,63,72,73) We find that adopting these strategies helps exploit the full potential of pretrained representations for GAN training.

---

3
Projected GANs
GANs aim to model the distribution of given training dataset. A generator G maps latent vectors sampled from a simple distribution 22 (typically a normal distribution) to corresponding generated samples Gzz). The discriminator D then aims to distinguish real samples x ~ Px from the generated samples Giz) ~ Pa(x). This basic idea results in the following minimax objective
min max D
[Ex[log D(x)] + [o[g( D(G())))I)
(1)
We introduce a set of feature projectors {Pi} which map real and generated images to the discrimina- tor's input space. Projected GAN training can thus be formulated follows
min max (Di)
Σ IEC
(Exllog Di(P(x))) (Ellog(1 (((P)(G()))))
where {Dr} } is set of independent discriminators operating on different feature projections. Note that we keep [Pi] fixed in (2) and only optimize the parameters of G and [Di}. The feature projectors {PI} should satisfy two necessary conditions: they should be differentiable and provide sufficient statistics of their inputs, i.e., they should preserve important information. Moreover, we aim to find feature projectors {P)} which turn the (difficult to optimize) objective in (I) into an objective more amenable to gradient- based optimization. We now show that a projected GAN indeed matches the distribution in the projected feature space, before specifying the details of our feature projectors.
3.1 Consistency
The projected GAN objective in (2) no longer optimizes directly to match the true distribution Pr- To understand the training properties under ideal conditions, we consider a more generalized form of the consistency theorem of [40]:
Theorem 1.
Let PT denote the density of the true data distribution and PP& the density of the distribution the Generator G produces. Let Pi o T and P: o G be the functional composition of the differentiable and fixed function Pi and the true/generated data distribution, and y be the transformed imput to the discriminator. For a fixed G. the optimal discriminators are given by
PPOT((y)
(y = Proor(y) + (y)
for all l E C. In this case, the optimal G under (2) is achieved iff P PorT = P.o for all I E L.
proof of the theorem provided in the appendix. From the theorem, we conclude that a feature projector Pt with its associated discriminator D; encourages the generator to match the true distribu- tion along the marginal through P,. Therefore, at convergence, matches the generated and true distributions in feature space. The theorem also holds when using stochastic data augmentations [25] before the deterministic projections P..
3.2 Model Overview
Projecting to and training in pretrained feature spaces opens up realm of new questions which we address below. This section will provide an overview of the general system and is followed by extensive ablations of each design choice. As feature projections affect the discriminator, focus on Pt and D, in this section and postpone the discussion of generator architectures to Section 5.
Multi-Scale Discriminators. We obtain features from four layers I.r of pretrained feature network F at resolutions (L1- = 642, Le = 32 L3 = 162, L, 82). We associate a separate discriminator Dj with the features at layer Lt. respectively. Each discriminator Dj uses simple convolutional architecture with spectral normalization [37] at each convolutional layer. We observe better perfor- mance if all discriminators output logits at the same resolution (42). Accordingly, we use fewer down- down-sampling blocks for lower resolution inputs. Following common practice, we sum all logits for computing the overall loss. For the generator pass, we the losses of all discriminators. More complex strategies [1, 13] did not improve performance in our experiments.
Random Projections. We observe that features at deeper layers are significantly harder to cover, as evidenced by our experiments Section We hypothesize that a discriminator can focus on subset of the feature space while wholly disregarding other parts. This problem might be especially
3

---

prominent in the deeper, more semantic layers. Therefore, we propose two different strategies to dilute prominent features, encouraging the discriminator utilize all available information equally. Common to both strategies is that they mix features using differentiable random projections which are fixed, i.e., after random initialization, the parameters of these layers are not trained.
Cross- Caas-Chaneel Mixing (CCM). Empirically, we found two proper- ties to be desirable: (i) the random projection should be information preserving to leverage the full representational power of F, and (ii) should not be trivially invertible. The easiest way mix across channels is a 1 Ixl 1 convolution. A 1 XX] I convolution with an equal number of output and input channels is a generalization of a permu- tation [28] and consequently preserves information about its input. In practice, we find that more output channels lead to better perfor- mance as the mapping remains injective and therefore information preserving. Kingma et al. [28] initialize their convolutional layers as random rotation matrix
Feature Network
good starting point for optimization. We do not find this to improve GAN performance (see Appendix), arguably since it violates (ii). We therefore randomly initialize the weights of the convolutional layer via Kaiming initialization [16]. Note that we do not add any activation functions. We apply this ran- dom projection at each of the four scales and feed the transformed feature to the discriminator depicted in Fig. 2.
Real/Generated Image
Figure 2: CCM (dashed blue arrows) employs 1 1x1 1 convo- lutions with random weights.
Cross-Scale Mixing (CSM). To encourage feature mixing across scales, CSM extends CCM with random 3x3 convolutions and bilinear upsampling, yielding a U-Net [50] architecture, see Fig. 3. However, our CSM block is simpler than vanilla U-Net [50]: we only use a single convolutional layer at each scale, As for CCM, we utilize Kaiming initialization for all weights.
Feature Network
D
Di
Prefrained Feature Networks. We ablate over varying feature net- works. First, we investigate different versions of EfficientNets, which allow for direct control over model size versus performance. Effi-
Real/Generated Image
cientNets are image classification models trained on ImageNet [7]
Figure 3: CSM (dashed red ar- rows) adds random 3x3 con- volutions and bilinear upsam- pling, yielding U-Network.
and designed to provide favorable accuracy-come comppte tradeoffs. Sec-
ond, we use ResNets of varying sizes. To analyze the dependency on ImageNet features (Section 4.3), we also consider R50-CLIP [46], a ResNet optimized with contrastive language-image objective on a dataset of 400 million (image, text) pairs. Lastly, we utilize a vision transformer architecture (ViT- Base) [9] and its efficient follow-up (DeiT-small distilled) [62]. We do not choose inception network [60] to avoid strong correlations with the evaluation metric FID [17]. In the appendix, we also evaluate several other neural and non non-neural metries to rule out correlations. These additional metrics reflect the rankings obtained by FID.
In the following, we conduct systematic ablation study to analyze the importance and best configu- ration of each component in our Projected GAN model, before comparing to the state-of-the-art
4 Ablation Study
To determine the best configuration of discriminators, mixing strategy, and pretrained feature network. we conduct experiments on LSUN-Church [67], which is medium-sized (126k images) and reasonably visually complex, using a resolution of 2562 pixels. For the generator G we use the generator architecture of FastGAN [35], consisting of several upsampling blocks, with additional skip-layer- excitation blocks. Using a hinge loss [33], we train with a batch size of 64 until million real images have been shown to the discriminator, a sufficient amount for G' to reach values close to convergence. not specified otherwise, we use an EfficientNet-Lice [61] feature network in this section. We found that discriminator augmentation (25,63, 72,73 consistently improves the performance of all methods, and is required to reach of performance. We leverage differentiable augmeennmentt [72] which we found to yield the best results in combination with the FastGAN generator.

---

Discriminator(s)
rel-FD, + rel-FD2 rel-FDs ret-FDa + rel FID
No Projection
L1
0.56
0.32
0.31
0.55
0.66
Lı, L2
0.35
0.21
0.23
0.47
0.53
on L1, L2, L3
0.42
0.26
0.28
0.64
0,90
on Lı, La, La, La
0.46
0.34
0.38
0.79
1.15
L2, La, La
0.95
0.67
0.71
1.19
1.99
La, La
2.14
1,41
1.18
1.99
3,46
10.92
5.74
2.56
2.79
5.08
Perceptual
2.98
1.76
1.20
1.89
2.73
CCM
TT I,, Ta
0.27
0.21
0.26
0.50
0.59
0.27
0.18
0.21
0.41
0.48
on L1, L2, L3
0.31
0.25
0.24
0.54
0.67
on Lı, L2, La. La
0.53
0.34
0.34
0.59
0.77
Perceptual
5.33
3.06
2.14
1.09
4.77
CCM CSM
0.34
0.25
0.19
0.35
0.44
on Lı, La
0.21
0.18
0.16
0.27
0.31
on L1, L2, Ls
0.41
0.26
0.17
0.23
0.29
L1, L2, Ls, La
0.26
0.16
0.13
0.16
0.24
Perceptual D
2.53
1.37
0.89
0.43
2.13
Table I: Feature Space Fréchet Distances. We aim to find the best combination of discriminators and random projections to the distributions feature network F. We show the relative FD at different layers of F (rel-F D between 50k generated and real images on LSUN-Church. rel-F D; normalized using the baseline Fréchet Distances for a model with a standard single RGB image discriminator. Hence, values 1 indicate worse performance than the RGB baseline. We report rel-FD for four layers of an EfficientNet (L1, Iz,, Ly and I. from shallow to deep), as well as relative Fréchet Inception Distance (FID) [17]. Note that rel-FD, D, should not be compared between different feature spaces, i.e., only within-column comparisons are meaningful. Blue boxes highlight the layers which supervise via independent discriminators. The green box corresponds perceptual discriminator [59], which takes in all feature maps at once.
4.1 Which feature network layers are most informative?
We first investigate the relevance of independent multi-scale discriminators. For this experiment, do not use feature mixing. To measure how well G fits a particular feature space, we employ the Fréchet Distance (FD) [12] on the spatially pooled features denoted as FD; for layer i. FDs across different feature spaces are not directly comparable. Therefore, we train GAN baseline with a standard RGB discriminator, record F D at each layer and quantify the relative improvement via the fraction rel-F D. FD;/ DD//FDRRC FDRGB We also investigate perceptual discriminator [59], where feature maps are fed into different layers of the same discriminator to predict a single logit.
The results in Table (No Projection) show that two discriminators are better than one and improve over the vanilla RGB baseline. Surprisingly, adding discriminators deep layers hurts performance. We conclude that these more semantic features do not respond well to direct adversarial losses. We also experimented with discriminators resized versions of the original image, but could not find setting of hyperparameters and architectures that improves over the single image baseline. Omitting discriminators on the shallow features decreases performance, which is anticipated, as these layers contain most of the information about the original image. A similar effect has been observed for feature inversion [11]- - the deeper the layer, the harder to reconstruct input. Lastly, we observe that independent discriminators outperform the perceptual discriminator by significant margin.
5

---

EfficientNet
ResNet
Transformer
lite0
litel
lite2
lite3
lite4
R18
R50
R50-CLIP
DeiT
ViT
Params (M) +
2.96
3.72
4.36
6.42
11.15
11.18
23.51
23.53
92.36
317.52
IN top-11
75.48
76.64
77.47
79.82
81.54
69.75
79.04
N/A
85,42
85.16
FID
2.53
1.65
1.69
1.79
2.35
4.16
4.40
3.80
2.46
12.38
Table 2: Pretrained Feature Networks Study. We train the projected GAN with different pretrained feature networks. We find that compact EfficientNets outperform both ResNets and Transformers.
4.2 How can we best utilize the pretrained features?
Given the insights from the previous section, aim to improve the utilization of deep features. For this experiment, we only investigate configurations that include discriminators at high resolutions. Table (CCM and CCM CSM) presents the results for both mixing strategies. CCM moderately decreases the FDs across all settings, confirming our hypothesis that mixing channels results in better feedback for the generator. When adding CSM, we achieve another notable improvement across all configurations. Especially rel-FD; at deeper layers are significantly decreased, demonstrating CSM's usefulness to leverage deep semantic features. Interestingly, we observe that the best performance is now obtained by combining all four discriminators. A perceptual discriminator is again inferior to multiple discriminators. We remark that integrating the original image, via an independent discriminator or CCM or CSM always resulted in worse performance. This failure suggests that naïvely combining non- non- pon-proeectdd with projected adversarial optimization impairs training dynamics.
4.3 Which feature network architecture is most effective?
Using the best setting determined by the experiments above (CCM + CSM with four discriminators), we study the effecctiveness of various perceptual feature network architectures for Projected GAN training. To ensure convergence, also for larger architectures, we train for 10 million images. Table 2 reports the FIDs achieved on LSUN-Church Surprisingly, we find that there is no correlation with ImageNet accuracy. On the contrary, we observe lower FIDs for smaller models (e.g.. EfficientNets- lite). This observation indicates that a more compact representation is beneficial while at the same time reducing computational overhead and consequently training time. R50-CLIP slightly outper- forms its R50 counterpart, indicating that ImageNet features are not required to achieve low FID. For the sake of completeness, we also train with randomly initialized feature networks, which, however, converge to much higher FID values (see Appendix). In the following, we thus use EfficientNet-Litt as our feature network.
5 Comparison to State-of-the-Art
This section conducts a comprehensive analysis demonstrating the advantages of Projected GANs with respect to state-of-the art models. Our experiments are structured into three sections: evaluation of convergence speed and data efficiency (5.1). and comparisons on large (5.2) and small (5.3) benchmark datasets. We cover a wide variety of datasets in terms of size (hundreds to millions of samples), resolution (256² to 1024²), and visual complexity (clip-art, paintings, and photographs).
Evaluation Protocol. We measure image quality using the Fréchet Inception Distance (FID) [17]. Following [26,27], we report the FID between 50k generated and all real images. We select the snapshot with the best FID for each method. In addition to image quality, we include a metric to evaluate convergence. As in [25], we measure training progress based on the number of real im- ages shown to the discriminator (Imgs). We report the number of images required by the model for the FID to reach values within 5% of the best FID over training. In the appendix, we also report other metrics that are less benchmarked in GAN literature: KID [3], SwAV-FID [39], precision and recall [5]]. Unless otherwise specified, we follow the evaluation protocol of [20] to facilitate fair comparisons. Specifically, we compare all approaches given the same fixed number of images (10 million). With this setting, each experiment takes roughly 100-200 GPU hours on a NVIDIA V100, for more details refer to the appendix.
Baselines. We use StyleGAN2-ADA [25] and FastGAN [35] as baselines. StyleGAN2-ADA the strongest model on most datasets in terms of sample quality, whereas FastGAN excels in training speed. We implement these baselines and our Projected GANs within the codebase provided by the authors of StyleGAN2-ADA [25]. For each model, we ran two kinds of data augmentation:

---

000
StylGAN2
80-
StylGGANZAADA
Projected StyleGAN ADA
50
PeeGAN
100
MeCGAN
Projected FeeGAN
TID (Oog scal))
Projected FoAGAN
FID (log
20-
10
10
10
14
Thining Progress maa non real images
Datnect Size (K Imagea]
Figure 4: Training Properties. Left: Projected FastGAN surpasses the best FID of StyleGAN2 (at 88 M images) after just 1.1 M images on LSUN-Church. Right: Projected FastGAN yields significantly improved FIID scores, even when using subsets of CLEVR with 1K and 10k samples.
100k 200k
3000
400G:
500%
600k
700K
S00G:
500l:
Figure 5: Training progress on LSUN church at 256 pixels. Shown are samples for a fixed noise vector z over k images. From top to bottom: FasIGAN, StyleGAN2-ADA, Projected GAN.
differentiable data-augmentation [72] and adaptive discriminator augmentation [25]. We select the better performing augmentation strategy per model. For all baselines and datasets, we perform data amplification through fllssss. Projected GANs use the same generator and discriminator architecture and training hyperparameters (learning rate and batch size) for all experiments. For high- high-resolution image generation, additional upsampling blocks are included in the generator to match the desired output resolution. We carefully tune all hyyerr-prametee that FastGAN is sensitive to the choice of batch size, and StyleGAN2-ADA to the learning rate and R1 penalty. The appendix documents additional implementation details used each of our experiments.
for both baselines for best results: we find
5.1 Convergence Speed and Data Efficiency
Following [20] and [68]. analyze the training properties of Projected GANs on LSUN-Church at an image resolution of 256 pixels and on the 70k CLEVR dataset [22]. In this section, we also train longer than 10 M images if necessary,
are interested in convergence properties.
Convergence Speed. We apply projected GAN training for both the style-based generator of Style- GAN2 and the standard generator with a single input noise vector of FastGAN. As shown in Fig. 4 (left), FastGAN converges quickly but saturates (88 M images) but reaches lower FID. Projected GAN training improves both generators. Par- ticularly for FastGAN, improvements in both convergence speed and final FID are significant while improvements for StyleGAN2 are less pronounced. Remarkably, Projected FasIGAN reaches the previously best FID of StyleGAN2 after experiencing only 1.1 M images as compared to 88 M of StyleGAN2. In wall clock time, this corresponds to less than hours instead of 5 days. Hence, from now on, we utilize the FastGAN generator and refer to this model simply as Projected GAN.
high FID. StyleGAN2 converges more slowly
Fig. 5 shows samples for fixed noise vector z during training on and StyleGAN, patches of texture gradually morph into a global structure. For Projected GAN, we directly observe the emergence of structure which becomes more detailed over time. Interestingly, the Projected GAN latent space appears to be very volatile, i.e., for fixed z the images undergo significant perceptual changes during training. In the non-projected cases, these changes are more gradual. We hypothesize that this induced volatility might be due to the discriminator providing more semantic feedback compared to conventional RGB losses. Such semantic feedback could introduce
For both FastGAN
7

---

more stochasticity during training which in turn improves convergence and performance. We also observed that the signed real logits of the discriminator remain the same level throughout training (see Appendix). Stable signed logits indicate that the discriminator does not suffer from overfitting.
Sample Efficiency. The use of pretrained models is generally linked to improved sample efficiency. To evaluate this property, we also created two subsets of the 70k CLEVR dataset by randomly sub- sampling 10k and Ik images from respectively. As depicted in Fig. 4 (right), our Projected GAN significantly improves over both baselines across all dataset splits.
5.2 Large Datasets
Besides CLEVR and LSUN--hhudee we benchmark Projected GANs against various state-of-the-an
models on three other large datasets: LSUN- [67] (3M indoor bedroom scenes), FFHQ [26] (70k images of faces) and Cityscapes [6] (25k driving scenes captured from vehicle). For all datasets, we use an image resolution of 256 pixels. As Cityscapes and CLEVR images are not of
aspect ratio 1:1 we resize them to 2562 for training. Besides StyleGAN2-ADA and FastGAN, we compare against SAGAN [69] and GANsformers [20]. All models were trained for 10 M images. For the large datasets, we also report numbers for StyleGAN2 trained for more than 10 M images to report the lowest FID values achieved in previous literature (denoted as StyleGAN2*). In the appendix, we report results on nine more large datasets.
Table 3 shows that the Projected GAN outperforms all stae-of--the-ar models in terms of FID values on all datasets by a large margin. For example, on LSUN-Bedroom,, achieves an FID value of 1 1.52 compared to 6.15 by GANsformer, the previously best model in this setting. Projected GAN achieves state-of-ther the-trr FID values remarkably fast, e.g., on LSUN-church, it achieves an FID value of 3.18 after 1.1 M Imgs. StyleGAN2 has obtained the previously lowest FID value of 3.39 after 88 M Imgs, 80 times as many as needed by Projected GAN. Similar speed-ups are also realized for all other large datasets shown in Table 3. Interestingly, when training longer FFHQ (39 M Imgs), we observe further improvements of Projected GAN to an FID of 2.2. Note that all five datasets represent very different objects in various scenes. This demonstrates that the performance gain is robust to the choice of the dataset, although the feature network is trained only on ImageNet. It is important to note that the main improvements are based on improved sample diversity indicated by recall which we report in the appendix. The improvement in diversity is most notable on large datasets, e.g., LSUN church, where the image fidelity appears to be similar to StyleGAN.
5.3 Small Datasets
To further evaluate our method in the few-shot setting, we compare against StyleGAN2-ADA and FastGAN on art paintings from WikiArt (1000 images; wikiart.org), Oxford Flowers (1360 images) [42]. photographs of landscapes (4319 images; flickr.com), AnimalFace-Doog (389 images) [57] and Pokemon (833 images; pokemon.com/ Further, we report results on high-resolution versions of Pokemon and Art-Painting (1024²). Lastly. we evaluate on AFHQ-Cat, -Dog and -Wild at 512² [5]. The AFHQ datasets contain ~5k closeups per category cat, dog, or wildlife. We do not have a license to re-distribute these datasets, but we provide the URLs to enable reproducibility, similar to [35].
Projected GAN outperforms all baselines in terms of FID values by a significant margin on all datasets and all resolutions as shown in Table 3. Remarkably, our model beats the prior state-of-the-art on all datasets (256²) after observing fewer than 600k images. For AnimalFace-Dog. the Projected GAN surpasses the previously best FID after only 20k images. One might argue that the EfficientNet used as feature network facilitates data generation for the animal datasets as EfficientNet is trained on ImageNet which contains many animal classes (e.g., 120 classes for dog breeds). However, it is interesting to observe that Projected GANs also achieve -tate-of-the-art FID on Pokemon and Art Painting though these datasets differ significantly from ImageNet. This evidences the generality of ImageNet features. For the high-resolutic datasets, Projected GANs achieve the same FID value many times faster than the best baselines, e.g., ten times faster than StyleGAN2-ADD on AFHQ- Cat or four times faster than FastGAN on Pokemon. We remark that F and D; generalize to any resolution they are fully convolutional.
8

---

Figure 6: Real samples (top rows) vs. samples by Projected GAN (bottom rows). Datasets (top left to bottom right): CLEVR (256)), LSUN church (256*), Art Painting (256²), Landscapes (256*),
AFHQ-wild (512²), Pokemon (256"), AFHQ-dog (512"), AFHQ-cat (512²).
FID
Imgs
FID
Imgs
FID
Imgs
FID
Imgs
FID
Imgs
Large Datasets (256*)
CLEVR
FFHQ
Cityscapes
Bedroom
Church
SAGAN [69]
26.04
M
16.21
10M
12.81
10 M
14.06
10
6.15
STYLEGAN2-AN ADA [25]
10.17
10 M
7.32
10 M
8.35
10 M
11.53
10 M
5.85
10
GANSFORMERS [20]
9.24
10 M
7.42
10 M
5.23
10
6.15
1
5.47
1
FASTGAN [35]
3.24
10 M
12.69
10
8.78
1.8 M
8.24
4.8 M
8.43
8.9
PROJECTED GAN
0.89
4.5 M
3.39
7.1 M
3.41
1.7 M
1.52
5.2
1.59
9.2
PROJECTED +AN*
3.39
0.5 M
3.56
7.0 M
4.60
1.1
2.58
1.5
3.18
1.1
STYLEGAN2 [25.26,68]
5.05
25 M
3.62
25 M
2.65
70 M
3.39
88 M
Small Datasets (2562)
Art Painting
Landscape
AnimalFace
Flowers
Pokemon
STYLEGAN2-ADA [25]
43.07
15.99
6.3
60.90
2.2
21.66
3.8
40.38
3.4 M
FASTGAN [35]
44.02
16.44
1.8
62.11
0.2
26.23
0.8
81.86
2.5 M
PROJECTED GAN
27.96
0.8 M
6.92
3.5 M
17.88
1
13,86
1.8 M
26.36
0.8 M
PROJECTED GAN*
40.22
14.99
0.6 M
58.07 0.02
21.60
0.2
36.57
0.3 M
10242
5122
Art Painting
Pokemon
AFHQ-Cat
AFHQ-Dog
AFHQ-Wild
STYLEGAN2-ADA [25]
41.69
1.0
56.76 0.6
3.55
10
7.40
10 M
3.05
M
FASTGAN [35]
46.71
0.8 M
56.46
0.8 M
4.69
1.1 M
13.09
1.6 M
3.14
1.6
PROJECTED GAN
32.07
0.9 M
33.96
1.3 M
2.16
3.7 M
4.52
3.8 M
2.17
5.4
PROJECTED GAN
40.33
0.2 M
53.74
0.2 M
3.53
1.0 M
7.10
0.9
3.03
1.6
Table 3: Quantitative Results. Projected GAN* reports the point where our approach surpasses the
thehe-tf- StyleGAN2* obtains the lowest FID in previous literature if trained long enough.

---

Discussion and Future Work
While we achieve low FID on all datasets, we also identify two systematic failure cases: As depicted in Fig. 7, we sometimes observe "floating heads" on AFHQ. imals appear in high quality but resemble cutouts on blurry or bland backgrounds. We hypothesize that generating a realis- tic background and image composition is less critical when prominent object already depicted. This hypothesis follows from the fact that we used image classification models for the projection, which have been shown to only marginally reduce in accuracy when applied on images of objects with removed background [66]. On FFHQ, projected GAN sometimes pro- duces poor-quality samples with wrong proportions and arti- facts, even at state-of satee-ofthhe of-the-art art FID, see Fig. 8.
few samples, the an-
Figure 7: "Floating Heads"
Figure 8: Artifacts FFHQ.
In terms of generators, StyleGAN is more challenging to tune and does not profit as much from projected training. The FastGAN generator is fast to optimize but simultaneously produced un- realistic samples in some parts of the latent space problem that could be solved by mapping network similar to StyleGAN. Hence, we speculate that unifying the strengths of both architectures in combination with projected training might improve performance further. Moreover, our study of different pretrained networks indicates that efficient models are especially suitable for projected GAN training. Exploring this connection in-depth, and in general, determining desirable feature space properties opens up exciting new research opportunities. Lastly, our work advances efficiency for generative models. More efficient models lower the barrier of computational effort needed for generating realistic images. A lower barrier facilitates malignant use of generative models (e.g.. "deep fakes") while simultancously also democratizing research in this area.
Acknowledgments and Disclosure of Funding
We acknowledge the financial support by the BMWi in the project KI Delta Learning (project number 19A190130), Andreas Geiger was supported by the ERC Starting Grant LEGO-3D (850533). Kashyap Chitta was supported by the German Federal Ministry of Education and Re- search (BMBF): Tübingen AI Center, FKZ: 01S118039 and the International Max Planck Re- search School for Intelligent Systems (IMPRS-IS). Jens Müller received funding by the Hcidel- berg Collaboratory for Image Processing (HCI). We thank the Center for Information Services and High Performance Computing (ZIH) at Dresden University of Technology for generous allo- cations of computation time. Lastly, would like thank Vanessa Sauer for her general sup- port.
References
[1] I. Albuquerque, J. Monteiro, T. Doan, B. Considine, T. Falk, and I. Mitliagkas. Multi-objective training of generative adversarial networks with multiple discriminators. In Proc. of the International Conf. on Machine learning (ICML), 2019. 2,3
[2] M. Arjovsky and L. Bottou. Towards principled methods for training generative adversarial networks. In Proc. of the International Conf. Learning Representations (ICLR), 2017. 2
[3] M. Binkowski, D. J. Sutherland, M. Arbel, and A. Gretton. Demystifying MMD gans. In Proc. of the International Conf. on Learning Representations (ICLR), 2018. 6
[4] A. Brock, J. Donahue, and K. Simonyan. Large scale GAN training for high fidelity natural image synthesis. In Proc. of the International Conf. on Learning Representations (ICLR), 2019. 1.2
[5] Y. Choi, Y. Uh, J. Yoo, and J.W. Ha. Stargan v2: Diverse image synthesis for multiple domains. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2020. 1,8
[6] M. Cordts, M. Omran, S. Ramos, T. Rehfeld, M. Enzweiler, R. Benenson, U. Franke, S. Roth, and B. Schiele. The cityscapes dataset for semantic urban scene understanding. Proc. IEEE
Conf. on Computer Vision and Pattern Recognition (CVPR), 2016. 8
10

---

[7] Deng, W. Dong, R. Socher, L. jiaa Li, K. Li, and L. Fei-fei. Imagenet: A lcrge-scale hierarchical image database. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2009. 4
[8] T. Doan, J. Monteiro, I. Albuquerque, B. Mazoure, A. Durand, J. Pineau, and R. D. Hjelm. On- line adaptative curriculum learning for gans. In Proc. of the Conf. on Artificial Intelligence
(AAAT), 2019. 2
[9] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby. An image is worth 16x 16 words:
Transformers for image recognition at scale. In Proc. of the International Conf. on Learning Representations (ICLR), 2021. 4
[10] A. Dosovitskiy and T. Brox. Generating images with perceptual similarity metrics based on deep networks. In Advances in Neural Information Processing Systems (NIPS), 2016.
[11] A. Dosovitskiy and T. Brox. Inverting visual representations with convolutional networks. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2016. 5
[12] D. Dowson and B. Landau. The fréchet distance between multivariate normal distributions.
Journal of multivariate analysis, 1982. 5
[13] I. Durugkar, I. Gemp, and S. Mahadevan. Generative multi-adversarial networks. In Proc. of the International Conf. on Learning Representations (ICLR), 2017. 2,
[14] P. Esser, R. Rombach, and B. Ommer. Taming transformers for high-resolution image synthesis.
In Proc. IEEE Conf. on Computer Vision and Puttern Recognition (CVPR), 2020. 2
[15] H. Ham, T. J. Jun, and D. Kim. Unbalanced gans: Pre-training the generator of generative
adversarial network using variational autoencoder. arXiv.org, 2002.02112. 12, 2020. 2
[16] K. He, X. Zhang, S. Ren, and J. Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proc. of the IEEE International Conf. on Computer Vision (ICCV), 2015.
[17] M. Heusel, H. Ramsauer, T. Unterthiner. B. Nessler, and S. Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium. In Advances in Neural Information
Processing Systems (NIPS), 2017, 4, 5,
[18] J. Howard and S. Ruder. Universal language model fine-tuning for text classification. In Association for Computational Linguisties (ACL), 2018.
[19] X. Huang, Y. Li, O. Poursaeed, J. Hopcroft, and S. Belongie. Stacked generative adversarial networks. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2017. 2 [20] D. A. Hudson and C. L.. Zitnick. Generative adversarial transformers. arXiv.org, 2103.01209,
2021. 6, 7, 9
[21] J. Johnson, A. Alahi, and L. Fei-Fei. Perceptual losses for real-time style transfer and super- resolution. Proc, of the European Conf. on Computer Vision (ECCV), 2016.
[22] Johnson, B. Hariharan, L. van der Maaten, L. Fei-Fei, C. Lawrence Zitnick, and R. Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2017. 7
[23] A. Karnewar and O. Wang. Msg-gan: Multi-scale gradients for generative adversarial networks.
In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2020. 2
[24] Karras, T. Aila, S. Laine, and J. Lehtinen. Progressive growing of GANs for improved quality, stability, and variation. In Proc. of the International Conf. on Learning Representations (ICLR), 2018.
[25] T. Karras, M. Aittala, J. Hellsten, S. Laine, J. Lehtinen, and T. Aila. Training generative adversarial networks with limited data, In Advances in Neural Information Processing Systems (NeurIPS), 2020. 2, 3, 4, 6, 7,
[26] T. Karras, S. Laine, and T. Aila. A style-based generator architecture for generative adversarial networks. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2019. 1,
2, 2.9
[27] T. Karras, S. Laine, M. Aittala, J. Hellsten, J. Lehtinen, and T. Aila. Analyzing and improving the image quality of StyleGAN. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2020. 2, 6
[28] D. P. Kingma and P. Dhariwal. Glow: Generative flow with invertible 1x1 convolutions. In Advances in Neural Information Processing Systems (NeurIPS), 2018.
[29] A. Kolesnikov, L.. Beyer, X. Zhai, J. Puigcerver, J. Yung. S. Gelly, and N. Houlsby. Big transfer (bit): General visual representation learning. In Proc. of the European Conf. on Computer
Vision (ECCV), 2020.
11

---

[30] S. Kornblith, J. Shlens, and Q. V. Le. Do better imagenet models transfer better? In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR). 2019. 1
[31] K. Kurach, M. Lucic, X. Zhai, M. Michalski, and S. Gelly. The gan landscape: Losses, architectures, regularization, and normalization. In Proc. of the International Conf. on Machine learning (ICML) Workshops, 2018.
[32] C. Ledig. L. Theis, F. Huszar, J. Caballero, A. Cunningham, A. Acosta, A. P. Aitken, A. Tejani,
J. Totz, Z. Wang. and W. Shi. Photo-realistic single image super-resolution using a generative adversarial network. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR),
2017. 2
[33] J. H. Lim and J. C. Ye. Geometric gan. arXiv.org, 1705.0 778944 2017.
[34] T. Lin, P. Dollár, R. B. Girshick, K. He, B. Hariharan, and S. J. Belongie. Feature pyramid
networks for object detection. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2017. 2
[35] B. Liu, Y. Zhu, K. Song, and A. Elgammal. Towards faster and stabilized gan training for high- fidelity few-shot image synthesis. In Proc. of the International Conf. on Learning Representa-
lions (ICLR), 2021. 4, 6, 8,
[36] L. Mescheder, A. Geiger, and S. Nowozin. Which training methods for gans do actually converge? In Proc. of the International Conf. on Machine learning (ICML), 2018. I
[37] T. Miyato, T. Kataoka, M. Koyama, and Y. Yoshida. Spectral normalization for generative adversarial networks. In Pruc. of the International Conf. on Leurning Representations (ICLR),
2018. 3
[38] Mo, M. Cho, and J. Shin. Freeze the discriminator: a simple baseline for fine-tuning gans. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR) Workshops, 2020. 2 [39] S. Morozov, A. Voynov, and A. Babenko. On self-supervised image representations for GAN evaluation. In Proc. of the International Conf. on Learning Representations (ICLR), 2021. [40] B. Neyshabur. S. Bhojanapalli, and A. Chakrabarti. Stabilizing gan training with multiple random projections. arXiv.org, 1705.0783, 2017. 2,
[41] A. Nguyen, J. Yosinski, Y. Bengio, A. Dosovitskiy, and J. Clune. Plug & play generative networks: Conditional iterative generation of images in latent space. arXiv.org, 1612.00005,
2016. 2
[42] M.-E. Nilsback and A. Zisserman. Automated flower classification over a large number of classes. In Proc. Indian Conf. on Computer Vision, Graphics & Imuge Processing, 2008. 8 [43] T. Park, M.-Y. Liu, T.-C. Wang, and J.-Y. Zhu. Semantic image synthesis with spatially-adaptive normalization. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2019.
2
[44] O. Patashnik, Z. Wu, E. Shechtman, D. Cohen-Or, and D. Lischinski. Styleclip: Text-driven
manipulation of stylegan imagery. arXiv.org, 2103.17249 2021. 2
[45] M. E. Peters, M. Neumann. M. lyyer. M. Gardner, C. Clark, K. Lee, and L. Zettlemoyer. Deep contextualized word representations. In Conf. of the North American Chapter of the Association
for Computational Linguistics (NAACL), 2018. 2
[46] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell,
P. Mishkin, J. Clark, et al. Learning transferable visual models from natural language supervision.
arXiv.org, 2103.00020, 2021. 4
[47] A. Radford, K. Narasimhan, T. Salimans, and Sutskever. Improving language understanding by generative pre-training, 2018. 2
[48] R. Ranftl, K. Lasinger, D. Hafner, K. Schindler, and V. Koltun. Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer. IEEE Trans. on Pattern Analysis and Machine Intelligence (PAMI), 2020. I
[49] S. R. Richter, H. A. AlHaija, and V. Koltun. Enhancing photorealism enhancement. arXiv.
2105.04619, 2021. 2
[50] O. Ronneberger, P. Fischer, and T. Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assister Intervention (MICCAI),
2015. 4
[51] M. S. Sajjadi, O. Bachem, M. Lucic, O. Bousquet, and S. Gelly. Assessing generative models via precision and recall. In Advances in Neural Information Processing Systems (NeurIPS). 2018. [52] M. S. Sajjadi, B. Scholkopf, and M. Hirsch. Enhancenet: Single image super-resolution through automated texture synthesis. In Proc. of the IEFE International Conf. on Computer Vision
(ICCV), 2017. 2
12

---

[53] C. N. d. Santos, Y. Mrouch, I. Padhi, and P. Dognin. Learning implicit generative models by matching perceptual features. In Proc. of the IEEE International Conf. on Computer Vision (ICCV), 2019. 2
[54] A. Sauer and A. Geiger. Counterfactual generative networks, In Proc. of the International Conf. on Learning Representations (ICLR), 2021. 2
[55] E. Schonfeld, B. Schiele, and A. Khoreva. A u-net based discriminator for generative adversarial networks. In Proc. IEEE Conf. on Computer Vision and Paltern Recognition (CVPR), 2020. 2 [56] A. Shocher, Y. Gandelsman, I. Mosseri, M. Yarom, M. Irani, W. T. Freeman, and T. Dekel. Semantic pyramid for image generation. In Proc. IEEE Conf. Computer Vision and Pattern Recognition (CVPR), 2020. 2
[57] Z. Si and S.-C. Zhu. Learning hybrid image templates (hit) by information projection. IEEE Truns. on Puttern Analysis and Machine Intelligence (PAMI), 2011. 8
[58] K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. In Proc. of the International Conf. on Learning Representations (ICLR), 2015. 2 [59] D. Sungatullina, E. Zakharov, D. Ulyanov, and V. Lempitsky. Image manipulation with perceptual discriminators. In Proc. of the European Conf. on Computer Vision (ECCV), 2018.
5
[60] C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. E. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich. Going deeper with convolutions. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2015. 4
[61] M. Tan and Q Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In Proc. of the International Conf. on Machine learning (ICML), 2019. 4
[62] H. Touvron, M. Cord, M. Douze, Massa, A. Sablayrolles, and H. Jégou. Training data- efficient image transformers & distillation through attention. arXiv.org, 2012.12877, 2020. 4 [63] N.-T. Tran, V.-H. Tran, N.-B. Nguyen, T.-K. Nguyen, and N.-M. Cheung. On data augmentation
for gan training. IEEE Trans. Image Processing (TTP), 2021. 2, 4
[64] T.-C. Wang. M.-Y. Liu, J.-Y. Zhu, A. Tao, J. Kautz, and B. Catanzaro. High-resolution image synthesis and semantic manipulation with conditional gans. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2018. 2
[65] Y. Wang, C. Wu, L.. Herranz, J. van de Weijer, A. Gonzalez-Gareiia, and B. Raducanu. Transfer- ring gans: generating images from limited data. In Proc. of the European Conf. on Computer
Vision (ECCV), 2018. 2
[66] K. Xiao, L. Engstrom, A. Ilyas, and A. Madry, Noise or signal: The role of image backgrounds in object recognition. In Proc. of the International Conf. on Learning Representations (ICLR),
2021. 10
[67] F. Yu, A. Seff,, Y. Zhang, S. Song, T. Funkhouser, and J. Xiao. Lsun: Construction of a large-scale image dataset using deep learning with humans in the loop. arXiv.org, 1506.03365, 2015. 4, [68] N. Yu, G. Liu, A. Dundar, A. Tao, B. Catanzaro, L. Davis, and M. Fritz. Dual contrastive loss
and attention for gans. arXiv.org, 16748, 2021. 7, 9
[69] H. Zhang, I. Goodfellow, D. Metaxas, and A. Odena. Self-attention generative adversarial networks. In Proc. of the International Conf. on Machine learning (ICML), 2019. 2, 8, 9
[70] J. Zhao, M. Mathieu, and Y. LeCun. Energy-based generative adversarial network. In Proc. of the International Conf. on Learning Representations (ICLR), 2017. 2
[71] M. Zhao, Y. Cong, and L. Carin. On leveraging pretrained gans for generation with limited data. In Proc. of the International Conf. on Machine learning (ICML.,, 2020. 2
[72] S. Zhao, Z. Liu, J. Lin, J.-Y. Zhu, and S. Han. Differentiable augmentation for data-efficient gan training. In Advances in Neural Information Processing Systems (NeurIPS), 2020. 2, 4, 7 [73] Z. Zhao, Z. Zhang, T. Chen, S. Singh, and H. Zhang. Image augmentations for gan training. arXiv.org, 2006.02595, 2020. 2, 2
[74] J.-Y. Zhu, T. Park, P. Isola, and A. A. Efros. Unpaired image-to-image translation using cycle- consistent adversarial networks. In Proc. of the IEEE International Conf. on Computer Vision (ICCV), 2017.
13


Reviews:
[{"review_id": "xYIh4MnyazW", "reviewer": "Reviewer_zAjH", "summary": "This paper investigates the use of pre-trained imagenet classifier networks as feature extractors for GAN discriminators. The author propose to use multiscale fixed random projections combined via an FPN-like approach, and explore which levels of the feature hierarchy are best fed into an ensemble of discriminators under different projection and mixing settings. Results are presented on a range of unimodal datasets (CLEVR, FFHQ, LSUN, etc) and compared against FastGAN and StyleGAN baselines, showing improvements particularly wrt sample efficiency.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "My take:\nThis is a decent paper. For the most part, it is clear (with a few sharp edges, noted in my detailed notes) and easy to follow. The experiments are sensible and the results are reasonably strong, particularly with respect to sample efficiency. \n\nMy primary concern with this paper would be that using pretrained features from classifiers in any way tends to strongly improve performance on *any metric which involves a pretrained classifier net,* (even one trained in a self-supervised fashion), but does not necessarily lead to actual sample quality improvements. E.g., for ImageNet GANs, this type of supervision leads to generating images which are more centred and emphasize the foreground, strongly improving both the rate of observed FID reduction and the final performance, but direct visual comparison makes it quite clear that the overall sample quality is not actually changed meaningfully.\n\nWhile I appreciate that the authors include multiple metrics aside from FID in the appendix, these metrics are still nonetheless based on pretrained features which are susceptible to this phenomenon. I think this manuscript would be made substantially more airtight if non-pretrained-net-based metrics were used: reporting LPIPS and the sliced Wasserstein distance would go a long way on this front. Normally I wouldn’t much care to see these metrics, but because of this specific concern I think it pertinent. Based on the presented samples I do not expect the relative ordering of the comparisons to change.  In particular it is clear that regardless of final performance, the sample complexity is sharply improved by this method, so this is to me more a matter of adhering to standards of scientific rigour by ensuring that one cannot explain away the improvements by appealing to the aforementioned confounding factor.\n\nUltimately I think this paper will be of interest to practitioners operating in low-resource (data or compute settings), but not of especial interest to the field of (deep) generative modeling at large. That’s not a negative per se, but it does bear thinning about when guessing at its potential impact at NeurIPS. I’m rating this paper around a 6.5, and I expect a healthy discussion phase; if the authors address my concerns wrt metrics I would likely be willing to move up to a solid 7.\n\n\nNotes:\n\n-Section 3 acknowledges that if the feature extractor tosses information then the generative modeling objective is changed; it would be good to devote slightly more discussion to this on a theoretical level, perhaps in future work. I appreciate the discussion in Section 6 which demonstrates artifacts and potential negative downsides of the approach in this context.\n\n“We observe that features at deeper layers are significantly harder to fit, as evidenced by our experiments in Section 4. We hypothesize that a discriminator can focus on a few descriptive feature channels while wholly disregarding others.”\n\nThis couplet needs a rework for several reasons. First, it seems the authors don’t mean that these features are harder to fit, but harder to *cover* (in the reverse KL sense); it sounds like these features are easier to overfit to. Second, this hypothesis is odd: focusing on a subset of the channels does not necessarily mean that D is overfitting, as it’s not only possible but highly likely that the feature space of any given pretrained network is not wholly utilized, so focusing just on a few channels may simply be because those channels are indeed the most informative. I think I understand the point the authors are trying to make, but they need to be careful and very specific with language here. Perhaps “focus on a subset of the feature space” would be a better choice\n\n-” We do not choose an inception network [59] to avoid strong correlations with the evaluation metric FID”\nWhile I understand the authors motivations here, this is misleading--using *any* classifier network will lead to strong correlations with the FID metric, regardless of if it is the same inception network used for the metric or not. This reviewer speaks directly from empirical experience, but evidence in the literature (that most classifier nets tend to learn approximately the same thing when trained on a given dataset) abounds. The supplementary provides several other metrics which should ameliorate this to some degree, but as mentioned in my main review the core issue is the use of classifier features to evaluate a model fit to classifier features, not the particular distance/divergence/Precision-recall curve used to evaluate the discrepancy between said features.\n\n\n-In Table 1, the rows are unlabeled, and even after spending quite a bit of time reading the text, the caption, and examining the table, I was unable to ascertain with certainty exactly what each row corresponds to. Please label the rows of table 1 or explicitly explain in clear and direct language what they represent. I *think* the table is only meant to show what happens when you apply discriminators to different feature layers (with the highlights indicating which of the seven stages in an EffNet are being used as features), but this is not explained and doesn’t explain the meaning of the different values as one goes down a column. The column labels for this table are also problematic: they’re labeled as “FD” but they actually represent the ratio of performance relative to an RGB only baseline; while this is explained in the text, this is impossible to infer from the table itself and further adds to the confusion.\n\n\n-The reduction in FD performance as discriminators are added to later layers is indeed interesting, but could potentially also be explained away by the known fact that (without contrary intervention) classifier networks tend to heavily rely on local structure and texture rather than global structure, and this is emphasized even in their deeper layers. While non-FID metrics are included in the appendix for the final models, it would be good to have them for this table as well.\n\n-The authors mention wallclock time in several places but do not provide direct comparisons in terms of per-step training latency. These should be measured and included: since the sample complexity of the proposed method appears to outstrip the baselines by a large margin, then even if the use of the pretrained model were to incur a modest relative compute overhead, this would still favor the authors. In other words, what is the value of training imgs/s on a given GPU for each of the different methods?  A small table in the supplementary showing this would suffice and shouldn’t require any additional experiments, just the authors checking their training logs.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "V1kx2f9IL18", "reviewer": "Reviewer_UArS", "summary": "The papers proposes a technique for faster training of GANs while achieving state-of-the-art FID scores on several datasets. The main idea is to use pre-trained features for the discriminator. Four improvements are recommended: 1) Multi-scale discriminators, 2) cross-channel mixing 3) cross-scale mixing 4) choice of pre-trained network.\n\nFID scores are shown for several datasets, CLEVR, FFHQ, Cityscapes, Bedroom and Church.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper presents very impressive FID results across numerous datasets. However, it is unclear why these improved results were achieved. It is possible the paper could be accepted if the results could be clarified, but in its current state it is below the bar for acceptance.\n\nPositives:\n\n\\++ Very good FID scores on many datasets.\n\n\\+ Very fast to train when compared to other approaches.\n\n\\+ Good job in calling out the approach's limitations.\n\nNegatives:\n\n\\- It is unclear why CCM would make any difference in the network. A linear transformation before a linear layer is equivalent to a single linear layer. Why would CCM help training? This is particularly confusing since the labels in Table 1a are missing.\n\n\\- For CSM, did the authors compare against a discriminator that has access to all feature layers? It is unclear if this would help if the discriminators weren't independent.\n\n\\- The y-axis labels are missing in Table 1a and 1b. Without these labels it is hard to determine what the numbers actually mean.\n\n\\- Why are the values in Table 2 different from Table1? I.e. the FID scores are >1 in Table 2 but below 0.5 in Table 1. Is it because training was done for 10M images instead of 1M? If so, why are the results worse for larger training set sizes?\n\n\\- Is the reason the FID scores are so high due to the discriminator features being essentially classifier features? If so, the use of FID scores is suspect for evaluation. To be convincing, it would be better to run human studies or another metric that indicates the generated images are actually better. It is not clear from the supplementary material that the images generated are actually improved.\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The paper did a great job of calling out their limitations in Section 6. No societal impact was provided beyond line 326. ", "review_text": "", "extra_scores": {"ethical_concerns": "None", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2.5 hours"}, "extra_text": {}}, {"review_id": "UnrttPn9Ywa", "reviewer": "Reviewer_CqSv", "summary": "This paper proposes to leverage pretrained image feature representations to improve GANs by training the discriminator on random projections of the pretrained features instead of directly on RGB images. This simple substitution results in improved image quality and diversity while also significantly reducing training time compared to existing state-of-the-art methods.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- Paper is technically sound. Authors do a good job of first motivating their solution by demonstrating that existing methods cannot leverage deep features from pretrained representations, and then demonstrate that their proposed solution successfully bypasses these limitation. \n- Ablation study compares several different SOTA feature representations.\n- Generation quality is evaluated on a wide variety of datasets, compared to several other SOTA GAN architectures using a variety of GAN evaluation metrics.\n- While the idea of leveraging pretrained feature representations to improve GAN training is certainly not new, the key insight that these representations need to be diluted in order to work well appears to be a novel contribution.\n- The reduction in training time afforded by the techniques proposed in this paper could have significant positive impact in several ways, such as: lower barrier to entry for those who would like to train high resolution GANs but don't have the compute budget, faster iteration time for researchers, and less energy consumption/carbon emissions for training models. This is very promising as one of the main challenges with GANs currently is their excessive training time. \n- The paper is nicely organized and well written for the most part. \n\nWeaknesses:\n- Some clarity issues - see below.\n\nUnfortunately, I had a very difficult time understanding Section 4.1 and Table 1. I was not entirely clear on the experimental setup, particularly in relation to the rows in Table 1, which I could not find descriptions for. \nThe discriminator architecture could also be described in more detail, even if only in the appendix. Currently the discriminator is described as \"a simple convolutional architecture with spectral normalization\" (lines 105-106), which is quite ambiguous. Considering that the discriminator is the focus of this paper, I think it is important to include sufficient detail to allow readers to replicate it. Releasing code would certainly help in this regard, but it is nice when a paper is self-contained.\n\nOther questions/comments:\n- I found it interesting that most of the performance improvement came in the form of better sample diversity (i.e. recall), rather than just image quality. I think this is a significant result and could be worth mentioning in the main paper rather than having it hidden in the appendix.\n- How important is it to use multiple discriminators? i.e., how well would a perceptual discriminator perform if combined with CCM? Or alternatively, how well would FastGAN or StyleGAN2 perform with a separate discriminator for each resolution? (This information may have been in Table 1 but I couldn't tell).\n- Considering that it was found that compact representations worked better than high performing classifier representations it would be interesting to test pretrained VGG, especially since it has been so successful for perceptual tasks in the past. (No need to run this experiment now, I am just curious).\n- Do you have any intuition as to why FastGAN benefits more from projection compared to StyleGAN2? ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Authors briefly mention benefits of lower computational requirements and the possibility of malicious use such as in deep fakes.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "6ciYVjiTsz", "reviewer": "Reviewer_gvfM", "summary": "The demonstrate that effectiveness of the results on a wide variety of datasets and outperform current SOTA on the several benchmarks. They show that they get faster convergence than competing works like StyleGAN2 and show excellent qualitative results on very small datasets like the provided pokemon.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "# Originality: \nI am not aware of any GANs that use pre-trained ImageNet embeddings in a GAN like so. Projects real and generated samples into a fix pre-trained feature space. There are some work that has used pretrained imagenet features to create knn clusters that are later used an auxilary classification task for the discriminator, but this approach seems sufficiently different to those approaches. The authors\n\n# Quality:\n Is the submission technically sound? Are claims well supported (e.g., by theoretical analysis or experimental results)? Are the methods used appropriate? Is this a complete piece of work or work in progress? Are the authors careful and honest about evaluating both the strengths and weaknesses of their work?\n\nMost of the claims in the paper are technically sound. The one issue I have is the entanglement between using ImageNet feature and entanglement with FID score. It's true they use a different architecture, EfficientNet than the InceptionNet does help a bit in that regard as stated in the paper. \"We do not choose an inception network [59] to avoid strong150correlations with the evaluation metric FID.\" Inherently, EfficientNet is trained on the same data as InceptionNet though, so I would prefer if a non-ImageNet trained feature space was used for evaluation. Even another similar dataset that could be use for classification like Google OpenImages, MS CoCo classification, etc. That would strengthen the results in the paper. I do appreciate that they include other metrics like SwAV-FID (although this should be moved to the main paper as it's the most compelling quantitative result. \n\nGiven the use of self-supervised FID assuages my concerns though. Also the Pokemon dataset results are pretty compelling. I'd also like to point that ImageNet and StyleGAN's architecture have similarly object centric biases (the objects are always in the center of the frame even in the Pokemon example). \n\n# Clarity: \nThe submission is fairly well written and the qualitative examples are very convincing. The paper is very well written and the supplement is detailed.\n\n\n# Significance:\nThe paper proposes a pre-trained feature layer to accelerate GAN training and improve results on small datasets. They show impressive results and demonstrates which techniques work and some that don't on guiding the GAN's using the feature spaces. It also shows how to utilize feature space like ImageNet on categories that are not included within. It also demonstrates some of the limitations of the approach like that fact that it actually hampers the results on certain datasets like FFHQ.\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"ethical_concerns": "No", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The submission discusses projected GANs, i.e., GANs which project generated and real samples into a fixed, pretrained feature space. The initial reviewer assessment is generally positive. The reviewers provided further clarifications in the rebuttal which clarified questions. Reviewers remained assured about their initial assessment. AC thinks that this is a valuable study that will be of interest.
Decision: Accept (Poster)