Supplementary Material for
Projected GANs Converge Faster1
2

Axel Sauer1,2 Kashyap Chitta1,2 Jens Müller3 Andreas Geiger1,2
University of Tübingen 2 Max Planck Institute for Intelligent Systems, Tübingen
3
Computer Vision and Learning Lab, University Heidelberg{firstname.lastname}@tue.mpg.de3{firstname.lastname}@iwr.uni-heidelberg.de

In this supplementary document, we first prove the theorem presented in the paper in Section 1.
Section 2 provides additional evaluation metrics for StyleGAN-ADA [12], FastGAN [20], andProjected GAN, and FID of Projected GAN on nine more datasets. Section 3 presents uncurated
samples for both baselines and our approach. Section 4 reports additional experiments. Lastly,
we provide details on training configurations, hyperparameters, and compute in Section 5. The
supplementary videos show interpolations between random samples of Projected GAN on all
datasets. Code, models, and supplementary videos can be found on the project page https://sites.
google.com/view/projected-gan.

1

ProofsAs described in the paper, Projected GAN training can be formulated as follows
X
min max
Ex [log Dl (Pl (x))] + Ez [log(1 − Dl (Pl (G(z))))]
G

{Dl }

(1)

l∈L

where {Dl } is a set of independent discriminators operating on different feature projections. In thefollowing, we first prove Theorem 1 for a deterministic projection. The second proof demonstrates
the theorem’s validity when training with stochastic differentiable augmentations.
Proof (deterministic). The following proof follows the consistency proofs in [23] and [7]. Let {Pl }l∈L
be a set of fixed feature projectors. Furthermore, let PT be the density of the true data distribution
and PG the density of the distribution the generator G produces. As in Theorem 1, Pl ◦ T and Pl ◦ G
are functional compositions of Pl and the true/generated data distribution. The minimax objective in(1) is then defined via
X
min max
Vl (Dl , G)
(2)
G

{Dl }

l∈L

where
Vl (Dl , G)

= Ex∼PT [log Dl (Pl (x))] + Ex∼PG [log (1 − Dl (Pl (x)))]= REy∼PPl ◦T [log Dl (y)] + Ey∼PPl ◦G [log (1 − Dl (y))]= y PPl ◦T (y) log(Dl (y)) + PPl ◦G (y) log(1 − Dl (y))dy(3)

In the following we derive the optimal discriminator for a fixed G. For any (a, b) ∈ R2 \ {(0, 0)}, theafunction t → a log(t) + b log(1 − t) obtains its maximum in [0, 1] at a+b[7]. Since the discriminators
do not need to be defined outside supp (PPl ◦T ) ∪ supp (PPl ◦G ), the maximum max{Dl } Vl (Dl , G)is achieved for
PPl ◦T (y)
∗
Dl,G
(y) =
(4)
PPl ◦T (y) + PPl ◦G (y)where G is fixed. Assuming a perfect discriminator, the minimax objective can be reformulated as
X
X∗
C(G) = max
Vl (G, Dl ) =Vl (G, Dl,G
)
(5)
{Dl }

l

l

35th Conference on Neural Information Processing Systems (NeurIPS 2021).

Following the arguments of [7] and in [23], we obtain
X
C(G) = −2|L| log(2) + 2
JSD(PPl ◦T ||PPl ◦G )(6)lwhere JSD is the Jensen-Shannon divergence. Since the Jensen-Shannon divergence is non-negativeand zero only in case of equality, the minimum is achieved iff PPl ◦T = PPl ◦G for all l. This shows,that we achieve minG C(G) iff PPl ◦T = PPl ◦G for all l.
Proof (stochastic). We now show that the result above still holds when applying stochastic differentiable augmentations before the feature projections. Utilizing a stochastic augmentation fθ,l before
the projection through Pl , can be viewed as a functional composition, i.e. Pθ,l = Pl ◦ fθ,l . The parameter θ ∼ PΘ encompasses both the probability of applying the augmentation and its parameters, e.g.,translation direction and magnitude. As in the deterministic case, the minimax objective is defined asX
min max
Vl (Dl , G)
(7)
G

where
Vl (Dl , G)

{Dl }

l∈L

= Ex∼PT [Eθ∼PΘ [log Dl (Pθ,l (x))]] + Ex∼PG [Eθ∼PΘ [log (1 − Dl (Pθ,l (x)))]]
= Eθ∼PΘ Ex∼PT [log Dl (Pθ,l (x))] + Ex∼PG [log (1 − Dl (Pθ,l (x)))]= Eθ∼PΘ Ey∼PPθ,l ◦T [log Dl (y)] + Ey∼PPθ,l ◦G [log (1 − Dl (y))]
R= Eθ∼PΘ y PPθ,l ◦T (y) log(Dl (y)) + PPθ,l ◦G (y) log(1 − Dl (y))dyR= y Eθ∼PΘ [PPθ,l ◦T (y)] log(Dl (y)) + Eθ∼PΘ [PPθ,l ◦G (y)] log(1 − Dl (y))dy(8)

Using the same arguments as above, we obtain that the maximum max{Dl } Vl (Dl , G) is achieved for
∗
Dl,G
(y) =

Eθ∼PΘ [PPθ,l ◦T (y)]Eθ∼PΘ [PPθ,l ◦T (y)] + Eθ∼PΘ [PPθ,l ◦G (y)](9)

where G is fixed. Note that Eθ∼PΘ [PPθ,l ◦T ] and Eθ∼PΘ [PPθ,l ◦G ] are densities. Similar to above, we
obtain minG C(G) iff Eθ∼PΘ,l [PPl ◦T ] = Eθ∼PΘ [PPθ,l ◦G ] for all l.

2Additional Metrics and Datasets

Metrics. In addition to FID and Imgs reported in the paper, we compute the following metrics:• Kernel Inception Distance (KID) [2]. KID is an unbiased alternative to FID, hence,
especially suitable for small datasets.
• Precision and Recall [28]. Precision measures the quality of samples, and recall measures
image diversity. Generally, GANs produce high-quality samples while being prone tomode collapse (high precision, low recall), compared to VAEs [16], which generate lower
quality but more diverse samples (low precision, high recall). This observation is evidenced
empirically in [28]. For our evalution, we use the improved formulation by [18].
• SwAV-FID [22]. Instead of utilizing an image classifier feature space, SwAV-FID uses selfsupervised representations. More specifically, SwAV-FID computes the Fréchet distancein the penultimate layer of a ResNet-50 trained with the contrastive SwAV objective [3].
Morozov et al. [22] show that in some cases, SwAV-FID is more consistent with humanjudgment of visual quality than FID.
• CLIP-FID & Virtex-FID. FID uses an Inception network trained on ImageNet. Our feature
network F has also been trained on ImageNet. To rule out training data as a confounding
factor between F and the evaluation metric, we propose to use FID with non-ImageNet
features. We evaluate CLIP-FID (using a ResNet50 trained with the CLIP objective on thedataset collected by [25]) and VirTex-FID (using a ResNet50 trained on COCO Captions
with the VirTex objective [5]).
• Sliced Wasserstein Distance (SWD). SWD is a non-neural metric that computes the
Wasserstein distance between local image patches drawn from a Laplacian pyramid. We
follow the evaluation protocol proposed by [11].
2In addition to the metrics above, we conduct a human preference study with 28 participants
unfamiliar with our method. The structure of the study is as follows:• For each dataset, we first show samples of the real dataset for context.
• We then present two sample sheets and ask the participants to rank the sheets relative toeach other based on sample fidelity and diversity. We define these as follows: (i) Fidelity is
the degree to which the generated samples resemble the real ones. (ii) Diversity measures
whether the generated samples cover the full variability of the real samples.
• Each sheet contains nine random samples; we present three sample sheet pairs per dataset.
• Samples and comparison pairs are randomized per participant. We make sure that all possible
pairings are equally represented.
• Evaluated Models: StyleGAN2-ADA, FastGAN, Projected GAN, and real data (for control)• Evaluated datasets: all 256x256 datasets• We count how often a given model wins a comparison and report the relative amount of wins.
Results. On all datasets, KID, SwaV-FID, CLIP-FID, VirTex-FID, and SWD mirror the ranking
obtained via FID. The low SwAV-FIDs indicate that Projected GAN’s low FIDs are not due tocorrelations between the feature network used for projection and the inception network used in FID.
On the large datasets, the baselines only outperform Projected GAN in precision on FFHQ, Cityscapes,and LSUN Church (Table 1). However, when comparing recall in these cases, it is apparent that thebaselines suffer from mode collapse. The high recall generally obtained by Projected GAN hints at
the reason for its superior performance on FID, KID, and SwAV-FID: the generated images are very
diverse. Hence, we conclude that Projected GANs alleviate mode collapse. The sample diversity isalso evident in the qualitative comparisons in Section 3. On small datasets (Table 2), Projected GAN
is outperformed in precision on Art Painting by FastGAN; however, FastGANs very low recall of
0.044 hints at mode collapse. Only on flowers, Projected GAN appears to cover fewer modes thanthe baselines as indicated by lower recall.
At high resolutions (Table 3), Projected GAN performs slightly worse in precision. It appears that
Projected GAN incurs small losses in image quality while obtaining a better mode coverage, which
can be observed in the quantitative comparisons, e.g., on AFHQ-Cat, some samples exhibit artifacts.
These artifacts indicate that projected GAN training at higher resolutions warrants closer inspection.
On small datasets, overfitting is a problem that is not detected well by FID and other metrics [27].
Therefore, it is instructive to inspect latent interpolations for which we refer the reader to the
supplementary videos. Projected GAN generates smooth interpolations between random samples on
all datasets suggesting that it generalizes rather than memorizing training samples.
The results of the human preference study are shown in Table 4. The study results largely agree withthe results obtained via FID. On FFHQ, the study demonstrates our reported failure case for projected
GANs. Interestingly, on AnimalFace projected GAN outperforms real data. We hypothesize that thisis because for AnimalFace there is a significant portion of low-quality images (blurry, compression
artifacts) in the dataset, and possibly projected GAN generates fewer of those. Of course, humanstudies are not optimal, as it is not straightforward to evaluate sample diversity - which is a strong
suit of projected GANs - given only a few samples.
Table 5 reports the FID achieved by Projected GAN for nine more datasets, all at a resolution of
2562 . We compare on LSUN cat and horse [31], ADE indoor (a subset of ADE [34] proposed in [1]),the full Oxford flowers dataset with 8k images [24], KITTI-fisheye (a subset of KITTI-360 [19],consisting of fisheye images), STL-10 [4], CUB200 [30], Stanford Dogs [14], and Stanford Cars [17].
We do not change the hyperparameters of Projected GAN. On each dataset, we report the lowest FID
achieved in previous literature. We train FastGAN as a baseline for ADE indoor and KITTI-fisheye.

3

Large Datasets (2562 )
CLEVR

FFHQ

Cityscapes

Bedroom

Church

11.53
8.24
1.52

5.85
8.43
1.59

7.42
5.90
0.36

4.70
4.61
0.50

0.429
0.602
0.614

0.565
0.645
0.612

0.202
0.189
0.346

0.416
0.207
0.438

8.47
5.38
1.44

2.51
3.64
1.01

42.12
31.10
11.97

15.85
35.47
13.71

2.20
2.66
0.81

1.10
3.61
0.82

12.53
14.68
6.83

14.62
14.42
8.37

F ID ↓S TYLE GAN2-ADA [12]FAST GAN [20]P ROJECTED GAN

10.17
3.24
0.89

7.32
12.69
3.08

8.35
8.78
3.41
KID × 103 ↓S TYLE GAN2-ADA [12]FAST GAN [20]P ROJECTED GAN

8.15
2.64
0.51

1.49
5.34
0.44

3.34
5.45
0.91
P recision ↑S TYLE GAN2-ADA [12]FAST GAN [20]P ROJECTED GAN

0.373
0.600
0.640

0.669
0.716
0.654

0.649
0.557
0.619
Recall ↑S TYLE GAN2-ADA [12]FAST GAN [20]P ROJECTED GAN

0.569
0.650
0.735

0.445
0.184
0.464

0.146
0.227
0.361
SwAV − F ID ↓S TYLE GAN2-ADA [12]FAST GAN [20]P ROJECTED GAN

3.50
1.46
0.56

1.24
2.55
0.85

1.35
1.29
0.60
CLIP − F ID ↓S TYLE GAN2-ADA [12]FAST GAN [20]P ROJECTED GAN

4.70
4.24
0.80

10.3
19.23
7.55

5.88
6.46
2.96
V irT ex − F ID ↓S TYLE GAN2-ADA [12]FAST GAN [20]P ROJECTED GAN

0.78
0.64
0.35

1.20
2.47
0.64

1.15
1.48
0.49
SW D × 10−3 ↓S TYLE GAN2-ADA [12]FAST GAN [20]P ROJECTED GAN

17.50
28.51
12.90

7.42
10.19
6.41

10.71
9.45
7.27

Table 1: Metrics on Large Datasets (2562 ). Projected GAN compares favorably on most metrics.
Exceptions are precision on FFHQ, Cityscapes, and LSUN Church. As argued by [13], shifting fromprecision to recall is generally desirable, since recall can be traded into precision via truncation.

4

Small Datasets (2562 )Art
Painting

Landscape AnimalFace Flowers

PokemonF ID ↓S TYLE GAN2-ADA [12]FAST GAN [20]P ROJECTED GAN

43.07
44.02
27.96

15.99
16.44
6.92

60.90
62.11
17.88

21.66
26.23
13.86

40.38
81.86
26.36

3.56
6.61
0.38

13.49
80.30
1.32

0.731
0.611
0.816

0.735
0.731
0.809

0.095
0.100
0.058

0.197
0.004
0.259

5.02
7.45
2.70

6.71
9.25
2.04

26.30
40.12
15.83

13.96
87.65
9.93

3.25
4.08
2.19

3.69
17.49
2.55

14.04
17.39
9.61

14.73
46.81
11.65

KID × 103 ↓S TYLE GAN2-ADA [12]FAST GAN [20]P ROJECTED GAN

10.23
13.00
1.25

4.39
3.40
1.30

22.52
22.11
0.03
P recision ↑S TYLE GAN2-ADA [12]FAST GAN [20]P ROJECTED GAN

0.691
0.858
0.762

0.709
0.768
0.774

0.841
0.849
0.998
Recall ↑S TYLE GAN2-ADA [12]FAST GAN [20]P ROJECTED GAN

0.218
0.044
0.239

0.213
0.160
0.258

0.036
0.015
0.095
SwAV − F ID ↓S TYLE GAN2-ADA [12]FAST GAN [20]P ROJECTED GAN

3.32
3.29
2.25

2.98
2.42
1.42

16.26
15.07
4.22
CLIP − F ID ↓S TYLE GAN2-ADA [12]FAST GAN [20]P ROJECTED GAN

44.13
40.47
22.91

24.89
19.84
13.71

46.18
54.69
16.89
V irT ex − F ID ↓S TYLE GAN2-ADA [12]FAST GAN [20]P ROJECTED GAN

4.15
5.72
3.53

2.78
3.86
1.98

8.83
9.41
3.79
SW D × 10−3 ↓S TYLE GAN2-ADA [12]FAST GAN [20]P ROJECTED GAN

25.55
21.94
11.44

19.06
29.87
15.38

22.31
29.23
14.34

Table 2: Metrics on Small Datasets (2562 ). Projected GAN performs best on most metrics.

5

10242
Art
Painting

5122
AHFQCatPokemon

AFHQDog

AFHQWild

7.40
13.09
4.52

3.05
3.14
2.17

1.21
5.51
0.80

0.47
0.74
0.12

0.753
0.746
0.718

0.765
0.761
0.705

0.470
0.380
0.643

0.137
0.201
0.292

1.98
3.07
1.12

1.89
1.77
1.08

F ID ↓S TYLE GAN2-ADA [12]FAST GAN [20]P ROJECTED GAN

41.69
46.71
32.07

56.76
56.46
33.96

3.55
4.69
2.16
KID × 103 ↓S TYLE GAN2-ADA [12]FAST GAN [20]P ROJECTED GAN

26.59
12.70
1.70

15.31
29.40
7.76

0.63
1.72
0.16
P recision ↑S TYLE GAN2-ADA [12]FAST GAN [20]P ROJECTED GAN

0.619
0.776
0.706

0.791
0.777
0.780

0.767
0.784
0.693
Recall ↑S TYLE GAN2-ADA [12]FAST GAN [20]P ROJECTED GAN

0.168
0.033
0.235

0.053
0.080
0.215

0.411
0.305
0.565
SwAV − F ID ↓S TYLE GAN2-ADA [12]FAST GAN [20]P ROJECTED GAN

3.68
3.41
2.28

5.03
5.14
3.83

1.23
1.73
0.68

Table 3: Metrics on Small Datasets (5122 and 10242 ). The results are in line with the findings at aresolution of 2562 . Only with respect to precision, the baselines slightly outperform Projected GAN.

CLEVR

FFHQ

Cityscapes

Bedroom

ChurchArtPainting

LandscapeAnimalFace

Flowers

PokemonAll Datasets

21 %
8%
17 %
54 %

4%
7%
55 %
34 %

17 %
4%
25 %
54 %

10 %
0%
38 %
52 %

15 %
6%
25 %
54 %

25 %
13 %
16 %
46 %

9%
9%
49 %
33 %

17 %
17 %
29 %
37 %

6%
0%
40 %
54 %

14 %
7%
27 %
52 %

Fidelity
S TYLEGAN 2-ADA [12]FAST GAN [20]P ROJECTED GAN
DATA

14 %
7%
42 %
37 %

32 %
2%
5%
61 %

17 %
15 %
15 %
53 %

5%
3%
16 %
76 %

16 %
9%
9%
66 %

16 %
0%
28 %
56 %

S TYLEGAN 2-ADA [12]FAST GAN [20]P ROJECTED GAN
DATA

13 %
13 %
31 %
43 %

24 %
4%
12 %
60 %

9%
11 %
21 %
59 %

10 %
2%
21 %
67 %

19 %
0%
14 %
67 %

11 %
4%
27 %
58 %

Diversity

Table 4: Human Preference Study. The obtained results largely agree with the rankings of othermetrics.

6

Prev. SotA
(Approach)
Projected GAN

Prev. SotA
(Approach)
Projected GAN

LSUN cat

LSUN horse

ADE indoor

Flowers Full

KITTI fisheye

5.57(ADM [6])2.57(ADM [6])30.33(FastGAN [20])19.60(MSG-StyleGAN [10])6.64(FastGAN [20])

3.89

2.17

6.70

3.86

2.72

STL-10

CUB200Stanford Dogs

Stanford Cars

25.32
(TransGAN [9])11.25
(FineGAN [29])25.66(FineGAN [29])16.03(FineGAN [29])

13.68

2.79

11.75

2.09

Table 5: FID on Additional Datasets (2562 ) Without any hyperparameter changes, Projected GAN
outperforms the previous state-of-the art on all evaluated datasets.

7

3

Qualitative Comparisons

We show generated images for CLEVR (Fig. 1), FFHQ (Fig. 2), Cityscapes (Fig. 3), Bedroom(Fig. 4), Church (Fig. 5), Art Painting (Fig. 6, 11), Landscape (Fig. 7), AnimalFace Dog (Fig. 8),
Flowers (Fig. 9), Pokemon (Fig. 10, 12), AFHQ-Cat (Fig. 13), AFHQ-Dog (Fig. 14), and AFHQ-Wild(Fig. 15). Following [12], we select a global seed per dataset. We do not perform truncation on anyof the models. Projected GAN produces convincing results on all datasets. The sample diversity inparticular is apparent in comparison to the baselines, e.g., on AFHQ-Cat or AFHQ-Wild, all baselines
generate high-quality samples, but Projected GAN captures more variability of the training data.

StyleGAN2-ADA

FastGAN

Projected GAN (ours)

FID 10.17 – Recall 0.569

FID 3.24 – Recall 0.650

FID 0.89 – Recall 0.735

2Figure 1: Uncurated Results for CLEVR (256 ). The images are selected randomly given one
global random seed. We recommend zooming in for comparison.

StyleGAN2-ADA

FastGAN

Projected GAN (ours)

FID 7.32 – Recall 0.445

FID 12.69 – Recall 0.184

FID 3.39 – Recall 0.464

2Figure 2: Uncurated Results for FFHQ (256 ). The images are selected randomly given one global
random seed. We recommend zooming in for comparison.
8

StyleGAN2-ADA

FastGAN

Projected GAN (ours)

FID 8.35 – Recall 0.146

FID 8.78 - Recall 0.227

FID 3.41 - Recall 0.361

Figure 3: Uncurated Results for Cityscapes (2562 ). The images are selected randomly given one
global random seed. We recommend zooming in for comparison.

StyleGAN2-ADA

FastGAN

Projected GAN (ours)

FID 11.53 – Recall 0.202

FID 8.24 - Recall 0.189

FID 1.52 - Recall 0.346

Figure 4: Uncurated Results for LSUN bedroom (2562 ). The images are selected randomly given
one global random seed. We recommend zooming in for comparison.

StyleGAN2-ADA

FastGAN

Projected GAN (ours)

FID 5.85 – Recall 0.416

FID 8.43 - Recall 0.207

FID 1.59 - Recall 0.438

Figure 5: Uncurated Results for LSUN church (2562 ). The images are selected randomly given
one global random seed. We recommend zooming in for comparison.

9StyleGAN2-ADA

FastGAN

Projected GAN (ours)

FID 43.07 – Recall 0.218

FID 44.02 - Recall 0.044

FID 27.96 - Recall 0.239

Figure 6: Uncurated Results for Art Painting (2562 ). The images are selected randomly given one
global random seed. We recommend zooming in for comparison.

StyleGAN2-ADA

FastGAN

Projected GAN (ours)

FID 15.99 – Recall 0.213

FID 16.44 - Recall 0.160

FID 6.92 - Recall 0.258

2Figure 7: Uncurated Results for Landscape (256 ). The images are selected randomly given one
global random seed. We recommend zooming in for comparison.

10

StyleGAN2-ADA

FastGAN

Projected GAN (ours)

FID 60.90 – Recall 0.036

FID 62.11 - Recall 0.015

FID 17.88 - Recall 0.095

Figure 8: Uncurated Results for AnimalFace-Dog (2562 ). The images are selected randomly given
one global random seed. We recommend zooming in for comparison.

StyleGAN2-ADA

FastGAN

Projected GAN (ours)

FID 21.66 – Recall 0.095

FID 26.23 - Recall 0.100

FID 13.86 - Recall 0.058

Figure 9: Uncurated Results for Flowers (2562 ). The images are selected randomly given one
global random seed. We recommend zooming in for comparison.

11

StyleGAN2-ADA

FastGAN

Projected GAN (ours)

FID 40.38 – Recall 0.197

FID 81.86 - Recall 0.004

FID 26.36 - Recall 0.259

Figure 10: Uncurated Results for Pokemon (2562 ). The images are selected randomly given one
global random seed. We recommend zooming in for comparison.

StyleGAN2-ADA

FastGAN

Projected GAN (ours)

FID 41.69 – Recall 0.168

FID 46.71 - Recall 0.033
2

FID 32.07 - Recall 0.235

Figure 11: Uncurated Results for Art Painting (1024 ). The images are selected randomly given
one global random seed. We recommend zooming in for comparison.

12

StyleGAN2-ADA

FastGAN

Projected GAN (ours)

FID 56.76 – Recall 0.053

FID 56.46 - Recall 0.080

FID 33.96 - Recall 0.215

Figure 12: Uncurated Results for Pokemon (10242 ). The images are selected randomly given one
global random seed. We recommend zooming in for comparison.

StyleGAN2-ADA

FastGAN

Projected GAN (ours)

FID 3.55 – Recall 0.411

FID 4.69 - Recall 0.305

FID 2.16 - Recall 0.565

2Figure 13: Uncurated Results for AFHQ-Cat (512 ). The images are selected randomly given one
global random seed. We recommend zooming in for comparison.

13

StyleGAN2-ADA

FastGAN

Projected GAN (ours)

FID 7.40 – Recall 0.470

FID 13.09 - Recall 0.380

FID 4.52 - Recall 0.643

Figure 14: Uncurated Results for AFHQ-Dog (5122 ). The images are selected randomly given one
global random seed. We recommend zooming in for comparison.

StyleGAN2-ADA

FastGAN

Projected GAN (ours)

FID 3.05 – Recall 0.137

FID 3.14 - Recall 0.201

FID 2.17 - Recall 0.292

Figure 15: Uncurated Results for AFHQ-Wild (5122 ). The images are selected randomly given
one global random seed. We recommend zooming in for comparison.

14

4

Additional Experiments

This section presents additional experiments referenced in the paper. The experiments explore
alternative setups to our final configuration entailing a pretrained, fixed feature network F , fixed 1x1
convolutions for cross-channel mixing (CCM), and fixed convolutions for cross-scale mixing (CSM).
We follow the setup of Section 4 of the paper: training on LSUN church at a resolution of 2562 , witha batch size of 64 for 1 million Imgs, four discriminators, and a EfficientNet-Lite1 as feature network
F . Again, we report FID normalized by the FID obtained by a model with a standard single RGB
image discriminator. Values > 1 indicate worse performance than the RGB baseline. The results are
summarized in Table 6.
No Projection. In all experiments, we utilize pretrained representations. As a sanity check, it is instructive to test if the architectural bias of F alone is helpful. The results in Table 6 demonstrate that randomly initializing F results in much higher FID.
CCM. We explore three different options for CCM. The first
option, Feature Norm, does not mix features; rather, it normalizes the features to exhibit zero mean and a standard deviationof 1. This option investigates the importance of input scaling.
The two other options utilize random convolution with different initializations. CCM-rotation is initialized with a random
rotation matrix, CCM-Kaiming utilizes Kaiming initialization.
As shown in Table 6, CCM-Kaiming (0.77) improves over thebaselines (Feature Norm (1.27), CCM-rotation (1.27)), over the
RGB baseline (1.0), and pretrained F without projection (1.15).
This supports our hypothesis, that we need a sufficient amountof channel mixing.

Configuration

FID

No Projection
Random F
Pretrained F

11.03
1.15

CCM
Feature Norm
CCM-rotation
CCM-Kaiming

1.27
1.27
0.77

CCM + CSM
Denoising AE (t0 )Denoising AE (t1 )Denoising AE (t2 )Denoising AE (t3 )

0.24
0.37
0.44
0.59

Train F , Train Prand 3.09
CSM. We investigate if training the random projections Prand
Fixed F , Train Prand 0.97
in CCM and CSM is advantageous for FID. We consider two
Fixed F , Fixed Prand 0.24
options: training Prand before or during GAN training. First,
we train Prand before GAN training, the feature network F
Table 6: Ablations.
remains fixed. For this purpose, we add a head on the last CSM
layer to map back to full resolution and three output channels, forming an autoencoder architecture.
This model trains with a denoising autoencoder loss on ImageNet: the input image is augmented
with gaussian blur, JPEG compression, coarse and fine region dropout, and conversion to grayscale.
All augmentations are applied with a probability of 0.5. The target for reconstruction is the nonaugmented image. We keep four models, at the beginning of training (t0 ), at convergence (t3 ), andtwo in between with equal spacing in terms of reconstruction loss. After autoencoder training, we
use the model (without the additional head) for projected GAN training, denoted as Denoising AE(ti ). Interestingly, the longer the AE trained, the higher the FID, see Table 6. For the second ablation,different parts of the projection stay fixed during GAN training: (i) training both F and Prand , (ii)
fixed F , training only Prand , (iii) both F and Prand stay fixed. Again, the results in Table 6 suggestthat training any part of the projection results in worse performance. We conclude that training therandom projection is not advantageous, neither before nor during GAN training.
The signed real logits of the discriminator sign(D(x)) are the portion of the training set thatgets positive discriminator outputs. Karras et al. [12] find this a helpful heuristic for quantifying
discriminator overfitting. The signed logits should remain constant, which they achieve via adapting
the augmentation probability during training. Fig. 16 shows that the logits of the RGB baseline
steadily increase throughout training, whereas the logits remain mostly constant for Projected GAN
training. This observation coincides with the finding that adaptive augmentation is unnecessary for
Projected GANs as the logits are already stable.

15

0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0
-0.1
-0.2
-0.3
0.0 M

0.2 M

0.4 M

0.6 M

0.8 M

1.0 M

1.2 M

1.4 M

Imgs

Figure 16: Signed Discriminator Logits. For this experiment, we project through F and train withup to four discriminators; we leave the augmentation probability constant. (|Di | = 1: red, |Di | = 2:
blue, |Di | = 3: pink, |Di | = 4: green, RGB baseline: orange). For projected GAN training, thelogits remain stable throughout training.

5

Implementation Details

This section highlights the codebases we used and details hyperparameters and training configurations.
5.1

Code and Compute

For dataset preparation, training, and evaluation, we build on top of the Stylegan2-ADA codebase1 .
The evaluation employs the official pretrained Inception network to compute FID and KID. For
SwAV-FID, we integrate the model of the SwaV-FID codebase2 using weights by [3]. SAGAN [32]and Gansformers [8] are trained in the Gansformers codebase3 , we use their default hyperparameters.
For the feature networks’ implementation and weights, we use timm4 , except for R50-CLIP5 . Lastly,
we utilize official implementations for differentiable augmentation6 and FastGAN7 .
We conduct our experiments on an internal cluster with several nodes, each with up to 8 Quadro RTX
6000 or NVIDIA V100 using PyTorch 1.7.1 and CUDA 11.0.
5.2

Wall Clock-Time

On images at resolution 2562 , the wall-clock training times
Modelsec/kimg
measured in sec/kimg using 8 Quadro RTX 6000 are shown in
5.6
Table 7. StyleGAN2 is the fastest overall, which is expected as S TYLE GAN2-ADA
7.2
we enable mixed-precision and use the custom CUDA kernels FAST GAN
10.1
provided by the authors. These are not available for the Fast- P ROJECTED GAN
GAN generator; hence, Projected GAN can only be compared
Table 7: Training Speed.
to FastGAN in a fair manner. FastGANs wall-clock times arehigher because it uses a reconstruction loss on the discriminator features. This reconstruction loss
adds computational overhead. In contrast, projected GANs exhibit lower wall-clock times as we do
not need any regularization other than spectral normalization.
5.3

Hyperparameters

Below, we describe the hyperparameter search for each method. For optimization, we always use
Adam [15] with β1 = 0, β2 = 0.99, and  = 10−8 . All models employ exponential moving averagefor the generator weights [11].
StyleGAN2-ADA. The official codebase supplies standard configurations of architectures and hyperparameters for different resolutions. Furthermore, an automatic configuration option is available,
1

https://github.com/NVlabs/stylegan2-ada-pytorch
https://github.com/stanis-morozov/self-supervised-gan-eval/
3
https://github.com/dorarad/gansformer
4
https://github.com/rwightman/pytorch-image-models
5
https://github.com/openai/CLIP6
https://github.com/mit-han-lab/data-efficient-gans
7
https://github.com/odegeasslbc/FastGAN-pytorch
2

16entailing several heuristics for different hyperparameters. We find the automatic option very robust
for both architecture and most hyperparameters. The exception is the R1 gradient penalty [21], which
is highly dependent on the dataset. For all large datasets, the Gansformer codebase suggests suitable values for StyleGAN2. For the small datasets at 2562 and 10242 , we perform a grid search overγ ∈ {1, 10, 20, 50}. We first train for 1 M Imgs, then continue training only the best one. For all
AFHQ datasets, we use the same setting as [12]. All experiments employ adaptive discriminatoraugmentation with the default target value of 0.6.
FastGAN. We replicate the generator and discriminator architecture of the official FastGAN codebase.
FastGAN is robust to most hyperparameters; we always use a learning rate of 0.0002 and train with ahinge loss. The only sensitive hyperparameter with direct impact on performance is the batch size.
Interestingly, FastGAN profits of smaller batch sizes. The default suggested by [20] is a batch sizeof 8. We conduct a search over 8, 16, 32, and 64, a batch size of 16 further improves the results, whilelarger batch sizes decrease performance and even result in divergence in some cases. We employ
differentiable augmentation [33] of color, translation, and cutout.
Projected GAN. We use the same architecture, learning rate (0.0002), batch size (64), and hinge
loss for all experiments at all resolutions. Compared to FastGAN, we see a slight improvement
when increasing model capacity; we double the channel count in each dimension, from a basevalue of 64 to 128. The multipliers for the base value are as follows (resolution: multiplier):
42 : 16, 82 : 8, 162 : 4, 322 : 2, 642 : 2, 1282 : 1, 2562 : 0.5, 5122 : 0.25, 10242 : 0.125. We did notobserve similar improvements for FastGAN when increasing capacity. We employ differentiable
augmentation [33] of color, translation, and cutout. Lastly, to extract feature maps of intermediate
layers of the feature networks, both CNNs and visual transformers, we follow the protocol presented
in the MiDAS [26] codebase8 . For all EfficientNets and ResNets, we use features at spatial resolutionsr = {642 , 322 , 162 , 82 }, for DeiT and ViT, we use layers l = {3, 6, 9, 12}. The CSM blocks follow a
residual design typically used in architectures for dense prediction [26]. The lower-resolution feature
is passed through a residual 3x3 convolution block; the higher-resolution feature is added, followed
by a second residual block and bilinear upsampling, followed by a 1x1 convolution.
The discriminator architectures are shown in Table 8 where ni are the channels of the different feature
network stages, cin and cout are the input and output channels of the DownBlock DB. A DownBlockconsists of a convolution with k size k = 4 and stride s = 2, BatchNorm, and LeakyReLU with aslope of 0.2. We apply spectral normalization on all convolution layers Conv.
Discriminator L1

Discriminator L2

Discriminator L3

Discriminator L4

DB(ci n = c1 , co ut = 64)
DB(cin = 64, cout = 128)
DB(cin = 128, cout = 256)
DB(cin = 256, cout = 512)
Conv(cin = 512, cout = 1, k = 4)

DB(cin = c2 , cout = 128)
DB(cin = 128, cout = 256)
DB(cin = 256, cout = 512)
Conv(cin = 512, cout = 1, k = 4)

DB(cin = c3 , cout = 256)
DB(cin = 256, cout = 512)
Conv(cin = 512, cout = 1, k = 4)

DB(cin = c4 , cout = 512)
Conv(cin = 512, cout = 1, k = 4)

Table 8: Discriminator Architectures.

Acknowledgments and Disclosure of Funding
We acknowledge the financial support by the BMWi in the project KI Delta Learning (project
number 19A19013O). Andreas Geiger was supported by the ERC Starting Grant LEGO-3D(850533). Kashyap Chitta was supported by the German Federal Ministry of Education and Research (BMBF): Tübingen AI Center, FKZ: 01IS18039B and the International Max Planck Research School for Intelligent Systems (IMPRS-IS). Jens Müller received funding by the Heidelberg Collaboratory for Image Processing (HCI). We thank the Center for Information Services
and High Performance Computing (ZIH) at Dresden University of Technology for generous allocations of computation time. Lastly, we would like to thank Vanessa Sauer for her general support.

8

https://github.com/intel-isl/MiDaS

17

References[1] S. Azadi, M. Tschannen, E. Tzeng, S. Gelly, T. Darrell, and M. Lucic. Semantic bottleneck
scene generation. arXiv.org, 2019. 3[2] M. Binkowski, D. J. Sutherland, M. Arbel, and A. Gretton. Demystifying MMD gans. In Proc.
of the International Conf. on Learning Representations (ICLR), 2018. 2[3] M. Caron, I. Misra, J. Mairal, P. Goyal, P. Bojanowski, and A. Joulin. Unsupervised learning ofvisual features by contrasting cluster assignments. arXiv.org, 2006.09882, 2020. 2, 16[4] A. Coates, A. Ng, and H. Lee. An analysis of single-layer networks in unsupervised featurelearning. In Proc. of the Conf. on Artificial Intelligence (AAAI), 2011. 3[5] K. Desai and J. Johnson. Virtex: Learning visual representations from textual annotations. In
Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2021. 2[6] P. Dhariwal and A. Nichol. Diffusion models beat gans on image synthesis. arXiv.org, 2021. 7[7] I. J. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. C. Courville,
and Y. Bengio. Generative adversarial nets. In Advances in Neural Information Processing
Systems (NIPS), 2014. 1, 2
[8] D. A. Hudson and C. L. Zitnick. Generative adversarial transformers. arXiv.org, 2103.01209,
2021. 16
[9] Y. Jiang, S. Chang, and Z. Wang. Transgan: Two transformers can make one strong gan.
arXiv.org, 2021. 7[10] A. Karnewar and O. Wang. Msg-gan: Multi-scale gradients for generative adversarial networks.
In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2020. 7[11] T. Karras, T. Aila, S. Laine, and J. Lehtinen. Progressive growing of GANs for improved quality,
stability, and variation. In Proc. of the International Conf. on Learning Representations (ICLR),
2018. 2, 16[12] T. Karras, M. Aittala, J. Hellsten, S. Laine, J. Lehtinen, and T. Aila. Training generative
adversarial networks with limited data. In Advances in Neural Information Processing Systems(NeurIPS), 2020. 1, 4, 5, 6, 8, 15, 17[13] T. Karras, S. Laine, M. Aittala, J. Hellsten, J. Lehtinen, and T. Aila. Analyzing and improving
the image quality of StyleGAN. 2020. 4[14] A. Khosla, N. Jayadevaprakash, B. Yao, and F.-F. Li. Novel dataset for fine-grained image
categorization: Stanford dogs. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition(CVPR) Workshops, 2011. 3[15] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. In Proc. of theInternational Conf. on Learning Representations (ICLR), 2015. 16[16] D. P. Kingma and M. Welling. Auto-encoding variational bayes. arXiv.org, abs/1312.6114,
2013. 2[17] J. Krause, M. Stark, J. Deng, and L. Fei-Fei. 3D object representations for fine-grained
categorization. In Proc. of the IEEE International Conf. on Computer Vision (ICCV) Workshops,
2013. 3[18] T. Kynkäänniemi, T. Karras, S. Laine, J. Lehtinen, and T. Aila. Improved precision and recall
metric for assessing generative models. Advances in Neural Information Processing Systems(NeurIPS), 2019. 2[19] Y. Liao, J. Xie, and A. Geiger. Kitti-360: A novel dataset and benchmarks for urban scene
understanding in 2d and 3d. arXiv.org, 2109.13410, 2021. 3[20] B. Liu, Y. Zhu, K. Song, and A. Elgammal. Towards faster and stabilized gan training for highfidelity few-shot image synthesis. In Proc. of the International Conf. on Learning Representations (ICLR), 2021. 1, 4, 5, 6, 7, 17[21] L. Mescheder, A. Geiger, and S. Nowozin. Which training methods for gans do actuallyconverge? In Proc. of the International Conf. on Machine learning (ICML), 2018. 17[22] S. Morozov, A. Voynov, and A. Babenko. On self-supervised image representations for GAN
evaluation. In Proc. of the International Conf. on Learning Representations (ICLR), 2021. 2[23] B. Neyshabur, S. Bhojanapalli, and A. Chakrabarti. Stabilizing gan training with multiple
random projections. arXiv.org, 1705.0783, 2017. 1, 2
[24] M.-E. Nilsback and A. Zisserman. Automated flower classification over a large number ofclasses. In Proc. Indian Conf. on Computer Vision, Graphics & Image Processing, 2008. 3
18

[25] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell,
P. Mishkin, J. Clark, et al. Learning transferable visual models from natural language supervision.
arXiv.org, 2103.00020, 2021. 2[26] R. Ranftl, K. Lasinger, D. Hafner, K. Schindler, and V. Koltun. Towards robust monoculardepth estimation: Mixing datasets for zero-shot cross-dataset transfer. IEEE Trans. on PatternAnalysis and Machine Intelligence (PAMI), 2020. 17[27] E. Robb, W.-S. Chu, A. Kumar, and J.-B. Huang. Few-shot adaptation of generative adversarial
networks. arXiv.org, 2010.11943, 2020. 3[28] M. S. Sajjadi, O. Bachem, M. Lucic, O. Bousquet, and S. Gelly. Assessing generative models via
precision and recall. In Advances in Neural Information Processing Systems (NeurIPS), 2018. 2[29] K. K. Singh, U. Ojha, and Y. J. Lee. Finegan: Unsupervised hierarchical disentanglement forfine-grained object generation and discovery. In Proc. IEEE Conf. on Computer Vision and
Pattern Recognition (CVPR), 2019. 7[30] C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie. The Caltech-UCSD Birds-2002011 Dataset. Technical Report CNS-TR-2011-001, California Institute of Technology, 2011. 3[31] F. Yu, A. Seff, Y. Zhang, S. Song, T. Funkhouser, and J. Xiao. Lsun: Construction of a largescale image dataset using deep learning with humans in the loop. arXiv.org, 1506.03365, 2015. 3[32] H. Zhang, I. Goodfellow, D. Metaxas, and A. Odena. Self-attention generative adversarial
networks. In Proc. of the International Conf. on Machine learning (ICML), 2019. 16[33] S. Zhao, Z. Liu, J. Lin, J.-Y. Zhu, and S. Han. Differentiable augmentation for data-efficientgan training. In Advances in Neural Information Processing Systems (NeurIPS), 2020. 17[34] B. Zhou, H. Zhao, X. Puig, S. Fidler, A. Barriuso, and A. Torralba. Scene parsing through ade20k
dataset. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2017. 3

19