Supplementary Material for
StyleGAN-XL: Scaling StyleGAN to Large Diverse Datasets
Axel Sauer

Katja Schwarz

Andreas Geiger

University of Tübingen
Germanya.sauer@uni-tuebingen.de

University of Tübingen
Germany
katja.schwarz@uni-tuebingen.de

University of Tübingen
Germanya.geiger@uni-tuebingen.de

ACM Reference Format:
Axel Sauer, Katja Schwarz, and Andreas Geiger, . 2022. Supplementary
Material for StyleGAN-XL: Scaling StyleGAN to Large Diverse Datasets.
In Special Interest Group on Computer Graphics and Interactive Techniques
Conference Proceedings (SIGGRAPH ’22 Conference Proceedings), August 7–11,
2022, Vancouver, BC, Canada. ACM, New York, NY, USA, 10 pages. https://doi.org/10.1145/3528233.3530738

In this supplemental document, we elaborate on the current limitations, describe the procedure to increase the resolution of ImageNet
to one megapixel, and specify the implementation details of ourapproach. The supplemental video shows additional samples andinterpolations. We use the same mathematical notation as in thepaper.

ALIMITATIONSArchitectural Limitations. First, StyleGAN-XL is three times larger
than StyleGAN3, constituting a higher computational overheadwhen used as a starting point for finetuning. Therefore, it will beworth exploring GAN distillation methods [Chang and Lu 2020] that
trade-off performance for model size. Second, StyleGAN-XL uses
translation-equivariant layers of StyleGAN3-T. As described above,
StyleGAN3-R tends to produce overly symmetrical images andadds significant computational overhead. Finding a more efficientrotational-equivariant architecture is an important future direction. Second, we find StyleGAN3, and consequently, StyleGAN-XL,
harder to edit, e.g., high-quality edits via W are noticeably easierto achieve with StyleGAN2. As already observed in [Karras et al.
2021], StyleGAN3’s semantic controllability is reduced for the sakeof equivariance. However, techniques using the StyleSpace [Wu et al.
2021], e.g., StyleMC [Kocasari et al. 2022], tend to yield better resultsin our experiments, confirming the findings of concurrent workby [Alaluf et al. 2022]. Furthermore, we remark that our framework
can also easily be used with StyleGAN2 layers.
Comparison to Diffusion Models. Our model is larger than earlier
StyleGANs, yet it is still several orders of magnitudes faster thanADM; we compare inference speeds in the supplementary. Low
data coverage is a known problem of GANs, and StyleGAN-XL
makes notable headway on this issue. However, StyleGAN-XL is
still outperformed by diffusion models regarding data coverage.
Furthermore, classes of unaligned humans and human faces arePublication rights licensed to ACM. ACM acknowledges that this contribution wasauthored or co-authored by an employee, contractor or affiliate of a national government. As such, the Government retains a nonexclusive, royalty-free right to publish orreproduce this article, or to allow others to do so, for Government purposes only.
SIGGRAPH ’22 Conference Proceedings, August 7–11, 2022, Vancouver, BC, Canada© 2022 Copyright held by the owner/author(s). Publication rights licensed to ACM.
ACM ISBN 978-1-4503-9337-9/22/08. . . $15.00
https://doi.org/10.1145/3528233.3530738

Figure 1: Imagenet Classes Containing Humans. Samples for
BigGAN and ADM are taken from [Dhariwal and Nichol 2021].

particularly challenging for all compared approaches, likely due to
ImageNet’s emphasis on non-human objects [Dhariwal and Nichol2021]. For such classes, we observe that ADM [Dhariwal and Nichol2021] generates more convincing human faces than BigGAN [Brocket al. 2019] or StyleGAN-XL. Both GANs can synthesize realistic
faces; however, the main challenge in this setting is that the dataset
is unstructured, and the humans are not aligned. [Brock et al. 2019]
remarked the particular challenge of classes containing details towhich human observers are more sensitive. We show examplesin Fig. 1. Whether the points above are a general limitation ofGANs remains an interesting open question for future research.

BPREPROCESSING IMAGENET

An initial challenge is the lack of high-resolution data; the mean
resolution of ImageNet is 469 × 387. Similar to the procedure used
for generating CelebA-HQ[Karras et al. 2018], we preprocess thewhole dataset with SwinIR-Large [Liang et al. 2021], a recent model
for real-world image super-resolution. Of course, a trivial way ofachieving good performance on this dataset would be to draw samples from a 2562 generative model and passing it through SwinIR.
However, SwinIR adds significant computational overhead as it
is 60 times slower than our upsampling stack. Furthermore, thisway, StyleGAN-XL’s weights can be used for initialization whenfinetuning on other high-resolution datasets. Lastly, combining
StyleGAN-XL and SwinIR would impair translation equivariance.

CINFERENCE SPEEDGANs generate samples in a single forward pass, unlike diffusion
models that must be applied several hundred or thousand times togenerate a sample. Table 1 compares StyleGAN-XL to ADM. Wefind that StyleGAN-XL is several orders of magnitude faster. Indefense of diffusion models, speeding up their sampling is an active
area of research, and novel techniques [Watson et al. 2021] may beable to reduce this gap in the future.

SIGGRAPH ’22 Conference Proceedings, August 7–11, 2022, Vancouver, BC, CanadaSauer et al.

Table 1: Inference speed comparison.. We measure the time required for a forward pass with batch size 1 in V100-seconds. ADM
uses classifier guidance.

Table 3: Inversion Results. The metrics are computed between theinversions obtained by the model and the reconstruction targets.

ModelInference Time ↓Res. 1282

Res. 2562

Res. 5122

27.07
0.05

40.26
0.07

91.54
0.10

ADM
StyleGAN-XL

Table 2: Results on Unimodal Datasets..
ModelFID

FFHQ 10242
StyleGAN2
StyleGAN3
StyleGAN-XL

DModel

FID

2.70
2.79
2.02

FastGAN
Projected GAN
StyleGAN-XL

56.46
33.96
25.47

RESULTS ON UNIMODAL DATASETSADDITIONAL QUALITATIVE RESULTS

In the following, we present additional qualitative results. Fig. 2shows additional interpolations between samples from different
classes. Fig. 4 and Fig. 5 show samples on FFHQ 10242 and Pokemon 10242 respectively. Lastly, we compare BigGAN, ADM, andStyleGAN-XL on different ImageNet classes. For a fair comparison,
we do not use truncation or classifier guidance. Instead, we show
images with the largest logits given by a VGG16 which corresponds
to individual image quality.

FMSE ↓PSNR ↑SSIM ↑

FID ↓BigGAN
StyleGAN-XL

0.10
0.06

10.85
13.45

0.26
0.33

47.48
21.73

Table 4: Results on ImageNet at Lower Resolutions..
Model
StyleGAN-XLFID ↓Res. 162

Res. 322

Res. 642

0.73

1.10

1.51

Pokémon 10242

StyleGAN-XL is designed to enable training on large and diverse
datasets. However, applying it to big and small unimodal datasetsis straightforward. In contrast to the configuration for ImageNet,
we begin with ten layers at the lowest stage and add two layers per
resolution stage. Furthermore, we do not employ classifier guidance.
Table 2 reports the results for both datasets at resolution 10242 ,StyleGAN-XL achieves state-of-the-art performance on both.

EModelIMPLEMENTATION DETAILSInversion. Following [Karras et al. 2020], we use basic latent optimization in W for inversion. Given a target image, we first computeits average style code w̄ by running 10000 random latent codes z
and target specific class samples c through the mapping network.
As the class label of the target image is unknown, we pass it to a pretrained classifier. We then use the classifier logits as a multinomial
distribution to sample c. In our experiments, we use Deit-base [Touvron et al. 2021] as a classifier, but other choices are possible. At thebeginning of optimization , we initialize w = w̄. The components
of w are the only trainable parameters. The optimization runs for1000 iterations using the Adam optimizer [Kingma and Ba 2015]
with default parameters. We optimize the LPIPS [Zhang et al. 2018]
distance between the target image and the generated image. For
StyleGAN-XL, the maximum learning rate is 𝜆𝑚𝑎𝑥 = 0.05. It isramped up from zero linearly during the first 50 iterations andramped down to zero using a cosine schedule during the last 250
iterations. For BigGAN, we empirically found 𝜆𝑚𝑎𝑥 = 0.001 anda ramp-down over the last 750 iterations to yield the best results.
All inversion experiments are performed at resolution 5122 andcomputed on 5𝑘 images (10% of the validation set). We report theresults in Table 3 and show qualitative results in Fig. 3.
Training StyleGAN3 on ImageNet. For training StyleGAN3, we
use the official PyTorch implementation1 . The results in Fig. 1 arecomputed with the StyleGAN3-R configuration on resolution 2562
until the discriminator has seen 10 million images. We find that
StyleGAN3-R and StyleGAN3-T converge to similar FID without
any changes to their training paradigm. The run with the best FID
score was selected from three runs with different random seeds. We
use a channel base of 16384 and train on 8 GPUs with total batch
size 256, 𝛾 = 0.256. The remaining settings are chosen according tothe default configuration of the code release. For the ablation study
in Table 1 , we use the StyleGAN3-T configuration as baseline since
StyleGAN-XL builds upon the translational-equivariant layers ofStyleGAN3. We train on 4 GPUs with total batch size 256 and batch
size 32 per GPU, 𝛾 = 0.25, and disable augmentation.
Training & Evaluation. For all our training runs, we do not use
data amplification via x-flips following [Karras et al. 2020]. Furthermore, we evaluate all metrics using the official StyleGAN3 codebase.
For the baseline values in Table 2, we report the numbers of [Dhariwal and Nichol 2021]. The official codebase of ADM2 provides files
containing 50k samples for ADM and BigGAN. We utilize the provided samples to compute rFID. Following [Dhariwal and Nichol2021], we compute precision and recall between 10k real samplesand 50k generated samples. Table 4 reports the results on ImageNetat lower resolutions.
Layer configurations. We start progressive growing at resolution
162 using 11 layers. The layer specifications are computed according
to [Karras et al. 2021] and remain fixed for the remaining training.
For the next stage, at resolution 322 , we discard the last 2 layers andadd 7 new ones. The specifications for the new layers are computed
according to [Karras et al. 2021] for a model with resolution 322 and1 https://github.com/NVlabs/stylegan3.git
2 https://github.com/openai/guided-diffusionSupplementary Material for
StyleGAN-XL: Scaling StyleGAN to Large Diverse Datasets

16 layers. Continuing this strategy up to resolution 10242 yields theflexible layer specification of StyleGAN-XL in Fig. 9.

REFERENCES
Yuval Alaluf, Or Patashnik, Zongze Wu, Asif Zamir, Eli Shechtman, Dani Lischinski,
and Daniel Cohen-Or. 2022. Third Time’s the Charm? Image and Video Editing
with StyleGAN3. arXiv.org abs/2201.13433 (2022). https://arxiv.org/abs/2201.13433
Andrew Brock, Jeff Donahue, and Karen Simonyan. 2019. Large Scale GAN Training
for High Fidelity Natural Image Synthesis. In Proc. of the International Conf. on
Learning Representations (ICLR). OpenReview.net. https://openreview.net/forum?
id=B1xsqj09Fm
Ting-Yun Chang and Chi-Jen Lu. 2020. TinyGAN: Distilling BigGAN for Conditional
Image Generation. In Proc. of the Asian Conf. on Computer Vision (ACCV) (Lecture
Notes in Computer Science, Vol. 12625), Hiroshi Ishikawa, Cheng-Lin Liu, Tomás
Pajdla, and Jianbo Shi (Eds.). 509–525. https://doi.org/10.1007/978-3-030-695385_31
Prafulla Dhariwal and Alex Nichol. 2021. Diffusion Models Beat GANs on ImageSynthesis. (2021).
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. 2018. Progressive Growingof GANs for Improved Quality, Stability, and Variation. In Proc. of the International
Conf. on Learning Representations (ICLR). OpenReview.net. https://openreview.net/forum?id=Hk99zCeAb
Tero Karras, Miika Aittala, Samuli Laine, Erik Härkönen, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. 2021. Alias-Free Generative Adversarial Networks. In Advancesin Neural Information Processing Systems (NeurIPS).
Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and TimoAila. 2020. Analyzing and Improving the Image Quality of StyleGAN. In Proc.
IEEE Conf. on Computer Vision and Pattern Recognition (CVPR). Computer Vision

SIGGRAPH ’22 Conference Proceedings, August 7–11, 2022, Vancouver, BC, Canada
Foundation / IEEE, 8107–8116. https://doi.org/10.1109/CVPR42600.2020.00813
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A Method for Stochastic Optimization.
In Proc. of the International Conf. on Learning Representations (ICLR).
Umut Kocasari, Alara Dirik, Mert Tiftikci, and Pinar Yanardag. 2022. StyleMC: MultiChannel Based Fast Text-Guided Image Generation and Manipulation. Proc. ofthe IEEE Winter Conference on Applications of Computer Vision (WACV) (2022).
https://arxiv.org/abs/2112.08493
Jingyun Liang, Jiezhang Cao, Guolei Sun, Kai Zhang, Luc Van Gool, and Radu Timofte.
2021. SwinIR: Image Restoration Using Swin Transformer. In Proc. of the IEEE
International Conf. on Computer Vision (ICCV) Workshops. IEEE, 1833–1844. https://doi.org/10.1109/ICCVW54120.2021.00210
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. 2021. Training data-efficient image transformers & distillation through attention. In Proc. of the International Conf. on Machine learning(ICML) (Proceedings of Machine Learning Research, Vol. 139). PMLR, 10347–10357.
http://proceedings.mlr.press/v139/touvron21a.html
Daniel Watson, Jonathan Ho, Mohammad Norouzi, and William Chan. 2021. Learning
to Efficiently Sample from Diffusion Probabilistic Models. CoRR abs/2106.03802(2021). https://arxiv.org/abs/2106.03802
Zongze Wu, Dani Lischinski, and Eli Shechtman. 2021. StyleSpace Analysis:
Disentangled Controls for StyleGAN Image Generation. In Proc. IEEE Conf. on
Computer Vision and Pattern Recognition (CVPR). Computer Vision Foundation /IEEE, 12863–12872. https://openaccess.thecvf.com/content/CVPR2021/html/Wu_StyleSpace_Analysis_Disentangled_Controls_for_StyleGAN_Image_Generation_
CVPR_2021_paper.html
Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shechtman, and Oliver Wang. 2018.
The Unreasonable Effectiveness of Deep Features as a Perceptual Metric. In Proc.
IEEE Conf. on Computer Vision and Pattern Recognition (CVPR). Computer Vision
Foundation / IEEE Computer Society, 586–595. https://doi.org/10.1109/CVPR.2018.
00068

SIGGRAPH ’22 Conference Proceedings, August 7–11, 2022, Vancouver, BC, Canada

Sauer et al.

Figure 2: Interpolations. StyleGAN-XL generates smooth interpolations between samples of different classes.

Figure 3: Inversion of a Given Source Image. For BigGAN, we invert to its latent space z, for StyleGAN-XL we invert to style codes w.

Supplementary Material for
StyleGAN-XL: Scaling StyleGAN to Large Diverse Datasets

SIGGRAPH ’22 Conference Proceedings, August 7–11, 2022, Vancouver, BC, Canada

Figure 4: Samples on FFHQ 10242 .

SIGGRAPH ’22 Conference Proceedings, August 7–11, 2022, Vancouver, BC, Canada

Figure 5: Samples on Pokemon 10242 .

Sauer et al.

Supplementary Material for
StyleGAN-XL: Scaling StyleGAN to Large Diverse Datasets

SIGGRAPH ’22 Conference Proceedings, August 7–11, 2022, Vancouver, BC, Canada

Figure 6: Qualitiative Comparison on ImageNet 2562 .. We compare BigGAN (left column), ADM (middle column), and StyleGAN-XL (rightcolumn). Classes from top to bottom: pizza, valley, daisy, dough, comic book.

SIGGRAPH ’22 Conference Proceedings, August 7–11, 2022, Vancouver, BC, Canada

Sauer et al.

Figure 7: Qualitiative Comparison on ImageNet 2562 .. We compare BigGAN (left column), ADM (middle column), and StyleGAN-XL (rightcolumn). Classes from top to bottom: bulbul, nematode, jack-o’-lantern, balloon, crossword puzzle.

Supplementary Material for
StyleGAN-XL: Scaling StyleGAN to Large Diverse Datasets

SIGGRAPH ’22 Conference Proceedings, August 7–11, 2022, Vancouver, BC, Canada

Figure 8: Qualitiative Comparison on ImageNet 2562 .. We compare BigGAN (left column), ADM (middle column), and StyleGAN-XL (rightcolumn). Classes from top to bottom: agaric, orange, Tibetian mastiff, espresso, paddlewheel.

SIGGRAPH ’22 Conference Proceedings, August 7–11, 2022, Vancouver, BC, Canada

01234Log2 frequency
56Sauer et al.

78910L0L1L2L3L4Cutoff
Min. stopband
Sampling rate
StopbandL5L6L7L8L9L10L11L12L13L14L15L16L17L18L19L20L21L22L23L24L25L26
L27L28L29L30L31L32L33
L34
L35
L36
L37
L38Figure 9: Flexible Layer Specification of Stylegan-XL. StyleGAN-XL consists of 39 layers at resolution 10242 . Cutoff (blue) and minimum
acceptable stopband frequency (orange) obey geometric progression over the layers; sampling rate (red) and actual stopband (green) are computed
according to our design constraints.