StyleGAN-T: Unlocking the Power of GANs for
Fast Large-Scale Text-to-Image Synthesis

Axel Sauer 1 2 Tero Karras 2 Samuli Laine 2 Andreas Geiger 1 Timo Aila 2

Abstract

LAFITE

Diffusion
Autoregressive
GAN

Text-to-image synthesis has recently seen significant progress thanks to large pretrained language
models, large-scale training data, and the introduction of scalable model families such as diffusion and autoregressive models. However, thebest-performing models require iterative evaluation to generate a single sample. In contrast, generative adversarial networks (GANs) only need asingle forward pass. They are thus much faster,
but they currently remain far behind the stateof-the-art in large-scale text-to-image synthesis.
This paper aims to identify the necessary steps
to regain competitiveness. Our proposed model,
StyleGAN-T, addresses the specific requirementsof large-scale text-to-image synthesis, such aslarge capacity, stable training on diverse datasets,strong text alignment, and controllable variation
vs. text alignment tradeoff. StyleGAN-T significantly improves over previous GANs and outperforms distilled diffusion models — the previous
state-of-the-art in fast text-to-image synthesis —in terms of sample quality and speed.

1. Introduction
In text-to-image synthesis, novel images are generated basedon text prompts. The state-of-the-art in this task has recently taken dramatic leaps forward thanks to two key ideas.
First, using a large pretrained language model as an encoder
for the prompts makes it possible to condition the synthesis based on general language understanding (Ramesh et al.,2022; Saharia et al., 2022). Second, using large-scale training data consisting of hundreds of millions of image-caption
pairs (Schuhmann et al., 2022) allows the models to synthesize almost anything imaginable.
Training datasets continue to increase rapidly in size andcoverage. Consequently, text-to-image models must be scal1
University of Tübingen, Tübingen AI Center 2 NVIDIA. Correspondence to: Axel Sauer <a.sauer@uni-tuebingen.de>.

Zero-shot FID30k

25
20
15

StyleGAN-T

GLIDE
Make-a-Scene

10
Stable Diffusion
5

LDM

0

1

Parti-3B
Imagen

3
5
10
15 20
Time to generate one image [s]eDiff-I
25 30 35

Figure 1. Quality vs. speed in large-scale text-to-image synthesis.
StyleGAN-T greatly narrows the quality gap between GANs andother model families while generating samples at a rate of 10 FPS
on an NVIDIA A100. The y-axis corresponds to zero-shot FID on
MS COCO at 256×256 resolution; lower is better.

able to a large capacity to absorb the training data. Recent
successes in large-scale text-to-image generation have been
driven by diffusion models (DM) (Ramesh et al., 2022; Saharia et al., 2022; Rombach et al., 2022) and autoregressivemodels (ARM) (Zhang et al., 2021; Yu et al., 2022; Gafni
et al., 2022) that seem to have this property built in, alongwith the ability to deal with highly multi-modal data.
Interestingly, generative adversarial networks (GAN) (Goodfellow et al., 2014) — the dominant family of generativemodels in smaller and less diverse datasets — have not been
particularly successful in this task (Zhou et al., 2022). Our
goal is to show that they can regain competitiveness.
The primary benefits offered by GANs are inference speed
and control of the synthesized result via latent space manipulations. StyleGAN (Karras et al., 2019; 2020; 2021)
in particular has a thoroughly studied latent space, whichallows principled control of generated images (Bermano
et al., 2022; Härkönen et al., 2020; Shen et al., 2020; Abdal
et al., 2021; Kafri et al., 2022). While there has been notable progress in speeding up DMs (Salimans & Ho, 2022;
Karras et al., 2022; Lu et al., 2022), they are still far behind
GANs that require only a single forward pass.
We draw motivation from the observation that GANs laggedStyleGAN-T: Unlocking the Power of GANs for Fast Large-Scale Text-to-Image Synthesis

similarly behind diffusion models in ImageNet (Deng et al.,2009; Dhariwal & Nichol, 2021) synthesis until the discriminator architecture was redesigned in StyleGAN-XL (Sauer
et al., 2021; 2022), which allowed GANs to close the gap.
In Section 3, we start from StyleGAN-XL and revisit thegenerator and discriminator architectures, considering therequirements specific to the large-scale text-to-image task:large capacity, extremely diverse datasets, strong text alignment, and controllable variation vs. text alignment tradeoff.
We have a fixed training budget of 4 weeks on 64 NVIDIA
A100s available for training our final model at scale. This
constraint forces us to set priorities because the budget islikely insufficient for state-of-the-art, high-resolution results(CompVis, 2022). While the ability of GANs to scale tohigh resolutions is well known (Wang et al., 2018; Karras
et al., 2020), successful scaling to the large-scale text-toimage task remains undocumented. We thus focus primarily
on solving this task in lower resolutions, dedicating only alimited budget to the super-resolution stages.
Our StyleGAN-T achieves a better zero-shot MS COCO
FID (Lin et al., 2014; Heusel et al., 2017) than current stateof-the-art diffusion models at a resolution of 64×64. At
256×256, StyleGAN-T halves the zero-shot FID previously
achieved by a GAN but continues to trail SOTA diffusion
models. The key benefits of StyleGAN-T include its fast
inference speed and smooth latent space interpolation in
the context of text-to-image synthesis, illustrated in Fig. 1
and Fig. 2, respectively. We will make our implementation
available at https://github.com/autonomousvision/stylegan-t

2. StyleGAN-XL
Our architecture design is based on StyleGAN-XL (Sauer
et al., 2022) that — similar to the original StyleGAN (Karras et al., 2019) — first processes the normally distributed
input latent code z by a mapping network to produce an intermediate latent code w. This intermediate latent is thenused to modulate the convolution layers in a synthesis network using the weight demodulation technique introducedin StyleGAN2 (Karras et al., 2020). The synthesis networkof StyleGAN-XL uses the alias-free primitive operationsof StyleGAN3 (Karras et al., 2021) to achieve translationequivariance, i.e., to enforce the synthesis network to haveno preferred positions for the generated features.
StyleGAN-XL has a unique discriminator design wheremultiple discriminator heads operate on feature projections (Sauer et al., 2021) from two frozen, pretrained feature extraction networks: DeiT-M (Touvron et al., 2021a)
and EfficientNet (Tan & Le, 2019). Their outputs are fed
through randomized cross-channel and cross-scale mixing
modules. This results in two feature pyramids with four resolution levels each that are then processed by eight discrim-

StyleGAN-XL
New generator
New discriminator
LCLIP

Zero-shot FID30k ↓CLIP score ↑

51.88
45.10
26.77
20.52

5.58
6.02
9.78
11.72

Table 1. Architecture ablation. Our architectural changes notably improve sample quality and text alignment. Here, we use thelightweight training configuration described in Appendix A.

inator heads. An additional pretrained classifier network isused to provide guidance during training.
The synthesis network of StyleGAN-XL is trained progressively, increasing the output resolution over time by introducing new synthesis layers once the current resolution
stops improving. In contrast to a previous progressive growing approach (Karras et al., 2018), the discriminator structure does not change during training. Instead, the early lowresolution images are upsampled as necessary to suit thediscriminator. In addition, the already trained synthesis layers are frozen as further layers are added.
For class-conditional synthesis, StyleGAN-XL concatenates
an embedding of a one-hot class label to z and uses a projection discriminator (Miyato & Koyama, 2018).

3. StyleGAN-T
We choose StyleGAN-XL as our baseline architecture because of its strong performance in class-conditional ImageNet synthesis (Sauer et al., 2022). In this section, we
modify this baseline piece by piece, focusing on the generator (Section 3.1), discriminator (Section 3.2), and variation
vs. text alignment tradeoff mechanisms (Section 3.3) in turn.
Throughout the redesign process, we measure the effect ofour changes using zero-shot MS COCO. For practical reasons, the tests use a limited compute budget, smaller models,and a smaller dataset than the large-scale experiments in Section 4; see Appendix A for details. We quantify sample quality using FID (Heusel et al., 2017) and text alignment using
CLIP score (Hessel et al., 2021). Following prior art (Balaji
et al., 2022), we compute the CLIP score using a ViT-g-14
model trained on LAION-2B (Schuhmann et al., 2022).
To change the class conditioning to text conditioning in ourbaseline model, we embed the text prompts using a pretrained CLIP ViT-L/14 text encoder (Radford et al., 2021)
and use them in place of the class embedding. Accordingly,
we also remove the training-time classifier guidance. This
simple conditioning mechanism matches the early text-toimage models (Reed et al., 2016a;b). As shown in Table 1,
this baseline reaches a zero-shot FID of 51.88 and CLIP
score of 5.58 in our lightweight training configuration. Note

StyleGAN-T: Unlocking the Power of GANs for Fast Large-Scale Text-to-Image Synthesis

Figure 2. Example images and interpolations. StyleGAN-T generates diverse samples matching the text prompt and allows for smoothinterpolations between prompts, illustrated as a single continuous interpolation in scanline order. Generating these 56 samples at 512×512
takes 6 seconds on an NVIDIA RTX 3090, while a comparable grid takes up to several minutes with current diffusion models. The
accompanying video further demonstrates interpolations and contrasts them with diffusion models.

StyleGAN-T: Unlocking the Power of GANs for Fast Large-Scale Text-to-Image Synthesis
Residual block

Generator block

Generator

Text prompt

DINO encoder“Castle, fauvism”

ModConv

Upsample

Fully-connected

ModConv

Fully-connected

Residual

ConcatGroupNorm

Residual

ModConv
Layer
scale

Residual✕

Residual＋

ToRGB

Modulated convolution (ModConv)✕

Weights

AffineSplit

CLIPtext encoder

Discriminator

Tokenize

D1
Transformer⨯3

CLIP
guidance

Fourier features
8⨯8 blockD2
Transformer⨯3
D3
Transformer⨯3

16⨯16 block＋32⨯32 block＋64⨯64 block＋128⨯128 block＋256⨯256 block＋

512⨯512 block＋

Conv 3⨯3

D4
Transformer⨯3
CLIP
image encoder

D5

Discriminator head
Upsample

Conv 1
Random64⨯64 crop

✕

Demod

Augment & crop

1D sequence of tokens

BatchNorm

Conv 9

Leaky ReLU

BatchNorm
Leaky ReLU

Bias & noise

✕

＋

＋

Trainable
Pretrained

Leaky ReLU

AffineConv 1⋅

Fixed-function

Generated image(b) Generator details

Per-token hinge loss(a) Generator architecture(d) Text encoding(c) Discriminator architecture

Figure 3. Overview of StyleGAN-T. (a) Our generator architecture (Sec. 3.1) is closely related to StyleGAN2, with the learned constantreplaced with Fourier features and conditioning applied in a slightly different place. (b) For each resolution, a generator block is executedand its contribution is accumulated to the image via a dedicated ToRGB layer. The generator blocks employ residual connections anda new 2nd order style mechanism (Eq. 1). (c) Our discriminator (Sec. 3.2) processes the intermediate tokens of a DINO-trained visiontransformer using 5 identical discriminator heads. Text conditioning is done using projection at the end. (d) Text prompt is embedded
using CLIP and supplied to the generator and discriminator. We also employ a guidance term to further improve text alignment (Sec. 3.3).

that we use a different CLIP model for conditioning the generator and for computing the CLIP score, which reduces therisk of artificially inflating the results.
3.1. Redesigning the Generator
StyleGAN-XL uses StyleGAN3 layers to achieve translational equivariance. While equivariance can be desirable
for various applications, we do not expect it to be necessary
for text-to-image synthesis because none of the successful
DM/ARM-based methods are equivariant. Additionally, theequivariance constraint adds computational cost and posescertain limitations to the training data that large-scale image
datasets typically violate (Karras et al., 2021).
For these reasons, we drop the equivariance and switch to
StyleGAN2 backbone for the synthesis layers, including
output skip connections and spatial noise inputs that facilitate stochastic variation of low-level details. The high-levelarchitecture of our generator after these changes is shown in
Fig. 3a. We additionally propose two changes to the detailsof the generator architecture (Fig. 3b).
Residual convolutions. As we aim to increase the model
capacity significantly, the generator must be able to scale inboth width and depth. However, in the basic configuration, asignificant increase in the generator’s depth leads to an earlymode collapse in training. An important building block inmodern CNN architectures (Liu et al., 2022; Dhariwal &Nichol, 2021) is an easily optimizable residual block that
normalizes the input and scales the output. Following these
insights, we make half the convolution layers residual andwrap them by GroupNorm (Wu & He, 2018) for normalization and Layer Scale (Touvron et al., 2021b) for scaling theircontribution. A layer scale of a low initial value of 10−5
allows gradually fading in the convolution layer’s contribution, stabilizing the early training iterations significantly.
This design allows us to increase the total number of layers
considerably — by approximately 2.3× in the lightweight
configuration and 4.5× in the final model. For fairness, wematch the parameter count of the StyleGAN-XL baseline.

StyleGAN-T: Unlocking the Power of GANs for Fast Large-Scale Text-to-Image Synthesis

Stronger conditioning. The text-to-image setting is challenging because the factors of variation can vastly differ per
prompt. Consider the prompts “a close-up of a face” and “abeautiful landscape.” The first prompt should generate faces
with varying eye color, skin color, and proportions, whereas
the second should produce landscapes from different areas,seasons, and daytime. In a style-based architecture, all of
this variation has to be implemented by the per-layer styles.
Thus the text conditioning may need to affect the styles
much more strongly than was necessary for simpler settings.
In early tests, we observed a clear tendency of the input
latent z to dominate over the text embedding ctext in ourbaseline architecture, leading to poor text alignment. To
remedy this, we introduce two changes that aim to amplify
the role of ctext . First, we let the text embeddings bypass the
mapping network, following the observations by Härkönen
et al. (2022). A similar design was also used in L AFITE(Zhou et al., 2022), assuming that the CLIP text encoder
defines an appropriate intermediate latent space for the text
conditioning. We thus concatenate ctext directly to w anduse a set of affine transforms to produce per-layer styless̃. Second, instead of using the resulting s̃ to modulate theconvolutions as-is, we further split it into three vectors ofequal dimension s̃1,2,3 and compute the final style vector ass = s̃1s̃2 + s̃3 .

(1)

The crux of this operation is the element-wise multiplication that effectively turns the affine transform into a 2ndorder polynomial network (Chrysos et al., 2020; Chrysos& Panagakis, 2021), increasing its expressive power. The
stacked MLP-based conditioning layers in DF-GAN (Taoet al., 2022) implicitly include similar 2nd order terms.
Together, our changes to the generator improve FID and
CLIP score by ∼10%, as shown in Table 1.
3.2. Redesigning the Discriminator
We redesign the discriminator from scratch but retain
StyleGAN-XL’s key ideas of relying on a frozen, pretrainedfeature network and using multiple discriminator heads.
Feature network. For the feature network, we choose
a ViT-S (Dosovitskiy et al., 2021) trained with the selfsupervised DINO objective (Caron et al., 2021). The network is lightweight, fast to evaluate, and encodes semantic
information at high spatial resolution (Amir et al., 2021).
An additional benefit of using a self-supervised feature network is that it circumvents the concern of potentially compromising FID (Kynkäänniemi et al., 2022).
Architecture. Our discriminator architecture is shownin Fig. 3c. ViTs are isotropic, i.e., the representation size
(tokens × channels) and receptive field (global) are thesame throughout the network. This isotropy allows us to
use the same architecture for all discriminator heads, which
we space equally between the transformer layers. Multiple
heads are known to be beneficial (Sauer et al., 2021), andwe use five heads in our design.
Our discriminator heads are minimalistic, as detailed inFig. 3c, bottom. The residual convolution’s kernel width
controls the head’s receptive field in the token sequence.
We found that 1D convolutions applied on the sequence of
tokens performed just as well as 2D convolutions applied on
spatially reshaped tokens, indicating that the discrimination
task does not benefit from whatever 2D structure remains
in the tokens. We evaluate a hinge loss (Lim & Ye, 2017)independently for each token in every head.
Sauer et al. (2021) use synchronous BatchNorm (Ioffe &Szegedy, 2015) to provide batch statistics to the discriminator. BatchNorm is problematic when scaling to a multinode setup, as it requires communication between nodes andGPUs. We use a variant that computes batch statistics on
small virtual batches (Hoffer et al., 2017). The batch statistics are not synchronized between devices but are calculated
per local minibatch. Furthermore, we do not use runningstatistics, and thus no additional communication overhead
between GPUs is introduced.
Augmentation. We apply differentiable data augmentation(Zhao et al., 2020) with default parameters before the feature
network in the discriminator. We use random crops whentraining at a resolution larger than 224×224 pixels (ViT-S
training resolution).
As shown in Table 1, these changes significantly improve
FID and CLIP score by further ∼40%. This considerable
improvement indicates that a well-designed discriminatoris critical when dealing with highly diverse datasets. Compared to the StyleGAN-XL discriminator, our simplified redesign is ∼2.5× faster, leading to ∼1.5× faster training.
3.3. Variation vs. Text Alignment Tradeoffs
Guidance (Dhariwal & Nichol, 2021; Ho & Salimans, 2022)
is an essential component of current text-to-image diffusion
models. It trades variation for perceived image quality in a
principled way, preferring images that are strongly alignedwith the text conditioning. In practice, guidance drastically
improves the results; thus, we want to approximate its behavior in the context of GANs.
Guiding the generator. StyleGAN-XL uses a pretrained
ImageNet classifier to provide additional gradients during
training, guiding the generator toward images that are easy
to classify. This method improves results significantly. In
the context of text-to-image, “classification” involves captioning the images. Thus, a natural extension of this ap-

StyleGAN-T: Unlocking the Power of GANs for Fast Large-Scale Text-to-Image Synthesis

proach is to use a CLIP image encoder instead of a classifier.
Following Crowson et al. (2022), at each generator update,
we pass the generated image through the CLIP image encoder to obtain caption cimage , and minimize the squared
spherical distance to the normalized text embedding ctext :ψ = 1.00
CS = 0.33

ψ = 0.60

LCLIP = arccos2 (cimage · ctext )(2)

This additional loss term guides the generated distribution
towards images that are captioned similarly to the input text
encoding ctext . Its effect is thus similar to the guidance in
diffusion models. Fig. 3d illustrates our approach.
CLIP has been used in prior work to guide a pretrained
generator during synthesis (Nichol et al., 2022; Crowson
et al., 2022; Liu et al., 2021). In contrast, we use it as a partof the loss function during training. It is important to notethat overly strong CLIP guidance during training impairs
FID, as it limits the distribution diversity and ultimately
starts introducing image artifacts. Therefore, the weight of
LCLIP in the overall loss needs to strike a balance between
image quality, text conditioning, and distribution diversity;
we set it to 0.2. We further observed that guidance is helpful
only up to 64×64 pixel resolution. At higher resolutions,
we apply LCLIP to random 64×64 pixel crops.
As shown in Table 1, CLIP guidance improves FID and
CLIP scores by further ∼20%.
Guiding the text encoder. Interestingly, the earlier methods listed above that use a pretrained generator did not reportencountering low-level image artifacts. We hypothesize thatthe frozen generator acts as a prior that suppresses them. We
build on this insight to further improve the text alignment.
In our primary training phase, the generator is trainable andthe text encoder is frozen. We then introduce a secondary
phase, where the generator is frozen and the text encoder
becomes trainable instead. We only train the text encoder as
far as the generator conditioning is concerned; the discriminator and the guidance term (Eq. 2) still receive ctext fromthe original frozen encoder. This secondary phase allows avery high CLIP guidance weight of 50 without introducing
artifacts and significantly improves text alignment withoutcompromising FID (Section 4.2). Compared to the primaryphase, the secondary phase can be much shorter. After convergence, we continue with the primary phase.
Explicit truncation. Typically variation has been traded tohigher fidelity in GANs using the truncation trick (Marchesi,
2017; Brock et al., 2019; Karras et al., 2019), where a sampled latent w is interpolated towards its mean with respect
to the given conditioning input. This way, truncation pushes
w to a higher-density region where the model performs better. In our implementation, w = [f (z), ctext ], where f (·)
denotes the mapping network, so the per-prompt mean isgiven by w̃ = Ez [w] = [f̃ , ctext ], where f̃ = Ez [f (z)]. We

CS = 0.36

ψ = 0.10
CS = 0.39

Figure 4. Truncation. Four samples for the prompt “a graphite
sketch of Eva Longoria” with different random z. Increasing
truncation (decreasing ψ) improves the text alignment according
to mean CLIP score per row (CS) at the cost of lower variation.

thus implement truncation by tracking f̃ during training andinterpolating between w̃ and w according to scaling parameter ψ ∈ [0, 1] at inference time.
We illustrate the impact of truncation in Fig. 4. In practice,
we rely on the combination of CLIP guidance and truncation.
Guidance improves the model’s overall text alignment, andtruncation can further boost quality and alignment for agiven sample, trading away some variation.

4. Experiments
Using the final configuration developed in Section 3, we
scale the model size, dataset, and training time. Our final
model consists of ∼1 billion parameters; we did not observe any instabilities when increasing the model size. We
train on a union of several datasets amounting to 250M textimage pairs in total. We use progressive growing similar
to StyleGAN-XL, except that all layers remain trainable.
The hyperparameters and dataset details are listed in Appendix A.
The total training time was four weeks on 64 A100 GPUs
using a batch size of 2048. We first trained the primary phasefor 3 weeks (resolutions up to 64×64), then the secondary
phase for 2 days (text embedding), and finally the primaryphase again for 5 days (resolutions up to 512×512). For
comparison, our total compute budget is about a quarter ofStable Diffusion’s (CompVis, 2022).
4.1. Quantitative Comparison to State-of-the-Art
We use zero-shot MS COCO to compare the performanceof our model to the state-of-the-art quantitatively at 64×64
pixel output resolution in Table 2 and 256×256 in Table 3.
At low resolution, StyleGAN-T outperforms all other approaches in terms of output quality, while being very fast toevaluate. In this test we use the model before the final train-

StyleGAN-T: Unlocking the Power of GANs for Fast Large-Scale Text-to-Image Synthesis
Model typeZero-shot FID30k

Speed [s]28Stable Diffusion *eDiff-I
LDM *
GLIDE

Diffusion
Diffusion
Diffusion
Diffusion

8.40
7.60
7.59
7.40

–
26.0
–
10.9

26

LAFITE *StyleGAN-T

GAN
GAN

14.80
7.30

∼0.01
0.06

* downsampled to 64×64 pixels using Lanczos

Zero-shot FID5k

Model

24
22
20

– not available

18

Table 2. Comparison of FID on MS COCO 64×64. Inference
speeds are measured on an A100. For LAFITE we estimate what
its speed would be at a native 64×64 resolution.
Model typeZero-shot FID30k

Speed [s]

LDM
GLIDE
DALL·E 2
Stable Diffusion *ImageneDiff-I

Diffusion
Diffusion
Diffusion
Diffusion
Diffusion
Diffusion

12.63
12.24
10.39
8.59
7.27
6.95

3.7
15.0
–
3.7
9.1
32.0

DALL·E
Ernie-ViLG
Make-A-Scene *Parti-3B
Parti-20B

Autoregressive
Autoregressive
Autoregressive
Autoregressive
Autoregressive

27.50
14.70
11.84
8.10
7.23

–
–
25.0
6.4
–

LAFITE
StyleGAN-T *

GAN
GAN

26.94
13.90

0.02
0.10
– not available

Table 3. Comparison of FID on MS COCO 256×256. Inference speeds are measured on an A100, except for Imagen and Parti
that use a faster TPUv4 accelerator. The Stable Diffusion numbers are from (Balaji et al., 2022; Lambda Labs, 2022); the othernumbers are obtained from the respective papers or through correspondence with the authors.

ing phase, i.e., one that produces 64×64 images natively. At
high resolution, StyleGAN-T still significantly outperforms
LAFITE but lags behind DMs and ARMs in terms of FID.
These results lead us to two conclusions. First, GANs
can match or even beat current DMs in large-scale textto-image synthesis at low resolution. Second, a powerfulsuperresolution model is crucial. While FID slightly decreases in eDiff-I when moving from 64×64 to 256×256(7.60→6.95), it currently almost doubles in StyleGAN-T.
Therefore, it is evident that StyleGAN-T’s superresolution
stage is underperforming, causing a gap to the current stateof-the-art high-resolution results. Whether this gap can be
bridged simply with additional capacity or longer training
is an open question.
4.2. Evaluating Variation vs. Text Alignment
We report FID–CLIP score curves in Fig. 5. We compare
StyleGAN-T to a strong DM baseline (CLIP-conditioned
variant of eDiff-I) and a fast, distilled DM baseline (SDdistilled) (Meng et al., 2022).

0.27

0.28
0.29
0.30
CLIP score (ViT-g-14)

0.31

Figure 5. Comparing text alignment tradeoffs. We compare
FID–CLIP score curves of StyleGAN-T, distilled Stable Diffusion(SD-distilled), and eDiff-I. We report values of SD-distilled at a
guidance scale of w = 4. For a fair comparison, we report numbers
for CLIP-conditioned eDiff-I disabling additional conditioning
on T5-XXL text embeddings. The models use different methods
to increase the CLIP score (i.e., text alignment): StyleGAN-T
decreases truncation ψ = {1.0 . . . 0.0}, SD-distilled increases thenumber of sampling steps {2, 4, 8}, eDiff-I increases guidancescale w = {0 . . . 10}.

24
Zero-shot FID5k

Model* downsampled to 256×256 pixels using Lanczos

SD-distilled
eDiff-I
StyleGAN-T

TE frozen
TE trained

22

20
0.285

0.290
0.295
0.300
CLIP score (ViT-g-14)

0.305

Figure 6. Text encoder training. Training the CLIP text encoder(TE) pushes the entire FID–CLIP score curve to the right, hence,increasing overall text alignment.

Using Truncation, StyleGAN-T can push the CLIP score to0.305, successfully improving text alignment. StyleGAN-T
outperforms SD-distilled in both FID and CLIP scores yet
remains behind eDiff-I. Regarding speed, eDiff-I requires
32.0 seconds to generate a sample. SD-distilled is significantly faster and only needs 0.6 seconds at its best performance at eight sampling steps. StyleGAN-T beats bothbaselines, generating a sample in 0.1 seconds.
To isolate the impact of text encoder training, we evaluate
FID–CLIP score curves in Fig. 6. For this experiment, we
utilize the same generator network and only swap the text
encoder. As the generator has been frozen in the secondary
phase, it can handle both the original and fine-tuned CLIP
text embeddings as evidenced by their equal performancemeasured by FID. Fine-tuning the text encoder significantly
improves the CLIP score without compromising FID.

StyleGAN-T: Unlocking the Power of GANs for Fast Large-Scale Text-to-Image Synthesis“a victorian house”“a cute puppy”→→“a modern house”“real photo”“cubism painting”“made of beads and yarn”“chalk art”

“van Gogh painting”“anime”“a cute blue puppy, Madhubani painting”“a landscape in winter”→“a landscape in fall”

Figure 8. Styles. Samples generated by StyleGAN-T for a fixed
random seed and the caption “astronaut, {X}”, where X is denoted
below each image.

Figure 7. Latent manipulation. Samples (first column) can be
manipulated by following semantic directions in latent space.

4.3. Qualitative Results
Fig. 2 shows example images produced by StyleGAN-T,
along with interpolations between them. The accompanying
video shows this in animation and compares it to diffusion
models, demonstrating that the interpolation properties of
GANs continue to be considerably smoother.
Interpolating between different text prompts is straightforward. For an image generated by an intermediate latent
w0 = [f (z), ctext0 ], we substitute the text condition ctext0
with a new text condition ctext1 . We then interpolate w0
towards the new latent w1 = [f (z), ctext1 ] as shown in
Fig. 7. This approach is similar to DALL·E 2’s text diff operation that interpolates between CLIP embeddings. Previous work for manipulating GAN-generated images (Patashnik et al., 2021) typically discovers these latent directions
via a training process that needs to be repeated per prompt
and is, therefore, expensive. Meaningful latent directions
are a built-in property of our model, and no extra training isneeded.
By appending different styles to a prompt, StyleGAN-T cangenerate a wide variety of styles as shown in Fig. 8. Subjects
tend to be aligned for a fixed latent z, which we showcasein the accompanying video.

“a red cube on
a blue cube”“astronaut, child’s
drawing”“a sign that says
deep learning”

Figure 9. Failure cases. StyleGAN-T can struggle to bind attributes to objects, and to produce coherent text.

Guidance via CLIP loss is vital for good text alignment, buthigh guidance strength results in image artifacts. A possible
solution could be to retrain CLIP on higher-resolution datathat does not suffer from aliasing or other image quality
issues. In this context, the conditioning mechanism in thediscriminator may also be worth revisiting.
Truncation improves text alignment but differs from guidance in diffusion models in two important ways. While truncation is always towards a single mode, guidance can atleast theoretically be arbitrarily multi-modal. Also, truncation sharpens the distribution before the synthesis network,
which can reshape the distribution in arbitrary ways, thus,
possibly undoing any prior sharpening. Therefore, alternative methods to truncation might further improve the results.

5. Limitations and Future Work

Improved super-resolution stages (i.e., high-resolution layers) through higher capacity and longer training are an obvious avenue for future work.

Similarly to DALL·E 2 that also uses CLIP as the underlying language model, StyleGAN-T sometimes struggles interms of binding attributes to objects as well as producing
coherent text in images (Fig. 9). Using a larger language
model would likely resolve this issue at the cost of slower
runtime (Saharia et al., 2022; Balaji et al., 2022).

Methods for “personalizing” diffusion models have become
popular (Ruiz et al., 2022; Gal et al., 2022). They finetunea pretrained model to associate a unique identifier with agiven subject, allowing it to synthesize novel images of thesame subject in novel contexts. Such approaches could be
similarly applied to GANs.

StyleGAN-T: Unlocking the Power of GANs for Fast Large-Scale Text-to-Image Synthesis

Acknowledgements
We would like to thank Tim Brooks, Miika Aittala, andJaakko Lehtinen for helpful discussions; Yogesh Balaji and
Seungjun Nah for computing additional metrics for eDiff-I;
Tuomas Kynkäänniemi for extensive feedback on an earlier draft; Tero Kuosmanen, Samuel Klenberg, and Janne
Hellsten for maintaining the compute infrastructure; andDavid Luebke and Vanessa Sauer for their general support.

References
Abdal, R., Zhu, P., Mitra, N. J., and Wonka, P. StyleFlow: Attribute-conditioned exploration of StyleGANgenerated images using conditional continuous normalizing flows. ACM Trans. Graph., 40(3), 2021. 1
Amir, S., Gandelsman, Y., Bagon, S., and Dekel, T.
Deep VIT features as dense visual descriptors. CoRR,
abs/2112.05814, 2021. 5
Balaji, Y., Nah, S., Huang, X., Vahdat, A., Song, J., Kreis,
K., Aittala, M., Aila, T., Laine, S., Catanzaro, B., et al.
eDiff-I: Text-to-image diffusion models with an ensembleof expert denoisers. CoRR, abs/2211.01324, 2022. 2, 7, 8
Bermano, A. H., Gal, R., Alaluf, Y., Mokady, R., Nitzan,
Y., Tov, O., Patashnik, O., and Cohen-Or, D. State-ofthe-art in the architecture, methods and applications of
StyleGAN. CoRR, abs/2202.14020, 2022. 1
Brock, A., Donahue, J., and Simonyan, K. Large scale GAN
training for high fidelity natural image synthesis. Proc.
ICLR, 2019. 6

Crowson, K., Biderman, S., Kornis, D., Stander, D., Hallahan, E., Castricato, L., and Raff, E. VQGAN-CLIP:
Open domain image generation and editing with natural
language guidance. In Proc. ECCV, 2022. 6
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei,
L. Imagenet: A large-scale hierarchical image database.
In Proc. CVPR, 2009. 2
Desai, K., Kaul, G., Aysola, Z., and Johnson, J. RedCaps:
web-curated image-text data created by the people, forthe people. In Proc. Neurips (Datasets and Benchmarks),2021. 12Dhariwal, P. and Nichol, A. Diffusion models beat GANs
on image synthesis. In Proc. NeurIPS, 2021. 2, 4, 5Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn,D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M.,
Heigold, G., Gelly, S., et al. An image is worth 16x16
words: Transformers for image recognition at scale. In
Proc. ICLR, 2021. 5
Gafni, O., Polyak, A., Ashual, O., Sheynin, S., Parikh, D.,
and Taigman, Y. Make-a-scene: Scene-based text-toimage generation with human priors. In Proc. ECCV,
2022. 1
Gal, R., Alaluf, Y., Atzmon, Y., Patashnik, O., Bermano,
A. H., Chechik, G., and Cohen-Or, D. An image is worth
one word: Personalizing text-to-image generation using
textual inversion. CoRR, abs/2208.01618, 2022. 8
Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B.,
Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y.
Generative adversarial networks. In Proc. NIPS, 2014. 1

Caron, M., Touvron, H., Misra, I., Jégou, H., Mairal, J.,
Bojanowski, P., and Joulin, A. Emerging properties inself-supervised vision transformers. In Proc. ICCV, 2021.
5

Härkönen, E., Hertzmann, A., Lehtinen, J., and Paris, S.
GANSpace: Discovering interpretable GAN controls. In
Proc. NeurIPS, 2020. 1

Changpinyo, S., Sharma, P., Ding, N., and Soricut, R. Conceptual 12M: Pushing web-scale image-text pre-training
to recognize long-tail visual concepts. In Proc. CVPR,
2021. 12Härkönen, E., Aittala, M., Kynkäänniemi, T., Laine, S., Aila,
T., and Lehtinen, J. Disentangling random and cyclic
effects in time-lapse sequences. ACM Trans. Graph., 41(4), 2022. 5

Chrysos, G., Moschoglou, S., Bouritsas, G., Deng, J., Panagakis, Y., and Zafeiriou, S. Deep polynomial neural networks. In Proc. CVPR, 2020. 5

Hessel, J., Holtzman, A., Forbes, M., Bras, R. L., and Choi,
Y. CLIPScore: A reference-free evaluation metric forimage captioning. In Proc. EMNLP, 2021. 2

Chrysos, G. G. and Panagakis, Y. CoPE: conditional image generation using polynomial expansions. In Proc.
NeurIPS, 2021. 5

Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., andHochreiter, S. GANs trained by a two time-scale update
rule converge to a local Nash equilibrium. Proc. NeurIPS,
2017. 2

CompVis. Stable diffusion model card, 2022. URL
https://github.com/CompVis/stable-diffusion/blob/main/
Stable Diffusion v1 Model Card.md. 2, 6

Ho, J. and Salimans, T. Classifier-free diffusion guidance.
CoRR, abs/2207.12598, 2022. 5

StyleGAN-T: Unlocking the Power of GANs for Fast Large-Scale Text-to-Image Synthesis

Hoffer, E., Hubara, I., and Soudry, D. Train longer, generalize better: closing the generalization gap in large batch
training of neural networks. In Proc. NeurIPS, 2017. 5

Marchesi, M. Megapixel size image creation using generative adversarial networks. CoRR, abs/1706.00082, 2017.
6

Ioffe, S. and Szegedy, C. Batch normalization: Accelerating
deep network training by reducing internal covariate shift.
In Proc. ICML, 2015. 5

Meng, C., Gao, R., Kingma, D. P., Ermon, S., Ho, J., andSalimans, T. On distillation of guided diffusion models.
CoRR, abs/2210.03142, 2022. 7

Kafri, O., Patashnik, O., Alaluf, Y., and Cohen-Or, D. StyleFusion: A generative model for disentangling spatial segments. ACM Trans. Graph., 41(5):1–15, 2022. 1

Miyato, T. and Koyama, M. cGANs with projection discriminator. In Proc. ICLR, 2018. 2

Karras, T., Aila, T., Laine, S., and Lehtinen, J. Progressive
growing of GANs for improved quality, stability, andvariation. In Proc. ICLR, 2018. 2

Nichol, A., Dhariwal, P., Ramesh, A., Shyam, P., Mishkin,P., McGrew, B., Sutskever, I., and Chen, M. GLIDE:
Towards photorealistic image generation and editing with
text-guided diffusion models. In Proc. ICML, 2022. 6

Karras, T., Laine, S., and Aila, T. A style-based generator
architecture for generative adversarial networks. In Proc.
CVPR, 2019. 1, 2, 6
Karras, T., Laine, S., Aittala, M., Hellsten, J., Lehtinen, J.,
and Aila, T. Analyzing and improving the image qualityof StyleGAN. In Proc. CVPR, 2020. 1, 2
Karras, T., Aittala, M., Laine, S., Härkönen, E., Hellsten, J.,
Lehtinen, J., and Aila, T. Alias-free generative adversarial
networks. In Proc. NeurIPS, 2021. 1, 2, 4
Karras, T., Aittala, M., Aila, T., and Laine, S. Elucidating
the design space of diffusion-based generative models. In
Proc. NeurIPS, 2022. 1
Kynkäänniemi, T., Karras, T., Aittala, M., Aila, T., andLehtinen, J. The role of ImageNet classes in Fréchet
inception distance. CoRR, abs/2203.06026, 2022. 5
Lambda Labs. All you need is one gpu: Inference benchmark for stable diffusion, 2022. URL https://lambdalabs.
com/blog/inference-benchmark-stable-diffusion. 7
Lim, J. H. and Ye, J. C. Geometric Gan. CoRR, 2017. 5
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P.,
Ramanan, D., Dollár, P., and Zitnick, C. L. Microsoft
COCO: Common objects in context. In Proc. ECCV,2014. 2
Liu, X., Gong, C., Wu, L., Zhang, S., Su, H., and Liu,
Q. FuseDream: Training-free text-to-image generation
with improved clip+ gan space optimization. CoRR,
abs/2112.01573, 2021. 6
Liu, Z., Mao, H., Wu, C.-Y., Feichtenhofer, C., Darrell, T.,
and Xie, S. A convnet for the 2020s. In Proc. CVPR,
2022. 4
Lu, C., Zhou, Y., Bao, F., Chen, J., Li, C., and Zhu, J. DPMSolver: A fast ODE solver for diffusion probabilistic
model sampling in around 10 steps. In Proc. NeurIPS,
2022. 1

Patashnik, O., Wu, Z., Shechtman, E., Cohen-Or, D., andLischinski, D. StyleCLIP: Text-driven manipulation of
StyleGAN imagery. In Proc. ICCV, 2021. 8
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G.,
Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J.,et al. Learning transferable visual models from natural
language supervision. In Proc. ICML, 2021. 2
Ramesh, A., Dhariwal, P., Nichol, A., Chu, C., and Chen,
M. Hierarchical text-conditional image generation with
CLIP latents. CoRR, abs/2204.06125, 2022. 1
Reed, S., Akata, Z., Mohan, S., Tenka, S., Schiele, B., andLee, H. Learning what and where to draw. In Proc.
NeurIPS, 2016a. 2
Reed, S., Akata, Z., Yan, X., Logeswaran, L., Schiele, B.,
and Lee, H. Generative adversarial text to image synthesis.
In Proc. ICML, 2016b. 2
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., andOmmer, B. High-resolution image synthesis with latentdiffusion models. In Proc. CVPR, 2022. 1
Ruiz, N., Li, Y., Jampani, V., Pritch, Y., Rubinstein, M.,
and Aberman, K. Dreambooth: Fine tuning text-to-image
diffusion models for subject-driven generation. CoRR,
abs/2208.12242, 2022. 8
Saharia, C., Chan, W., Saxena, S., Li, L., Whang, J., Denton, E., Ghasemipour, S. K. S., Ayan, B. K., Mahdavi,
S. S., Lopes, R. G., et al. Photorealistic text-to-image
diffusion models with deep language understanding. In
Proc. NeurIPS, 2022. 1, 8
Salimans, T. and Ho, J. Progressive distillation for fastsampling of diffusion models. CoRR, abs/2202.00512,
2022. 1
Sauer, A., Chitta, K., Müller, J., and Geiger, A. Projected
GANs converge faster. In Proc. NeurIPS, 2021. 2, 5

StyleGAN-T: Unlocking the Power of GANs for Fast Large-Scale Text-to-Image Synthesis

Sauer, A., Schwarz, K., and Geiger, A. StyleGAN-XL:
Scaling StyleGAN to large diverse datasets. In Proc.
SIGGRAPH, 2022. 2
Schuhmann, C., Beaumont, R., Vencu, R., Gordon, C.,
Wightman, R., Cherti, M., Coombes, T., Katta, A., Mullis,C., Wortsman, M., et al. LAION-5B: An open large-scale
dataset for training next generation image-text models. In
Proc. NeurIPS, 2022. 1, 2, 12
Sharma, P., Ding, N., Goodman, S., and Soricut, R. Conceptual captions: A cleaned, hypernymed, image alt-text
dataset for automatic image captioning. In Proc. ACL,
volume 1, pp. 2556–2565, 2018. 12
Shen, Y., Gu, J., Tang, X., and Zhou, B. Interpreting thelatent space of GANs for semantic face editing. In Proc.
CVPR, 2020. 1
Singh, A., Hu, R., Goswami, V., Couairon, G., Galuba,
W., Rohrbach, M., and Kiela, D. Flava: A foundational
language and vision alignment model. In Proc. CVPR,
2022. 12Tan, M. and Le, Q. EfficientNet: Rethinking model scaling
for convolutional neural networks. In Proc. ICML, 2019.
2
Tao, M., Tang, H., Wu, F., Jing, X.-Y., Bao, B.-K., and Xu,
C. DF-GAN: A simple and effective baseline for text-toimage synthesis. In Proc. CVPR, 2022. 5
Thomee, B., Shamma, D. A., Friedland, G., Elizalde, B., Ni,
K., Poland, D., Borth, D., and Li, L.-J. Yfcc100m: The
new data in multimedia research. Communications of theACM, 59(2):64–73, 2016. 12
Touvron, H., Cord, M., Douze, M., Massa, F., Sablayrolles,
A., and Jegou, H. Training data-efficient image transformers & distillation through attention. In Proc. ICML,
2021a. 2
Touvron, H., Cord, M., Sablayrolles, A., Synnaeve, G., andJégou, H. Going deeper with image transformers. In Proc.
ICCV, 2021b. 4
Wang, X., Yu, K., Wu, S., Gu, J., Liu, Y., Dong, C., Qiao, Y.,
and Change Loy, C. Esrgan: Enhanced super-resolution
generative adversarial networks. In Proc. ECCV, 2018. 2
Wu, Y. and He, K. Group normalization. In Proc. CVPR,
2018. 4Yu, J., Xu, Y., Koh, J. Y., Luong, T., Baid, G., Wang, Z., Vasudevan, V., Ku, A., Yang, Y., Ayan, B. K., et al. Scaling
autoregressive models for content-rich text-to-image generation. CoRR, abs/2206.10789, 2022. 1

Zhang, H., Yin, W., Fang, Y., Li, L., Duan, B., Wu, Z.,
Sun, Y., Tian, H., Wu, H., and Wang, H. ERNIE-ViLG:
Unified generative pre-training for bidirectional visionlanguage generation. CoRR, abs/2112.15283, 2021. 1
Zhao, S., Liu, Z., Lin, J., Zhu, J.-Y., and Han, S. Differentiable augmentation for data-efficient GAN training. In
Proc. NeurIPS, 2020. 5
Zhou, Y., Zhang, R., Chen, C., Li, C., Tensmeyer, C., Yu, T.,
Gu, J., Xu, J., and Sun, T. Towards language-free training
for text-to-image generation. In Proc. CVPR, 2022. 1, 5

StyleGAN-T: Unlocking the Power of GANs for Fast Large-Scale Text-to-Image Synthesis

A. Configuration Details
Table 4 lists the training and network architecture hyperparameters for our two configurations: lightweight (used for
ablations) and the full configuration (used for main results). Table 5 details the training schedules.
Lightweight training configuration. We train using the CC12M dataset (Changpinyo et al., 2021) at 64×64 resolution,
without using progressive growing.
Full training configuration.
We train using a union of several datasets: CC12m (Changpinyo et al., 2021),
CC (Sharma et al., 2018), YFCC100m (filtered) (Thomee et al., 2016; Singh et al., 2022), Redcaps (Desai et al., 2021),
LAION-aesthetic-6+ (Schuhmann et al., 2022). This amounts to a total of 250M text-image pairs. We use progressive growing similar to StyleGAN-XL, except that all layers remain trainable. The vast majority of the training budget is spent on
resolutions up to 64×64.

B. Truncation Grids
Fig. 10 shows additional examples of truncation.

Generator channel base
Generator channel max
Number of residual blocks per generator block
Generator parameters
Text encoder parameters
Latent (z) dimension
Discriminator’s feature network
Discriminator head’s input feature space size
Discriminator head’s feature space size at text conditioning
Dataset size
Number of GPUs
Batch size
Optimizer
Generator learning rate
Generator Adam betas
Discriminator learning rate
Discriminator Adam betas
EMA
CLIP guidance weight
Progressive growing

Lightweight
32768
512
3
75 million
123 million
64
DINO ViT-S/16
384
64
12M
8
2048
Adam
0.002
(0, 0.99)
0.002(0, 0.99)
0.9978
0.2
No

Full
65536
2048
4
1.02 billion
123 million
64
DINO ViT-S/16
384
64
250M
64
2048
Adam
0.002
(0, 0.99)
0.002(0, 0.99)
0.99780.2 (primary phase), 50 (secondary phase)
Yes

Table 4. Generator, discriminator, and training hyperparameters for the two setups used in this paper: Lightweight and Full configuration.
LightweightPrimary Phase
64x64 for 50 A100 days (25 million iterations)

Full
Primary Phase
16x16 for 450 A100 days (118,000 iterations)
32x32 for 450 A100 days ( 78,000 iterations)
64x64 for 450 A100 days ( 57,000 iterations)
Secondary Phase
190 A100 days (20,000 iterations)Primary Phase
128x128 for 96 A100 days (10,000 iterations)
256x256 for 70 A100 days ( 6,000 iterations)
512x512 for 30 A100 days ( 3,000 iterations)

Table 5. Training schedules for the two training configurations used in this paper. The times are listed as the number of days it would
have taken on a single NVIDIA A100 GPU. An iteration corresponds to 2048 real and generated examples.

StyleGAN-T: Unlocking the Power of GANs for Fast Large-Scale Text-to-Image Synthesis

ψ = 1.00

ψ = 1.00

CS = 0.34

CS = 0.30

ψ = 0.60

ψ = 0.60

CS = 0.36

CS = 0.31

ψ = 0.10

ψ = 0.10

CS = 0.36

CS = 0.32“A painting of a fox in the style of starry night”“A surrealist dream-like oil painting by Salvador Dalı́ of a cat playing checkers”

ψ = 1.00

ψ = 1.00

CS = 0.26

CS = 0.32

ψ = 0.60

ψ = 0.60

CS = 0.28

CS = 0.33

ψ = 0.10

ψ = 0.10

CS = 0.31

CS = 0.34“Robots meditating in a vipassana retreat”“A teddy bear on a skateboard in times square”

ψ = 1.00

ψ = 1.00

CS = 0.28

CS = 0.39

ψ = 0.60

ψ = 0.60

CS = 0.29

CS = 0.40

ψ = 0.10

ψ = 0.10

CS = 0.30

CS = 0.40“A still of Kermit The Frog in WALL-E (2008)”“A transformer robot with legs and arms made out of vegetation and leaves”

Figure 10. Additional truncation grids. We show samples for 6 different prompts and 5 different random latents, shared between theprompts. Increasing truncation (decreasing ψ), improves the text alignment according to mean CLIP score per row, CS, at the cost oflower variation.