Scalable GANs with Transformers
Authors: name: Sangeek Hyun, name: MinKyu Lee, name: Jae-Pil Heo

Scalability has driven recent advances in generative modeling, yet its
principles remain underexplored for adversarial learning. We investigate the
scalability of Generative Adversarial Networks (GANs) through two design
choices that have proven to be effective in other types of generative models:
training in a compact Variational Autoencoder latent space and adopting purely
transformer-based generators and discriminators. Training in latent space
enables efficient computation while preserving perceptual fidelity, and this
efficiency pairs naturally with plain transformers, whose performance scales
with computational budget. Building on these choices, we analyze failure modes
that emerge when naively scaling GANs. Specifically, we find issues as
underutilization of early layers in the generator and optimization instability
as the network scales. Accordingly, we provide simple and scale-friendly
solutions as lightweight intermediate supervision and width-aware learning-rate
adjustment. Our experiments show that GAT, a purely transformer-based and
latent-space GANs, can be easily trained reliably across a wide range of
capacities (S through XL). Moreover, GAT-XL/2 achieves state-of-the-art
single-step, class-conditional generation performance (FID of 2.96) on
ImageNet-256 in just 40 epochs, 6x fewer epochs than strong baselines.