The GAN is dead; long live the GAN! A Modern GAN Baseline
Authors: Nick Huang
Aaron Gokaslan
Volodymyr Kuleshov
James Tompkin
Venue: NeurIPS 2024 poster (2024)
Keywords: GAN
Primary area: generative_models
TL;DR: Makes GAN training stable with math (!), then uses this to create simple modern baseline architecture with SOTA-competitive performance.

Abstract: There is a widely-spread claim that GANs are difficult to train, and GAN architectures in the literature are littered with empirical tricks. We provide evidence against this claim and build a modern GAN baseline in a more principled manner. First, we derive a well-behaved regularized relativistic GAN loss that addresses issues of mode dropping and non-convergence that were previously tackled via a bag of ad-hoc tricks. We analyze our loss mathematically and prove that it admits local convergence guarantees, unlike most existing relativistic losses. Second, this loss allows us to discard all ad-hoc tricks and replace outdated backbones used in common GANs with modern architectures. Using StyleGAN2 as an example, we present a roadmap of simplification and modernization that results in a new minimalist baseline---R3GAN. Despite being simple, our approach surpasses StyleGAN2 on FFHQ, ImageNet, CIFAR, and Stacked MNIST datasets, and compares favorably against state-of-the-art GANs and diffusion models. Code: https://www.github.com/brownvc/R3GAN

The GAN is dead; long live the GAN! A Modern Baseline GAN
Yiwen Huang Brown University
Aaron Gokaslan Cornell University
Volodymyr Kuleshov Cornell University
James Tompkin Brown University
Abstract
There is widely-spread claim that GANs are difficult to train, and GAN archi- tectures in the literature are littered with empirical tricks. We provide evidence against this claim and build a modern GAN baseline in more principled manner. First, we derive a cell-behaved regularized relativistic GAN loss that addresses issues of mode dropping and non-convergence that were previously tackled via a bag of ad-hoe tricks. We analyze our loss mathematically and prove that admits local convergence guarantees, unlike most existing relativistic losses. Second, this loss allows us to discard all ad-hoc tricks and replace outdated backbones used in common GANs with modern architectures. Using StyleGAN2 as an example, we present a roadmap of simplification and modernization that results in a new minimalist baseline- R3GAN ("Re-GAN") Despite being simple, our approach surpasses StyleGAN2 on FFHQ, ImageNet, CIFAR, and Stacked MNIST datasets, and compares favorably against state-of-the-as GANs and diffusion models. Code: http::/wwwwwwww...b/bbbo/w////// htt..//..ooooooo//////NNN
Introduction
Generative adversarial networks (GANs) let generate high-quality images in single forward pass. However, the original objective in Goodfellow et al. [13], is notoriously difficult to optimize due to its minimax nature. This leads fear that training might diverge at any point due instability, and fear that generated images might lose diversity through mode collapse. While there has been progress in GAN objectives [14, 22, 81, 52, 64], practically, the effects of brittle losses are still regularly felt. This notoriety has had a lasting negative impact on GAN research.
A complementary issue partly motivated by this instability is that existing popular GAN backbones like StyleGAN [29, 31, 30, 32] use many worlyyunddrstood empirical tricks with little theory. For instance, StyleGAN uses gradient penalized non-saturating loss [52] to increase stability (affecting sample diversity), but then employs a minibatch standard deviation trick [28] to increase sample diversity. Without tricks, the StyleGAN backbone still resembles DCGAN [60] from 2015, yet is still the common backbone of SOTA GANs such as GigaGAN [26) and StyleGAN-T [70]. Advances in GANs have been conservative compared to other generative models such diffusion models [20, 78, 33, 34], where modern computer vision techniques such as multi-headed self attention [87] and backbones such as preactivated ResNet [17], U-Net [63] and vision transformers (ViTs) [9] are the norm. Given outdated backbones, it is not surprising that there is widely-spread belief that GANs do not scale in terms of quantitative metrics like Frechet Inception Distance [19].
We reconsider this situation: we show that by combining progress in objectives into a regularized training loss, GANs gain improved training stability, which allows us upgrade GANs with modern backbones. First, we propose a novel objective that augments the relativistic pairing GAN loss (RpGAN; [22]) with zero-centered gradient penalties [52, 64], improving stability [14, 64, 52]. We show mathematicalyy that gradient-penalizer RpGAN enjoys the same guarantee of local convergence as regularized classic GANs, and that removing our regularization scheme induces non-convergence.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Once we have well-behaved loss, none of the GAN tricks are necessary [28, 311, and we are free to engineer modern SOTA backbone architecture. We strip StyleGAN of all its features. identify those that are essential, then borrow new architecture designs from modern ConvNets and transformers [48, 97]. Briefly, we find that proper ResNet design [17, 67], initialization [99], and resampling [29, 31, 32, 100] are important, along with grouped convolution [95, 5] and no normalization [31, 34, 14, 88, 4]. This leads to design that is simpler than StyleGAN and improves FID performance for the same network capacity (2.75 vs. 3.78 on FFHQ-256).
In summary, our work first argues mathematically that GANs need not be tricky to train via an improved regularized loss. Then, it empirically develops a simple GAN baseline that, without any tricks, compares favorably by FID to StyleGAN [29, 31, 321, other SOTA GANs [3, 42, 94], and diffusion models [20, 78, 86] across FFHQ, ImageNet, CIFAR, and Stacked MNIST datasets.
2 Serving Two Masters: Stability and Diversity with RpGAN +R + Rs
In defining a GAN objective, we tackle two challenges: stability and diversity. Some previous work deals with stability [29, 31, 32] and other previous work deals with mode collapse [22]. To make progress both, we combine a stable method with simple regularizer that is grounded by theory.
2.1 Traditional GAN
traditional GAN [13, 57] is formulated as a minimax game between a discriminator (or critic) D and generator Gg. Given real data x ~ pD and fake data x ~ pe produced by Gg. the most general form of a GAN is given by:
[(0, v) = Ezzp, [/ (D& + Ra-pp [s (x))
(I)
where tries to minimize f. while tries to maximize it. The choice of f is flexible [50, 44]. In particular, f(t) = log(1 + e-t) recovers the classic GAN by Goodfellow et al. [13]. For the rest of this work, this will be our choice of f [57].
It has been shown that Equation 1 has convex properties when pa can be optimized directly [13, 81]. However, in practical implementations, the empirical GAN loss typically shifts fake samples beyond the decision boundary set by D, as opposed to directly updating the density function Pg. This deviation leads to a significantly more challenging problem, characterized by susceptibility to two prevalent failure scenarios: mode collapse/dropping and non-convergence.
2.2 Relativistic f-GAN
We employ a slightly different minimax game named relativistic pairing GAN (RpGAN) by Jolicoeur- Martineau et al. [22] to address mode dropping. The general RpGAN is defined as:
[(0, v) = Erpp x~pp
(D+ - -<<())))D(((()))
Although Eq. differs only slightly from Eq. 1, evaluating this critic difference has fundamental impact on the landscape of L. Since Eq. 1 merely requires D to separate real and fake data, in the scenario where all real and fake data can be separated by a single decision boundary, the empirical GAN loss encourages G to simply move all fake samples barely past this single boundary- this degenerate solution is what we observe as mode collapse/dropping. Sun el al. [81] characterize such degenerate solutions as bad local minima in the landscape of L, and show that Eq. I has exponentially many bad local minima. The culprit is the existence of a single decision boundary that naturally arises when real and fake data are considered in isolation. RpGAN introduces a simple solution by coupling real and fake data, i.e. a fake sample is critiqued by its realness relative to a real sample, which effectively maintains decision boundary in the neighborhood of each real sample and hence forbids mode dropping. Sun et al. [81] show that the landscape of Eq. 2 contains no local minima that correspond mode dropping solutions, and that every basin is global minimum.
While mode collapse and mode dropping are technically distinct issues, they are used interchangeably in this context to describe the common problem where supp(pe) does not comprehensively cover supp(pp). Mode collapse refers to the generator producing a limited diversity of samples (i.e., one image for the entire distribution). whereas mode dropping involves the generator failing to represent certain modes of the data distribution (ignoring entire subsets of the training distribution).

---

2.3 Training Dynamics of RpGAN
Although the RpGAN landscape result [81] allows us to address mode dropping, the training dynamics of RpGAN have yet be studied. The ultimate goal of Eq. 2 is find the equilibrium (0* xb*) such that po- pD and De- is constant everywhere on pD- Sun et al. [81] show that 0* is globally reachable along non-increasing trajectory in the landscape of Eq. 2 under reasonable assumptions. However, the existence of such a trajectory does not necessarily mean that gradient descent will find it. Jolicoeur-Martineau al. show empirically that unregularized RpGAN does not perform well [22].
Proposition I. (Informal) Unregularized RpGAN does not always converge using gradient descent.
We confirm this proposition with a proof in Appendix B. We show analytically that RpGAN does not converge for certain types of pD. such as ones that approach delta distribution. Thus, further regularization is necessary to fill in the missing piece of a well-behaved loss.
Zero-centered gradient penalties. To tackle RpGAN non-convergence, explore gradient penalties as the solution since it is proven that zero-centered gradient penalties (0-GP) facilitate convergent training for classic GANs [52]. The two most commonly-used O-GPs are R₃ and R2:
R1 (v) = 7 EEEPP
V
2
R2(0, = " Exnppo
D+1?
2
Ri penalizes the gradient norm of on real data, and Rg penalizes the gradient norm of D on fake data. Analysis on the training dynamics of GANs has thus far focused on local convergence [55, 51, 52]. ie., whether the training at least converges when (f, a/-) are in a neighborhood of ,/"). In such a scenario, the convergence behavior can be analyzed [55, 51, 52] by examining the spectrum of the Jacobian of the gradient vector field Vo, v. at (0", 25*) The key insight here is that when G already produces the true distribution, from its optimal state, and thus the training does not oscillate. R4 and R2 impose such a constraint when pe = PD- This also explains why earlier attempts at gradient penalties, such the one-centered gradient penalty (1-GP) in WGAN-GP [14], fail to achieve convergent training [52] as they still encourage D have a non-zero slope when G has reached optimality.
want D O, that G is not pushed away
Since the same insight also applies to RpGAN, we extend our previous analysis and show that:
Proposition II. (Informal) RpGAN with Ri or R2 regularization locally convergent subject to similar assumptions as in Mescheder et al. [52].
In Appendix C, our proof similarly analyzes the eigenvalues the Jacobian of the regularized RpGAN gradient vector field at (8", *) We show that all eigenvalues have negative real part; thus, regularized RpGAN is convergent in a neighborhood of (#*, 1²)) for small enough learning rates [52].
Discussion. Another line of work [64] links R1 and R2 to instance noise [75] as its analytical approximation. Roth et al. [64] showed that for the classic GAN [13] by Goodfellow er al., R approximates convolving PD with the density function of (0, 7I). up to additional weighting and a Laplacian error term. R2 likewise approximates convolving ps with (0, 77) up to similar error terms. The Laplacian error terms from R1, R2 cancel when D,s approaches Da... We do not extend Roth et al. 's proof [64] to RpGAN; however, this approach might provide complimentary insights to our work, which follows the strategy of Mescheder et al. [52].
2.4 A Practical Demonstration
We experiment with how well-behaved our loss on StackedMNIST [46] which consists of 1000 uniformly-de distriu-d modes. The network is a small ResNet [17] for G and D without any normal- ization layers [21, 91, 1, 85]. Through the use of a pretrained MNIST classifier, we can explicitly measure how many modes of pp are recovered by pg. Furthermore, we can estimate the reverse KL. divergence between the fake and real samples Dxt. (pe PD) via the KI. divergence between the categorical distribution of pe and the true uniform distribution.
3

---

RoGAN
GAN R; + Rz
RoGAN R:
Generator lots
a0*
Loss
# modest
DRL.
RpGAN +R1 + R
1000
0.0781
GAN +R1 + R
693
0.9270
RpGAN +R1
Fail
Fail
GAN +R1
Fail
Fail
Figure| Generator G' loss for different objec- tives over training. Regardless of which objective used, training diverges with only R, and suc- ceeded with both Ri and R2. Convergence failure with only R1 was noted by Lee et al. [42].
Table 1: StackedMNIST [46] result for each loss function. The maximum possible mode coverage is 1000. "Fail" indicates that training diverged early on.
conventional GAN loss with R1, as used by Mescheder et al. [52] and the StyleGAN series [29, 31, 32], diverges quickly (Fig. 1). Next, while theoretically sufficient for local convergence, RpGAN with only R1 regularization is also unstable and diverges quickly'. In each case, the gradient of D on fake samples explodes when training diverges. With both R1 and Rg, training becomes stable for both the classic GAN and RpGAN. Now stable, we can see that the classic GAN suffers from mode dropping, whereas RpGAN achieves full mode coverage (Tab. 1) and reduces DKI. from 0.9270 to 0.0781. As a point of contrast, StyleGAN [29, 31, 30, 32] uses the minibatch standard deviation trick to reduce mode dropping, improving mode coverage from 857 to 881 on StackedMNIST and with barely any improvement on DKL [28].
training. While a theoretical analysis of this is difficult, our small demonstration still provides insights into the assumptions of our convergence proof. In particular, the assumption that (0, 4)) are sufficiently close to (0", x/*)) is highly unlikely early in training. In this scenario, if D is sufficiently powerful, regularizing
Rj alone is not sufficient for globally-converge
solely on real data not likely to have much effect on D's behavior on fake data and so training can fail due to an ill- bebaeeeeedd D on fake data. This observation has been made by previous studies [84, 83] specifically for empirical GAN training, that regularizing an empirical discriminator with only Rt leads to gradient explosion on fake data due to the memorization of real samples.
Thus, the practical solution is to regularize D on both real and fake data. The benefit of doing so can be viewed from the insight of Roth et al. [64]: that applying Rí and Rg conjunction smooths both PD and Po which makes learning casier than only smoothing PD. We also find empirically that with [IV x Enwpe
[| even early in the training. Kolicoeur-Martinenn el al. [23] show that in this case D becomes maximum margin blassifier-but only one regularization term is applied, this does not hold. Additionally, having roughly the same gradient norm on real and fake data potentially reduces discriminator overfitting, as Fang et al. [10] observe that the gradient norm on real and fake data diverges when D starts to overfit.
both R₁ and R2 in place, D tends to satisfy Eg-pp
3 A Roadmap to a New Baseline - -RGAN
The well-behaved RpGAN + R₁ + Rs loss alleviates GAN optimization problems, and lets us proceed to build a minimalist
recent network backbone advances in mind [48, 97]. Rather than simply state the new approach, we will draw out a roadmap from the StyleGAN2 baseline [30]. This model (Config A; identical to [30]) consists of VGG-like [73] backbone for G, a ResNet D, a few techniques that facilitate style-based generation, and many tricks that serve as patches to the weak backbone. Then, we remove all non-essential features of StyleGAN2 (Config B), apply our loss function (Config C), and gradually modernize the network backbone (Config D-E).
VVarying > from 0.1 to 100 does not stabilize training.
"These numbers are from Karras et al. [28]. Table 4. "857" corresponds to a law-ccpycity version of a progressive GAN and "881" adds the minibatch standard deviation trick. Further comparisons via loss curves are difficult since progressive GAN is substantially different model than the small ResNet use for this experiment.

---

We evaluate each configuration on FFHQ 256 x 256 [29]. Network capacity is kept roughly the same for all configurations- both G and D have about 25 M trainable parameters. Each configuration is trained until 1) sees 5 M real images. We inherit training hyperparameters (e.g., optimizer settings, batch size, EMA decay length) from Config unless otherwise specified. We tune the training hyperparameters for our final model and show the converged result in Sec. 4.
Minimum baseline (Config B). We strip away all StyleGAN2 features, taining only the raw network backbone and basie image generation capability. The features fall into three categories:
Configuration
FIDI
G #params
D #params
A
StyleGAN2
7,516
24.767M
24.001M
B Stripped StyleGAN2
22 normalization
Minihatch stddev
Equalized learning rate
Style-based generation: mapping net- work [29], style injection [29], weight
Mapping network
Style injection
modullttion/demodullaiin [31], noise injection [29].
Weight mod/ demod
12.46
18.890M
23.996M
NNoie injection
Mixing regularization
Path length regularization
Image manipulation enhancements:
Lazy regularization
mixing regularization [29], path
C Well-behaved Loss
length regularization [31].
RpGAN loss
11.77
18.890M
23.996M
Tricks: z normalization [28], mini- batch stddev [28], equalized learning
Rs gradient penalty
11.65
D ConvNeXt-ify pt.
rate [28], lazy regularization [31].
ResNet-ify G & D
10.17
23.400M
23.282M
-Ottput skips
9.950
23.378M
Following [69, 70]. we reduce the dimen-
E ConvNeXt-ify pt.
sion of to 64. The absence of equal-
+ ResNeXt-ify G & D. 7.507
23.188M
23.091 M
ized learning rate necessitates lower learning rate, reduced from 2,5x10* to
+ Inverted bottleneck
7.045
23.058M
23.010M
Table Effect of our simplification and modernization efforts evaluted on FFHQ-256.
5x 10 Despite a higher FID of 12.46 than Co this simplified baseline produces reasonable sample quality and stable training. We compare this with DCGAN [60], an early attempt at image generation. Key differences include:
a) Convergent training objective with R1 regularization.
b) Smaller learning rate, avoiding momentum optimizer (Adam 31 0).
C) No normalization layer in G or D.
d) Proper resampling via bilinear interpolation instead of strided (transposed) convolution.
e) Leaky ReLU in both G' and D, no tanh in the output layer of G.
f) 4x4 constant input for G, output skips for G, ResNet D.
Experimental findings from StyleGAN. Violating a), b), c) often leads to training failures. Gidel er al. [1]] show that negative momentum can improve GAN training dynamics. Since optimal negative momentum is another challenging hyperparameter, we do not use any momentum to avoid worsening GAN training dynamics. Studies suggest normalization layers harm generative models [31, 34]. Batch normalization [21] often cripples training due to dependencies across multiple samples, and is incompatible with R, Rgg or RpGAN that assume independent handling of each sample. Weaker data-independent normalizations [31, 34] might help: we leave this for future work. Early GANs may succeed despite violating a) and c). possibly constituting a full-rank solution [52] to Eq.
Violations of d) or e) do not significantly impair training stability but negatively affect sample quality. Improper transposed convolution can cause checkerboard artifacts, unresolved even with subpixel convolution [72] or carefully tuned transposed convolution unless a low-pass filter is applied. Interpolation methods avoid this issue, varying from nearest neighbor [28] to Kaiser filters [32]. We use bilinear interpolation for simplicity. For activation functions, smooth approximations of (leaky) ReLU, such as Swish [61]. GELU [18], and SMU [2]. worsen FID. PReLU [15] marginally improves FID but increases VRAM usage, so we use leaky ReLU,
All subsequent configurations adhere to a) through e). Violation of f) is acceptable as it pertains to the network backbone of StyleGAN2 [31], modernized in Config D and E.
Well-behaved loss function (Config C). We use the loss function proposed in Section and this
reduces FID to 11.65. We hypothesize that the network backbone in Config B is the limiting factor.
General network modernization (Config D). First, we apply the 1-3-1 bottleneck ResNet archi- tecture [16, 17] to both G and D. This is the direct ancestor of all modern vision backbones [48, 97].
5

---

Plncftatio
To 13G
wooght
Taloew
Mitaor 3x3
Lowa
Mili
LeoksBell
Risew
Bis
Biar
Ilustaa Block
Ouey
Nitee:
[wiita Blluk
Eqea
Bick
Ru (D00
Hillowr
Ray
Styra 3
Tealsfuute
Biss
RyG 1
Nodiual Back
wigght
Miud
Nonteal Sect
EeCice 1
(a) Overall view
(b) StyleGAN2 architecture blocks [31] (Config A)
(c) Ours (Config E)
Figure 2: Architecture comparison. For image generation, G and D are often both deep ConvNets with either partially or fully symmetric architectures. (a) StyleGAN2 [31] G uses network to map noise vector 2 to an intermediate style space W. We use a traditional generator as style mapping is not necessary for a minimal working model. (b) StyleGAN2's s building blocks have intricate layers but are themselves simple, with ConvNet architecture from 2015 [38, 73, 16]. ResNet's identity mapping principle also violated in the discriminator.
(c) We remove tricks and modernize the architecture. Our design has clean layers with a more powerful ConvNet architecture.
We also incorporate principles discovered in Config B and various modernization efforts from ConvNeXt [48]. We categorize the roadmap of ConvNeXt as follows:
i. Consistently beneficial: i.1) increased width with depthwise convolution, i.2) inverted bottleneck, i.3) fewer activation functions, and i.4) separate resampling layers.
ii Negligible performance gain: ii.1) large kernel depthwise conv. with fewer channels, ii.2) swap ReLU with GELU, ii.3) fewer normalization layers, and ii.4) swap batch norm. with layer norm. ii.. Irrelevant to our setting: iii. 1) improved training recipe, iii.2) stage ratio, and iii.3) 'patchify' stem.
We aim to apply i) our model, specifically i.3 and i.4 for the classic ResNet, while reserving i. and i.2 for Config E. Many aspects of ii) were introduced merely to mimic vision transformers [47, 9] without yielding significant improvements [48]. ii.3 and ii.4 are inapplicable due to our avoidance of normalization layers following principle c). ii.2 contradicts our finding that GELU deteriorates GAN performance, thus we use leaky ReLU per principle e). Liu et al. emphasize large conv. kernels (ii.1) [48], but this results in slightly worse performance compared to wider 3 3x3 conv. layers, so we do not adopt this ConvNeXt design choice.
Neural network architecture details. Given i.3, i.4, and principles c), d), and e), we can replace the StyleGAN2 backbone with a modernized ResNet. We use a fully symmetric design for G and D with 25 M parameters each, comparable to Config-A. The architecture is minimalist: each resolution stage has one transition layer and two residual blocks. The transition layer consists of bilinear resampling and optional 1x1 conv. for changing spatial size and feature map channels. The residual block includes five operations: Convl x 1 Leaky ReLU -> Conv3x33- Leaky ReLU + Conv 1, with the final Convl x having no bias term. For the 4x4 resolution stage, the transition layer is replaced by a basis layer for G and a classifier head for D. The basis layer, similar to StyleGAN [29, 31], uses 4x4 learnable feature maps modulated by via linear layer. The classifier head uses global 4x4 depthwise conv. to remove spatial extent, followed by a linear layer to produce Dss output. We maintain the width ratio for each resolution stage as in Config A, making the stem width 3x as wide due to the efficient 1 Xx conv. The 3x3 conv. in the residual block has a compression ratio of 4, following [16, 17], making the bottleneck width 0.75x wide as Config A.
To avoid variance explosion due to the lack of normalization, we employ fix-up initialization [99]: We pero-initialize the last convolutional layer in each residual block and scale down the initialization of the other two convolutional layers in the block by I, 0.22. where is the number of residual blocks. We avoid other fix-up tricks, such as excessive bias terms and a learnable multiplier.

---

Bottleneck modernization (Config E). Now that we have settled on the overall architecture, we investigate how the residual block can be modemized, specifically i.l) and i.2). First, we explore i.1 and replace the 3 x convolution in the residual block with a grouped convolution. We set the group size to 16 rather than 1 (i.e. depthwise convolution as ConvNeXt) as depthwise convolution is highly inefficient on GPUs and is not much faster than using a larger group size. With grouped convolution, we can reduce the bottleneck compression ratio to two given the same model size. This increases the width of the bottleneck to 1,5x as wide as Config A. Finally, cost of grouped convolution is negligible compared to I I convolution, and so we seek to enhance the capacity of grouped convolution. We apply i.2), which inverts the bottleneck width and the stem width, and which doubles the width of grouped convolutions without any increase in model size. Figure 2 depicts our final design, which reflects modern CNN architectures.
notice that the compute
4 Experiments Details
4.1 Roadmap Insights on FFHQ-256 [29]
As per Table 2, Config A (vanilla StyleGAN2) achieves an FID 7.52 using the official implementa- tion on FFHQ-256. Config B with all tricks removed achieves an FID of 12.46 perforrancc drops as expected. Config C, with a well-behaved loss, achieves an FID of 11.65. But, now training is sufficiently stable to improve the architecture.
Config D which improves G and D based on the classic ResNet and ConvNeXt findings, achieves an FID of 9.95. The output skips of the StyleGAN2 generator are no longer useful given our new architecture: including them produces a worse FID of 10.17. Karras et al. find that the benefit of output skips is mostly related to gradient magnitude dynamics [32], and this has been addressed by our ResNet architecture. For StyleGAN2, Karras et al. conclude that ResNet architecture harmful to G IJ, but this is not true in our case as their ResNet implementation is considerably different from ours: 1) Karras et al. use one 3-3 residual block for each resolution stage, while we have a separate transition layer and two 1-3-1 residual blocks; 2) i.3) and i.4) are violated as they do not have a linear residual block [67] and the transition layer is placed on the skip branch of the residual block rather than the stem; 3) the essential principle of ResNet [16] [16]-identity mapping [17]-is violated as Karras et al. divide the output of the residual block by /2 to avoid variance explosion due to the absence of a proper initialization scheme.
For Config E, we conduct two experiments that ablate i.1 (increased width with depthwise conv.) and i.2 (an inverted bottleneck). We add GroupedConv and reduce the bottleneck compression ratio to two given the same model size. Each bottleneck is now 1.5x the width of Config A, and the FID drops to 7,51, surpassing the performance of StyleGAN2. By inverting the stem and the bottleneck dimensions to enhance the capacity of GroupedConv, our final model achieves an FID of 7.05, exceeding StyleGAN2.
Model
# modes?
DEL
4.2 Mode Recovery - StackedMNIST [53]
DCGAN [60]
99
3.40
VEEGAN [80]
150
2.95
We repeat the earlier experiment in 1000-mode con- vergence on StackedMNIST (unconditional genera- tion), but this time with our updated architecture and with comparisons to SOTA GANs and likelihood- based methods (Tab. 3, Fig. 5). One advantage brought up of likelihood based models such as dif- fusion over GANs is that they achieve mode cover- age [7]. We find that most GANs struggle to find all modes. But, PreSGAN [8], DDGAN [94], and our approach are successful. Further, our method outperforms all other tested GAN models in term of KL divergence.
WGAN-GP [14]
959
0.73
PacGAN [46]
992
0.28
StyleGAN2 [31]
940
0.42
PresGAN [8]
1000
0.12
Adv. DSM [24]
1000
1.49
VAEBM [93]
1000
0.087
DDGAN [94]
1000
0.071
MEG [39]
1000
0.031
Ours Config F
1000
0.029
Table 3: StackedMNIST 1000-mode coverage.
4.3 FID- FFHQ-256 [29] (Optimized)
We train Config E. model until convergence and with optimized hyperparameters and training schedule on FFHQ at 256x256 (unconditional generation) (Tab. 4, Figs. and 6). Please see our supplemental material for training details. Our model outperforms existing StyleGAN methods, plus four more
7

---

Model
NFE,
FIDU
StyleGAN2 [31]
3.78
StyleGAN3-T(332
4.81
StyleGAAN3RR [32]
3.92
LDM [62]
200
4.98
ADM (DDIM) [7.49]
500
8.41
Model
NFE,
FID
ADM (DPM-Solver) [7,49)
500
8.40
StyleGiA22 [31,
3.32
Diffusion Autoencoder [59, 491
500
5.81
MSG-GAN [27, 451
2.7
Ours Ours- Config E
2.75
Anycost GAN [45]
2.52
With ImageNet feature leakage [41]:
VE [78, 33]
79
25.95
PolyINR* [74]
2.72
VP 178, 33]
79
3.39
StyleGAN-XL* [69]
2.19
EDM [33]
79
2.39
StyleSAN-XL X [82]
1.68
Ours Config E
1
1.95
Table 4: FFHQ-256. * denotes models that leak ImageNet features.
Table 5: FFHQ-64.
recent diffusion based methods. On this common dataset experimental setting, many methods (not listed here) use the bCR [101] trick this has only been shown to improve performance on FFHQ-256 (not even at different resolutions of FFHQ) [101, 98]. We do not use this trick.
4.4 FID - FFHQ-64 [33]
To compare with EDM [33] directly, we evaluate our model on FFHQ at 64 resolution. For this, remove the two highest resolution stages of our 256x256 model, resulting in a generator that is less than half the number of parameters as EDM. Despite this, our model outperforms EDM on this dataset and needs one function evaluation only (Tab. 5).
4.5
FID CIFAR-10 [37]
Model
BigGAN [3]
14.73
We train Config model until convergence and with optimized hyperparameters and training sched-
TransGAN
9.26
VITGAN [42]
6.66
ule on CIFAR-10 (conditional generation) (Tab. 6, Fig. 8). Our method outperforms many other GANs by FID even though the model has relatively small capacity. For instance, StyLeGAN-XL [69] has 18 M parameters the generator and 125 M parame- ters in the discriminator, while our model has a 40 M parameters between the generator and discrim- inator combined (Fig. 3). Compared diffusion models like LDM or ADM, GAN inference is sig- nificantly cheaper as requires only one network function evaluation compared to the tens or hun- dreds of network function evaluations for diffusion models without distillation.
DDGAN
4
3.75
Diffusson StyleGAN2 [90]
StyleGAN2 ADA [30]
2.42
StyleGAN3-R [32,
DDPM [20]
1000
3.21
DDIM [76]
VE [78, 33]
3.11
(78. 33]
35
2.48
Ours Config F.
1.96
Winh ImageNer feature leakage [41]:
StyleGAN-XL [69]
Table 6: CIFAR-10 performance.
Many state-of-the-art GANs derived from Pro- jected GAN [68], including StyleGAN-XI. [69] and the concurrent work of StyleSAN-XL [82]. These methods use a pre-trained ImageNet classifier in the discriminator. Prior work has shown that pre- trained ImageNet discriminator can leak ImageNet features into the model [41], causing the model perform better when evaluating on FID since it re- lies on a pre-trained ImageNet classifier for the loss. But, this does not improve results in perceptual stud- ies [41]. Our model produces low FID without any ImageNet pre- pre-train
Figure 3: Millions of parameters vs. FID-50K (log scale) on CIFAR-10. Lower better.
8

---

Model
NFE
FID
BigGAN-ddepp [3]
4.06
DDPM [20]
250
11.0
DDIM [76]
50
13.7
Model
NFE,
FID
ADM [7]
250
2.91
DDPM++ [35]
1000
8.42
EDM [33]
79
2.23
VDM [36]
1000
7.41
CT [79]
2
11.1
MSGAN (27,56]
12.3
CD [79]
3
4.32
ADM [7]
1000
3.60
iCT-deep [77]
2
2.77
DDPM-IP [56]
1000
2.87
DMD [96]
2.62
Ours Our-CConfi E
1.27
Ours Config E
2.09
With ImageNet feature leakage
[41]:
With ImageNet feature leakage [41]:
StyleGAN-XI. [69]
1.10
StyleGAN-XI. [69]
1.52
Table ImageNet-32.
Table 8: 1mageNiat-44.5ddtemmmnnns sampling.
4.6 FID ImageNet-32 [6]
train Contig E model until convergence and with optimized hyperparameters and training schedule on ImageNet-32 (conditional generation). We compare against recent GAN models and recent diffusion models in Table 7. We adjust the number parameters in the generator our model to match StyleGAN-XL [69]'s generator (84M parameters). Specifically, significantly wider to match. Our method achieves comparable FID despite using a 60% smaller discriminator (Tab. 7) and despite not using a pre-trained ImageNet classifier.
make the model
4.7 FID ImageNet-64 [6]
We evaluate our model on ImageNet-64 to test its scalability. We stack another resolution stage on ImageNet-32 model, resulting in generator of 104 M parameters. This model is nearly 3x smaller than diffusion-like models [7, 33, 79, 77] that rely on the ADM backbone, which contains about 300 M parameters. Despite the smaller model size and that model generates samples in one step, it outperforms larger diffusion models with many NFEs on FID (Tab. 8).
4.8 Recall
We evaluate the recall [40] of our model on each dataset to quantify sample diversity. In general, our model achieves a recall that is similar to or marginally worse than the diffusion model counterpart, yet superior to existing GAN models. For CIFAR- 10, the recall of our model peaked at 0.57; as point of comparison, StyleGAN-XL [69] has a worse recall of 0.47 despite its lower FID. For FFHQ, we obtain recall of 0.53 64 and 0.49 at 256x256, whereas StyleGAN2 [31] achieved a recall of 0.43 on FFHQ-256. Our ImageNet-32 model achieved recall of 0.63; comparable to ADM [7]. Our ImageNet-64 model achieved recall 0.59. While this is slightly worse than =0063 that many diffusion models achieve, it is better than BigGAN-deep [3] which achieved a recall of 0.48.
5 Discussion and Limitations
We have shown that simplification of GANs is possible for image generation tasks, built upon more stable RpGAN+ R₁ + R2 objective with still provides diverse output. This stability is what lets re-engineer a modern network architecture without the tricks of previous methods, producing the R3GAN model with competitive FID on the common datasets of Stacked-MNISTT FFHQ, CIFAR-10, and ImageNet empirical demonstration of the mathematical benefits.
convergence properties that
The focus of our work is to elucidate the essential components of minimum GAN for image generation. As such, we prioritize simplicity over functionality-ww do not claim to beat the performance of every existing model every dataset task; merely to provide new simple

---

Figure 4: Qualitative examples sample generation from our Config on FFHQ-256.
baseline that converges easily. While this makes our model possible backbone for future GANs, also means that it is not suitable to apply our model directly to downstream applications such as image editing controllable generation, as model lacks dedicated features for easy image inversion or disentangled image synthesis. For instance, we remove style injection functionality from StyleGAN even though this has a clear We also omitted common techniques that have been shown in previous literature to improve FID considerably. Examples include some form adaptive normalization modulated by the latent code [7, 33, 29, 98, 58, 89, 66]. and using multiheaded self attention at lower resolution stages [7, 33, 34]. We aim to explore these techniques a future study.
Further, our work is limited in its evaluation of the scalability of R3GAN models. While they show promising results on 64x64 ImageNet, we are yet verify the scalability on higher resolution ImageNet data large-scale text to image generation tasks [12].
Finally, as a method that improve the quality of generative models, it would be amiss men- tion that generative models-especially of people-can cause direct harm (e.g., through personalized deep fakes) and societal harm through the spread of disinformation (c.g., fake influencers).
6 Conclusion
This work introduced R3GAN, new baseline GAN that features increased stability, leverages modern architectures, and does not require ad-hoc tricks that are commonplace in existing GAN models. Central to our approach is regularized relativistic loss that provably features local convergence and that improves the stability of GAN training. This stable loss enables us to ablate various tricks that were previously necessary in GANs, and incorporate in their place modern deep architectures. The resulting streamlined baseline achieves competitive performance to SOTA models within its parameter size class. We anticipate that our backbone will help drive future GAN research.
10

---

The authors thank Xinjie Jayden Yi for contributing to the proof and Yu Cheng for helpful discussion. For compute, the authors thank Databricks Mosaic Research. Yiwen Huang was supported by Brown University Division of Research Seed Award, and James Tompkin was supported by NSF CAREER 2144956. Volodymyr Kuleshov was supported by NSF CAREER 2145577 and NIH MIRA IR35GM15124301.
Acknowledgements.
References
[1] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv:1607.0644 00400000006.22 2016.
arXiv preprint
[2] Koushik Biswas, Sandeep Kumar, Shilpak Banerjee, and Ashish Kumar Pandey. Smu: smooth activation function for deep networks using smoothing maximum technique. arXiv preprint arXiv:2111.0444 2021.
[3] Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. arXiv preprint 1rXi::1809. 1006, 2018.
[4] Andy Brock, Soham De, Samuel L Smith, and Karen Simonyan. High-performance large-scale image recognition without normalization. International Conference on Machine Learning, pp. 1059-1071, PMLR, 2021.
[5] François Chollet. Xception: Deep learning with depthwise separable convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.
2017.
[6] Patryk Chrabaszez, Ilya Loshchilov, and Frank Hutter. A downsampled variant of imagenet an alternative to the cifar datasets. arXiv preprint arXiv: 1707.08819, 2017.
[7] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in neural information processing systems,
2021.
[8] Adji B Dieng, Francisco JR Ruiz, David M Blei, and Michalis K Titsias. Prescribed generative adversarial networks. arXiv preprint arXiv: 2019.
[9] Alexcy Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is Transformers for image recognition at scale. arXiv preprint arXiv:2010. 2020.
worth 16x16 words:
[10] Tiantian Fang. Ruoyu Sun, Alex Schwing, DigGAN: Discriminator gradient gap regularization for GAN training with limited data. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (cds.), Advances in Neural Information Processing Systems, 2022. URL https: // opwnreeeee.eeeeeeeeeee net/ forum? fomm?dddazBBii ddda2BBV77442
[11] Gauthier Gidel, Reyhane Askari Hemmat, Mohammad Pezeshki, Rémi Le Priol, Gabriel Huang, Simon Lacoste-Julien, and loannis Mitliagkas. Negative momentum for improved game dynamics. In The 22nd International Conference on Artificial Intelligence and Statistics, pp. 1802-1811, PMER, 2019.
[12] Aaron Gokaslan, A Feder Cooper, Jasmine Collins, Landan Seguin, Austin Jacobson, Mihir Patel, Jonathan Frankle, Cory Stephenson, and Volodymyr Kuleshov. Commoncanvas: Open diffusion models trained on creative-commons images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8250-8260, 2024.
[13] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley. Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications the ACM, 63(11): 139-144, 2020.
[14] Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron C Courville. Improved training of wasserstein gans. Advances in neural information processing systems, 30, 2017.
[15] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance imagenet classification. In Proceedings of the IEEE international conference on computer vision, pp. 1026 1034, 2015.
[16] Kaiming He, Xiangyu Zhang. Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 770-778, 2016.

---

[17] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part IV 14. pp. 630-645. Springer, 2016.
[18] Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08415. 2016.
[19] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale scale update rule converge to a local nash equilibrium. Advances neural information processing systems, 30, 2017.
[20] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models, Advances in neural information processing systems, 33:6840- 6851, 2020.
[21] Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pp. 448-456, pmlr, 2015.
[22] Alexia Marttnnaa. The relativistic discriminator: a key element missing from standard gan. arXiv preprint arXiv:1807.000 2018.
[23] Alexia Maliineuu-MMarinnau and Joannis Mitliagkas. Gradient penalty from maximum margin perspective. arXiv preprint arXiv: 191006922, 2019.
[24] Alexia JolicoerrMaarincca, Rémi Piché-Taillefer, Rémi Tachet des Combes, and Joannis Mitliagkas. Ad- versarial score matching and improved sampling for image generation. arXiv preprint (rXiv:2009005475 2020.
[25] Minguk Kang, Joonghyuk Shin, and Jaesik Park. Studiogan: a taxonomy and benchmark of gans for image synthesis. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.
[26] Minguk Kang, Jun- Yan Zhu, Richard Zhang, Jaesik Park, Eli Shechtman, Sylvain Paris, and Taesung Park. Scaling up gans for text-to-image synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 10124-10134, 2023.
[27] Animesh Karnewar and Oliver Wang. Msg-gan: Multi-scale gradients for generative adversarial networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 7799-7808, 2020.
[28] Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for improved quality, stability, and variation. arXiv preprint arXiv:1710. 10196, 2017.
[29] Tero Karras, Samuli Laine. and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEF/CVF conference computer vision and pattern recognition, pp. 4401-4410, 2019.
2019.
[30] Tero Karras, Miika Aittala, Janne Hellsten, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Training generative adversarial networks with limited data. Advances in neural information processing systems, 33:12104-121 14, 2020.
[31] Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of stylegan. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 8110-8119 19, 2020.
[32] Tero Karras, Miika Aittala, Samuli Laine, Erik Härkönen, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Alias-free generative adversarial networks. Advances in Neural Information Processing Systems, 863,, 2021.
[33] Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion- generative models. Advances in Neural Information Processing Systems, 35:26565-266 265777, 2022.
[34] Tero Karras, Miika Aittala, Jaakko Lehtinen, Janne Hellsten, Timo Aila, and Samuli Laine. Analyzing and improving the training dynamics of diffusion models. arXiv preprint arXiv:2112.02666, 2023.
[35] Dongjun Kim, Seungjae Shin, Kyungwoo Song, Wanmo Kang. and I1-Chul Moon. Soft truncation: A universal training technique of score-based diffusion model for high precision score estimation. arXiv preprint arXiv:2106.000 05527, 2021.
[36] Diederik Kingma. Tim Salimans, Ben Poole, and Jonathan Ho. Variational diffusion models. Advances in neural information processing systems, 34:21696-21777 21702 2021.
12

---

[37] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Thesis. 2009.
[38] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In F. Pereira, C.J. Burges, I. Bottou, and K.Q. Wein- berger (eds.), Advances in Neural Information Processing Systems, volume 25. Curran Asso- ciates. Inc., 2012. URL htttp:://proceediiiie wwwprooeeedinng..nriipppeeee cc/paper..flles/pppeeeeee 339964bbbbb66663332444446-----
.df.
[39] Rithesh Kumar, Sherjil Ozair, Anirudh Goyal, Aaron Courville, and Yoshua Bengio. Maximum entropy generators for energy-hased models. arXiv preprint arXiv: 085008 2019.
[40] Tuomas Kynkäänniemi, Tero Karras, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Improved precision and recall metric for assessing generative models. Advances in neural information processing systems. 32, 2019.
[41] Tuomas Kynkäänniemi, Tero Karras, Miika Aittala, Timo Aila, and Jaakko Lehtinen. The role of imagenet classes in fréchet inception distance. arXiv preprint arXiv:2203, 06006, 2022.
[42] Kwonjoon Lee, Huiwen Chang, Lu Jiang. Han Zhang, Zhuowen Tu, and Ce Liu. Vitgan: Training gans with vision transformers. arXiv preprint arXiv: 048594 2021.
[43] Bee Lim, Sanghyun Son, Heewon Kim, Seungjun Nah, and Kyoung Mu Lee. Enhanced deep residual networks for single image super-resolution. In Proceedings the IEEE conference computer vision and pattern recognition workshops, pp. 136-144, 2017.
[44] Jac Hyun Lim and Jong Chul Ye. Geometric gan. arXiv preprint 0etl::1700.00000 2017.
[45] Ji Lin, Richard Zhang, Frieder Ganz, Song Han, and Jun-Yan Zhu. Anycost gans for interactive image synthesis and editing. In Pruceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 14986-14996, 2021.
[46] Zinan Lin, Ashish Khetan, Giulia Fanti, and Sewoong Oh. Pacgan: The power of two samples in generative adversarial networks. Advances in neural information processing systems, 31, 2018.
[47] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 10012 10022, 2021.
[48] Zhuang Liu, Hanzi Mao, Chao- Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pp. 11976 1986, 2022.
[49] Hui Lu, Ronald Poppe, et al. Compensation sampling for improved convergence in diffusion models. arXiv preprint arXiv:23 /.2.02288, 2023.
[50] Xudong Mao, Qing Li, Haoran Xie, Raymond YK Lau, Zhen Wang, and Stephen Paul Smolley. Least squares generative adversarial networks. In Proceedings of the IEEE intermational conference on computer vision, pp. 2794-2802, 2017.
[51] Lars Mescheder, Sebastian Nowozin, and Andreas Geiger. The numerics of gans. Advances in neural information processing systems, 30, 2017.
[52] Lars Mescheder, Andreas Geiger, and Sebastian Nowozin. Which training methods for gans do actually converge? In International conference on machine learning, pp. 3481-3490. PMLR, 2018.
[53] Luke Metz, Ben Poole, David Pfau, and Jascha Sohl-Dickstein. Unrolled generative adversarial networks. In International Conference on Learning Representations, 2016.
[54] Takeru Miyato and Masanori Koyama. arXiv: 1802005637 2018
cgans with projection discriminator. arXiv preprint
[55] Vaishnavh Nagarajan and J Zico Kolter. Gradient descent gan optimization is locally stable. Advances in neural information processing systems, 30, 2017.
[56] Mang Ning, Enver Sangineto, Angelo Porrello, Simone Calderara, and Rita Cucchiara. Input perturbation reduces exposure bias diffusion models. arXiv preprint arXiv:2301.1 11766 2023.
[57] Sebastian Nowozin, Botond Cseke, and Ryota Tomioka. f-gan: Training generative neural samplers using variational divergence minimization. Advances in neural information processing systems, 29, 2016.
13

---

[58] William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4195-4205, 2023.
[59] Konpat Preechakul. Nattanat Chatthee, Suttisak Wizadwongsa, and Supasom Suwajanakorn. Diffusion autoencoders: Toward meaningful and decodable representation. In Proceedings of the IEEE/CVF Conference Computer Vision and Patter Recognition (CVPR), pp. 10619-10629, June 2022.
[60] Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learnngg with deep convolutional generative adversarial networks. arXiv preprint 1rXiv:/511.0444 2015.
[61] Prajit Ramachandran, Barret Zoph, and Quoe V Le. Searching for activation functions. arXiv preprint arXii:1710.05594,, 2017.
[62] Robin Romhach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 10684-10695, 2022.
[63] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical image computing and computer-assiste intervention MICCAI 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, part III 18, pp. 234-241. Springer, 2015.
[64] Kevin Roth, Aurelien Lucchi, Sebastian Nowozin, and Thomas Hofmann. Stabilizing training of generative adversarial networks through regularization. Advances in neural information processing systems, 30, 2017.
[65] Seyedmorteza Sadat, Jakob Buhmann, Derek Bradley, Otmar Hilliges, and Romann M Weber. Lite- vae: Lightweight and efficient variational autoencoders for latent diffusion models. arXiv preprint arXiv:2405. 14477, 2024.
[66] Subham Sekhar Sahoo, Aaron Gokaslan, Chris De Sa, and Volodymyr Kuleshov, Diffusion models with learned adaptive noise. arXiv preprint ar3iv:2312. 13236, 2023.
[67] Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chiae Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 45 4510-4520, 2018.
[68] Axel Sauer, Kashyap Chitta, Jens Müller, and Andreas Geiger. Projected gans converge faster. Advances Neural Information Processing Systems, 34: 1447000-17792, 2021.
[69] Axel Sauer, Katja Sehwarz, and Andreas Geiger. StyleGAN XL: Scaling stylegan to large diverse datasets. In ACM SIGGRAPH 2022 conference proceedings. pp. 1-10, 2022.
[70] Axel Sauer, Tero Karras, Samuli Laine, Andreas Geiger, and Timo Aila. Stylegan-t: Unlocking the power of gans for fast large-scale text-to-image synthesis. In International conference machine learning, pp. 30105 30118. PMLR, 2023.
[71] Wenzhe Shi, Jose Caballero, Ferene Huszár, Johannes Totz, Andrew P Aitken, Rob Bishop, Daniel Rueckert, and Zehan Wang. Real-time single image and video super-resolution using an efficient sub- pixel convolutional neural network. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1874-1883, 2016.
[72] Wenzhe Shi, Jose Caballero. Ferene Huszár, Johannes Totz. Andrew P Aitken, Rob Bishop, Daniel Rueckert, and Zehan Wang. Real-time single image and video super-resolution using an efficient sub- pixel convolutional neural network. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1874-1883, 2016.
[73] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: |4X9
2014.
[74] Rajhans Singh, Ankita Shukla, and Pavan Turaga. Polynomial implicit neural representations for large diverse datasets. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pp. 2041-2051, 2023.
[75] Casper Kaae Sønderby. Jose Caballero, Lucas Theis. Wenzhe Shi, and Ferenc Huszár. Amortised map inference for image super-resolution. arXiv preprini 1rXiv:1610.04490, 2016.
[76] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In International Conference Learning Representations, 2021.
14

---

[77] Yang Song and Prafulla Dhariwal. Improved techniques for training consistency models. In The Twelfth International Conference Learning Representations, 2024.
[78] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint urXiv:2011.13450 2020.
[79] Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency models. In International Conference on Machine Learning, pp- 3221 1-32252. PMLR, 2023.
[80] Akash Srivastava, Lazar Valkov, Chris Russell, Michael Gutmann, and Charles Sutton, Veegan: Reducing mode collapse in gans using implicit variational learning, Advances in neural information processing systems, 30, 2017.
[81] Ruoyu Sun, Tiantian Fang, and Alexander Schwing. Towards a better global loss landscape of gans. Advances in Neural Information Processing Systems, 33:10186 1111011161118,,,,,11111111111811 2020.
[82] Yuhta Takida, Masaaki Imaizumi, Takashi Shibuya, Chich-Hsin Lai, Toshimitsu Ucsaka, Naoki Murata, and Yuki Mitsufuji. SAN: Inducing metrizability of GAN with discriminative normalized linear layer. In The Twelfth International Conference on Learning Representations, 2024. URL https: //ppereveeeeeeeeeee/ net/forum? wowuummmmiie iid=eie/eeeEFBB
[83] Song Tao and Jia Wang. Alleviation of gradient exploding in gans: Fake can be real. In Procedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 1191-1200. 2020.
[84] Hoang Thanh- Tung. Truyen Tran, and Svetha Venkatesh. Improving generalization and stability of generative adversarial networks. In International Conference on Learning Representations, 2019. URL httpp:///ppereee
www//eettunnnee Pdd JCSQQ.
[85] Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Instance normalization The missing ingredient for fast stylization. arXiv preprint arXiv: 08002, 2016.
[86] Arash Vahdat, Karsten Kreis, and Jan Kautz. Score-based generative modeling latent space. Advances in neural information processing systems, 34:1 1 1287-11302, 2021.
[87] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lakasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017.
[88] Xintao Wang. Ke Yu, Shixiang Wu, Jinjin Gu, Yihao Liu, Chao Dong, Yu Qiao, and Chen Change Loy. Esrgan: Enhanced super-resolution generative adversarial networks. In Proceedings of the European conference on computer vision (ECCV) workshops, pp. 0-0, 2018.
[89] Yingheng Wang. Yair Schiff, Aaron Gokaslan, Weishen Pan, Fei Wang, Christopher De Sa, and Volodymyr Kuleshov. Infodiffusion: Representation learning using information maximizing diffusion models. In International Conference on Machine Learning, pp. 36336-36354. PMLR, 2023.
[90] Zhendong Wang, Huangjie Zheng, Pengcheng He, Weizhu Chen, and Mingyuan Zhou. Diffusion-gam Training gans with diffusion. In The Eleventh International Conference on Learning Representations, 2023.
[91] Yuxin Wu and Kaiming He. Group normalization. In Proceedings of the European conference on computer vision (ECCV), pp. 3-19, 2018
[92] Yuxin Wu and Kaiming He. Group normalization. In Proceedings of the European conference on computer vision (ECCV), pp. 3-19. 2018
[93] Zhisheng Xiao, Karsten Kreis, Jan Kautz, and Arash Vahdat. Vaebm: A symbiosis between variational autoencoders and energy-based models. arXiv preprint arXiv:2010000654, 2020.
[94] Zhisheng Xiao, Karsten Kreis, and Arash Vahdat. Tackling the generative learning trilemma with denoising diffusion gans, arXiv preprint 2 12.07804, 2021.
[95] Saining Xie. Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transforma- tions for deep neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1492-1500, 2017.
[96] Tianwei Yin, Michael Gharbi, Richard Zhang, Eli Shechtman, Fredo Durand, William T Freeman, and Taesung Park. One-step diffusion with distribution matching distillation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 6613 063366633644 2024.
15

---

[97] Weihao Yu, Mi Luo, Pan Zhou, Chenyang Si, Yichen Zhou, Xinchao Wang, Jiashi Feng, and Shuicheng Yan. Metaformer is actually what you need for vision. In Proceedings the IEEE/CVF conference computer vision and pattern recognition, pp. 10819-10829, 2022.
[98] Bowen Zhang. Shuyang Gu, Bo Zhang. Jianmin Bao, Dong Chen, Fang Wen, Yong Wang. and Baining Guo. Styleswin: Transformer- based gan for high-resolution image generation. In Proceedings of IEEE/CVF conference on computer vision and pattern recognition, pp. 11304-11314, 2022.
[99] Hongyi Zhang, Yann Dauphin, and Tengyu Ma. Fixup initialization: Residual learning without normalization. arXiv preprint (90.:0:99000092211 2019.
[100] Richard Zhang. Making convolutional networks shift-invariant again. International conference on machine learning, pp. 7324-7333. PMLR, 2019.
[101] Zhengli Zhao, Sameer Singh, Honglak Lee, Zizhao Zhang. Augustus Odena, and Han Zhang. Improved consistency regularization for gans. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pp. 11033-11041, 2021.
16

---

Appendices
A Local convergence
Following [52], GAN training can be formulated as dynamical system where the update operator is given by Fh (0,, (6) (0, v hv(O, vb). is the learning rate and denotes the gradient vector field:
( VeL(O, vt) v c(0,
v(0, m/)
Mescheder et al. [51] showed that local convergence near (0*, ())) can be analyzed by examining the spectrum of the Jacobian JF, at the equilibrium: the Jacobian has eigenvalues with absolute value bigger than 1, then training does not converge. On the other hand, if all eigenvalues have absolute value smaller than 1, then training will converge to (0", x)") at a linear rate. all eigenvalues have absolute value equal to 1, the convergence behavior is undetermined.
Given some calculations [52], we can show that the eigenvalues the Jacobian of the update operator AsF, can be determined by Ar.
Ala
hAs.
(5)
That is, given small enough [52], the training dynamics can instead be examined using Aj Le., the eigenvalues of the Jacobian of the gradient vector field. If all AT have a negative real part, the training will locally converge to (0", 1/") at linear rate. On the other hand, if some λ₁ have positive real part, the training is not convergent. If all λj, have a zero real part, the convergence behavior inconclusive.
B DiracRpGAN: A demonstration of non-convergence
Summary. To obtain DiracRpGAN, we apply Eq. 2 to the DiracGAN [52] problem setting, After simplification, DiracRpGAN and DiracGAN are different only by a constant. They have the same gradient vector field, therefore all proofs are identical to Mescheder et al. [52].
Definition B.1. The DiracRpGAN consists of a (univariate) generator distribution pe ôe and linear discriminator (z) = +p The true data distribution pp is given by a Dirac distribution concentrated
In this setup, the RpGAN training objective is given by:
C(0, 1b) f(v0).
(6)
We can now show analytically that DiracRpGAN does not converge without regularzation.
Lemma B.2. The unique equilibrium point of the training objective in Eq. is given by 6 + Moreover, the Jacobian of the gradient vector field at the equilibrium point has the two eigenvalues (O) which are both on the imaginary axis.
The gradient vector field u of Eq. 6 is given by:
VoL(O, - (1(8) v(0,v) vb) = V vx.20.
(
(
(7)
and the Jacobian of
(( J. (v0))
(1 ((0))
8² fm (v0))
(8)
Evaluating J, at the equilibrium point 0 + 0 gives
-M
Ju
=
('())
(9)
(0.0)
Therefore, the eigenvalues of J,, are \1/2 = + both of which have real part of 0. Thus, the convergence of DiracRpGAN is inconclusive and further analysis is required.
17

---

Lemma B.3. The integral curves of the gradient vector field v(O, 1) do not converge to the equilib- rium point. More specifically, every integral curve (0(t), (t)) of the gradient vector field v(0, 1) satisfies 0(1) + (b)² const for all E [0, oo).
Let R(0, 1p 2 (02 ) then:
(000))))0=00)000000000000 00000000000000000000000000000000
dt
0.
(10)
We that the distance between (0, 1b and the equilibrium point (0, 0) stays constant. Therefore, training runs in circles and never converges.
Next, we investigate the convergence behavior of DiracRpGAN with regulariaazi,... For DiracRpGAN, both R and Ry can reduced to the following form:
7 R(v) = 2 "
(11)
Lemma B.4. The eigenvalues of the Jacobian of the gradient field the gradient-regularized DiracRpGAN at the equilibrium point are given by
11/2 = 2 - (0)
(12)
In particular. for 7i > O all eigenvalues have negative real part. Hence, gradient descent is locally convergent for small enough learning rates.
With regularization, the gradient vector field becomes
5(0, ( (wbf((b) (v() - 796
(13)
the Jacobian of is then given by
Ja = ( + (w0)
3/2 20 (v'))
(b'( - -v0f"((0))
(14)
82 (w00) -
evaluating the Jacobian = = yields
= ((r0) -r(o))
J
(15)
(0.0)
given some calculations, we arrive at Eq.
C General Convergence Results
Summary. The proofs are largely the same as Mescheder el al. [52]. We use the same proving techniques, and only slightly modify the assumptions and proof details to adapt Mescheder er al.'s effort to RpGAN. Like in [52], our proofs do not rely unrealistic assumptions such as supp pD = Pe.
C.1 Assumptions
We closely follow [52] but modify the assumptions wherever necessary to tailor the proofs for RpGAN. Like in [52], we also consider the realizable case where there exists such that Go produces the true data distribution.
Assumption We have Pa- = PD. and D C. in some local neighborhood of supp Pp. where C is some arbitrary constant.
Since RpGAN defined on critic difference rather than raw logits, we no longer require Da- to produce 0 on supp PD. instead any constant C' would suffice.
18

---

Assumption II. We have (0) # 0 and (0) < 0.
This assumption is the satisfies this assumption.
in [52]. The choice f(1) log(1 e adopted in the main text
As discussed in [52], there generally is not single equilibrium point (0", ),", but submanifold of
equivalent equilibria corresponding to different parameterizations of the same function. is therefore necessary to represent the equilibrium as reparameterization manifolds MM and Mp. We modify the reparameterization h as follows:
h ((b)) = Ezapp
[ID((o) - (D Dv)
(16)
to account for the fact that Di-- now allowed have any constant value on supp pD. The reparameterization manifolds are then given by:
MG = {0) Po = PD}
(17)
MD { h(1b) 0)
(18)
We assume the same regularity properties as in [52] for Ma and MD near the equilibrium. To state these assumptions, we need:
g(0) En-po
[ S.DDllmms
(19)
which leads to:
Assumption III. There are e-balls (8*) and (.(+*) around 6* and v² so that Mg n. (.
and MD n (.(b)) define C³ -mmnnfoodds Moreover, the following holds:
(i) if I2" is not in Tw Mp. then (h((v)) + 0.
(ii) if E Rm is not in To-Ma: then (wr((")) O.
These two conditions have exactly the same meanings as in [52]: the first condition indicates the geometry of MD can be locally described by the second derivative of h. The second condition implies that D is strong enough that it can detect any deviation from the equilibrium generator distribution. This the only assumption we have about the expressiveness of D.
C.2 Convergence
We can now show the general convergence result for gradient penalized RpGAN, consider the gradient vector field with either Rí Ra regularization:
(,(0, v)) (v. VoL(O, v))
(20)
(O,,
note that the convergence result can also be trivially extended to the case where both R₁ and R are applied. We omit the proof for this case as it is redundant once the convergence with either regularization is proven.
Theorem. Assume Assumption I, and III hold for (0*, 3² For small enough learning rates, gradient descent for in and i2 are both convergent to MG x MD in a neighborhood of (0", x²) Moreover, the rate of convergence is at least linear.
We extend the convergence proof by Mescheder et al. [52] to our setting. We first prove lemmas necessary to our main proof.
Lemma C.2.1. Assume J R ((n+m))((+m)) is of the following form:
J= ( -Q )
B
(21)
where Raxxm is symmetric positive definite matrix and B E RRmx has full column rank. Then all eigenvalues of satisfy R(A) < O.
Proof. See Mescheder et al. [52], Theorem A.7.
19

---

Lemma C.2.2. The gradient of [(0, 16) w.r. 0 and vo are given by:
VoL(O, 1/) = E gnpp
(.(D(( x-pp
(22)
V L(0, 1b) = pepp (D - - Ve
(23)
Proof. This is just the chain rule.
Lemma C.2.3. Assume that (6", x²*) satisfies Assumption 1. The Jacobian of the gradient vector field v(0, 1) (0", x6 is then
-Ko
-K KDD
(24)
KDG
(8',5-)
the terms KDD and KD are given by
KDD ( (0) Dw-
(25)
KD (0) Exeops D.DD-(x)) (m-)
(26)
Proof. Note that
- V1,200 (0)) 0,, 16*)
-V C2 v2)) 200 264)
J
(27)
(0+.0*)
By Assumption I, Dø- C' in some neighborhood supp PD. Therefore we also have D. = and V Dw. for x E supp pD. Using these two conditions, we see that V2L(O,, ,1'"00)))))))))) = 0.
To see 25.25 and 26, simply take the derivatives of Eq.23 and evaluate at (0", 16*))
Lemma C.2.4. The gradient V,,RR(B, v) of the regularization terms R, i E {1,2). w.r..
V (1,, +) ?Eupp V Des De) V (2,,0, bb = 71 D De
(28)
(29)
Proof. See Mescheder et al. [52], Lemma D.3.
Lemma C.2.5. The second derivatives V R2 x)** of the regularization terms R3 (1,2), wet. up at (f*,)"") ,/²) are both given by
LDD = 7 BBu-poleooo [AA"]
(30)
where V Vr Moreover, both regularization terms satisfy V Vo. Vo Ra(O", ++* 0.
Proof. See Mescheder et al. [52], Lemma D.4.
Given Lemma C.2.3, Lemma C.2.5 and Eq.20, we can now show that the Jacobian of the regularized gradient field the equilibrium point given by
-KT
DG MDD
Ja
(31)
KDG
(0*
where Mpn Knn Lnd. To prove our main theorem, we need to examine Ja when restricting it to the space orthogonal to T( M x MD.
Lemma C.2.6. Assume Assumptions and III hold. I + is not in Tox Mp. then v MDDU <
Proof. By Lemma C.2.3 and Lemma C.2.5, we have
T KDDV = fd( ('(0)Er<p y~po
[((V ((VvDe--<)) (x))
(32)
v T LDDU = =yEuppp
||
(33)
By Assumption II, we have (0) < Therefore T MADU & 0. Suppose T MADU = 0, this implies
(Ve Dv- (z)
(v) and Av
(34)
20

---

for all (I, E supp PD x Pp. Recall the definition of h(w) from Eq.16. 16. Using the fact that
De- C' and V,D i'- for supp pD. we see that the Hessian of h(b) at 26" is
Va((v") 2P (V D (x) Уーシр
De (v V De (1))
AA (35)
The second directional derivative (1h(1)) is therefore
(.) (v))) of | =0
2Ez=pp y>PD
(36)
By Assumption III, this can only hold if Tw-MD-
Lemma C.2.7. Assume Assumption III holds. w + 0 is not in Mc, then KDo +
Proof. See Mescheder et al. [52], Lemma D.6.
Proof for the main theorem. Given previous lemmas, by choosing local coordinates A(a, 7G) and 31,3,70 (D) for Mg and MD such that 6" = 0, v0" = 0 as well as
MG To- = {0)* x RR-
(37)
MD MD (0)' R--
(38)
our proof is exactly the same Mescheder el al. [52]. Theorem 4.1.
21

---

D Hyperparameters, training configurations, and compute
We implement our models on lop of the official StyleGAN3 code base. While the loss function and the models are implemented from scratch, we reuse support code from the existing implementation whenever possible. This includes exponential moving average (EMA) of generator weights [28], non-leaky data augmentation [30], and metric evaluation [32].
Training schedule. To speed up the convergence early in training, we specify a cosine schedule for the following hyperparameters before they reach their target values:
Learning rate
? for R and R2 regularization
Adam 32
EMA half-life
Augmentation probability
We call this early training stage the burn-in phase. Burn-in length and schedule for each hyperpa- rameter are listed in Table 9 for each experiment. A schedule for the EMA half-life can already be found in Karras al. [30]. albeit they use a linear schedule. A lower initial Adam B2 is crucial the initial large learning rate as it allows the optimizer to adapt to the gradient magnitude change much quicker. We use a large initial y to account for that early training: py and pp are far apart and a large smooths both distributions more aggressively which makes learning easier. Augmentation is not necessary until D starts to overfit later on; thus, we set the initial augmentation probability to O.
Dataset augmentation. We apply horizontal flips and non-leaky augmentation [30] to all datasets where augmentation is enabled. Following [30], we include pixel blitting, geometric transformations, and color transforms in the augmentation pipeline. We additionally include cutout augmentation which works particularly well with our model, although does not seem to have much effect on StyleGAN2. We also find it beneficial to apply color transforms less often and thus set their probability multiplier to 0.5 while retaining the multiplier for other types of augmentations. As previously mentioned, we apply fixed cosine schedule to the augmentation probability rather than adjusting adaptively as in [30]. We did not observe any performance degradation with this simplification.
Network capacity. We keep the capacity distribution for each resolution the same in [30, 32]. We place two residual blocks per resolution which makes our model roughly 3x as deep, 1.5-3x as wide as StyleGAN2 while maintaining the same model size on CIFAR-10 and FFHQ. For the ImageNet model, we double the number of channels which results in roughly 4x as many parameters as the default StyleGAN2 configuration.
Mixed precision training. We apply mixed precision training as in [30, 32] where all parameters are stored in FP32, but cast to lower precision along with the activation maps for the 4 highest resolutions. We notice that using FP16 as the low precision format cripples the training of our model. However, we see no problem when using BFloat16 instead.
Class conditioning. For class conditional models, we follow the same conditioning scheme as in [30]. For G, the conditional latent code z' is the concatenation of = and the embedding of the class label c, specifically z' concat(z, embed(c)). For D, we use a projection discriminator [54] which evaluates the dot product the class embedding and the feature vector D' D'(x) produced by the last layer of D, concretely D (ar) = embed (c) D' D'(x) T. We do not employ any normalization-bass
conditioning such as AdaIN [29], AdaGN [7, 331, AdaBN [3] or AdaLN [58] for simplicity, even though they improve FID considerably.
Stacked MNIST. We base this model off of the CIFAR-10 model but without class conditioning. We disable all data augmentation and shorten the burn-in phase considerably. We use a constant learning rate and did not observe any benefit of using lower learning rate later in the training.
Compute resources. We train the Stacked MNIST and CIFAR-10 models on an 8x NVIDIA L40 node. Training took 7 hours for Stacked MNIST and 4 days for CIFAR-10 10. The FFHQ model was trained on an 8x NVIDIA A6000 für roughly weeks. The ImageNet model was trained on NVIDIA A A100/H100 clusters and training took one day 32 H100s (about 5000 H100 hours).
22

---

Hyperparameter
Stacked MNIST
CIFAR-I0
FFHQ
ImageNet
Resolution
32 32
32
256 256
32 32
64
Class conditional
v
Number GPUs
32
64
Duration (Mimg)
10
250
200
100
1000
1000
Burn-in (Ming)
2
20
20
20
200
200
Minibatch size
512
512
256
256
4096
4096
Learning rate
10-4
2 10 - 5 10-5
x 10
5 10-5
2 10
- 10-5
2 10
5 10-3
2 10 - 10
Y R₁ and R2
1 0.1
0.05 0.005
150
0.5 0.05
0.1
Adam 02
0.99
0.9 - 0.99
0.9 -
0.99
0.99
EMA half-life (Ming)
-
5
50
Channels per resolution
768-768-768-768
768-768-768 768-766-768
96- 1-1--992---653 384-768-7668 768--666-7--666
384-768-768-766 768-768-768
1536- 1536-1536-111 1536
1536 1-66--566-3---11111111 1536-155366666
ResBlocks per resolution
2-2-2-2
2-2-2-2
2-2-2-2-2-2-2
2-2-2-2-2
2-2-2-2
2-2-2-2-2
Groups per resolution
96-96-96-96
96-96-96 96-96
12-24-48-988 96-96-96-96
18-96-96-96 96-99
96-96-9-96
96-96-96-96 96-96-96666666
G params
20.73M
20.78M
23.06M
22.43M
82.91M
103.57M
32
D params
20.68M
21.28M
23.01M
22.38M
86.55M
107.21M
Dataset z-flips
V
Augment probability
0.55
0.3
0.3
+ 0.5
0.4
Table 9: Hyperparameters for each experiment. The decay factor 8 of EMA can be obtained using the formula 3 212
0.5
Misitach
EMA baliilis
e.g. for CIFAR-10. EMA
B = 0.5 10 2 0.9999.

---

E
Negative Results and Future Work
Following the convention of Brock et al. [3], we report alternative design choices that did not make to our final model. Either because they failed to produce any quantitative improvement or because they would considerably complicate our minimalist design which might be better suited for future study.
We tried to apply GELU [18], Swish [61], and SMU [2] to G and D and found that doing so deteriorates FID considerably. We did not try on G xor D. We posit two independent factors:
ConvNeXt in general does not benefit much from GELU (and possibly similar ac- tivations). Table 10 and Table 11 in [48]: replacing ReLU with GELU gives little performance gain to ConvNeXt-T and virtually no performance gain to ConvNeXt-BB In the context of GANs, GELU and Swish have the same problem ReLU: that they have little gradient in the negative interval. Since G is updated from the gradient of D, having these activation functions in D could sparsify the gradient of D and as result G will not receive as much useful information from D compared to using leaky ReLU.
This does not explain the strange case of SMU [2]: SMU a smooth approximation of leaky ReLU and does not have the sparse gradient problem. It is unclear why it also underperformed and future work awaits.
We tried adding group normalization [92] G and D and it did not improve FID or training stability. We do not claim that all forms of normalizations are harmful. Our claim in principle e) only extends to normalization layers that explicitly standardizes the mean and standard deviation of the activation maps. This has been verified by prior studies [31, 34, 65J. The harm of normalization layers extends to the adjacent field of image restoration [43, 88). To the best of our knowledge, EDM2 [34] is currently the strongest diffusion UNet and it does not use normalization layers. However, it does apply normalization to the trainable weights and this improves performance considerably. We expect that applying the normalization techniques in EDM2 would improve our model's performance.
We tried removing the activation function after the 3 grouped convolution in each residual block as modern architectures [48, 97] typically do not apply non-linearity after depthwise convolution. This worsened FID performance.
We tried Unffeuuf [71] for changing the resolution of the activation maps, and found that without low-pass filtering, this led to high frequency artifacts similar to checkerboard artifacts even though Pixel-Shuffle does not have the uneven overlap problem that transposed convolution does. Note that bilinear resampling is equivalent to applying channel paaiiiaaiiooaveragnnnnn with Seae--SfffleeUsssuueu low- kernel. It might be interesting in future studies to explore inplace resampling filters that apply a low-pass filtered Baael--Suffle/Unnne that changes the number of channels.
in conjunction with a [1, 2, 1]
operation on top of a learned function
We tried scaling up our model size. We found that allocating more model capacity to lower resolution stages generally did not improve FID, but contributed to more rapid overfitting, Increasing model capacity at higher resolution stages always improves FID in our experiments, however scaling up higher resolution stages very computationally expensive. Capacity distribution for each resolution stage of the model might be an important topic explore in future studies.
For model simplicity, we did not conduct any experiment with transformer architecture attention mechanism in general. We are interested to see whether adding attention blocks to convolutional network (similar to BigGAN [3] and diffusion UNet [20, 33, 34J) or using a pure transformer architecture (similar to DiT [58]) will result in stronger performance. Given the impressive results of EDM2 [34] (which uses UNet), it seems the argument has not yet settled for generative modeling.
We experimented with Adam 82- = 0.999 following common practice in supervised learning and diffusion models, and found that doing so led to stability issues our ImageNet models. We expect that introducing proper normalization to our model will resolve this problem.
We tried mixed precision training with IEEE FP16 as this is the low precision format used in StyleGAN2-ADE [30], StyleGAN3 [32], and EDM2 [34]. This crippled the training of
24

---

our model and switching to 6 fixed the problem. We expect that introducing proper normalization our model will allow to use IEEE FP16 which offers more precision than BF
We tried lazy regularization [31] in our early experiments where Ri and R2 were applied once every 8 minibatches. This led to slightly worse FID performance real world datasets like FFHQ and CIFAR-10. However, it resulted in complete convergence failure on Stacked MNIST and several two dimensional toy datasets (line, circle, 25 Gaussians, etc.), indicating potential concerns regarding the mathematical legitimacy of this trick.
25

---

F
Qualitative Results
B
0
9
4
2
6
2
3
4
8
6
9
9
4
6
#4
I
4
D
4
4
Figure 5: Qualitative examples of sample generation from our Config E on Stacked-MNIST.
26

---

Figure 6: More qualitative examples of sample generation from our Config E on FFHQ-256.
27

---

Figure 7: Qualitative examples of sample generation from our Config on FFHQ-64.
28

---

Figure 8: Qualitative examples of sample generation from Config E on CIFAR-10.
29

---

Figure 9: Qualitative examples of sample generation from our Config E on ImageNet-32. 32.
30

---

Figure 10: Qualitative examples of sample generation from our Config E on ImageNet-64.
31

---

G
Training Curves
Figure 11: CIFAR-10 training curves.
Dlmm
0.21
Figure 12: FFHQ-64 training curves.
Clab
4.000
Figure 13: FFHQ-256 training curves.
32
Ah
12-
D0m
1810
Yea
Figure 14: ImageNet-32 training curves.
Dimn
Figure 15: ImageNet-64 training curves.
32

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Claim of stability is justified by Figure I and later experimental performance. Claim of convergence properties is justified in Appendices A,B,C. Claim of SOTA GAN is experimentally justified in Section 4, Claims bound specific datasets.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Please see Section 5.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-specfiiaatinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g. if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
33

---

Answer: [Yes]
Justification: Prior knowledge of Mescheder et al. [52] is required, but this is cited appropri- ately help the reader.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Supplemental table lists all hyperparamters, and supplemental section describes the training configurations.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel urchitecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which authors are welcome describe the particular way they provide for reproducibilityy In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers have some path reproducing verifying the results.
34

---

5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [No]
Justification: There is data. There is code at submission time. The authors will aim to release this by publication time, with instructions to faithfully reproduce the experiments. Code URL included in abstract.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
//nips. wttppscccppblliiii wwc/publi//ggiiiieeeeeeeeebeeeee gew.dess////eeeeeeeeoooc.ee.ee Codeeuumiis issisnnnn sssionolicyy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Supplemental table lists all hyperparamters, and supplemental section describes the training configurations.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Each experiment takes many days to compute, some take weeks. We do not have the compute time to provide variance bars on training executions.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
35

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
Question:
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification: Please see supplemental section on the experimental setting.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https : httper/iss.ccpppuicceeeeeeeeeeee
cc
Answer: [Yes]
Justification: Experimental settings are standard and within the norms of the community.
Guidelines:
The answer NA means that the authors have reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We mention it briefly in Section 5. The paper describes basic machine learning methodology, and so does not address a specific application with specific societal impacts. But, GANs do have potential social impact; it is clear that face generation has significant impact (e-g., deep fakes) and our paper does use face database for evaluation thanks to it being community norm.
36

---

Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (c.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploymemt of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [No]
Justification: There is no new data and much larger models produce higher fidelity images. The cost of training these large GANs is not prohibitive and is often done by hobbyists. As such, it is doubtful that these models will unlock any new capabilities for mis-use or dual-use.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (c.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All datasets are cited.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
37

---

The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g.. website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswi thcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's ' creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
No new assets are released.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the datasct/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: No human subjects are used and no crowdsourcing is used.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: No human subjects used and crowdsourcing is used.
Guidelines:
38

---

The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
39


Reviews:
[{"review_id": "JiVbQFJakv", "reviewer": "Reviewer_otaU", "summary": "The authors posit that the main reason GAN research has been slow in recent years is due to the most foundational StyleGAN2 not having undergone major architectural changes, essentially due to a lack of convergence guarantee in GAN objectives and being prone to mode collapse. This has limited the scope of architectural changes and further GAN research.\nTwo identifiable goals of the paper are to mathematically derive a well-behaved regularized loss which guarantees GAN convergence and diversity, provide empirical evidence for the same, and now being free to make unconstrained design choices attempt to develop a GAN baseline which inherits features from modern vision architectures. \n\nThe authors' methodology is inspired by two key works: (i) RpGAN [20], which addressed mode dropping in GANs with a better loss landscape, theoretically confirmed by Sun et al. [65] to contain only global minima; and (ii) Mescheder et al. [43], which provided evidence for local convergence with zero-centered gradient penalties in GAN objectives. By extending zero-centered gradient penalties to the RpGAN formulation, the authors address both mode dropping and unstable training. They offer theoretical proof for local convergence and present experimental evidence supporting global convergence, though no theoretical proof for the latter is provided.\nDesign changes in Config D are motivated by ResNet [14] and ConvNeXt [41], selectively applying aspects consistent with the authors' observations. Following findings in [31, 29], they eliminate normalization in StyleGAN in B and introduce fix-up initialization [77] in D to prevent variance explosion. Changes in E are inspired by ResNeXt [74], incorporating group convolutions to increase bottleneck width and followed by bottleneck inversion for efficient use. This simple baseline is empirically found to outperform previous GAN methods and some recent diffusion based models on several unconditional and conditional generation tasks.", "questions": "1. To clarify, on line 132 on page 4, “857” corresponds to StyleGAN backbone trained on WGAN-GP loss and “881” corresponds to the mini-batch standard deviation trick added to the same? Would be better to clarify the exact configuration and include the two cases in table 1 and figure 1.", "limitations": "-", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper presents a combination of well-known techniques by extending the RpGAN formulation with zero-centered gradient penalties. 2. While the individual components like RpGAN, zero-centered gradient penalties, and modern vision architectures (ResNet, ConvNeXt) are not new, their integration into a cohesive GAN framework is a notable contribution.\n3. The submission is technically sound with well-supported claims through theoretical analysis and experimental results.\n4. The paper is clearly written and well-organized.\n5. The results are significant, with improved performance over existing GANs and some diffusion models on various datasets.", "weaknesses": "1. While experimental results are well presented, there seems to be a lack of information about the training setup, hyperparameters used, number of inference steps, etc pertaining to the diffusion based models in Tables 4, 5 and 6. It would be useful to include this information in the supplementary or the tables itself for a transparent comparison.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback.\n\n> “there seems to be a lack of information about the training setup, hyperparameters used, number of inference steps, etc pertaining to the diffusion based models in Tables 4, 5 and 6.”\n\nThe diffusion model numbers in these tables are directly taken from reports in existing papers. Below, we have added the commonly-reported number of function evaluations (NFEs) for each. For the camera ready, we will add more detailed training setup descriptions and a hyperparameter table.\n\nTable 4:\n- LDM: from https://arxiv.org/abs/2112.10752, Table 18, NFE: 200\n- ADM (DDIM & DPM-Solver) & Diffusion Autoencoder: from https://arxiv.org/abs/2312.06285, Table 1, NFE: 500\n\nTable 5:\n- DDPM: from https://arxiv.org/abs/2006.11239, Table 1, NFE: 1000\n- DDIM: from https://arxiv.org/abs/2010.02502, Table 1, NFE: 50\n- VE & VP: from https://arxiv.org/abs/2206.00364, Table 2, NFE: 35\n\nTable 6:\n- ADM  -> https://arxiv.org/pdf/2301.11706 Table 3, NFE: 1000\n- DDPM-IP> https://arxiv.org/pdf/2301.11706 Table 3, NFE 1000\n- VDM -> https://proceedings.neurips.cc/paper_files/paper/2021/file/b578f2a52a0229873fefc2a4b0 -> NFE 1000\n- DDPM++ -> https://openreview.net/forum?id=PxTIG12RRHS -> NFE 1000\n\n> “To clarify, on line 132 on page 4, “857” corresponds to StyleGAN backbone trained on WGAN-GP loss and “881” corresponds to the mini-batch standard deviation trick added to the same? Would be better to clarify the exact configuration…”\n\nNot quite. Although the mini-batch standard deviation trick was popularized by StyleGAN, it was introduced in Progressive GAN (https://arxiv.org/abs/1710.10196). Thus, we took these numbers from Table 4 of the Progressive GAN paper. “857” corresponds to a low-capacity version of the progressive GAN trained with WGAN-GP loss and “881” adds the minibatch stddev trick. We will clarify the reference in the camera-ready paper.\n\n> “... and include the two cases in table 1 and figure 1.”\n\nWhile in principle this is a good idea to contextualize the gain, in practice it is a little tricky: ProGAN/StyleGAN are quite different models, and Table/Figure 1 aim to show without complication the effect of different losses on our small simple model. We cannot ‘strip’ ProGAN/StyleGAN meaningfully, and so their performance may be better than what our simple model achieves. We do not want to confuse the reader by adding new lines to the plots that are incomparable, obfuscating the point we are trying to make about stability. If the reviewer has other ideas for how to make this comparison, we welcome them.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "synu2zRICp", "reviewer": "Reviewer_2Hzz", "summary": "This paper introduces R3GAN as a GAN baseline that simplifies and modernizes the architecture by replacing ad-hoc tricks with modern designs. R3GAN utilized a regularized relativistic GAN loss coupled with zero-centered gradient penalties on both real and generated data, to addresses mode dropping and non-convergence, providing local convergence guarantees. The proposed loss allows for the elimination of unnecessary tricks and the adoption of advanced architectures. R3GAN demonstrates improved stability and competitive performance against state-of-the-art models on multiple datasets such as StackedMNIST, CIFAR, ImageNet-32, and FFHQ-256.", "questions": "How scalable is the proposed model? It is suggested to validate the effectiveness of the method on higher-resolution datasets and more complex tasks. This would align with the motivation to serve as a modern baseline, especially in the current context where both model parameter size and training data scale are increasing.", "limitations": "The authors have discussed the limitations and potential negative societal impact of their work.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "+ The writing of the paper is clear, making it easy to understand and easy to follow.\n+ One contribution of this paper is the simplification and optimization of the StyleGAN network architecture, which, as demonstrated by experiments, achieves considerable performance without the need for many elaborate tricks.\n+ The paper provides theoretical analysis to prove the convergence of the loss used.", "weaknesses": "- The novelty of the method is somewhat limited, as both relativistic pairing GAN (RpGAN) and zero-centered gradient penalties (0-GPs) are previously proposed and validated approaches in the field of GANs.\n- Small-scale experiments to validate the significant improvements in stability and diversity provided by the used loss are insufficient; more complex datasets should be used to verify its effectiveness. As shown in Table 2, the improvement from config C compared to config B is not significant. Generally, stability can allow the model to converge better, thereby enhancing the final results, and diversity can be measured by metrics such as recall or coverage.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback.\n\n> “The novelty of the method is somewhat limited, as both relativistic pairing GAN (RpGAN) and zero-centered gradient penalties (0-GPs) are previously proposed and validated approaches in the field of GANs.”\n\nWhile these components have been proposed separately, none matches our contributions:\n- **New theoretical and empirical contribution**: 0-GPs have been proposed in the context of classic GANs, but nobody has studied how they interact with the training dynamics of RpGANs. We provide both theoretical and empirical evidence on how they address the non-convergence of RpGANs, which ultimately allows us to derive a well-behaved GAN loss that is resilient to GAN optimization problems.\n- **New practical contribution**: We establish a clean and cohesive GAN framework with just the must-have components. With the new loss, we show that a simple non-StyleGAN based architecture can beat StyleGAN2 by a large margin.\n\n> “How scalable is the proposed model? It is suggested to validate the effectiveness of the method on higher-resolution datasets and more complex tasks. This would align with the motivation to serve as a modern baseline, especially in the current context where both model parameter size and training data scale are increasing.”\n\nWe present early evidence: To complement our ImageNet-32 result, we are currently running ImageNet-64 experiments by stacking another stage onto our ImageNet-32 architecture. Note that, to evaluate EDM, Karras et al. (2022) used ImageNet-64 as their biggest dataset. Without hyperparameter tuning, we achieved FID 2.56, which is in between ADM (FID: 2.91, 250 DDIM steps) and EDM (FID: 2.23, 79 steps with EDM sampler). We achieve this without techniques like adaGN and attention in ADM/EDM. Our result uses ~90M parameters while EDM and ADM use ~300M. This is evidence that our model is more efficient at its current scale and will hopefully scale well.\n\nFor more complex tasks, we are unsure what the reviewer would like to see - please suggest something. To the best of our knowledge, ImageNet is the one of the largest and best studied closed-world datasets. If the reviewer meant extending the setting to open-world text-to-image generation, then this requires significantly more compute (and monetary cost!). It also may require additional techniques like text encoders, self/cross-attention, and conditional modulation. \n\nMore generally, addressing scalability is secondary to our top priority of addressing the GAN “myths” that led to the shrinking (as YF9j puts it) of GAN research: GANs are difficult to train, GANs do not work without many tricks, GANs are fragile and do not work with powerful modern vision backbones. Scalability, while also highly important to GAN applications, is dependent upon proving the feasibility of a clean GAN framework, via a minimal model that beats prior work. Of course, we agree with the reviewer on the practical importance of scalability. We are not OpenAI or Meta with thousands of GPUs, but our added experiment will provide evidence for the scalability of our model given our compute constraints.\n\nFinally, we will open source our code and pretrained models and we welcome the community to test the scalability of our model.\n\n> \"Small-scale experiments to validate the significant improvements in stability and diversity provided by the used loss are insufficient; more complex datasets should be used to verify its effectiveness.\" \n\nWe acknowledge that stability and diversity are only directly validated on StackedMNIST (Figure 1 / Table 1). We have added diversity via the recall metric for all other experiments below. Concerning stability, we propose to add convergence plots for all datasets and for loss-ablated models, similar to Figure 1, in supplemental material, with discussion in the main paper.\n\nThat said, StackedMNIST itself should not be dismissed so easily: it is a challenging case for stability because of the limited data and that data having an all-black background. This makes D's job to reject fake samples extremely easy early on, causing instability. FFHQ is generally more stable and the original StyleGAN loss is likely to work in terms of stability, but we show that our approach gives better results in terms of FID.\n\n> [Following on; concerning the impact of stability on performance] “As shown in Table 2, the improvement from config C [well-behaved loss] compared to config B [no well-behaved loss] is not significant.”\n\nImproving performance may need both a well-behaved loss _and_ a more powerful backbone, as is the case here. Both Config-B and C have a weak DCGAN-style backbone. Assuming convergence is stable, G and D must still be sufficiently powerful for a given dataset, otherwise performance will saturate regardless of the loss used. With a weak backbone, it is not surprising that replacing the loss does not result in a drastic improvement. With a stronger backbone in D and E, performance increases. Adding convergence plots for this more complex FFHQ data will contextualize this result with respect to stability.\n\n> “... and diversity can be measured by metrics such as recall …”\n\nThanks - our oversight - we report recall numbers below.\n\nOn ImageNet-32, we obtain a recall of 0.63, comparable to ADM (recall 0.63). \n\nOn FFHQ-256, after gamma hyperparameter tuning, we achieved a lower FID of 2.77 and a recall of 0.50, outperforming StyleGAN2 (FID: 3.78, recall: 0.43).\n\nOn CIFAR-10, we achieved a slightly better FID of 1.97 and recall 0.57 after gamma hyperparameter tuning. By comparison, StyleGAN-XL has a slightly lower FID of 1.85 but a much worse recall of 0.47. StyleGAN2-Ada has a considerably worse FID of 2.42 but a slightly higher recall of 0.59. We attribute the recall gap to hyperparameter tuning, as we tune hyperparameters for best FID performance.\n\nFor ablations, Config-C, using our new loss, achieved recall 0.24, and Config-B with the original StyleGAN loss achieved a worse recall of 0.22.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GFqE6fAfBw", "reviewer": "Reviewer_YF9j", "summary": "A lot of GANs papers have shrunk in quality since diffusion arose as a strong method. The authors goes against the grain by focusing on modernizing GAN baseline in a principled manner and in doing so they obtain stable, convergent, diversity, and high-quality images comparable to diffusion models. The authors highlight the fact that a lot of the GAN literature is riled with empirical tricks with no theory to back them. They explain that StyleGAN without tricks is very close to the original DCGAN of 2015. Meanwhile a lot more progress as been done in diffusion models leveraging many architectural improves.\n\nThey instead start with a Relativistic pairing GAN baseline which, unlike regular GANs, have been shown to have no local-minima basins where the loss must worsen before reaching the true global minimum. However, they show that, just like regular GANs, RpGANs are not guaranteed to converge to the global minimum. They show that, just like regular GANs, gradient penalty on real data (R1) or fake data (R2) guarantee convergence. They suggests using both together. They then show empirically that on StackMNIST, GANs with R1 alone fails, but having both R1, R2, and RpGAN lead to perfect mode coverage and strong stability. As further justification, they also leverage the fact that from a constructive approach, when deriving a maximum-margin GAN, one obtains a gradient penalty with both R1, R2.\n\nThen they use a very careful ablation approach by starting from StyleGAN2, removing all the complicated empirical tricks but keeping R1 penalty, which reduce performance a bit, then moving to RpGAN, adding R2 penalty, then modernizing the architecture, reaching slightly better performance at similar number of parameters, but without tricks. The describe extensively and carefully the effect of different architecture choices on performance.\n\nExperiments on CIFAR-10, Imagenet-32, and FFHQ-256 are shown were the method proposed shows extremely low FID, better than diffusion models and do so while keeping the parameter count low. Results are very good, but work is still needed to scale these methods to text-to-image settings. However, the current results are very promising and the method has been made very clean and carefully designed (similar to ConvNext).", "questions": "- Any intuition to why worse performance with GELU and swish activations since these are very successful with transformers? Is this true for both G and D?\n- Have you consider a transformer architecture as alternative to convnext-style architecture?\n- You mention that normalization can harm GANs, but groupnorm is used in diffusion models and layernorm is used in language models, could this be useful for your approach? Has it been tried before? I saw the references you mentioned to say that normalization is bad for GANs, but I haven't looked at them individually.\n- You mention that PresGAN and DDGANs obtain good performances, do you think that some of their ideas could be useful for improve the GAN baseline?\n- What is EMA half-life? Is the EMA .999?\n- Why do you schedule Adam B2? This seems very atypical.\n- Assume $\\alpha$ is the lr. Rather than schedule the R1/R2 $\\gamma$ manually, why not make it $\\alpha\\gamma$ so that by decreasing the lr $\\alpha$ over time, $\\gamma$ is also reduces automatically (thus reducing the hyperparameter tuning). The same could be done with Beta2, by doing $1-((\\alpha/alpha_{init})*(1-beta2))$, so moving the 1-.9 = 0.1 to 0.01 (1-.99) when decreasing the lr to 1/10 of its value over training time. I just feel like choices like this would greatly reduce the arbitrary hyperparameter tuning used here which is quite complex in Table 7.", "limitations": "The authors have adequately addressed the limitations.", "rating": 9, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "Experiments on CIFAR-10, Imagenet-32, and FFHQ-256 are shown were the method proposed shows extremely low FID, better than diffusion models and do so while keeping the parameter count low. The derivation of their architecture is done carefully in a thorough manner.", "weaknesses": "Great results on image datasets, but it still need to be scaled to large-scale text-to-image setting.\nMany GPUs are used, even for CIFAR-10; this is also in line with diffusion models, but it seems a lot, are all these GPUs needed for CIFAR-10?\nAlthough cleaner, the new baseline still contains a lot of complicated hyperparameter tuning, which change from one dataset to another (in Table 7). It would be great to clean this set of hyperparameters, I suggested some potential ideas in the \"questions\".", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback.\n\n> “... still need to be scaled to large-scale text-to-image setting.”\n\nFor scaling, we are currently running ImageNet-64 experiments to include in our paper; please see our discussion in response to Reviewer 2Hzz. In general, we hope our work is a meaningful first step and that future works can better assess its scaling.\n\n> “Many GPUs are used, even for CIFAR-10; this is also in line with diffusion models, but it seems a lot, are all these GPUs needed for CIFAR-10?”\n\nUsing 8 GPUs to train CIFAR-10 is not required; our models can be trained with fewer GPUs at the cost of slower training speed. But, it is becoming a common practice, e.g., for diffusion models; Karras et al. used 8 GPUs to train their CIFAR-10 models in EDM. \n\n> “Any intuition to why worse performance with GELU and swish activations since these are very successful with transformers? Is this true for both G and D?”\n\nIn experiments, we tried to apply GELU/Swish/SMU to G and D and found that doing so deteriorates FID considerably. We did not try on G xor D. We posit two independent factors:\n- ConvNeXt in general does not benefit much from GELU (and possibly similar activations). Table 10 and Table 11 in https://arxiv.org/abs/2201.03545: replacing ReLU with GELU gives little performance gain to ConvNeXt-T and virtually no performance gain to ConvNeXt-B.\n- In the context of GANs, GELU and Swish have the same problem as ReLU: that they have little gradient in the negative interval. Since G is updated from the gradient of D, having these activation functions in D could sparsify the gradient of D and as a result G will not receive as much useful information from D compared to using leaky ReLU.\n\nThis does not explain the strange case of SMU (https://arxiv.org/abs/2111.04682): SMU is a smooth approximation of leaky ReLU and does not have the sparse gradient problem. It is unclear why it also underperformed and future work awaits. We are happy to add this discussion to supplemental.\n\n> “Have you considered a transformer architecture?”\n\nWe have not conducted any experiments, but are also interested to see how they perform. In particular, whether adding attention blocks to a ConvNet (similar to BigGAN and diffusion UNet) or using a pure transformer architecture (similar to DiT) will result in stronger performance. Given the impressive results of EDM2 (which uses UNet), it seems the argument has not yet settled for generative modeling.\n\n> “You mention that normalization can harm GANs, but groupnorm is used in diffusion models and layernorm is used in language models, could this be useful for your approach? Has it been tried before? I saw the references you mentioned to say that normalization is bad for GANs.”\n\nWe tried adding groupnorm to G and D and it did not improve FID or training stability. We would like to clarify that we do not intend to claim that all forms of normalizations are harmful. Our claim only extends to normalization layers that explicitly standardizes the mean and stddev of the activation maps. This has been verified by prior studies:\n- Instance norm harms GANs: https://arxiv.org/abs/1912.04958\n- Group norm harms diffusion models: https://arxiv.org/abs/2312.02696\n- Group norm harms VAEs: https://arxiv.org/abs/2405.14477, pg. 5\n\nThe harm of normalization layers extends to the adjacent field of image restoration (https://arxiv.org/abs/1707.02921, sec 3.1; https://arxiv.org/abs/1809.00219, supplemental sec 1). To the best of our knowledge, EDM2 is currently the strongest diffusion UNet and it does not use normalization layers. However, it does apply normalization to the trainable weights and this improves performance considerably. We expect that applying the normalization techniques in EDM2 would improve our model's performance.\n\n> “You mention that PresGAN and DDGANs obtain good performances, do you think that some of their ideas could be useful?”\n\nYes, it is possible. We excluded these ideas as they are not an indispensable part of a minimal GAN framework. DDGAN in particular combines GANs and diffusion models - a promising research direction. Prior work has explored this in various flavors: DDGAN formulates GAN as a fast diffusion solver; Diffusion GAN (https://arxiv.org/abs/2206.02262) applies diffusion to GANs as a non-leaky augmentation; Diffusion2GAN (https://arxiv.org/abs/2405.05967) distills a diffusion model into a GAN; PaGoDA (https://arxiv.org/abs/2405.14822) inverts a pretrained diffusion model to obtain the posterior p(z | x) and facilitates the combination of GANs and MLE.\n\n> “What is EMA half-life? Is the EMA .999?”\n\nWe follow the notation of Karras et al. in StyleGAN and EDM. EMA beta can be obtained using the formula ema_beta = 0.5 ** (batch_size / ema_half_life).\n\n> “Why do you schedule Adam B2? This seems very atypical.”\n\nIt is atypical. The gradient magnitude of G and D varies drastically early in training. This effect is further amplified when we use a large initial learning rate, leading to large loss oscillations early on. Using a constant Adam beta2 such as 0.99 produces a loss that always stabilizes in a few hundred steps, and the initial oscillations do not seem to have any long-term negative impact. \nHowever, by annealing Adam beta2, we can reduce the initial loss oscillations: now, we no longer need to wait a few hundred steps for the loss to settle. In particular, a lower initial Adam beta2 allows the optimizer to adapt to large gradient magnitude changes much quicker and so stabilize training earlier. Similar reasoning for tweaking Adam beta2 is given in EDM2 (https://arxiv.org/abs/2312.02696, pg. 18). The benefit of scheduling Adam beta2 is marginal, but since we already schedule multiple hyperparameters, we might as well schedule Adam beta2 too.\n\n> Hyperparameter scheduling (question truncated for space).\n\nWe agree that your suggestion will ease the hyperparameter tuning, and we plan to introduce something similar for the automatic configuration of our model.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Thank you everyone for your constructive feedback.

In summary, all reviewers found that the paper had strengths:
- The paper is clearly written.
- The claims are well supported with both theoretical and empirical evidence.
- The theoretical insights allow a method that is simpler than past GAN works with fewer tricks.
- The results are significant, being better than many existing GANs and some diffusion methods.

However, there are some weaknesses and questions:
- Novelty: Reviewer 2Hzz believes that novelty is somewhat limited as both RpGAN and zero-centered gradient penalty works both exist; reviewer otaU notes that this is true but that "their integration into a cohesive GAN framework is a notable contribution."
- Technical Retails: Reviewer otaU requests training details for comparison methods and a technical clarification, and reviewer YF9j poses numerous small questions to help us improve our technical explanation. We answer each of these questions.
- Stability/Diversity Evaluation: Reviewer 2Hzz states that "small-scale experiments to validate the significant improvements in stability and diversity ... are insufficient; more complex datasets should be used." We report the recall metric for all datasets in this rebuttal to evaluate diversity, and propose plots to show stability on our more complex datasets.
- Scalability: Reviewer 2Hzz asks "How scalable is the proposed model?" Reviewer YF9j notes this issue too - "work is still needed to scale these methods to text-to-image settings" - but is satisfied to support the work as the "current results are very promising." To provide evidence for scalability, we report preliminary results on ImageNet-64 to contrast to our ImageNet-32 results for this task. But, we defer more significant scalability evaluation to future work - we will open source our code and pretrained models to make this easier.

We hope to answer these questions in detail in response to each individual reviewer, and we look forward to further discussion with you all.

Meta-review: The reviews are overall positive and recognize the solid theoretical foundation, empirical performance, and contribution to modernizing GAN architectures. While there are some concerns about novelty, scalability, and the transparency of the experimental setup, the reviewers believe that it's a valuable contribution to the community given the sole recent focus on diffusion models. I urge the authors to incorporate the feedback in the manuscript.
Decision: Accept (poster)