The GAN is dead; long live the GAN! A Modern GAN Baseline

Authors: yiwen  huang
aaron  gokaslan
volodymyr  kuleshov
james  tompkin
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-12 17:21:55

Abstract: There is a widely-spread claim that GANs are difficult to train, and GAN architectures in the literature are littered with empirical tricks. We provide evidence against this claim and build a modern GAN baseline in a more principled manner. First, we derive a well-behaved regularized relativistic GAN loss that addresses issues of mode dropping and non-convergence that were previously tackled via a bag of ad-hoc tricks. We analyze our loss mathematically and prove that it admits local convergence guarantees, unlike most existing relativistic losses. Second, this loss allows us to discard all ad-hoc tricks and replace outdated backbones used in common GANs with modern architectures. Using StyleGAN2 as an example, we present a roadmap of simplification and modernization that results in a new minimalist baseline---R3GAN. Despite being simple, our approach surpasses StyleGAN2 on FFHQ, ImageNet, CIFAR, and Stacked MNIST datasets, and compares favorably against state-of-the-art GANs and diffusion models. Code: https://www.github.com/brownvc/R3GAN

Main paper: 
 introduction: 
 Generative adversarial networks (GANs) let us generate high-quality images in a single forward pass. However, the original objective in Goodfellow et al. [13] , is notoriously difficult to optimize due to its minimax nature. This leads to a fear that training might diverge at any point due to instability, and a fear that generated images might lose diversity through mode collapse. While there has been progress in GAN objectives [14, 22, 81, 52, 64] , practically, the effects of brittle losses are still regularly felt. This notoriety has had a lasting negative impact on GAN research.A complementary issue-partly motivated by this instability-is that existing popular GAN backbones like StyleGAN [29, 31, 30, 32] use many poorly-understood empirical tricks with little theory. For instance, StyleGAN uses a gradient penalized non-saturating loss [52] to increase stability (affecting sample diversity), but then employs a minibatch standard deviation trick [28] to increase sample diversity. Without tricks, the StyleGAN backbone still resembles DCGAN [60] from 2015, yet it is still the common backbone of SOTA GANs such as GigaGAN [26] and StyleGAN-T [70] . Advances in GANs have been conservative compared to other generative models such as diffusion models [20, 78, 33, 34] , where modern computer vision techniques such as multi-headed self attention [87] and backbones such as preactivated ResNet [17] , U-Net [63] and vision transformers (ViTs) [9] are the norm. Given outdated backbones, it is not surprising that there is a widely-spread belief that GANs do not scale in terms of quantitative metrics like Frechet Inception Distance [19] .We reconsider this situation: we show that by combining progress in objectives into a regularized training loss, GANs gain improved training stability, which allows us to upgrade GANs with modern backbones. First, we propose a novel objective that augments the relativistic pairing GAN loss (RpGAN; [22] ) with zero-centered gradient penalties [52, 64] , improving stability [14, 64, 52] . We show mathematically that gradient-penalized RpGAN enjoys the same guarantee of local convergence as regularized classic GANs, and that removing our regularization scheme induces non-convergence.Once we have a well-behaved loss, none of the GAN tricks are necessary [28, 31] , and we are free to engineer a modern SOTA backbone architecture. We strip StyleGAN of all its features, identify those that are essential, then borrow new architecture designs from modern ConvNets and transformers [48, 97] . Briefly, we find that proper ResNet design [17, 67] , initialization [99] , and resampling [29, 31, 32, 100] are important, along with grouped convolution [95, 5] and no normalization [31, 34, 14, 88, 4] . This leads to a design that is simpler than StyleGAN and improves FID performance for the same network capacity (2.75 vs. 3.78 on FFHQ-256).In summary, our work first argues mathematically that GANs need not be tricky to train via an improved regularized loss. Then, it empirically develops a simple GAN baseline that, without any tricks, compares favorably by FID to StyleGAN [29, 31, 32] , other SOTA GANs [3, 42, 94] , and diffusion models [20, 78, 86] across FFHQ, ImageNet, CIFAR, and Stacked MNIST datasets.2 Serving Two Masters: Stability and Diversity with RpGAN +R 1 + R 2In defining a GAN objective, we tackle two challenges: stability and diversity. Some previous work deals with stability [29, 31, 32] and other previous work deals with mode collapse [22] . To make progress in both, we combine a stable method with a simple regularizer that is grounded by theory. 
traditional gan: 
 A traditional GAN [13, 57] is formulated as a minimax game between a discriminator (or critic) D ψ and a generator G θ . Given real data x ∼ p D and fake data x ∼ p θ produced by G θ , the most general form of a GAN is given by:EQUATIONwhere G tries to minimize L while D tries to maximize it. The choice of f is flexible [50, 44] . In particular, f (t) = -log(1 + e -t ) recovers the classic GAN by Goodfellow et al. [13] . For the rest of this work, this will be our choice of f [57] .It has been shown that Equation 1 has convex properties when p θ can be optimized directly [13, 81] . However, in practical implementations, the empirical GAN loss typically shifts fake samples beyond the decision boundary set by D, as opposed to directly updating the density function p θ . This deviation leads to a significantly more challenging problem, characterized by susceptibility to two prevalent failure scenarios: mode collapse/dropping1 and non-convergence.We employ a slightly different minimax game named relativistic pairing GAN (RpGAN) by Jolicoeur-Martineau et al. [22] to address mode dropping. The general RpGAN is defined as:EQUATIONAlthough Eq. 2 differs only slightly from Eq. 1, evaluating this critic difference has a fundamental impact on the landscape of L. Since Eq. 1 merely requires D to separate real and fake data, in the scenario where all real and fake data can be separated by a single decision boundary, the empirical GAN loss encourages G to simply move all fake samples barely past this single boundary-this degenerate solution is what we observe as mode collapse/dropping. Sun et al. [81] characterize such degenerate solutions as bad local minima in the landscape of L, and show that Eq. 1 has exponentially many bad local minima. The culprit is the existence of a single decision boundary that naturally arises when real and fake data are considered in isolation. RpGAN introduces a simple solution by coupling real and fake data, i.e. a fake sample is critiqued by its realness relative to a real sample, which effectively maintains a decision boundary in the neighborhood of each real sample and hence forbids mode dropping. Sun et al. [81] show that the landscape of Eq. 2 contains no local minima that correspond to mode dropping solutions, and that every basin is a global minimum.Although the RpGAN landscape result [81] allows us to address mode dropping, the training dynamics of RpGAN have yet to be studied. The ultimate goal of Eq. 2 is to find the equilibrium (θ * , ψ * ) such that p θ * = p D and D ψ * is constant everywhere on p D . Sun et al. [81] show that θ * is globally reachable along a non-increasing trajectory in the landscape of Eq. 2 under reasonable assumptions. However, the existence of such a trajectory does not necessarily mean that gradient descent will find it. Jolicoeur-Martineau et al. show empirically that unregularized RpGAN does not perform well [22] .Proposition I. (Informal) Unregularized RpGAN does not always converge using gradient descent.We confirm this proposition with a proof in Appendix B. We show analytically that RpGAN does not converge for certain types of p D , such as ones that approach a delta distribution. Thus, further regularization is necessary to fill in the missing piece of a well-behaved loss.Zero-centered gradient penalties. To tackle RpGAN non-convergence, we explore gradient penalties as the solution since it is proven that zero-centered gradient penalties (0-GP) facilitate convergent training for classic GANs [52] . The two most commonly-used 0-GPs are R 1 and R 2 :EQUATIONR 1 penalizes the gradient norm of D on real data, and R 2 penalizes the gradient norm of D on fake data. Analysis on the training dynamics of GANs has thus far focused on local convergence [55, 51, 52] , i.e., whether the training at least converges when (θ, ψ) are in a neighborhood of (θ * , ψ * ). In such a scenario, the convergence behavior can be analyzed [55, 51, 52] by examining the spectrum of the Jacobian of the gradient vector field(-∇ θ L, ∇ ψ L) at (θ * , ψ * ).The key insight here is that when G already produces the true distribution, we want ∇ x D = 0, so that G is not pushed away from its optimal state, and thus the training does not oscillate. R 1 and R 2 impose such a constraint when p θ = p D . This also explains why earlier attempts at gradient penalties, such as the one-centered gradient penalty (1-GP) in WGAN-GP [14] , fail to achieve convergent training [52] as they still encourage D to have a non-zero slope when G has reached optimality.Since the same insight also applies to RpGAN, we extend our previous analysis and show that:Proposition II. (Informal) RpGAN with R 1 or R 2 regularization is locally convergent subject to similar assumptions as in Mescheder et al. [52] .In Appendix C, our proof similarly analyzes the eigenvalues of the Jacobian of the regularized RpGAN gradient vector field at (θ * , ψ * ). We show that all eigenvalues have a negative real part; thus, regularized RpGAN is convergent in a neighborhood of (θ * , ψ * ) for small enough learning rates [52] .Discussion. Another line of work [64] links R 1 and R 2 to instance noise [75] as its analytical approximation. Roth et al. [64] showed that for the classic GAN [13] by Goodfellow et al., R 1 approximates convolving p D with the density function of N (0, γI), up to additional weighting and a Laplacian error term. R 2 likewise approximates convolving p θ with N (0, γI) up to similar error terms. The Laplacian error terms from R 1 , R 2 cancel when D ψ approaches D ψ * . We do not extend Roth et al.'s proof [64] to RpGAN; however, this approach might provide complimentary insights to our work, which follows the strategy of Mescheder et al. [52] .We experiment with how well-behaved our loss is on StackedMNIST [46] which consists of 1000 uniformly-distributed modes. The network is a small ResNet [17] for G and D without any normalization layers [21, 91, 1, 85] . Through the use of a pretrained MNIST classifier, we can explicitly measure how many modes of p D are recovered by p θ . Furthermore, we can estimate the reverse KL divergence between the fake and real samples D KL (p θ ∥ p D ) via the KL divergence between the categorical distribution of p θ and the true uniform distribution. A conventional GAN loss with R 1 , as used by Mescheder et al. [52] and the StyleGAN series [29, 31, 32] , diverges quickly (Fig. 1 ). Next, while theoretically sufficient for local convergence, RpGAN with only R 1 regularization is also unstable and diverges quickly 2 . In each case, the gradient of D on fake samples explodes when training diverges. With both R 1 and R 2 , training becomes stable for both the classic GAN and RpGAN. Now stable, we can see that the classic GAN suffers from mode dropping, whereas RpGAN achieves full mode coverage (Tab. 1) and reduces D KL from 0.9270 to 0.0781. As a point of contrast, StyleGAN [29, 31, 30, 32] uses the minibatch standard deviation trick to reduce mode dropping, improving mode coverage from 857 to 881 on StackedMNIST 3 and with barely any improvement on D KL [28] .R 1 alone is not sufficient for globally-convergent training. While a theoretical analysis of this is difficult, our small demonstration still provides insights into the assumptions of our convergence proof.In particular, the assumption that (θ, ψ) are sufficiently close to (θ * , ψ * ) is highly unlikely early in training. In this scenario, if D is sufficiently powerful, regularizing D solely on real data is not likely to have much effect on D's behavior on fake data and so training can fail due to an ill-behaved D on fake data. This observation has been made by previous studies [84, 83] specifically for empirical GAN training, that regularizing an empirical discriminator with only R 1 leads to gradient explosion on fake data due to the memorization of real samples.Thus, the practical solution is to regularize D on both real and fake data. The benefit of doing so can be viewed from the insight of Roth et al. [64] : that applying R 1 and R 2 in conjunction smooths both p D and p θ which makes learning easier than only smoothing p D . We also find empirically that with both R 1 and R 2 in place, D tends to satisfyE x∼p D ∥∇ x D∥ 2 ≈ E x∼p θ ∥∇ x D∥ 2 even early inthe training. Jolicoeur-Martineau et al. [23] show that in this case D becomes a maximum margin classifier-but if only one regularization term is applied, this does not hold. Additionally, having roughly the same gradient norm on real and fake data potentially reduces discriminator overfitting, as Fang et al. [10] observe that the gradient norm on real and fake data diverges when D starts to overfit.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: Please see Section 5.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
a roadmap to a new baseline -r3gan: 
 The well-behaved RpGAN + R 1 + R 2 loss alleviates GAN optimization problems, and lets us proceed to build a minimalist baseline-R3GAN-with recent network backbone advances in mind [48, 97] . Rather than simply state the new approach, we will draw out a roadmap from the StyleGAN2 baseline [30] . This model (Config A; identical to [30] ) consists of a VGG-like [73] The features fall into three categories:• Style-based generation: mapping network [29] , style injection [29] , weight modulation/demodulation [31] , noise injection [29] . • Image manipulation enhancements:mixing regularization [29] , path length regularization [31] . • Tricks: z normalization [28] , minibatch stddev [28] , equalized learning rate [28] , lazy regularization [31] .Following [69, 70] , we reduce the dimension of z to 64. The absence of equalized learning rate necessitates a lower learning rate, reduced from 2.5×10 -3 to 5×10 -5 . Despite a higher FID of 12.46 than Config-A, this simplified baseline produces reasonable sample quality and stable training. We compare this with DCGAN [60] , an early attempt at image generation. Key differences include: Experimental findings from StyleGAN. Violating a), b), or c) often leads to training failures. Gidel et al. [11] show that negative momentum can improve GAN training dynamics. Since optimal negative momentum is another challenging hyperparameter, we do not use any momentum to avoid worsening GAN training dynamics. Studies suggest normalization layers harm generative models [31, 34] . Batch normalization [21] often cripples training due to dependencies across multiple samples, and is incompatible with R 1 , R 2 , or RpGAN that assume independent handling of each sample. Weaker data-independent normalizations [31, 34] might help; we leave this for future work. Early GANs may succeed despite violating a) and c), possibly constituting a full-rank solution [52] to Eq. 1.Violations of d) or e) do not significantly impair training stability but negatively affect sample quality. Improper transposed convolution can cause checkerboard artifacts, unresolved even with subpixel convolution [72] or carefully tuned transposed convolution unless a low-pass filter is applied. Interpolation methods avoid this issue, varying from nearest neighbor [28] to Kaiser filters [32] . We use bilinear interpolation for simplicity. For activation functions, smooth approximations of (leaky) ReLU, such as Swish [61] , GELU [18] , and SMU [2] , worsen FID. PReLU [15] marginally improves FID but increases VRAM usage, so we use leaky ReLU.All subsequent configurations adhere to a) through e). Violation of f) is acceptable as it pertains to the network backbone of StyleGAN2 [31] , modernized in Config D and E.Well-behaved loss function (Config C). We use the loss function proposed in Section 2 and this reduces FID to 11.65. We hypothesize that the network backbone in Config B is the limiting factor.General network modernization (Config D). First, we apply the 1-3-1 bottleneck ResNet architecture [16, 17] to both G and D. This is the direct ancestor of all modern vision backbones [48, 97] . We also incorporate principles discovered in Config B and various modernization efforts from ConvNeXt [48] . We categorize the roadmap of ConvNeXt as follows:i. Consistently beneficial: i.1) increased width with depthwise convolution, i.2) inverted bottleneck, i.3) fewer activation functions, and i.4) separate resampling layers. ii. Negligible performance gain: ii.1) large kernel depthwise conv. with fewer channels, ii.2) swap ReLU with GELU, ii.3) fewer normalization layers, and ii.4) swap batch norm. with layer norm. iii. Irrelevant to our setting: iii.1) improved training recipe, iii.2) stage ratio, and iii.3) 'patchify' stem.We aim to apply i) to our model, specifically i.3 and i.4 for the classic ResNet, while reserving i.1 and i.2 for Config E. Many aspects of ii) were introduced merely to mimic vision transformers [47, 9] without yielding significant improvements [48] . ii.3 and ii.4 are inapplicable due to our avoidance of normalization layers following principle c). ii.2 contradicts our finding that GELU deteriorates GAN performance, thus we use leaky ReLU per principle e). Liu et al. emphasize large conv. kernels (ii.1) [48] , but this results in slightly worse performance compared to wider 3×3 conv. layers, so we do not adopt this ConvNeXt design choice.Neural network architecture details. Given i.3, i.4, and principles c), d), and e), we can replace the StyleGAN2 backbone with a modernized ResNet. We use a fully symmetric design for G and D with 25 M parameters each, comparable to Config-A. The architecture is minimalist: each resolution stage has one transition layer and two residual blocks. The transition layer consists of bilinear resampling and an optional 1×1 conv. for changing spatial size and feature map channels. The residual block includes five operations: Conv1×1→ Leaky ReLU → Conv3×3→ Leaky ReLU → Conv1×1, with the final Conv1×1 having no bias term. For the 4×4 resolution stage, the transition layer is replaced by a basis layer for G and a classifier head for D. The basis layer, similar to StyleGAN [29, 31] , uses 4×4 learnable feature maps modulated by z via a linear layer. The classifier head uses a global 4×4 depthwise conv. to remove spatial extent, followed by a linear layer to produce D's output. We maintain the width ratio for each resolution stage as in Config A, making the stem width 3× as wide due to the efficient 1×1 conv. The 3×3 conv. in the residual block has a compression ratio of 4, following [16, 17] , making the bottleneck width 0.75× as wide as Config A.To avoid variance explosion due to the lack of normalization, we employ fix-up initialization [99] : We zero-initialize the last convolutional layer in each residual block and scale down the initialization of the other two convolutional layers in the block by L -0.25 , where L is the number of residual blocks. We avoid other fix-up tricks, such as excessive bias terms and a learnable multiplier.. Now that we have settled on the overall architecture, we investigate how the residual block can be modernized, specifically i.1) and i.2). First, we explore i.1 and replace the 3×3 convolution in the residual block with a grouped convolution. We set the group size to 16 rather than 1 (i.e. depthwise convolution as in ConvNeXt) as depthwise convolution is highly inefficient on GPUs and is not much faster than using a larger group size. With grouped convolution, we can reduce the bottleneck compression ratio to two given the same model size. This increases the width of the bottleneck to 1.5× as wide as Config A. Finally, we notice that the compute cost of grouped convolution is negligible compared to 1×1 convolution, and so we seek to enhance the capacity of grouped convolution. We apply i.2), which inverts the bottleneck width and the stem width, and which doubles the width of grouped convolutions without any increase in model size. 4 Experiments Details 4.1 Roadmap Insights on FFHQ-256 [29] As per Table 2 , Config A (vanilla StyleGAN2) achieves an FID of 7.52 using the official implementation on FFHQ-256. Config B with all tricks removed achieves an FID of 12.46-performance drops as expected. Config C, with a well-behaved loss, achieves an FID of 11.65. But, now training is sufficiently stable to improve the architecture.Config D, which improves G and D based on the classic ResNet and ConvNeXt findings, achieves an FID of 9.95. The output skips of the StyleGAN2 generator are no longer useful given our new architecture; including them produces a worse FID of 10.17. Karras et al. find that the benefit of output skips is mostly related to gradient magnitude dynamics [32] , and this has been addressed by our ResNet architecture. For StyleGAN2, Karras et al. conclude that a ResNet architecture is harmful to G [31] , but this is not true in our case as their ResNet implementation is considerably different from ours: 1) Karras et al. use one 3-3 residual block for each resolution stage, while we have a separate transition layer and two 1-3-1 residual blocks; 2) i.3) and i.4) are violated as they do not have a linear residual block [67] and the transition layer is placed on the skip branch of the residual block rather than the stem; 3) the essential principle of ResNet [16] -identity mapping [17] -is violated as Karras et al. divide the output of the residual block by √ 2 to avoid variance explosion due to the absence of a proper initialization scheme.For Config E, we conduct two experiments that ablate i.1 (increased width with depthwise conv.) and i.2 (an inverted bottleneck). We add GroupedConv and reduce the bottleneck compression ratio to two given the same model size. Each bottleneck is now 1.5× the width of Config A, and the FID drops to 7.51, surpassing the performance of StyleGAN2. By inverting the stem and the bottleneck dimensions to enhance the capacity of GroupedConv, our final model achieves an FID of 7.05, exceeding StyleGAN2. We repeat the earlier experiment in 1000-mode convergence on StackedMNIST (unconditional generation), but this time with our updated architecture and with comparisons to SOTA GANs and likelihoodbased methods (Tab. 3, Fig. 5 ). One advantage brought up of likelihood-based models such as diffusion over GANs is that they achieve mode coverage [7] . We find that most GANs struggle to find all modes. But, PresGAN [8] , DDGAN [94] , and our approach are successful. Further, our method outperforms all other tested GAN models in term of KL divergence.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Justification: Please see supplemental section on the experimental setting.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
fid -ffhq-256 [29] (optimized): 
 We recent diffusion-based methods. On this common dataset experimental setting, many methods (not listed here) use the bCR [101] trick-this has only been shown to improve performance on FFHQ-256 (not even at different resolutions of FFHQ) [101, 98] . We do not use this trick.To compare with EDM [33] directly, we evaluate our model on FFHQ at 64×64 resolution. For this, we remove the two highest resolution stages of our 256×256 model, resulting in a generator that is less than half the number of parameters as EDM. Despite this, our model outperforms EDM on this dataset and needs one function evaluation only (Tab. 5).Model NFE↓ FID↓ BigGAN [ We train Config E model until convergence and with optimized hyperparameters and training schedule on CIFAR-10 (conditional generation) (Tab. 6, Fig. 8 ). Our method outperforms many other GANs by FID even though the model has relatively small capacity. For instance, StyleGAN-XL [69] has 18 M parameters in the generator and 125 M parameters in the discriminator, while our model has a 40 M parameters between the generator and discriminator combined (Fig. 3 ). Compared to diffusion models like LDM or ADM, GAN inference is significantly cheaper as it requires only one network function evaluation compared to the tens or hundreds of network function evaluations for diffusion models without distillation. Many state-of-the-art GANs are derived from Projected GAN [68] , including StyleGAN-XL [69] and the concurrent work of StyleSAN-XL [82] . These methods use a pre-trained ImageNet classifier in the discriminator. Prior work has shown that a pretrained ImageNet discriminator can leak ImageNet features into the model [41] , causing the model to perform better when evaluating on FID since it relies on a pre-trained ImageNet classifier for the loss. But, this does not improve results in perceptual studies [41] We train Config E model until convergence and with optimized hyperparameters and training schedule on ImageNet-32 (conditional generation). We compare against recent GAN models and recent diffusion models in Table 7 . We adjust the number of parameters in the generator of our model to match StyleGAN-XL [69] 's generator (84M parameters). Specifically, we make the model significantly wider to match. Our method achieves comparable FID despite using a 60% smaller discriminator (Tab. 7) and despite not using a pre-trained ImageNet classifier.We evaluate our model on ImageNet-64 to test its scalability. We stack another resolution stage on our ImageNet-32 model, resulting in a generator of 104 M parameters. This model is nearly 3× smaller than diffusion-like models [7, 33, 79, 77] that rely on the ADM backbone, which contains about 300 M parameters. Despite the smaller model size and that our model generates samples in one step, it outperforms larger diffusion models with many NFEs on FID (Tab. 8).We evaluate the recall [40] of our model on each dataset to quantify sample diversity. In general, our model achieves a recall that is similar to or marginally worse than the diffusion model counterpart, yet superior to existing GAN models. For CIFAR-10, the recall of our model peaked at 0.57; as a point of comparison, StyleGAN-XL [69] has a worse recall of 0.47 despite its lower FID. For FFHQ, we obtain a recall of 0.53 at 64×64 and 0.49 at 256×256, whereas StyleGAN2 [31] achieved a recall of 0.43 on FFHQ-256. Our ImageNet-32 model achieved a recall of 0.63; comparable to ADM [7] .Our ImageNet-64 model achieved recall 0.59. While this is slightly worse than ≈0.63 that many diffusion models achieve, it is better than BigGAN-deep [3] which achieved a recall of 0.48. 
discussion and limitations: 
 We have shown that a simplification of GANs is possible for image generation tasks, built upon a more stable RpGAN+R 1 + R 2 objective with mathematically-demonstrated convergence properties that still provides diverse output. This stability is what lets us re-engineer a modern network architecture without the tricks of previous methods, producing the R3GAN model with competitive FID on the common datasets of Stacked-MNIST, FFHQ, CIFAR-10, and ImageNet as an empirical demonstration of the mathematical benefits.The focus of our work is to elucidate the essential components of a minimum GAN for image generation. As such, we prioritize simplicity over functionality-we do not claim to beat the performance of every existing model on every dataset or task; merely to provide a new simple baseline that converges easily. While this makes our model a possible backbone for future GANs, it also means that it is not suitable to apply our model directly to downstream applications such as image editing or controllable generation, as our model lacks dedicated features for easy image inversion or disentangled image synthesis. For instance, we remove style injection functionality from StyleGAN even though this has a clear use. We also omitted common techniques that have been shown in previous literature to improve FID considerably. Examples include some form of adaptive normalization modulated by the latent code [7, 33, 29, 98, 58, 89, 66] , and using multiheaded self attention at lower resolution stages [7, 33, 34] . We aim to explore these techniques in a future study.Further, our work is limited in its evaluation of the scalability of R3GAN models. While they show promising results on 64×64 ImageNet, we are yet to verify the scalability on higher resolution ImageNet data or large-scale text to image generation tasks [12] .Finally, as a method that can improve the quality of generative models, it would be amiss not to mention that generative models-especially of people-can cause direct harm (e.g., through personalized deep fakes) and societal harm through the spread of disinformation (e.g., fake influencers). 
conclusion: 
 This work introduced R3GAN, a new baseline GAN that features increased stability, leverages modern architectures, and does not require ad-hoc tricks that are commonplace in existing GAN models. Central to our approach is a regularized relativistic loss that provably features local convergence and that improves the stability of GAN training. This stable loss enables us to ablate various tricks that were previously necessary in GANs, and incorporate in their place modern deep architectures. The resulting streamlined baseline achieves competitive performance to SOTA models within its parameter size class. We anticipate that our backbone will help to drive future GAN research. 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?Answer: [Yes] Justification: Experimental settings are standard and within the norms of the community.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes]Justification: We mention it briefly in Section 5. The paper describes a basic machine learning methodology, and so does not address a specific application with specific societal impacts. But, GANs do have potential social impact; it is clear that face generation has a significant impact (e.g., deep fakes) and our paper does use a face database for evaluation thanks to it being a community norm.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [NA]Justification: No new assets are released.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA] Justification: No human subjects are used and no crowdsourcing is used.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA] Justification: No human subjects are used and no crowdsourcing is used.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: Following [52] , GAN training can be formulated as a dynamical system where the update operator is given by F h (θ, ψ) = (θ, ψ) + hv(θ, ψ). h is the learning rate and v denotes the gradient vector field:EQUATIONMescheder et al. [51] showed that local convergence near (θ * , ψ * ) can be analyzed by examining the spectrum of the Jacobian J F h at the equilibrium: if the Jacobian has eigenvalues with absolute value bigger than 1, then training does not converge. On the other hand, if all eigenvalues have absolute value smaller than 1, then training will converge to (θ * , ψ * ) at a linear rate. If all eigenvalues have absolute value equal to 1, the convergence behavior is undetermined.Given some calculations [52] , we can show that the eigenvalues of the Jacobian of the update operator λ J F h can be determined by λ J v :EQUATIONThat is, given small enough h [52] , the training dynamics can instead be examined using λ J v , i.e., the eigenvalues of the Jacobian of the gradient vector field. If all λ J v have a negative real part, the training will locally converge to (θ * , ψ * ) at a linear rate. On the other hand, if some λ J v have a positive real part, the training is not convergent. If all λ J v have a zero real part, the convergence behavior is inconclusive.Summary. To obtain DiracRpGAN, we apply Eq. 2 to the DiracGAN [52] problem setting. After simplification, DiracRpGAN and DiracGAN are different only by a constant. They have the same gradient vector field, therefore all proofs are identical to Mescheder et al. [52] .Definition B.1. The DiracRpGAN consists of a (univariate) generator distribution p θ = δ θ and a linear discriminator D ψ (x) = ψ • x. The true data distribution p D is given by a Dirac distribution concentrated at 0.In this setup, the RpGAN training objective is given by:EQUATIONWe can now show analytically that DiracRpGAN does not converge without regularzation.Lemma B.2. The unique equilibrium point of the training objective in Eq. 6 is given by θ = ψ = 0. Moreover, the Jacobian of the gradient vector field at the equilibrium point has the two eigenvalues ±f ′ (0)i which are both on the imaginary axis.The gradient vector field v of Eq. 6 is given by:EQUATIONand the Jacobian of v:EQUATIONEvaluating J v at the equilibrium point θ = ψ = 0 gives us:EQUATIONTherefore, the eigenvalues of J v are λ 1/2 = ±f ′ (0)i, both of which have a real part of 0. Thus, the convergence of DiracRpGAN is inconclusive and further analysis is required.Lemma B.3. The integral curves of the gradient vector field v(θ, ψ) do not converge to the equilibrium point. More specifically, every integral curve (θ(t), ψ(t)) of the gradient vector field v(θ, ψ) satisfies θ(t) 2 + ψ(t) 2 = const for all t ∈ [0, ∞).Let R(θ, ψ) = 1 2 (θ 2 + ψ 2 ), then:EQUATION)We see that the distance between (θ, ψ) and the equilibrium point (0, 0) stays constant. Therefore, training runs in circles and never converges.Next, we investigate the convergence behavior of DiracRpGAN with regularization. For DiracRpGAN, both R 1 and R 2 can be reduced to the following form:EQUATIONLemma B.4. The eigenvalues of the Jacobian of the gradient vector field for the gradient-regularized DiracRpGAN at the equilibrium point are given byEQUATIONIn particular, for γ > 0 all eigenvalues have a negative real part. Hence, gradient descent is locally convergent for small enough learning rates.With regularization, the gradient vector field becomesEQUATIONthe Jacobian of ṽ is then given byEQUATIONevaluating the Jacobian at θ = ψ = 0 yieldsEQUATIONgiven some calculations, we arrive at Eq.12.Summary. The proofs are largely the same as Mescheder et al. [52] . We use the same proving techniques, and only slightly modify the assumptions and proof details to adapt Mescheder et al.'s effort to RpGAN. Like in [52] , our proofs do not rely on unrealistic assumptions such as supp p D = supp p θ .We closely follow [52] but modify the assumptions wherever necessary to tailor the proofs for RpGAN. Like in [52] , we also consider the realizable case where there exists θ such that G θ produces the true data distribution.Assumption I. We have p θ * = p D , and D ψ * = C in some local neighborhood of supp p D , where C is some arbitrary constant.Since RpGAN is defined on critic difference rather than raw logits, we no longer require D ψ * to produce 0 on supp p D , instead any constant C would suffice.Assumption II. We have f ′ (0) ̸ = 0 and f ′′ (0) < 0.This assumption is the same as in [52] . The choice f (t) = -log(1 + e -t ) adopted in the main text satisfies this assumption.As discussed in [52] , there generally is not a single equilibrium point (θ * , ψ * ), but a submanifold of equivalent equilibria corresponding to different parameterizations of the same function. It is therefore necessary to represent the equilibrium as reparameterization manifolds M G and M D . We modify the reparameterization h as follows:EQUATIONto account for the fact that D ψ * is now allowed to have any constant value on supp p D . The reparameterization manifolds are then given by:EQUATIONWe assume the same regularity properties as in [52] for M G and M D near the equilibrium. To state these assumptions, we need:EQUATIONAssumption III. There are ϵ-balls B ϵ (θ * ) and B ϵ (ψ * ) around θ * and ψ * so that M G ∩ B ϵ (θ * ) and M D ∩ B ϵ (ψ * ) define C 1 -manifolds. Moreover, the following holds:(i) if v ∈ R n is not in T ψ * M D , then ∂ 2 v h(ψ * ) ̸ = 0. (ii) if w ∈ R m is not in T θ * M G , then ∂ w g(θ * ) ̸ = 0.These two conditions have exactly the same meanings as in [52] : the first condition indicates the geometry of M D can be locally described by the second derivative of h. The second condition implies that D is strong enough that it can detect any deviation from the equilibrium generator distribution. This is the only assumption we have about the expressiveness of D.We can now show the general convergence result for gradient penalized RpGAN, consider the gradient vector field with either R 1 or R 2 regularization:EQUATIONnote that the convergence result can also be trivially extended to the case where both R 1 and R 2 are applied. We omit the proof for this case as it is redundant once the convergence with either regularization is proven.Theorem. Assume Assumption I, II and III hold for (θ * , ψ * ). For small enough learning rates, gradient descent for ṽ1 and ṽ2 are both convergent to M G × M D in a neighborhood of (θ * , ψ * ). Moreover, the rate of convergence is at least linear.We extend the convergence proof by Mescheder et al. [52] to our setting. We first prove lemmas necessary to our main proof.Lemma C.2.1. Assume J ∈ R (n+m)×(n+m) is of the following form:EQUATIONwhere Q ∈ R m×m is a symmetric positive definite matrix and B ∈ R m×n has full column rank. Then all eigenvalues λ of J satisfy ℜ(λ) < 0.Proof. See Mescheder et al. [52] , Theorem A.7.for all (x, y) ∈ supp p D × supp p D . Recall the definition of h(ψ) from Eq.16. Using the fact that D ψ * = C and ∇ x D ψ * = 0 for x ∈ supp p D , we see that the Hessian of h(ψ) at ψ * is∇ 2 ψ h(ψ * ) = 2Ex∼pD y∼p D [(∇ ψ D ψ * (x) -∇ ψ D ψ * (y))(∇ ψ D ψ * (x) -∇ ψ D ψ * (y)) ⊤ + AA ⊤ ] (35)The second directional derivative ∂ 2 v h(ψ) is thereforeEQUATIONBy Assumption III, this can only hold if v ∈ T ψ * M D .Lemma C.2.7. Assume Assumption III holds. If w ̸ = 0 is not in T θ * M G , then K DG w ̸ = 0.Proof. See Mescheder et al. [52] , Lemma D.6.Proof for the main theorem. Given previous lemmas, by choosing local coordinates θ(α, γ G ) and ψ(β, γ D ) for M G and M D such that θ * = 0, ψ * = 0 as well asEQUATIONEQUATION)our proof is exactly the same as Mescheder et al. [52] , Theorem 4.1.We implement our models on top of the official StyleGAN3 code base. While the loss function and the models are implemented from scratch, we reuse support code from the existing implementation whenever possible. This includes exponential moving average (EMA) of generator weights [28] , non-leaky data augmentation [30] , and metric evaluation [32] .Training schedule. To speed up the convergence early in training, we specify a cosine schedule for the following hyperparameters before they reach their target values:• Learning rate • γ for R 1 and R 2 regularization • Adam β 2 • EMA half-life • Augmentation probabilityWe call this early training stage the burn-in phase. Burn-in length and schedule for each hyperparameter are listed in Table 9 for each experiment. A schedule for the EMA half-life can already be found in Karras et al. [30] , albeit they use a linear schedule. A lower initial Adam β 2 is crucial to the initial large learning rate as it allows the optimizer to adapt to the gradient magnitude change much quicker. We use a large initial γ to account for that early in training: p θ and p D are far apart and a large γ smooths both distributions more aggressively which makes learning easier. Augmentation is not necessary until D starts to overfit later on; thus, we set the initial augmentation probability to 0.Dataset augmentation. We apply horizontal flips and non-leaky augmentation [30] to all datasets where augmentation is enabled. Following [30] , we include pixel blitting, geometric transformations, and color transforms in the augmentation pipeline. We additionally include cutout augmentation which works particularly well with our model, although it does not seem to have much effect on StyleGAN2. We also find it beneficial to apply color transforms less often and thus set their probability multiplier to 0.5 while retaining the multiplier 1 for other types of augmentations. As previously mentioned, we apply a fixed cosine schedule to the augmentation probability rather than adjusting it adaptively as in [30] . We did not observe any performance degradation with this simplification.Network capacity. We keep the capacity distribution for each resolution the same as in [30, 32] . We place two residual blocks per resolution which makes our model roughly 3× as deep, 1.5-3× as wide as StyleGAN2 while maintaining the same model size on CIFAR-10 and FFHQ. For the ImageNet model, we double the number of channels which results in roughly 4× as many parameters as the default StyleGAN2 configuration.We apply mixed precision training as in [30, 32] where all parameters are stored in FP32, but cast to lower precision along with the activation maps for the 4 highest resolutions. We notice that using FP16 as the low precision format cripples the training of our model. However, we see no problem when using BFloat16 instead.Class conditioning. For class conditional models, we follow the same conditioning scheme as in [30] . For G, the conditional latent code z ′ is the concatenation of z and the embedding of the class label c, specifically z ′ = concat(z, embed(c)). For D, we use a projection discriminator [54] which evaluates the dot product of the class embedding and the feature vector D ′ (x) produced by the last layer of D, concretely D(x) = embed(c) • D ′ (x) ⊤ . We do not employ any normalization-based conditioning such as AdaIN [29] , AdaGN [7, 33] , AdaBN [3] or AdaLN [58] for simplicity, even though they improve FID considerably.Stacked MNIST. We base this model off of the CIFAR-10 model but without class conditioning. We disable all data augmentation and shorten the burn-in phase considerably. We use a constant learning rate and did not observe any benefit of using a lower learning rate later in the training. -2-2-2 2-2-2-2 2-2-2-2-2-2-2 2-2-2-2-2 2-2-2-2 2-2-2-2-- ✓ ✓ ✓ ✓ ✓ Augment probability - 0 → 0.55 0 → 0.3 0 → 0.3 0 → 0.5 0 → 0.4Table 9 : Hyperparameters for each experiment. The decay factor β of EMA can be obtained using the formula β = 0.5EMA half-life , e.g. for CIFAR-10, EMA β = 0.5 512 5×10 6 ≈ 0.9999.Following the convention of Brock et al. [3] , we report alternative design choices that did not make to our final model. Either because they failed to produce any quantitative improvement or because they would considerably complicate our minimalist design which might be better suited for future study.• We tried to apply GELU [18] , Swish [61] , and SMU [2] to G and D and found that doing so deteriorates FID considerably. We did not try on G xor D. We posit two independent factors:-ConvNeXt in general does not benefit much from GELU (and possibly similar activations). Table 10 and Table 11 in [48] : replacing ReLU with GELU gives little performance gain to ConvNeXt-T and virtually no performance gain to ConvNeXt-B. -In the context of GANs, GELU and Swish have the same problem as ReLU: that they have little gradient in the negative interval. Since G is updated from the gradient of D, having these activation functions in D could sparsify the gradient of D and as a result G will not receive as much useful information from D compared to using leaky ReLU. This does not explain the strange case of SMU [2] : SMU is a smooth approximation of leaky ReLU and does not have the sparse gradient problem. It is unclear why it also underperformed and future work awaits.• We tried adding group normalization [92] to G and D and it did not improve FID or training stability. We do not claim that all forms of normalizations are harmful. Our claim in principle c) only extends to normalization layers that explicitly standardizes the mean and standard deviation of the activation maps. This has been verified by prior studies [31, 34, 65] . The harm of normalization layers extends to the adjacent field of image restoration [43, 88] . To the best of our knowledge, EDM2 [34] is currently the strongest diffusion UNet and it does not use normalization layers. However, it does apply normalization to the trainable weights and this improves performance considerably. We expect that applying the normalization techniques in EDM2 would improve our model's performance. • We tried removing the activation function after the 3×3 grouped convolution in each residual block as modern architectures [48, 97] typically do not apply non-linearity after depthwise convolution. This worsened FID performance. • We tried Pixel-Shuffle/Unshuffle [71] for changing the resolution of the activation maps, and found that without low-pass filtering, this led to high frequency artifacts similar to checkerboard artifacts even though Pixel-Shuffle does not have the uneven overlap problem that transposed convolution does. Note that bilinear resampling is equivalent to applying channel duplication/averaging with Pixel-Shuffle/Unshuffle in conjunction with a [1, 2, 1] low-pass kernel. It might be interesting in future studies to explore inplace resampling filters that apply a low-pass filtered Pixel-Shuffle/Unshuffle operation on top of a learned function that changes the number of channels. • We tried scaling up our model size. We found that allocating more model capacity to lower resolution stages generally did not improve FID, but contributed to more rapid overfitting. Increasing model capacity at higher resolution stages always improves FID in our experiments, however scaling up higher resolution stages is very computationally expensive. Capacity distribution for each resolution stage of the model might be an important topic to explore in future studies. • For model simplicity, we did not conduct any experiment with a transformer architecture or attention mechanism in general. We are interested to see whether adding attention blocks to a convolutional network (similar to BigGAN [3] and diffusion UNet [20, 33, 34] ) or using a pure transformer architecture (similar to DiT [58] ) will result in stronger performance.Given the impressive results of EDM2 [34] (which uses UNet), it seems the argument has not yet settled for generative modeling. • We experimented with Adam β 2 = 0.999 following common practice in supervised learning and diffusion models, and found that doing so led to stability issues on our ImageNet models. We expect that introducing proper normalization to our model will resolve this problem. • We tried mixed precision training with IEEE FP16 as this is the low precision format used in StyleGAN2-ADA [30] , StyleGAN3 [32] , and EDM2 [34] . This crippled the training of our model and switching to BFloat16 fixed the problem. We expect that introducing proper normalization to our model will allow us to use IEEE FP16 which offers more precision than BFloat16. • We tried lazy regularization [31] in our early experiments where R 1 and R 2 were applied once every 8 minibatches. This led to slightly worse FID performance on real world datasets like FFHQ and CIFAR-10. However, it resulted in complete convergence failure on Stacked MNIST and several two dimensional toy datasets (line, circle, 25 Gaussians, etc.), indicating potential concerns regarding the mathematical legitimacy of this trick. Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.Justification: There is no new data and much larger models produce higher fidelity images. The cost of training these large GANs is not prohibitive and is often done by hobbyists.As such, it is doubtful that these models will unlock any new capabilities for mis-use or dual-use.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes]Justification: All datasets are cited.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators.While mode collapse and mode dropping are technically distinct issues, they are used interchangeably in this context to describe the common problem where supp(p θ ) does not comprehensively cover supp(pD). Mode collapse refers to the generator producing a limited diversity of samples (i.e., one image for the entire distribution), whereas mode dropping involves the generator failing to represent certain modes of the data distribution (ignoring entire subsets of the training distribution).Varying γ from 0.1 to 100 does not stabilize training.These numbers are from Karras et al.[28], Table"857" corresponds to a low-capacity version of a progressive GAN and "881" adds the minibatch standard deviation trick. Further comparisons via loss curves are difficult since progressive GAN is a substantially different model than the small ResNet we use for this experiment. 


Decision: The reviews are overall positive and recognize the solid theoretical foundation, empirical performance, and contribution to modernizing GAN architectures. While there are some concerns about novelty, scalability, and the transparency of the experimental setup, the reviewers believe that it's a valuable contribution to the community given the sole recent focus on diffusion models. I urge the authors to incorporate the feedback in the manuscript.

Reviews: review: paper_summary: paper_summary: The authors posit that the main reason GAN research has been slow in recent years is due to the most foundational StyleGAN2 not having undergone major architectural changes, essentially due to a lack of convergence guarantee in GAN objectives and being prone to mode collapse. This has limited the scope of architectural changes and further GAN research.
Two identifiable goals of the paper are to mathematically derive a well-behaved regularized loss which guarantees GAN convergence and diversity, provide empirical evidence for the same, and now being free to make unconstrained design choices attempt to develop a GAN baseline which inherits features from modern vision architectures. 

The authors' methodology is inspired by two key works: (i) RpGAN [20], which addressed mode dropping in GANs with a better loss landscape, theoretically confirmed by Sun et al. [65] to contain only global minima; and (ii) Mescheder et al. [43], which provided evidence for local convergence with zero-centered gradient penalties in GAN objectives. By extending zero-centered gradient penalties to the RpGAN formulation, the authors address both mode dropping and unstable training. They offer theoretical proof for local convergence and present experimental evidence supporting global convergence, though no theoretical proof for the latter is provided.
Design changes in Config D are motivated by ResNet [14] and ConvNeXt [41], selectively applying aspects consistent with the authors' observations. Following findings in [31, 29], they eliminate normalization in StyleGAN in B and introduce fix-up initialization [77] in D to prevent variance explosion. Changes in E are inspired by ResNeXt [74], incorporating group convolutions to increase bottleneck width and followed by bottleneck inversion for efficient use. This simple baseline is empirically found to outperform previous GAN methods and some recent diffusion based models on several unconditional and conditional generation tasks.
strength_weakness: strength_weakness: 1. While experimental results are well presented, there seems to be a lack of information about the training setup, hyperparameters used, number of inference steps, etc pertaining to the diffusion based models in Tables 4, 5 and 6. It would be useful to include this information in the supplementary or the tables itself for a transparent comparison.
questions: questions: 1. To clarify, on line 132 on page 4, “857” corresponds to StyleGAN backbone trained on WGAN-GP loss and “881” corresponds to the mini-batch standard deviation trick added to the same? Would be better to clarify the exact configuration and include the two cases in table 1 and figure 1.
limitations: limitations: -
paper_summary: paper_summary: This paper introduces R3GAN as a GAN baseline that simplifies and modernizes the architecture by replacing ad-hoc tricks with modern designs. R3GAN utilized a regularized relativistic GAN loss coupled with zero-centered gradient penalties on both real and generated data, to addresses mode dropping and non-convergence, providing local convergence guarantees. The proposed loss allows for the elimination of unnecessary tricks and the adoption of advanced architectures. R3GAN demonstrates improved stability and competitive performance against state-of-the-art models on multiple datasets such as StackedMNIST, CIFAR, ImageNet-32, and FFHQ-256.
strength_weakness: strength_weakness: - The novelty of the method is somewhat limited, as both relativistic pairing GAN (RpGAN) and zero-centered gradient penalties (0-GPs) are previously proposed and validated approaches in the field of GANs.
- Small-scale experiments to validate the significant improvements in stability and diversity provided by the used loss are insufficient; more complex datasets should be used to verify its effectiveness. As shown in Table 2, the improvement from config C compared to config B is not significant. Generally, stability can allow the model to converge better, thereby enhancing the final results, and diversity can be measured by metrics such as recall or coverage.
questions: questions: How scalable is the proposed model? It is suggested to validate the effectiveness of the method on higher-resolution datasets and more complex tasks. This would align with the motivation to serve as a modern baseline, especially in the current context where both model parameter size and training data scale are increasing.
limitations: limitations: The authors have discussed the limitations and potential negative societal impact of their work.
paper_summary: paper_summary: A lot of GANs papers have shrunk in quality since diffusion arose as a strong method. The authors goes against the grain by focusing on modernizing GAN baseline in a principled manner and in doing so they obtain stable, convergent, diversity, and high-quality images comparable to diffusion models. The authors highlight the fact that a lot of the GAN literature is riled with empirical tricks with no theory to back them. They explain that StyleGAN without tricks is very close to the original DCGAN of 2015. Meanwhile a lot more progress as been done in diffusion models leveraging many architectural improves.

They instead start with a Relativistic pairing GAN baseline which, unlike regular GANs, have been shown to have no local-minima basins where the loss must worsen before reaching the true global minimum. However, they show that, just like regular GANs, RpGANs are not guaranteed to converge to the global minimum. They show that, just like regular GANs, gradient penalty on real data (R1) or fake data (R2) guarantee convergence. They suggests using both together. They then show empirically that on StackMNIST, GANs with R1 alone fails, but having both R1, R2, and RpGAN lead to perfect mode coverage and strong stability. As further justification, they also leverage the fact that from a constructive approach, when deriving a maximum-margin GAN, one obtains a gradient penalty with both R1, R2.

Then they use a very careful ablation approach by starting from StyleGAN2, removing all the complicated empirical tricks but keeping R1 penalty, which reduce performance a bit, then moving to RpGAN, adding R2 penalty, then modernizing the architecture, reaching slightly better performance at similar number of parameters, but without tricks. The describe extensively and carefully the effect of different architecture choices on performance.

Experiments on CIFAR-10, Imagenet-32, and FFHQ-256 are shown were the method proposed shows extremely low FID, better than diffusion models and do so while keeping the parameter count low. Results are very good, but work is still needed to scale these methods to text-to-image settings. However, the current results are very promising and the method has been made very clean and carefully designed (similar to ConvNext).
strength_weakness: strength_weakness: Great results on image datasets, but it still need to be scaled to large-scale text-to-image setting.
Many GPUs are used, even for CIFAR-10; this is also in line with diffusion models, but it seems a lot, are all these GPUs needed for CIFAR-10?
Although cleaner, the new baseline still contains a lot of complicated hyperparameter tuning, which change from one dataset to another (in Table 7). It would be great to clean this set of hyperparameters, I suggested some potential ideas in the "questions".
questions: questions: - Any intuition to why worse performance with GELU and swish activations since these are very successful with transformers? Is this true for both G and D?
- Have you consider a transformer architecture as alternative to convnext-style architecture?
- You mention that normalization can harm GANs, but groupnorm is used in diffusion models and layernorm is used in language models, could this be useful for your approach? Has it been tried before? I saw the references you mentioned to say that normalization is bad for GANs, but I haven't looked at them individually.
- You mention that PresGAN and DDGANs obtain good performances, do you think that some of their ideas could be useful for improve the GAN baseline?
- What is EMA half-life? Is the EMA .999?
- Why do you schedule Adam B2? This seems very atypical.
- Assume $\alpha$ is the lr. Rather than schedule the R1/R2 $\gamma$ manually, why not make it $\alpha\gamma$ so that by decreasing the lr $\alpha$ over time, $\gamma$ is also reduces automatically (thus reducing the hyperparameter tuning). The same could be done with Beta2, by doing $1-((\alpha/alpha_{init})*(1-beta2))$, so moving the 1-.9 = 0.1 to 0.01 (1-.99) when decreasing the lr to 1/10 of its value over training time. I just feel like choices like this would greatly reduce the arbitrary hyperparameter tuning used here which is quite complex in Table 7.
limitations: limitations: The authors have adequately addressed the limitations.

Comments: title: response
comment: Thanks for your response. After going through the response and other response to other reviewers, I have decided to keep my current ratings.
I appreciate the author’s response. Considering that the authors reply to my questions and addressed some of my concerns, I decided to increase my score to Borderline Accept. However, I still believe that conducting experiments on higher resolution datasets (such as ImageNet-128, ImageNet-256, ImageNet-512) would significantly enhance the contribution of this paper to the GAN research field.
Thank you for addressing my questions and concerns. Having some of these short discussions in the appendix would indeed be useful to the reader. The extra imagenet-64 experiment will also help.

I will increase my score by 1. This method is particularly good and helps bring GANs extremely close to diffusion performance which is quite important for the field of GANs to regain usefulness. The design choices are quite interesting. This reminds me a bit of the very useful experiments detailed discussions in the BigGAN paper which really helped pave the way to better GANs.