user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title:  The relativistic discriminator: a key element missing from standard GAN

Paper content:
In standard generative adversarial network (SGAN), the discriminator estimates the probability that the input data is real.
The generator is trained to increase the probability that fake data is real.
We argue that it should also simultaneously decrease the probability that real data is real because
1) this would account for a priori knowledge that half of the data in the mini-batch is fake,
2) this would be observed with divergence minimization, and
3) in optimal settings, SGAN would be equivalent to integral probability metric (IPM) GANs.
We show that this property can be induced by using a relativistic discriminator which estimate the probability that the given real data is more realistic than a randomly sampled fake data.
We also present a variant in which the discriminator estimate the probability that the given real data is more realistic than fake data, on average.
We generalize both approaches to non-standard GAN loss functions and we refer to them respectively as Relativistic GANs (RGANs) and Relativistic average GANs (RaGANs).
We show that IPM-based GANs are a subset of RGANs which use the identity function.
Empirically, we observe that
1) RGANs and RaGANs are significantly more stable and generate higher quality data samples than their non-relativistic counterparts,
2) Standard RaGAN with gradient penalty generate data of better quality than WGAN-GP while only requiring a single discriminator update per generator update (reducing the time taken for reaching the state-of-the-art by 400%), and
3) RaGANs are able to generate plausible high resolutions images (256x256) from a very small sample (N=2011), while GAN and LSGAN cannot; these images are of significantly better quality than the ones generated by WGAN-GP and SGAN with spectral normalization.
The code is freely available on https://github.com/AlexiaJM/RelativisticGAN.
Generative adversarial networks (GANs) BID7 form a broad class of generative models in which a game is played between two competing neural networks, the discriminator D and the generator G. D is trained to discriminate real from fake data, while G is trained to generate fake data that D will mistakenly recognize as real.
In the original GAN by BID4 , which we refer to as Standard GAN (SGAN), D is a classifier, thus it is predicting the probability that the input data is real.
When D is optimal, the loss function of SGAN is approximately equal to the Jensen-Shannon divergence (JSD) BID4 .
SGAN has two variants for the generator loss functions: saturating and non-saturating.
In practice, the former has been found to be very unstable, while the latter has been found to more stable BID4 .
Under certain conditions, proved that, if real and fake data are perfectly classified, the saturating loss has zero gradient and the non-saturating loss has non-zero, but volatile gradient.
In practice, this means that the discriminator in SGAN often cannot be trained to optimality or with a too high learning rate; otherwise, gradients may vanish and, if so, training will stop.
This problem is generally more noticeable in high-dimensional setting (e.g., high resolution images and discriminator architectures with high expressive power) given that there are enough degrees of freedom available to perfectly classify the training set.To improve on SGAN, many GAN variants have been suggested using different loss functions and discriminators that are not classifiers (e.g., LSGAN BID14 , WGAN ).
Although these approaches have partially succeeded in improving stability and data quality, the large-scale study by BID13 suggests that these approaches do not consistently improve on SGAN.
Additionally, some of the most successful approaches, such as WGAN-GP BID5 , are much more computationally demanding than SGAN.Many of the recent successful GANs variants have been based on Integral probability metrics (IPMs) BID18 ) (e.g., WGAN , WGAN-GP, Fisher GAN , Sobolev GAN ).
In IPM-based GANs, the discriminator is real-valued and constrained to a specific class of function which regularize the discriminator.
See for a review of the different IPMs.These IPM constraints have been shown to be beneficial even in non-IPM based GANs.
Spectral normalization BID15 improves the stability of various GANs and it consists in making the discriminator Lipschitz-1, which is the constraint of WGAN.
Similarly, the gradient penalty of WGAN-GP also provides improve the stability of SGAN BID3 .
Although this shows that certain IPM constraints improve the stability of GANs, it does not explain why IPM-based GANs generally provide increased stability over other metrics/divergences in GANs (e.g., JSD for SGAN, f -divergences for f -GANs BID19 ).Note
that although powerful, IPM-based GANs tend to more computationally demanding than other GANs. Certain
IPM-based GANs use a gradient penalty (e.g. WGAN-GP, Sobolev GAN) which is very computationally costly and most IPM-based GANs need more than one discriminator update per generator update (WGAN-GP requires at least 5 BID5 ). Assuming
equal training time for D and G, every additional discriminator update increase training time by a significant 50%.In this paper
, we argue that non-IPM-based GANs are missing a key ingredient, a relativistic discriminator, which IPM-based GANs already possess. We show that
a relativistic discriminator is necessary to make GANs analogous to divergence minimization and produce sensible predictions based on the a priori knowledge that half of the samples in the mini-batch are fake. We provide empirical
evidence showing that GANs with a relativistic discriminator are more stable and produce data of higher quality.
In this paper, we proposed the relativistic discriminator as a way to fix and improve on standard GAN.
We further generalized this approach to any GAN loss and introduced a generally more stable variant called RaD.
Our results suggest that relativism significantly improve data quality and stability of GANs at no computational cost.
Furthermore, using a relativistic discriminator with other tools of the trade (spectral norm, gradient penalty, etc.) may lead to better state-of-the-art.Future research is needed to fully understand the mathematical implications of adding relativism to GANs.
Furthermore, our experiments were limited to certain loss functions using only one seed, due to computational constraints.
More experiments are required to determine which relativistic GAN loss function is best over a wide-range of datasets and hyper-parameters.
We greatly encourage researchers and machine learning enthusiasts with greater computing power to experiment further with our approach.
Table 3 : An illustrative example of the discriminator's output in standard GAN as traditionally defined (P (x r is real) = sigmoid(C(x r ))) versus the Relativistic average Discriminator (RaD) (P (x r is real|C(x f )) = sigmoid(C(x r ) − C(x f ))).
Breads represent real images, while dogs represent fake images.
Scenario Absolute probability Relative probability (Standard GAN) (Relativistic average Standard GAN)Real image looks real and fake images look fake DISPLAYFORM0 Real image looks real but fake images look similarly real on average DISPLAYFORM1 Real image looks fake but fake images look more fake on average DISPLAYFORM2 P (x r is bread|C(x f )) = .88

assistant: text: The paper proposes a “relativistic discriminator”, whic helps in some settings, although a bit sensitive to hyperparameters, architectures, and datasets.