Title: NOISE-TEMPERED GENERATIVE ADVERSARIAL NET-WORKS

Abstract: We present a novel method to stabilize the training of generative adversarial networks. The training stability is often undermined by the limited and lowdimensional support of the probability density function of the data samples. To address this problem we propose to simultaneously train the generative adversarial networks against different additive noise models, including the noise-free case. The benefits of this approach are that: 1) The case with noise added to both real and generated samples extends the support of the probability density function of the data, while not compromising the exact matching of the original data distribution, and 2) The noise-free case allows the exact matching of the original data distribution. We demonstrate our approach with both fixed additive noise and with learned noise models. We show that our approach results in a stable and well-behaved training of even the original minimax GAN formulation. Moreover, our technique can be incorporated in most modern GAN formulations and leads to a consistent improvement on several common datasets.

1. INTRODUCTION:

Since the seminal work of Goodfellow et al. (2014) , generative adversarial networks (GAN) have been widely used and analyzed due to the quality of the samples that they produce, in particular when applied to the space of natural images. Unfortunately, GANs still prove difficult to train. In fact, a vanilla implementation does not converge to a high-quality sample generator and heuristics used to improve the generator often exhibit an unstable behavior. This has led to a substantial work to better understand GANs (see, for instance, Sønderby et al. (2017) ; Roth et al. (2017) ; Arjovsky & Bottou (2017) ). In particular, Arjovsky & Bottou (2017) point out how the unstable training of GANs is due to the (limited and low-dimensional) support of the data and model distributions. In the original GAN formulation, the generator is trained against a discriminator in a minimax optimization problem. The discriminator learns to distinguish real from fake samples, while the generator learns to generate fake samples that can fool the discriminator. When the support of the data and model distributions is disjoint or lies on a low-dimensional manifold, the generator stops improving as soon as the discriminator achieves perfect classification, because this prevents the propagation of useful information to the generator through gradient descent. To address this limitation, some research focused on novel formulations, as, for example, in Gulrajani et al. (2017) ; Mao et al. (2017) and on techniques to improve the training, such as, for example Miyato et al. (2018) ; Arjovsky et al. (2017) . However, a recent large-scale evaluation by Lucic et al. (2017) shows that obtaining improvements over the seminal work of Goodfellow et al. (2014) is extremely challenging.

The recent work by Arjovsky & Bottou (2017) proposes to extend the support of the distributions by adding noise to both generated and real images before they are fed as input to the discriminator. This procedure results in a smoothing of both data and model probability distributions, which indeed increases their support extent and dimensionality. In fact, samples x = x + , obtained by adding noise ∼ p to the data samples x ∼ p d , are also instances of the probability density function p d, = p * p d , where * denotes the convolution operator. The support of p d, is the Minkowski sum of the supports of p and p d and thus larger than the support of p d . Similarly, adding noise to the samples from the generator probability density p g leads to the smoothed probability density p g, = p * p g . Adding noise is a quite well-known technique that has been used in maximum likelihood methods, but is considered undesirable as it yields approximate generative models that produce low-quality blurry samples. Indeed, most formulations with additive noise boil down to finding the model distribution p g that best solves p d, = p g, . As shown in Figure 1 (a), this results in a low quality estimate p g because p d * p has lost the high frequency content of p d . Therefore, Arjovsky and Bottou propose a form of noise annealing, where the noise variance is initially high and is then reduced gradually during the iterations so that the original distributions, rather than the smooth ones, are eventually matched. This results in an improved training, but as the noise variance approaches zero, the optimization problem converges to the original formulation and the algorithm may be subject to the usual unstable behavior.

In this work, we build on the findings of Arjovsky et al. (2017) to design a novel noise-based procedure that yields stable and accurate training. Our method is based on the following two observations: 1) If we add noise to the real and fake samples in a GAN training, we achieve p d, = p g, , i.e., p * (p d -p g ) = 0, which is a necessary (but not sufficient) condition for p d = p g ; 2) p d, = p g, is a sufficient condition for p d = p g , if it holds for any p . Based on the first observation, we propose not to anneal noise, but rather sustain it, as it does not prevent the desired solution. Based on the second observation, we propose to vary the noise probability density. In fact, as shown in Figure 1 (b) if we train a GAN on data with additive noise from two Gaussian noise instances p 1 = N (0, σ 1 I d ) and p 2 = N (0, σ 2 I d ), the matching of both p d, 1 = p g, 1 and p d, 2 = p g, 2 yields a much more accurate approximate probability density p g . Both noise instances help the estimation of p g : the low-noise instance preserves high-frequency content of p d , while the high-noise instance extends the support of p d , which then helps the training of the generator. Therefore, we suggest to obtain a generative model G by solving a multi-objective optimization

min G max D E ∼p E x∼p d [log D(x + )] + E z∼N (0,I d ) [log(1 -D(G(z) + ))] , ∀p ∈ S (1)

where we introduced a space S of probability density functions, and D denotes the discriminator.

In fact, if we solve the innermost optimization problem, then we obtain the optimal discriminator D(x) = p d, (x) /(p d, (x)+pg, (x)), where we have defined p g (x) δ(x -G(z))p(z)dz, with δ the Dirac delta and p(z) = N (z; 0, I d ). If we substitute this in the problem above we have

EQUATION

)

where JSD is the Jensen-Shannon divergence. Under suitable assumptions on the space S and the dimensionality of z, the optimal solution of equation 2 is unique and p g = p d regardless of the support of p d . Thus, our formulation enjoys the following properties

• It defines a fitting of probability densities that is not affected by their support

• It promotes, rather than prevents, the exact matching of the data probability density function

• It can be easily applied to other GAN formulations.

In the next sections we introduce our analysis more in detail and then devise a computationally feasible approximation of the problem formulation equation 1. Our method is evaluated quantitatively on CIFAR-10 (Krizhevsky ( 2009)), STL-10 (Coates et al. (2011) ), and CelebA (Liu et al. (2015) ), and qualitatively on ImageNet (Russakovsky et al. (2015) ) and LSUN bedrooms (Yu et al. (2015) ).

2. NOISE-TEMPERED ADVERSARIAL TRAINING:



2.1. MATCHING PROBABILITY DENSITY FUNCTIONS THROUGH NOISE:

We are interested in finding a formulation that yields as optimal generator G a sampler of the data probability density function (pdf) p d . The main difficulty in dealing with p d is that its support often does not extend to the whole space where the data lives (see, for instance, Arjovsky et al. (2017) ). This allows adversarial training to converge to degenerate solutions, where the model pdf p g only partially overlaps with p d (a scenario called mode collapse). It has been noticed that introducing additive noise on both real and fake data samples during training helps reduce this issue. In fact, additive noise ∼ p corresponds to blurring the original pdfs to p d * p and p g * p , an operation that is known to increase the support of both original pdfs and thus their likelihood to overlap.

However, even if the optimization of equation 2 yielded a generator G such that p g * p = p d * p , there would be no guarantee that p g = p d . As a consequence, prior work such as Arjovsky et al. (2017) starts the training of G with a large additive noise and then decreases it with iteration time.

Unfortunately, as the noise variance is reduced, the model converges to the original formulation and may exhibit an unstable behavior. In this work, we notice that p g * p = p d * p includes p g = p d as a solution for any additive noise ∼ p . This means that the additive noise does not compromise our ability to still match p g to the original p d . Thus, rather than reducing the noise variance, we use the principle that matching convolutions of probability density distributions for all possible additive noise distributions is equivalent to matching directly the probability density distributions without additive noise. If we consider the range of image intensities Ω to be finite, then the support of the pdf p d is bounded and contained in Ω. Then, we can prove the following:1 

Lemma 1 If for any ω ∈ Ω and any bounded and continuous function p

EQUATION

then we have p d (ω) = p g (ω) for all ω ∈ Ω.

Now recall the problem formulation in equation 2, which looks for the pdf p g that minimizes the expectation of JSD(p d, , p g, ) over p ∈ S. The minimum value of the Jensen-Shannon divergence is 0 and it is achieved when p d, (ω) = p g, (ω) for all ω ∈ Ω. The global minimum is then achieved when JSD(p d, , p g, ) = 0 for all p . Thus, by exploiting the above lemma, the optimal solution of equation 2 is still p d (ω) = p g (ω) for all ω ∈ Ω, as in the original GAN formulation.

2.2. FORMULATION:

There are two practical issues with the proposed formulation equation 1. One is that the multiobjective optimization requires, in principle, testing infinite pdfs. Another is that the use of constrained models (such as neural networks) for both the discriminator and the generator may prevent the optimal solution from reaching the global minimum. If we scalarize the multi-objective with some fixed weights then the optimization will only find a tradeoff across all noise distributions p . This means that the case with no or little noise may have a small relevance compared to all other cases and this could lower the accuracy of the estimated p g . Moreover, the case with no noise should be given the highest priority as it is the only one that ensures the exact matching between the data and model distributions. Our proposed solution is to define S with only two terms, the Dirac delta distribution (no noise) and a pdf with a large variance, and to scalarize the multi-objective with a random variable µ ∼ U(0, 1), where U(0, 1) is the uniform distribution between 0 and 1, which we sample at every iteration during training. This is then equivalent to

min G max D E ∼µδ+(1-µ)p E x∼p d [log D(x + )] + E z∼N (0,I d ) [log(1 -D(G(z) + ))] . (4)

We then consider two configurations: 

L r D (φ, ω) = m i=1 µ ln(D(x i )) + (1 -µ) ln(D(x i + i )); L f D (φ, ω) = m i=1 µ ln(1 -D(x i )) + (1 -µ) ln(1 -D(x i + i )); L (ω) = m i=1 | i | 2 ; φ ← φ + α∇ φ (L r D (φ, ω) + L f D (φ, ω)); ω ← ω -α∇ ω L r D (φ, ω) + L f D (φ, ω) + λL (ω) ; end Sample fake examples {x 1 , . . . , xm } ∼ p g ; Sample noise { 1 , . . . , m } ∼ p and µ ∼ U(0, 1) ; L f G (θ) = m i=1 µ ln(D(x i )) + (1 -µ) ln(D(x i + i )); θ ← θ + α∇ θ L f G (θ); end 1.

Gaussian noise with a fixed/learned standard deviation σ: S = {δ( ), N ( ; 0, σI d )}; 2. A learned noise generator: S = δ( ), δ( -N (w))p(w)dw , with N a noise generator network (to be jointly trained) and p(w) = N (w; 0, I d ).

The proposed approximations result in the following problem formulations Gaussian noise

min G min σ max D E ∼µδ+(1-µ)N (0,σ) E x log D(x + ) + E z log(1 -D(G(z) + )) + λσ 2 (5)

Noise generator

EQUATION

)

In both formulations we may learn some optimal parameters of the noise distribution. We do so by minimizing the cost function after the maximization with respect to the discriminator. The minimization would encourage an infinite noise since this would make p d, (ω) more similar to p g, (ω) regardless of p d and p g . Such term however would not be very useful to the training. Therefore, to limit the noise magnitude we introduce as a regularization term the noise variance σ 2 or the Euclidean norm of the noise N output image, and multiply it by a positive scalar λ, which we tune.

2.3. IMPLEMENTATION:

Implementing our algorithm only requires a few minor modifications of the standard GAN framework. We perform the update for the noise-generator and the discriminator in the same iteration. Mini-batches for the discriminator are formed by collecting all the fake and real samples in two separate batches, i.e., {x 1 , . . . , x m , x 1 + 1 , . Application to Different GAN Models. We investigate the possibility of applying our proposed algorithm to several standard GAN models. The network architectures are the same as proposed in the original works with only the necessary adjustments to the given image-resolutions of the datasets (i.e., truncation of the network architectures). Note that for the GAN with minimax loss (MMGAN) and WGAN-GP we use the architecture of DCGAN. Hyper-parameters are kept at their default values for each model. The models are evaluated on two common GAN benchmarks: CIFAR-10 (Krizhevsky ( 2009)) and CelebA (Liu et al. (2015) ). The image resolution is 32 × 32 for CIFAR-10 and 64 × 64 for CelebA. All models are trained for 100K generator iterations. For the alternative objective function of LSGAN we set the loss of the noise generator to be the negative of the discriminator loss, as is the case in our standard model. The results are shown in Table 3 . We can observe that applying our method improves performance in most cases and even enables the training with the original saturation-prone minimax GAN objective, which is very unstable otherwise. We show random CelebA reconstructions from models trained with and without NT in Figure 2 .

Robustness to Hyperparameters. We test the robustness of NTGANs with respect to various hyperparamters by training on CIFAR-10 with the settings listed in Table 4 . The network is the same as specified in Table 1 . The noise penalty term is set to λ = 0.1. We compare to a model without stabilization (Standard), a model with the gradient penalty regularization proposed by Roth et al. (2017) (GAN+GP) and a model with spectral normalization (SNGAN). To the best of our knowledge, these methods are the current state-of-the-art in terms of GAN stabilization. Figure 3 shows that our NTGAN is stable and accurate across all settings.

Qualitative Results. We trained NTGANs on 128 × 128 images from two large-scale datasets: Ima-geNet (Russakovsky et al. (2015) ) and LSUN bedrooms (Yu et al. (2015) ). The network architecture is similar to the one in Table 1 with one additional layer in both networks. We trained the models for 100K iterations on LSUN and 300K iterations on ImageNet. Random samples of the models are shown in Figure 4 . 

4. RELATED WORK:

The inherent instability of GAN training was first addressed through a set of techniques and heuristics (Salimans et al. (2016) ) and careful architectural design choices and hyper-parameter tuning (Radford et al. (2015) ). Salimans et al. (2016) for example propose the use of one-sided label smoothing and the injection of Gaussian noise into the layers of the discriminator. A theoretical analysis of the unstable training and the vanishing gradients phenomena was introduced by Arjovsky & Bottou (2017). They argue that the main source of instability stems from the fact that the real and the generated distributions have disjoint supports or lie on low-dimensional manifolds. In the case of an optimal discriminator this will result in zero gradients that then stop the training of the generator. More importantly, they also provide a way to avoid such difficulties by introducing noise and considering "softer" metrics such as the Wasserstein distance. Sønderby et al. ( 2017) made similar observations and also proposed the use of "instance noise" as a way to overcome these issues. Arjovsky et al. (2017) build on the work of Arjovsky & Bottou (2017) and introduce the Wasserstein GAN (WGAN). The WGAN optimizes an integral probability metric that is the dual to the Wasserstein distance. This formulation requires the discriminator to be Lipschitz-continuous, which is realized through weight-clipping. Gulrajani et al. (2017) present a better way to enforce the Lipschitz constraint via a gradient penalty over interpolations between real and generated data (WGAN-GP). Roth et al. (2017) introduced a stabilizing regularizer based on a gradient norm penalty similar to that by Gulrajani et al. (2017) . Their formulation however is in terms of f-divergences and is derived via an analytic approximation of adversarial training with additive Gaussian noise on the datapoints. Another recent GAN regularization technique that bounds the Lipschitz constant of the discriminator is the spectral normalization introduced by Miyato et al. (2018) . This method demonstrated state-of-the-art in terms of robustness in adversarial training. Several alternative loss functions and GAN models have been proposed over the years, claiming superior stability and sample quality over the original GAN (e.g., Mao et al. (2017) 4 and performed on CIFAR-10. We compare the standard GAN (1st column), a GAN with gradient penalty (2nd column), a GAN with spectral normalization (3rd column) and a GAN with our proposed method (4th column).

Results are reported in Fréchet Inception Distance FID (top) and Inception Score IS (bottom). 

5. CONCLUSION:

We have introduced a novel method to stabilize generative adversarial training that results in accurate generative models. Our method is rather general and can be applied to other GAN formulations with an average improvement in generated sample quality and variety, and training stability. Since GAN training aims at matching probability density distributions, we exploit additive noise to extend the support of the densities and thus facilitate the matching through gradient descent. More importantly, we show that using multiple loss terms with different additive noise (including the no-noise case) is necessary to achieve a highly accurate match of the original data distribution. We demonstrate the proposed training method on several common datasets of real images.

6. APPENDIX 6.1 EXPERIMENTS ON SYNTHETIC DATA:

We performed experiments with a standard GAN and a NTGAN using Gaussian noise on synthetic 2-D data. The generator and discriminator architectures are both MLPs consisting of three fullyconnected layers with a hidden-layer size of 512. We use ReLU activations and batch-normalization (Ioffe & Szegedy (2015) ) in all but the first discriminator layer and the output layers. The Adam optimzer (Kingma & Ba ( 2014)) was used with a learning rate of 10 -4 and we trained for 20K iterations. The results are shown in Figure 5 . We can observe how the matching of both clean and blurred distribution leads to a better fit in the case of NTGAN. 

6.2. EXAMPLES OF THE NOISE GENERATOR:

Examples of the noise generated by the noise generator of a NTGAN trained on CIFAR-10 are illustrated in Figure 6 . We can observe noise patterns that resemble some of the commonly observed artifacts produced by a degenerate GAN generator. The noise patterns vary over time and therefore encourage the fitting of the distributions under different blurs.

6.3. IMPLEMENTATION DETAILS:

Noise Generator. The noise-generator architecture in all our experiments is equivalent to the generator architecture with the number of filters reduced by a factor of eight. The output of the noisegenerator has a tanh activation scaled by a factor of two to allow more noise if necessary. We also experimented with a linear activation but didn't find a significant difference in performance.

GAN+GP. For the comparisons to the GAN regularizer proposed by Roth et al. (2017) we used the same settings as used in their work in experiments with DCGAN.

SNGAN+NT. We used the standard GAN loss (same as DCGAN) in all our experiments with models using spectral normalization. When combining SNGAN with NT we achieved the best results when batch-normalizing the inputs to the discriminator. For models with batch-normalization we found no benefit by adding this input normalization.

6.4. QUALITATIVE RESULTS FOR EXPERIMENTS:

We provide qualitative results for some of the ablation experiments in Figure 7 and for the robustness experiments in Figure 8 . As we can see in Figure 8 , none of the tested settings led to degenerate solutions in the case of NTGAN while the other methods would show failure cases in some settings. 

:

We rely on the same principle as in Calculus of Variations(Gelfand & Fomin (1964)), whereby the necessary condition for an extremum initially written in its weak form, as the integral of the functional derivative against an arbitrary function, is then transformed into its strong form by the Fundamental Lemma of the Calculus of Variations.

Figures and Tables:

Figure 1: Illustration of uses of additive noise as a regularizer for generative models. Adding noise to samples from a probability density p d results in a smoothing of p d . (a) Shows that matching smooth distributions results in a low-quality probability density estimate p g . (b) Shows that matching smooth probability densities for two noise instances results already in a much higher quality estimate p g .

Figure 2: Left column: Random reconstructions from models trained on CelebA without noisetempering (NT). Right column: Random reconstructions with our proposed method.

,Zhao et al. (2016),Berthelot et al. (2017),Arjovsky et al.  (2017),Zhao et al. (2016),Kodali et al. (2017)).

Figure 3: Results of the robustness experiments as specified in Table4and performed on CIFAR-10. We compare the standard GAN (1st column), a GAN with gradient penalty (2nd column), a GAN with spectral normalization (3rd column) and a GAN with our proposed method (4th column). Results are reported in Fréchet Inception Distance FID (top) and Inception Score IS (bottom).

Figure 4: Reconstructions from NTGANs trained on 128 × 128 images from the LSUN bedrooms dataset (top) and ImageNet (bottom).

Figure 5: We performed experiments on synthetic 2D data with a standard GAN (top) and a NTGAN (bottom). The ground truth data is shown in red and the model generated data is shown in blue. For NTGAN we also show samples from the blurred data distribution p d, in green and the blurred model distribution p g, in purple.

Figure 6: We show examples of the generated noise (top row) and corresponding noisy training examples (rows 2 to 4). The different columns correspond to different iterations.

Figure 7: We show random reconstructions for some of the ablation experiments listed in Table 2. The left column shows results on CIFAR-10 and the right column shows results on STL-10.

Algorithm 1: Noise-Tempered GAN (NTGAN) Input: Training set D ∼ p d , number of discriminator updates n disc , number of training iterations N , batch-size m, learning rate α, noise penalty λ Output: Generator parameters θ Initialize generator parameters θ, discriminator parameters φ and noise-generator parameters ω ; for 1 . . . N do for 1 . . . n disc do Sample mini-batches {x 1 , . . . , x m } ∼ p d and fake examples {x 1 , . . . , xm } ∼ p g ; Sample noise { 1 , . . . ,

Table content: <table/>

. . , x m + m } is the batch with real examples and {x 1 , . . . , xm , x1 + 1 , . . . , xm + m } the fake examples batch. The complete procedure is outlined Network architectures used for experiments on CIFAR-10 and STL-10. Images are assumed to be of size 32 × 32 for CIFAR-10 and 64 × 64 for STL-10. We set M = 512 for CIFAR-10 and M = 1024 for STL-10. Layers in parentheses are only included for STL-10. The noise-generator network follows the generator architecture with the number of channels reduced by a factor of 8. compare and evaluate our model using two common GAN metrics: the Inception score IS (Salimans et al. (2016)) and the Fréchet Inception distance FID(Heusel et al. (2017)). Throughout this Section we use 10K generated and real samples to compute IS and FID. In order to get a measure of the stability of the training we report the mean and standard deviation of the last five checkpoints for both metrics (obtained in the last 10% of training). More reconstructions, experiments and details are provided in the appendix. (d)-(i) Both noisy and clean samples: The second set of experiments consists of variants of our proposed model. Experiments (d) and (e) use a simple Gaussian noise model; in (e) the standard deviation of the noise σ is learned. We observe a drastic improvement in the quality of the generated examples even with this simple modification. The other experiments show results of our full model with a separate noise-generator network. We vary the weight λ of the L 2 norm of the noise in experiments (f)-(h). Ablation (i) uses the alternative minibatch construction with faster runtime as described in Section 2.3;

Table content: <table><tr><td>Generator CIFAR-10/(STL-10)</td><td>Discriminator CIFAR-10/(STL-10)</td></tr><tr><td>z ∈ R 128 ∼ N (0, I)</td><td>conv 3 × 3 stride=1 lReLU 64</td></tr><tr><td>fully-connected BN ReLU 4 × 4 × M</td><td>conv 4 × 4 stride=2 BN lReLU 64</td></tr><tr><td>(deconv 4 × 4 stride=2 BN ReLU 512)</td><td>conv 4 × 4 stride=2 BN lReLU 128</td></tr><tr><td>deconv 4 × 4 stride=2 BN ReLU 256</td><td>conv 4 × 4 stride=2 BN lReLU 256</td></tr><tr><td>deconv 4 × 4 stride=2 BN ReLU 128</td><td>conv 4 × 4 stride=2 BN lReLU 512</td></tr><tr><td>deconv 4 × 4 stride=2 BN ReLU 64</td><td>(conv 4 × 4 stride=2 BN lReLU 1024)</td></tr><tr><td>deconv 3 × 3 stride=1 tanh 3</td><td>fully-connected sigmoid 1</td></tr><tr><td>3 EXPERIMENTS</td><td/></tr></table>

We perform ablation experiments on CIFAR-10 and STL-10 to demonstrate the effectiveness of our proposed algorithm. Experiments (a)-(c) show results where only noisy examples are fed to the discriminator. Experiment (c) corresponds to previously proposed noise-annealing and results in an improvement over the standard GAN training. Our approach of feeding both noisy and clean samples to the discriminator shows a clear improvement over the baseline.

Table content: <table><tr><td>Experiment</td><td>FID</td><td colspan="2">CIFAR-10</td><td>IS</td><td>FID</td><td>STL-10</td><td>IS</td></tr><tr><td>Standard GAN</td><td colspan="2">46.1 ± 0.7</td><td colspan="2">6.12 ± .09</td><td colspan="2">78.4 ± 6.7</td><td>8.22 ± .37</td></tr><tr><td>(a) Noise only: ∼ N (0, I)</td><td colspan="2">94.9 ± 4.9</td><td colspan="2">4.68 ± .12</td><td colspan="2">107.9 ± 2.3</td><td>6.48 ± .19</td></tr><tr><td>(b) Noise only: learned</td><td colspan="2">69.0 ± 3.4</td><td colspan="2">5.05 ± .14</td><td colspan="2">107.2 ± 3.4</td><td>6.39 ± .22</td></tr><tr><td>(c) Noise only: ∼ N (0, σI), σ → 0</td><td colspan="2">44.5 ± 3.2</td><td colspan="2">6.85 ± .20</td><td colspan="2">75.9 ± 1.9</td><td>8.49 ± .19</td></tr><tr><td>(d) Clean + noise: ∼ N (0, I)</td><td colspan="2">29.7 ± 0.6</td><td colspan="2">7.16 ± .05</td><td colspan="2">66.5 ± 2.3</td><td>8.64 ± .17</td></tr><tr><td>(e) Clean + noise: ∼ N (0, σI) with learnt σ</td><td colspan="2">28.8 ± 0.7</td><td colspan="2">7.23 ± .14</td><td colspan="2">71.3 ± 1.7</td><td>8.30 ± .12</td></tr><tr><td>(f) NTGAN (λ = 0.1)</td><td colspan="2">27.7 ± 0.8</td><td colspan="2">7.31 ± .06</td><td colspan="2">63.9 ± 1.7</td><td>8.81 ± .07</td></tr><tr><td>(g) NTGAN (λ = 1)</td><td colspan="2">26.5 ± 0.6</td><td colspan="2">7.49 ± .04</td><td colspan="2">64.0 ± 1.4</td><td>8.52 ± .16</td></tr><tr><td>(h) NTGAN (λ = 10)</td><td colspan="2">29.8 ± 0.4</td><td colspan="2">6.55 ± .08</td><td colspan="2">66.9 ± 3.2</td><td>8.38 ± .20</td></tr><tr><td>(i) NTGAN alt. mini-batch (λ = 1)</td><td colspan="2">28.7 ± 0.6</td><td colspan="2">7.3 ± .05</td><td colspan="2">67.8 ± 3.2</td><td>8.30 ± .11</td></tr></table>

We apply our proposed method to various previous GAN models trained on CIFAR-10 and CelebA. The same network architectures and hyperparameters as in the original works are used. We can observe that using our approach increases performance in most cases even with the suggested hyperparameter settings. Note that our algorithm also allows successful training with the original minimax GAN loss as opposed to the commonly used heuristic (e.g., in DCGAN).

Table content: <table><tr><td>Model</td><td>FID</td><td colspan="2">CIFAR-10</td><td>IS</td><td>FID</td><td>CelebA</td><td>IS</td></tr><tr><td>MMGAN (Goodfellow et al. (2014))</td><td colspan="2">&gt; 450</td><td/><td>∼ 1</td><td colspan="2">&gt; 350</td><td>∼ 1</td></tr><tr><td>DCGAN (Radford et al. (2015))</td><td colspan="2">33.4 ± 0.5</td><td colspan="2">6.73 ± .07</td><td colspan="2">25.4 ± 2.6</td><td>2.42 ± .06</td></tr><tr><td>WGAN-GP (Gulrajani et al. (2017))</td><td colspan="2">37.7 ± 0.4</td><td colspan="2">6.55 ± .08</td><td colspan="2">15.5 ± 0.2</td><td>2.57 ± .02</td></tr><tr><td>LSGAN (Mao et al. (2017))</td><td colspan="2">38.7 ± 1.8</td><td colspan="2">6.73 ± .12</td><td colspan="2">21.4 ± 1.1</td><td>2.55 ± .06</td></tr><tr><td>SNGAN (Miyato et al. (2018))</td><td colspan="2">29.1 ± 0.4</td><td colspan="2">7.26 ± .06</td><td colspan="2">13.2 ± 0.3</td><td>2.31 ± .02</td></tr><tr><td>MMGAN +NT (λ = 0.1)</td><td colspan="2">33.1 ± 0.7</td><td colspan="2">6.91 ± .05</td><td colspan="2">16.6 ± 1.9</td><td>2.44 ± .02</td></tr><tr><td>DCGAN + NT (λ = 10)</td><td colspan="2">31.2 ± 0.3</td><td colspan="2">6.95 ± .11</td><td colspan="2">14.7 ± 1.0</td><td>2.57 ± .05</td></tr><tr><td>LSGAN + NT (λ = 10)</td><td colspan="2">36.7 ± 1.2</td><td colspan="2">6.63 ± .17</td><td colspan="2">19.9 ± 0.4</td><td>2.60 ± .06</td></tr><tr><td>SNGAN + NT (λ = 1)</td><td colspan="2">28.0 ± 1.6</td><td colspan="2">7.39 ± .06</td><td colspan="2">11.0 ± 0.4</td><td>2.39 ± .03</td></tr><tr><td>NTGAN (same generator as SNGAN)</td><td colspan="2">26.5 ± 0.6</td><td colspan="2">7.49 ± .04</td><td colspan="2">11.5 ± 0.9</td><td>2.54 ± .04</td></tr></table>

Hyperparameter settings used to evaluate the robustness of our proposed GAN stabilizer.

Table content: <table><tr><td colspan="4">Exp. Learning Rate BatchNorm in G Optimizer</td><td>Activation</td><td>n disc</td><td>#Filters G</td><td>#Samples</td></tr><tr><td>a)</td><td>2 • 10 -4</td><td>FALSE</td><td>ADAM</td><td>ReLU / lReLU</td><td>1</td><td>[512, 256, 128, 64]</td><td>50K</td></tr><tr><td>b)</td><td>2 • 10 -4</td><td>TRUE</td><td>ADAM</td><td>tanh</td><td>1</td><td>[512, 256, 128, 64]</td><td>50K</td></tr><tr><td>c)</td><td>1 • 10 -3</td><td>TRUE</td><td>ADAM</td><td>ReLU / lReLU</td><td>1</td><td>[512, 256, 128, 64]</td><td>50K</td></tr><tr><td>d)</td><td>1 • 10 -2</td><td>TRUE</td><td>SGD</td><td>ReLU / lReLU</td><td>1</td><td>[512, 256, 128, 64]</td><td>50K</td></tr><tr><td>e)</td><td>2 • 10 -4</td><td>TRUE</td><td>ADAM</td><td>ReLU / lReLU</td><td>5</td><td>[512, 256, 128, 64]</td><td>50K</td></tr><tr><td>f)</td><td>2 • 10 -4</td><td>TRUE</td><td>ADAM</td><td>ReLU / lReLU</td><td>1</td><td>[64, 64, 64, 64]</td><td>50K</td></tr><tr><td>g)</td><td>2 • 10 -4</td><td>FALSE</td><td>ADAM</td><td>ReLU / lReLU</td><td>1</td><td>[512, 256, 128, 64]</td><td>5K</td></tr></table>

Peer review (iclr2019):
 In this work, a new method for (adaptively) choosing the noise distribution in a noise-tempered generative adversarial network is proposed. Noise tempering in training GANs has been around for a while. The main argument in those existing work (and also in this paper) is that GAN training gets stuck when gradient vanishes due to non-intersecting support of the real and generated distributions. Hypothetically, if such phenomenon is happening, then adding noise to the samples ensures that the support overlap and hence gradient vanishing is expected to be prevented. Typical scenarios of adding noise has been studied, for example adding a noise with variance vanishing with training iterations. This ensures that later in the training, if the generator is close to the real data, the error from having noisy samples do not propagate to the generator training. This paper starts off from this assumption, asking the question of why we need to reduce the noise. This suggests training with non-vanishing noise. Two heuristic reasonings in support of such approach are (i) in existing approach of vanishing noise, it is not clear how to schedule the reduction of noise variance, leading to inefficiency or non-convergence, and (ii) minimizing noisy divergence can also achieve convergence to the true distribution, as long as the noise is not degenerate. In the latter the non-degeneracy can be achieved by using multiple noise distributions. The main contribution of this paper is that a neural network based adaptive noise generator is proposed. During the training, the noise (from a family of noise distributions) that makes the real and generated distributions as similar as possible is chosen, while constrained to having a bounded variance. The only evidence that this improves the training is via numerical experiments run by the authors (with no (anonymized) reproducible implementation references, e.g.~github). The FID and IS scores on several datasets are presented, which suggest that NTGAN achieves better scores. In table 3, specific architectural and hyper-parameter choices have been made for competing baselines. For example, for SNGAN, the authors of SNGAN report IS score of 8.22 on CIFAR-10 with ResNET and 7.58 with standard CNN. If the settings are fair, then these improves upon the reported IS of NTGAN. Given that no code is available, how does one interpret the resulting table of scores? How do you ensure fairness in different choices made for different architectures? The proposed approach of minimizing the loss over the noise with regularizer for the variance is surprising, as the theory in Lemma 1 seems to suggest an alternative (but more natural) approach. That is maximize the loss over the noise with perhaps regularizer that restricts the choice of no-noise or small variance. Can the authors elaborate on that choice? The metric of IS and FID score is too coarse to measure the subtle improvements that are made by the proposed approach. Given that the paper is motivated by the paths the gradients are taking during the training, it seems that analysis of the gradients (with and without the noise) is in order. Such thorough analyses will settle several myths regarding gradient vanishing, such as does gradient vanishing happen in the training of typical GANs, and does noise adding mitigate it. This is particularly interesting, as if WGAN is used with Wasserstein distance, there should be no gradient vanishing due to non-overlapping support. Checking these conjectures numerically is an important task, and is particularly critical for this paper.