dzabdul/MediumArticleDataset huggingface dataset:
tical transformations made Wasserstein distance evaluable
 and a neural network is capable of approximating this metric. Whenever an approximately optimal discriminator is obtained
Category:  which is also highly correlated with the quality of the generated examples. The author offered extensive empirical results for the WGAN framework.

 minimizing the Wasserstein distance can pull the two distributions together. Not only does WGAN make the training process stable
,5185,Machine Learning,Peiyuan (Alexander) Liao,https://medium.com/s/story/the-wonderful-wasserstein-gan-def614a8aacc,Just FYI
 since their first appearance in 2014
Category:  as well as collapsing generated samples.

 GAN’s implementations have suffered from training difficulties
Scholars did attempt to improve
 yet most of the papers made little improvement

 like the enumerative methodology introduced by the DCGAN paper
Understanding the proofs and theorems in the original paper requires rather extensive knowledge on measure theory and topology
 so instead this paper will focus on a more intuitive aspect

 where the interpretations are based off lower-dimensional examples to help readers to ponder the ideas behind the scene while maintaining mathematical rigor.
To recap
 original GAN paper proposes that the goal of the discriminator is to minimize the following loss function

 where a sample from the model distribution has a positive effect and vice versa.
In the equation
 $P_{r}$ refers to the model distribution

 and $P_{r}$ refers to the implicitly defined generative network distribution.
For the generative network
 Goodfellow initially presented a loss function

 to which a refined version was also proposed.
The readers are welcomed to take a deep breath at this point and see the conclusions we have made so far: According to the generative network loss defined in the original GAN
 we can obtain an optimal discriminator

 and thus rewrite the generative loss as the JS divergence between the model distribution $P_r$ and the generative distribution $P_g$.
In other words
 no matter how far they are apart from each other
Category:  which gives a gradient 0 for the gradient descent method.

 as long as the distributions are not “touching each other with a considerably large intersection
If the parameters of the generative network are fixed
 though the distribution is defined in a high-dimensional space (like 4096)

 the variation of which is merely restricted by the low-dimensional support (like the 100-D “seed”).
Thus
 the support of this 4096-D generative distribution can only be a low-dimensional manifold whose dimension can be no more than 100

 which makes the distribution “unable to extend” to the entire space.
Thus we can arrive at another conclusion: if the generative network is initialized randomly
 $P_g$ is highly unlikely to have any relation to $P_r$
Category:  hence having a measure of 0.

 and their supports either do not intersect at all
Then we can obtain the first argument on the vanishing gradient in the generative network of the original GAN: given an (approximately) optimal discriminator
 minimizing the loss of the generative network is equivalent to minimizing the JS divergence between Pr and Pg
Category:  their JS divergence is always $\log2$
Author:  which results in (approximately) zero gradient.

 and since they are highly unlikely to have any intersections
As a universal approximator
 an artificial neural network can approximate this hyperplane to infinite precision

 so an optimal discriminator do exist that give almost all model examples 1 and all generative examples 0.
The normalized possibilities that the optimal discriminator provide on both the model distribution and the generative distribution are all constant (1/0)
 so the gradient of the generative network loss is always 0

 hence the vanishing gradient.
Summary on part 1: When the original GAN obtains an (approximately) optimal discriminator
 the first generator loss suffers from vanishing gradient while the second loss faces the menace of absurd optimization goal

 unstable gradient
Again
 the problem with the original GAN is twofold: the unreasonable optimization goal produced by KL/JS divergence metrics

 and the difficulty for an arbitrary distribution to overlap with the model distribution.
Thinking reversely
 we can obtain the JS divergence between the two noisy distributions from the loss of the optimal discriminator

 which is in a way the “distance” between them.
The motivation of the noise addition only roots from the second problem with the original GAN
 so although the training process is now stabilized

 we still don’t have a reliable metric to indicate the training process.
However
 by rooting from the more fundamental first problem of GAN

 WGAN replaces JS divergence with Wasserstein distance
And for all possible joint distributions
 one could observe to obtain a model sample $x$ and a generative sample $y$

 and thus calculate the distance.
Similarly
 in a high-dimensional case

 neither KL/JS divergence can provide information on distance
But now
 we have WGAN using discriminator fw approximating the Wasserstein distance

 which is apparently a regression problem
The last point of nuance
 something that the original paper didn’t mention

 is that the author claimed that the approximated Wasserstein distance could guide researchers to adjust hyperparameters of the network when compared to different sessions.
The first form has an equivalent form of minimizing the JS divergence when an optimal discriminator is obtained
 and since a randomly initialized distribution is highly likely to overlap significantly with the model distribution

 the jump discontinuity of JS divergence causes the gradient to vanish.
The second form has an absurd goal of minimizing KL divergence while maximizing JS divergence when the optimal discriminator is obtained
 which destabilizes the gradient

 and the asymmetric nature of KL divergence caused the collapse mode in generative networks.
By adding noises to the distributions
 the forced overlaps can stabilize the training process

 where obtaining an optimal discriminator is no longer problematic.