# A Generative Adversarial Network Structure for Learning with Small Numerical Data Sets
## 2 Literature Review  
### The Generative Adversarial Networks  
[PERSON] et al. [13] proposed generative adversarial nets (GANs); the architecture is shown in Figure 1. This consists of the generative network (Generator) and the discriminative network (Discriminator). The former aims to generate the virtual sample, based on a set of real data as its input, whereas the latter discriminates whether the output of the Generator, the virtual sample, is real or fake. By continuously training and optimizing the min-max function, it is expected that the Generator can ultimately produce a set of virtual data that would be recognized as real data.  
Assume that a random variable \(z\), drawn from \(P_{z}(z)\), is an input of the Generator network (with parameter \(\theta_{g}\)), \(G(z;\theta_{g})\), and the Discriminator network (with parameter \(\theta_{d}\)), \(D(x;\theta_{d})\), identifies whether \(x\) is fake or real by its outputs: 0 or 1, where \(x\) represents the training data. The framework of GAN is to train \(D\) so that the accuracy of recognizing the real and virtual data is maximized, and to train \(G\) so that the loss function is minimized, \(\log(1-D(G(z)))\). The framework can be concluded with a two-player minmax function, as shown in Formula (1).  
[PERSON] et al. [21] proposed the Wasserstein GAN (WGAN), whose purpose is to improve the problem of GAN gradient vanishing. As mentioned above, when a GAN is trained to a certain level or when the generative and the discriminative networks are unbalanced, the gradient gradually becomes much smaller and the loss function is gradually saturated. This kind of unstable training process can lead to a collapse of the model. The concept of W distance (Wasserstein Distance) was introduced into the GAN algorithm to address the above problems. This new version of the algorithm completely overcomes the problem of instability in GAN training. It is no longer necessary to carefully balance the training range of the generative and discriminative networks. It also resolves the problem of mode collapse and ensures the diversity of the generated virtual samples.  
\[\underset{G}{min}\underset{||D||_{L}<1}{max}V(D,G)=\mathrm{E}_{x\sim p_{data}( x)}[logD(x)]-\mathrm{E}_{z\sim p_{z}(z)}[D(G(z))] \tag{1}\]  
Numerous variations of GANs have been proposed to improve its performance. The cGAN ([PERSON] et al., 2014) adds extended labels to the samples as the input of the Generator and Discriminator, noted as G and D respectively. The Wasserstein GAN [21] uses Wasserstein loss in the loss function to address the vanishing gradient problem. Moreover, the WGAN avoids unstable training and prevents the collapse of the model.  
The WGAN introduced several important concepts into this formula: (1) removing the sigmoid from the last layer of the adversarial network; (2) not taking the log value for the loss functions of both the generative and adversarial networks; (3) adopting the weight clipping method to limit the gradient, of adversarial networks, to a certain range, and fixing the maximum and minimum weights; and (4) trimming the absolute value of the adversarial network weights after iterative updating, within a fixed constant c.