# Improvement of Learning Stability of Generative Adversarial Network Using Variational Learning
## 2 Preliminary Works
### Variational Autoencoder  
The variational autoencoder (VAE) is a deep generative model that consists of two neural networks, that is, an encoder and a decoder. The objective of VAE is to find a parameter \(\theta\) that maximizes the likelihood of the approximate distribution \(P_{\theta}(x)\), which approximates the original data distribution \(P_{data}\). To solve this problem, it is assumed that \(P_{\theta}(x)\) is a latent variable model with a latent variable \(z\). Accordingly, it can be expressed as a joint distribution, as follows:  
\[P_{\theta}(x)=\int P_{\theta}(x,z)dz=\int P_{\theta}(x|z)P(z)dz. \tag{4}\]  
To allow for the application of the logarithm on both sides of Equation (4), an arbitrary distribution \(Q(z)\) is introduced for \(z\) to make the calculation easier. Using \(\log P_{\theta}(x,z)=\log P_{\theta}(z|x)+\log P_{\theta}(x)\), Equation (4) can be decomposed as follows. \[\log P_{\theta}(x)=\int Q(z)\log\bigg{\{}\frac{P_{\theta}(x,z)}{Q(z)}\bigg{\}} dz-\int Q(z)\log\bigg{\{}\frac{P_{\theta}(z|x)}{Q(z)}\bigg{\}}dz \tag{5}\]  
The second term on the right side of Equation (5) can be expressed according to Equation (7) while using the KLD, whereby \(KLD(Q(z)||P_{\theta}(z|x))\)[35] indicates how similar \(P_{\theta}(z|x)\) and \(Q(z)\) are. \[-\int Q(z)\log\bigg{\{}\frac{P_{\theta}(z|x)}{Q(z)}\bigg{\}}dz=KLD(Q(z)||P_{ \theta}(z|x)) \tag{6}\]  
The KLD is always \(\geq 0\), and it has a characteristic of being zero when \(P_{\theta}(z|x)=Q(z)\). Therefore, the first term on the right side of Equation (5) can be considered as the lower bound of \(\log P_{\theta}(x,z)\) and can be expressed, as follows:  
\[\int Q(z)\log\bigg{\{}\frac{P_{\theta}(x,z)}{Q(z)}\bigg{\}}dz=L(Q(z),\theta). \tag{7}\]  
Using Equations (6) and (7), Equation (5) can be rewritten as follows. \[\log P_{\theta}(x)=L(Q(z),\theta)+KLD(Q(z)||P_{\theta}(z|x)) \tag{8}\]  
If \(P_{\theta}(z|x)\) is tractable, \(Q(z)\) can be replaced by \(P_{\theta}(z|x)\), the KLD becomes zero, then \(logP_{\theta}(x)\) is equal to \(L(Q(z),\theta)\). In this case, the solution of Equation (8) can be obtained while using the expectation-maximization (EM) algorithm [36]. However, if \(P_{\theta}(z|x)\) is intractable, then the solution cannot be obtained. In this case, \(Q(z)\) is assumed to be the distribution that approximates \(P_{\theta}(z|x)\). This is called variational inference. It approximates \(P_{\theta}(z|x)\), which is necessary for the calculation of \(L(Q(z),\theta)\) based on the use of the artificial neural network \(Q_{\phi}(z|x)\) with parameter \(\phi\) as \(Q(z)\), and with periodic updating of \(\phi\)Given that \(P_{\theta}(z|x)\) is intractable in variational inference, it is not possible to update \(\phi\) while directly minimizing \(KLD(Q_{\phi}(z|x)||P_{\theta}(z|x))\), yet the solution can be obtained by updating \(\phi\) to maximize the lower bound \(L(Q_{\phi}(z|x),\theta)=L(\phi,\theta)\). Therefore, the loss function of VAE is as follows. \[L(\phi,\theta) =\int Q_{\phi}(z|x)\log\bigg{\{}\frac{P_{\theta}(x,z)}{Q_{\phi}(z| x)}\bigg{\}}dz\] \[=\int Q_{\phi}(z|x)\log\bigg{\{}\frac{P(z)}{Q_{\phi}(z|x)}\bigg{\}} dz+\int Q_{\phi}(z|x)\log\{P_{\theta}(x|z)\}dz\] \[=-KLD(Q_{\phi}(z|x)||P(z))+E_{Q_{\phi}(z|x)}[\log P_{\theta}(x|z)] \tag{9}\]  
The second term on the right side of Equation (9) is a metric that shows how well the input \(x\) is restored owing to the reconstruction loss in the autoencoder. The first term shows how similar the distribution of the inference model \(Q_{\phi}(z|x)\) and the given prior \(P(z)\) are. Minimizing these two terms at the same time is equivalent to minimizing \(-\log P_{\theta}(x)\). The VAE learnt in this way generates new data by inputting \(z\sim P(z)\) into the decoder. Experimentally, the method of multiplication of the regularization term with the use of the appropriate weight \(\beta\) in VAE [37] is known to be helpful for the learning of VAE. Thus, the loss function of VAE can be finally written, as follows.