[Review] Legacy GAN by Goodfellow et al in 2014

LINK: https://arxiv.org/pdf/1406.2661.pdf
## Foreword
It's been long time since I have firstly heard about the word "GAN".
GAN stands for "Generative Adversarial Network", which is a novel architecture of neural networks in generative models. Honestly I have strong interest in this area though, I couldn't save a time to do research very well before. So in this article I would like to walk you through the review of the paper and the implementation as well.

## Introduction
According to [LeCun](http://yann.lecun.com/) https://www.kdnuggets.com/2016/08/yann-lecun-quora-session.html
> The most important one, in my opinion, is adversarial training (also called GAN for Generative Adversarial Networks). This is an idea that was originally proposed by Ian Goodfellow when he was a student with Yoshua Bengio at the University of Montreal (he since moved to Google Brain and recently to OpenAI).
This, and the variations that are now being proposed is the most interesting idea in the last 10 years in ML, in my opinion.

Since the invention of this novel approach, many researchers have spent huge time on this to find and refine this network. You may have heard about GAN already. This family of networks are designed to achieve correctly representing the dataset distribution through the learning step. Simply saying, it is a maximum likelihood in the probabilistic theory, which defines the parametrically approximated distribution of the dataset. Prior to GAN, still there were some approaches, for instance legacy DBN(deep belief network), DBM(deep boltzman machine) or even more advanced models, which was using Markov Chain model. Especially prominent recent work in this area includes the generative stochastic network (GSN) framework, which improves the generalised denoising auto-encoder.

## Adversarial Network Framework in learning
Firstly please do allow me quote the nice and simple explanation on GAN here by Rowel Atienza(Associate Professor, University of the Philippines-Diliman. PhD Robotics, The Australian National University)
> GAN is almost always explained like the case of a counterfeiter (Generative) and the police (Discriminator). Initially, the counterfeiter will show the police a fake money. The police says it is fake. The police gives feedback to the counterfeiter why the money is fake. The counterfeiter attempts to make a new fake money based on the feedback it received. The police says the money is still fake and offers a new set of feedback. The counterfeiter attempts to make a new fake money based on the latest feedback. The cycle continues indefinitely until the police is fooled by the fake money because it looks real.
https://towardsdatascience.com/gan-by-example-using-keras-on-tensorflow-backend-1a6d515a60d0

Goodfellow et al have applied the adversarial architecture to neural network, which is zero-sam game in Game Theory. In this game, there are two agents cooperating or negotiating together to maximise own reward. So in GAN, following this architecture, Goodfellow et al have separated the learning steps into two independent components, a discriminator and a generator. 
<img width="692" alt="Screen Shot 2018-06-10 at 8.21.22.png" src="https://qiita-image-store.s3.amazonaws.com/0/240801/7c4b8c88-1fc9-771a-fd9d-88f7938714c6.png">
source: https://www.slideshare.net/xavigiro/deep-learning-for-computer-vision-generative-models-and-adversarial-training-upc-2016

To prove the convergence of this architecture, they have used two important theorem from the domain of information theory.
1. KL divergence $KL(P||Q) = \sum P \log \frac{P}{Q}$
2. Jensen–Shannon divergence $JSD(P||Q) = \frac{1}{2}KL(P||M) + \frac{1}{2}KL(Q||M)$
Regarding the mathematical definition of both theories, please refer to here!
https://qiita.com/Rowing0914/items/2f3fcb114ae671c7c394

## Mathematical Proof of Convergence of learning
So assuming that P is representing the generator's distribution and Q is indicating discriminator's distribution. We could confirm that optimal Q is proportional to P. Hence, as for theorem 1, we can state as below.

- Theorem 1

```math
\begin{align}
&D^*_G(x) = \frac{p_{data}}{p_{data} + p_g}\\
&where\\
&p_g = generator's \space distribution\\
&p_{data} = dataset's \space distribution
\end{align}
```

Also, based on this theorem 1, finally we can properly define the learning objective of discriminator as below, which is indicating the maximising the log-likelihood for estimating the binary conditional probability.

```math
V(G, D) = \int_x p_{data}(X) \log(D(X))dx + \int_z p_z(z) \log(1 - D(Z))dz\\
= \int_x \big[ p_{data}(X) \log(D(X)) + p_g(x) \log(1 - D(X)) \big]dx
```
Hence, the criteria in learning will be like below.
<img width="691" alt="Screen Shot 2018-06-10 at 9.43.58.png" src="https://qiita-image-store.s3.amazonaws.com/0/240801/d37a71c6-2c53-231d-92e7-8e552386d357.png">


- Theorem 2
By now, we understand the optimal situation of learning, which is $p_{data} = p_g$. Actually, this can be interpreted as $p_{data} = p_g = \frac{1}{2}$.
So based on this assumption, we now move on to seeing the proof of the convergence of the learning.

In fact, substituting the assumption above to $C(G)$, it will transform like below.

<img width="438" alt="Screen Shot 2018-06-10 at 9.45.12.png" src="https://qiita-image-store.s3.amazonaws.com/0/240801/97191648-99ec-b4de-3f95-8228d356ea37.png">
So the optimality can be likely to occur in learning stands on the case where we can achieve $p_{data} = p_g = \frac{1}{2}$.
So, the goal of the learning is to asymptotically push $C(G)$ closer to $-\log(4)$ as below.
<img width="376" alt="Screen Shot 2018-06-10 at 9.46.27.png" src="https://qiita-image-store.s3.amazonaws.com/0/240801/2cf07205-7973-305f-e3ef-e56ec43b3d34.png">

## Algorithm
<img width="925" alt="Screen Shot 2018-06-10 at 9.48.34.png" src="https://qiita-image-store.s3.amazonaws.com/0/240801/e48aacb6-b56b-9f17-9913-3563a110025a.png">

## Implementation
I have used this on Google colab to check if it's buggy.

```python
import os
import numpy as np
import matplotlib.pyplot as plt
import keras
from keras.models import Sequential
from keras.layers import Dense, Activation, BatchNormalization, LeakyReLU
from keras.optimizers import SGD, Adam, RMSprop
from keras.datasets import mnist
from keras.regularizers import l1_l2

def make_sample(n_sample, z_input_size):
	sample = np.random.uniform(0, 1, n_sample*z_input_size)
	sample = sample.reshape([n_sample, z_input_size])
	return sample

def plot_generated(noise, Generator):
	image_fake = Generator.predict(noise)
	plt.figure(figsize=(10,8))
	plt.show()
	plt.close()

def plot_metircs(metrics, epoch=None):
	plt.figure(figsize=(10,8))
	plt.plot(metrics['d'], label='discriminative loss', color='b')
	plt.legend()
	plt.show()
	plt.close()

	plt.figure(figsize=(10,8))
	plt.plot(metrics['g'], label='generative loss', color='r')
	plt.legend()
	plt.show()
	plt.close()

def Generator():
	model = Sequential()
	model.add(Dense(128, input_dim=100))
	model.add(LeakyReLU(alpha=0.2))
	model.add(BatchNormalization(momentum=0.8))
	model.add(Dense(512))
	model.add(LeakyReLU(alpha=0.2))
	model.add(BatchNormalization(momentum=0.8))
	model.add(Dense(1024))
	model.add(LeakyReLU(alpha=0.2))
	model.add(BatchNormalization(momentum=0.8))
	model.add(Dense(input_dim=128, units=784, activation='tanh'))
	return model

def Discriminator():
	model = Sequential()
	LeakyReLU = keras.layers.advanced_activations.LeakyReLU(alpha=0.2)
	model.add(Dense(input_dim=784, units=128, activation=LeakyReLU, name='d_input'))
	model.add(Dense(input_dim=128, units=1, activation='sigmoid', name='d_output'))
	model.compile(loss='binary_crossentropy', optimizer='Adam')
	return model

def Generative_Adversarial_Network(Generator, Discriminator):
	model = Sequential()
	model.add(Generator)
	model.add(Discriminator)
	# train only generator in the entire GAN architecture
	Discriminator.trainable = False
	model.compile(loss='binary_crossentropy', optimizer='Adam')
	return model

def Training(z_input_size, Generator, Discriminator, GAN, loss_dict, X_train, epoch, batch, smooth, n_sample):
	for e in range(epoch):
		# z: noise, used for input of G to generate fake image based on this noise! it's like a seed 
		noise = np.random.uniform(0, 1, size=[batch, z_input_size])
		image_fake = Generator.predict_on_batch(noise)

		# sampled real_image from dataset
		rand_train_index = np.random.randint(0, X_train.shape[0], size=batch)
		image_real = X_train[rand_train_index, :]

		# concatenate real and fake images
		"""
		X = [
			image_real => label : 1 (we can multiply a smoothing factor)
			image_fake => label : 1 (honestly this should be 0, but the purpose of G in training is cheating D, so I set this as 1)
			]
		"""
		X = np.vstack((image_real, image_fake))
		y = np.zeros(len(X))

		# putting label "1" to image_real
		y[len(image_real):] = 1*(1 - smooth)
		y = y.astype(int)

		# train only discriminator
		d_loss = Discriminator.train_on_batch(x=X, y=y)

		# NOTE: remember?? we set discriminator OFF during the training of GAN!
		# So, we can safely train only generator, the weight of discriminator set fixed!
		g_loss = GAN.train_on_batch(x=noise, y=y[len(noise):])

		# learning log!
		loss_dict['d'].append(d_loss)
		loss_dict['g'].append(g_loss)

		# show the images, the result of leanring!
		if e%1000 == 0:
			plt.imshow(image_fake)
			plt.show()
			sample = make_sample(n_sample, z_input_size)
			plot_generated(sample, Generator)

	# display the history of learning
	plot_metircs(loss_dict)
	return "done!"
		  

Gen = Generator()
Dis = Discriminator()
GAN = Generative_Adversarial_Network(Gen, Dis)
GAN.summary()
Gen.summary()
Dis.summary()

gan_losses = {"d":[], "g":[], "f":[]}
epoch = 30000
batch = 1000
smooth = 0.9
z_input_size = 100
row, col = 28, 28
n_sample = 10

(X_train, y_train), (X_test, y_test) = mnist.load_data()
X_train, X_test = X_train.reshape(X_train.shape[0], row*col), X_test.reshape(X_test.shape[0], row*col)
X_train.astype('float32')
X_test.astype('float32')
X_train, X_test = X_train/255, X_test/255
print('X_train shape: ', X_train.shape)
print(X_train.shape[0], 'train samples')
print(X_test.shape[0], 'test samples')

Training(z_input_size, Gen, Dis, GAN, loss_dict=gan_losses, X_train=X_train, epoch=epoch, batch=batch, smooth=smooth, n_sample=n_sample)
```

## Nice References of Implementation
tensorflow version used in Udacity course
https://github.com/mchablani/deep-learning/blob/master/gan_mnist/Intro_to_GANs_Solution.ipynb

this is good!
https://github.com/eriklindernoren/Keras-GAN/blob/master/gan/gan.py