Gated CNN の実装に関するメモ（２）

##ポイント
* Gated CNN を実装し、Sequential MNIST でパフォーマンスを検証。
* Residual Block、Weight Normalization、Gradient Clipping の効果をチェック。
* 今後、LSTM ( Batch Normalization、Zoneout ) と比較検証。

##レファレンス
[1. Language Modeling with Gated Convolutional Networks](https://arxiv.org/pdf/1612.08083.pdf)
[2. An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling](https://arxiv.org/abs/1803.01271)
[3. Gated CNN に関するメモ](https://qiita.com/takeshikondo/items/f3c1eb1cd40e217b8f68)


##検証方法
* Sequential MNIST を使用　（ [14*14, 1] に縮尺 ） 。
* BASE : GLU 
* RES : BASE + Residual Block
* RES+WN : RES + Weight Normalization
* RES+WN+GL : RES+WN + Gradient Clipping (by Norm, 0.5)

##データ
MNIST handwritten digits

```python
from tensorflow.examples.tutorials.mnist import input_data
mnist = input_data.read_data_sets('***/mnist', \
                                     one_hot = True)

train_images = np.reshape(mnist.train.images, [-1, 28, 28])
train_images = skimage.measure.block_reduce(train_images, \
                    block_size = (1, 2, 2), func = np.max)
train_images = np.reshape(train_images, [-1, 14*14])
```
##検証結果
数値計算例：

* n_in = 14*14
* filter_size = 5
* n_units = 32
* n_out = 10
* n_layers = 4
* residual_step = 2
* learning_rate = 0.01
* batch_size = 64

BASE
![image.png](https://qiita-image-store.s3.amazonaws.com/0/254604/8266454d-3fe9-ff09-6aad-1651e7bd3819.png)
<br>

RES
![image.png](https://qiita-image-store.s3.amazonaws.com/0/254604/668be5e7-fe8d-c9c4-6589-91b15b75423c.png)
<br>

RES+WN
![image.png](https://qiita-image-store.s3.amazonaws.com/0/254604/fbbd4953-c12a-e852-76b6-faa8602014d1.png)
<br>

RES+WN+GC
![image.png](https://qiita-image-store.s3.amazonaws.com/0/254604/42e58aa6-dac3-7b70-908c-4fcee6c8e6c3.png)


##サンプルコード
```python
  def dilated_gated_conv(self, x, shape, rate):
    w = self.weight_variable('w', shape)
    b = self.bias_variable('b', shape[-1])
    v = self.weight_variable('v', shape)
    c = self.bias_variable('c', shape[-1])
    
    f = tf.add(tf.nn.atrous_conv2d(x, w, rate, \
                          padding = 'VALID'), b)
    g = tf.add(tf.nn.atrous_conv2d(x, v, rate, \
                          padding = 'VALID'), c)
    
    output = tf.multiply(f, tf.sigmoid(g))  # GLU
    #output = tf.multiply(tf.tanh(f), tf.sigmoid(g))  # GTU
    #output = tf.nn.relu(f)  # ReLU
    #output = tf.tanh(f)  # Tanh
    
    return output

  def dilated_gated_conv_wn(self, x, shape, rate):
    v_1 = self.weight_variable('v_1', shape)
    g_1 = self.weight_variable('g_1', shape[-1])
    b_1 = self.weight_variable('b_1', shape[-1])
    v_2 = self.weight_variable('v_2', shape)
    g_2 = self.weight_variable('g_2', shape[-1])
    b_2 = self.weight_variable('b_2', shape[-1])
    
    v1_norm = tf.nn.l2_normalize(v_1, axis = [0, 1, 2])
    w_1 = g_1 * v1_norm
    v2_norm = tf.nn.l2_normalize(v_2, axis = [0, 1, 2])
    w_2 = g_2 * v2_norm
    
    
    f = tf.add(tf.nn.atrous_conv2d(x, w_1, rate, \
                               padding = 'VALID'), b_1)
    g = tf.add(tf.nn.atrous_conv2d(x, w_2, rate, \
                               padding = 'VALID'), b_2)
    
    output = tf.multiply(f, tf.sigmoid(g))  # GLU
    #output = tf.multiply(tf.tanh(f), tf.sigmoid(g))  # GTU
    #output = tf.nn.relu(f)  # ReLU
    #output = tf.tanh(f)  # Tanh
    
    return output

  # With Residual Block
  def inference_2(self, x, filter_size, n_in, n_units, \
                        n_out, n_layers, residual_step):
    width = n_in
    channel = 1
    x = tf.reshape(x, [-1, 1, width, channel])
    
    x = tf.nn.max_pool(x, ksize = [1, 1, 2, 1], \
              strides = [1, 1, 2, 1], padding = 'SAME')
    
    shape = [1, 1, channel, n_units]
    with tf.variable_scope('initial'):
      y = self.conv(x, shape)
    
    inputs = y

    shape = [1, filter_size, n_units, n_units]
    for i in range(n_layers):
      with tf.variable_scope('layer_{}'.format(i + 1)):
        y = tf.pad(y, [[0, 0], [0, 0], \
                 [2**i * (filter_size - 1), 0], [0, 0]])
        
        # Without Weight Normalization
        y = self.dilated_gated_conv(y, shape, 2**i)
        # With Weight Normalization
        #y = self.dilated_gated_conv_wn(y, shape, 2**i)
                
        if (i + 1) % residual_step == 0:
          y += inputs
          inputs = y
        
    y = y[:, :, -1, :]
    y = tf.squeeze(y, axis = 1)
    
    with tf.variable_scope('final'):
      w = self.weight_variable('w', [n_units, n_out])
      b = self.bias_variable('b', [n_out])
      
      y = tf.add(tf.matmul(y, w), b)
      y = tf.nn.softmax(y, axis = 1)
        
    return y

  def training_clipped(self, loss, learning_rate, clip_norm):
    optimizer = tf.train.AdamOptimizer(learning_rate \
                                    = learning_rate)

    grads_and_vars = optimizer.compute_gradients(loss)
    clipped_grads_and_vars = [(tf.clip_by_norm(grad, \
                  clip_norm = clip_norm), var) for grad, \
                  var in grads_and_vars]
    train_step = \
       optimizer.apply_gradients(clipped_grads_and_vars)
    
    return train_step
```