Gated CNN x Separable Convolution の実装に関するメモ

##ポイント
* Gated CNN において、Regular Convolution を Separable Convolution に置き換え。
* Sequential MNIST を使用。
* 今後、パラメータ数の削減効果、Dilation と Filter size のトレードオフを追加検証。

##レファレンス
[1. Language Modeling with Gated Convolutional Networks](https://arxiv.org/abs/1612.08083)
[2. Depthwise Separable Convolutions for Neural Machine Translation](https://arxiv.org/abs/1706.03059)
[3. Gated CNN に関するメモ](https://qiita.com/takeshikondo/items/f3c1eb1cd40e217b8f68)


##検証方法
* Sequential MNIST を [14*14, 1] に縮尺。
* パラメータ数 ( Regular : 約16,000、Separable : 約4,000 )
 
　　　  

##データ
MNIST handwritten digits

```python
from tensorflow.examples.tutorials.mnist import input_data
mnist = input_data.read_data_sets('***/mnist', \
                                     one_hot = True)

train_images = np.reshape(mnist.train.images, [-1, 28, 28])
train_images = skimage.measure.block_reduce(train_images, \
                    block_size = (1, 2, 2), func = np.max)
train_images = np.reshape(train_images, [-1, 14*14])
```
##検証結果
数値計算例：

* n_in = 14*14
* filter_size = 5
* n_units = 32
* n_out = 10
* n_layers = 4
* residual_step = 2
* learning_rate = 0.01
* batch_size = 64

Regular Convolution
![image.png](https://qiita-image-store.s3.amazonaws.com/0/254604/d0a392e1-ef62-69f6-10f1-295a07abc644.png)
<br>

Separable Convolution
![image.png](https://qiita-image-store.s3.amazonaws.com/0/254604/55a0243d-165b-d835-df4c-d840591f42af.png)
<br>


##サンプルコード
```python
  def dilated_gated_separable_conv(self, x, shape_d, shape_p, rate):
    w_d = self.weight_variable('w_d', shape_d)
    w_p = self.weight_variable('w_p', shape_p)
    b = self.weight_variable('b', shape_p[-1])
    v_d = self.weight_variable('v_d', shape_d)
    v_p = self.weight_variable('v_p', shape_p)
    c = self.weight_variable('c', shape_p[-1])
    
    f = tf.add(tf.nn.separable_conv2d(x, w_d, w_p, \
            strides = [1, 1, 1, 1], padding = 'VALID', \
            rate = [1, rate]), b)
    g = tf.add(tf.nn.separable_conv2d(x, v_d, v_p, \
            strides = [1, 1, 1, 1], padding = 'VALID', \
            rate = [1, rate]), c)
    
    output = tf.multiply(f, tf.sigmoid(g))  # GLU
    #output = tf.multiply(tf.tanh(f), tf.sigmoid(g))  # GTU
    #output = tf.nn.relu(f)  # ReLU
    #output = tf.tanh(f)  # Tanh
    
    return output

  def inference_2(self, x, filter_size, n_in, n_units, \
                    n_out, n_layers, residual_step):
   
    width = n_in
    channel = 1
    x = tf.reshape(x, [-1, 1, width, channel])
    
    x = tf.nn.max_pool(x, ksize = [1, 1, 2, 1], \
               strides = [1, 1, 2, 1], padding = 'SAME')
    
    shape = [1, 1, channel, n_units]
    with tf.variable_scope('initial'):
      y = self.conv(x, shape)
    
    inputs = y

    # for regular convolution
    #shape = [1, filter_size, n_units, n_units]
    # for separable convolution
    shape_d = [1, filter_size, n_units, 1]
    shape_p = [1, 1, n_units, n_units]
    for i in range(n_layers):
      with tf.variable_scope('layer_{}'.format(i + 1)):

        # Dilated
        y = tf.pad(y, [[0, 0], [0, 0], \
                 [2**i * (filter_size - 1), 0], [0, 0]])
        # Separable Convolution Without Weight Normalization
        y = self.dilated_gated_separable_conv(y, \
                 shape_d, shape_p, 2**i)
                
        if (i + 1) % residual_step == 0:
          y += inputs
          inputs = y
        
    y = y[:, :, -1, :]
    y = tf.squeeze(y, axis = 1)
    
    with tf.variable_scope('final'):
      w = self.weight_variable('w', [n_units, n_out])
      b = self.bias_variable('b', [n_out])
      
      y = tf.add(tf.matmul(y, w), b)
      y = tf.nn.softmax(y, axis = 1)
        
    return y

```