Gated CNN x Positional Encoding の実装に関するメモ

##ポイント
* Gated CNN において、Positional Encoding を追加。
* Regular Convolution と Separable Convolution で効果をチェック。
* Sequential MNIST を使用。

##レファレンス
[1. Language Modeling with Gated Convolutional Networks](https://arxiv.org/abs/1612.08083)
[2. Depthwise Separable Convolutions for Neural Machine Translation](https://arxiv.org/abs/1706.03059)
[3. Gated CNN x Separable Convolution に関するメモ](https://qiita.com/takeshikondo/items/be6138d12df0817ce8bd)


##検証方法
* Sequential MNIST を [14*14, 1] に縮尺。
* パラメータ数 ( Regular : 約16,000、Separable : 約4,000 )
 
　　　  

##データ
MNIST handwritten digits

```python
from tensorflow.examples.tutorials.mnist import input_data
mnist = input_data.read_data_sets('***/mnist', \
                                     one_hot = True)

train_images = np.reshape(mnist.train.images, [-1, 28, 28])
train_images = skimage.measure.block_reduce(train_images, \
                    block_size = (1, 2, 2), func = np.max)
train_images = np.reshape(train_images, [-1, 14*14])
```
##検証結果
数値計算例：

* n_in = 14*14
* filter_size = 5
* n_units = 32
* n_out = 10
* n_layers = 4
* residual_step = 2
* learning_rate = 0.01
* batch_size = 64

Regular Convolution without Positional Encoding
![image.png](https://qiita-image-store.s3.amazonaws.com/0/254604/eb9f109c-c7b8-c617-75f5-1ee9f091bc19.png)
<br>

Regular Convolution with Positional Encoding
![image.png](https://qiita-image-store.s3.amazonaws.com/0/254604/987db50a-afae-74c4-f62f-21684e733ffc.png)
<br>

Separable Convolution without Positional Encoding
![image.png](https://qiita-image-store.s3.amazonaws.com/0/254604/d78b7874-6100-fb50-d9d0-2b096fb04890.png)
<br>

Separable Convolution with Positional Encoding
![image.png](https://qiita-image-store.s3.amazonaws.com/0/254604/ec651c39-ddd8-4f2b-36cb-4e5f34ce5beb.png)
<br>

##サンプルコード
```python
  def get_lookup_table(self, length, n_units):
    lt = np.array([[pos / np.power(10000, 2.0 * i / \
               n_units) for i in range(int(n_units / 2))] \
               for pos in range(length)])
    lt = np.repeat(lt, 2, axis = 1)

    lt[:, 0::2] = np.sin(lt[:, 0::2])
    lt[:, 1::2] = np.cos(lt[:, 1::2])

    return tf.convert_to_tensor(lt)
  
  def positional_encoding(self, batch_size, length, n_units):
    pos = tf.tile(tf.expand_dims(tf.range(length), 0), \
                [batch_size, 1])

    lt = self.get_lookup_table(length, n_units)

    return tf.nn.embedding_lookup(lt, pos)

  def inference(self, x, filter_size, n_in, n_units, \
                    n_out, n_layers, residual_step):
   
    width = n_in
    channel = 1
    x = tf.reshape(x, [-1, 1, width, channel])
    
    x = tf.nn.max_pool(x, ksize = [1, 1, 2, 1], \
               strides = [1, 1, 2, 1], padding = 'SAME')
    
    shape = [1, 1, channel, n_units]
    with tf.variable_scope('initial'):
      y = self.conv(x, shape)
    
    # positional encoding
    pe = tf.expand_dims(self.positional_encoding( \
            batch_size, int(width / 2), n_units), axis = 1)
    pe = tf.cast(pe, dtype = tf.float32)
    y += pe

    inputs = y

    # for regular convolution
    #shape = [1, filter_size, n_units, n_units]
    # for separable convolution
    shape_d = [1, filter_size, n_units, 1]
    shape_p = [1, 1, n_units, n_units]
    for i in range(n_layers):
      with tf.variable_scope('layer_{}'.format(i + 1)):

        # Dilated
        y = tf.pad(y, [[0, 0], [0, 0], \
                 [2**i * (filter_size - 1), 0], [0, 0]])
        # Separable Convolution Without Weight Normalization
        y = self.dilated_gated_separable_conv(y, \
                 shape_d, shape_p, 2**i)
                
        if (i + 1) % residual_step == 0:
          y += inputs
          inputs = y
        
    y = y[:, :, -1, :]
    y = tf.squeeze(y, axis = 1)
    
    with tf.variable_scope('final'):
      w = self.weight_variable('w', [n_units, n_out])
      b = self.bias_variable('b', [n_out])
      
      y = tf.add(tf.matmul(y, w), b)
      y = tf.nn.softmax(y, axis = 1)
        
    return y

```