TensorFlowで input:100, output:100のネットワークでの学習を検討 v0.7 / v0.7 > 1000の英単語の正立画像から倒立画像への変換の学習 

```txt:動作環境
GeForce GTX 1070 (8GB)
ASRock Z170M Pro4S [Intel Z170chipset]
Ubuntu 14.04 LTS desktop amd64
TensorFlow v0.11
cuDNN v5.1 for Linux
CUDA v8.0
Python 2.7.6
IPython 5.1.0 -- An enhanced Interactive Python.
gcc (Ubuntu 4.8.4-2ubuntu1~14.04.3) 4.8.4
GNU bash, version 4.3.8(1)-release (x86_64-pc-linux-gnu)
```

前回 http://qiita.com/7of9/items/470b260b01c794562bd9
画素値をランダムに作成した10x10の倒立画像の学習を行ったが、誤差が0.1くらいから下がらず失敗。

画素値のランダム性が高すぎて、学習が難しいのだろうかと考えて、今回は英単語を画像化して、その正立画像をinput nodeに、倒立画像をoutput nodeにすることにした。

### データ準備

以下のスクリプトを用いて`test_in.csv`と`test_out.csv`を生成した。

関連 [bash > lookを用いて英単語を抽出し、正立画像と倒立画像を作成し、そのBlue成分をcsv出力する実装 > v0.1,v0.2](http://qiita.com/7of9/items/1a04107f8af1e38e3969)

```bash:prep_data_exec_170304
#!/usr/bin/env bash

#
# v0.3 Mar. 5, 2017
#   - change [IMG_SIZE] and [FONT_SIZE]
# v0.2 Mar. 5, 2017
#   - add extract_blue()
# v0.1 Mar. 4, 2017
#   - add [OUTPUT_INFILE], [OUTPUT_OUTFILE]
#   - add toCsvText()
#   - flip vertically the image
#   - make upright image for the word
#   - loop for all the words in the [look] dictionary (99,154 words)
#

cnt=1

# for TensorFlow Learning
IMG_SIZE=10x10
FONT_SIZE=6
# for debug
#IMG_SIZE=30x30
#FONT_SIZE=20
#
OUTPUT_INFILE="test_in.csv"
OUTPUT_OUTFILE="test_out.csv"

extract_blue() {
	# input: ( (R,G,B), (R,G,B), ... )
	((cnt=1))
	for aword in $@;do
		mdl=$((cnt % 3))
		if [ $mdl -eq 0 ];then  # (1:R, 2:G, 0:B)
			echo $aword
		fi
		((cnt++))
	done	
}

toCsvText() {
	convert -depth 8 -compress none $1 tmp.ppm
	res=$(extract_blue $(tail -n +4 tmp.ppm) | tr '\n' ' ' | sed 's/ /,/g')
	# remove comma at the end of the line
	echo $res | sed 's/,$//g'
}

rm -f $OUTPUT_INFILE
rm -f $OUTPUT_OUTFILE

for x in {a..z}; do
	list=$(look $x)
	for aword in $list; do
		echo $aword # for progress display

		# image for the word
		# - upright
		convert -background lightblue -fill blue -size $IMG_SIZE -gravity center -pointsize $FONT_SIZE label:$aword label_uprt.gif
		# - upside down
		convert label_uprt.gif -flip label_updn.gif		

		# to csv file
		toCsvText label_uprt.gif >> $OUTPUT_INFILE
		toCsvText label_updn.gif >> $OUTPUT_OUTFILE

		# --- exit for debug ---
		((cnt++))
		if [ $cnt -gt 1000 ]; then
			exit 0
		fi
	done
done
```

### 学習コード

```learn_in100out100.py
#!/usr/bin/env python
# -*- coding: utf-8 -*-
import sys
import tensorflow as tf
import tensorflow.contrib.slim as slim
import numpy as np

'''
v0.7 Feb. 22, 2017
    - treat each hidden layer to handle dropout instead of slim.stack()
v0.6 Feb. 19, 2017
    - add sys.stdout.flush() to immediately print
v0.5 Feb. 18, 2017
    - add fc_drop()
v0.4 Feb. 15, 2017
    - fix bug > ZeroDivisionError: float division by zero @ shuffle_batch()
v0.3 Feb. 15, 2017
    - tweak [batch_size] for shuffle_batch()
v0.2 Feb. 15, 2017
    - fix bug > different dimensions for placeholder and network
v0.1 Feb. 06, 2017
    - read [test_in.csv],[test_out.csv]
'''

'''
codingrule:PEP8
'''


def fc_drop(inputs, *args, **kwargs):
    # Thanks to: http://qiita.com/shngt/items/f532601b4f059ce8584f
    net = slim.fully_connected(inputs, *args, **kwargs)
    return slim.dropout(net, 0.9)

filename_inp = tf.train.string_input_producer(["test_in.csv"])
filename_out = tf.train.string_input_producer(["test_out.csv"])
NUM_INP_NODE = 100
NUM_OUT_NODE = 100

# parse csv
# a. input node
reader = tf.TextLineReader()
key, value = reader.read(filename_inp)
deflist = [[0.] for idx in range(NUM_INP_NODE)]
input1 = tf.decode_csv(value, record_defaults=deflist)
# b. output node
key, value = reader.read(filename_out)
deflist = [[0.] for idx in range(NUM_OUT_NODE)]
output1 = tf.decode_csv(value, record_defaults=deflist)
# c. pack
# inputs = tf.pack([input1])
inputs = input1
# outputs = tf.pack([output1])
outputs = output1

batch_size = 2
inputs_batch, output_batch = tf.train.shuffle_batch(
    [inputs, outputs], batch_size, capacity=10, min_after_dequeue=batch_size)

input_ph = tf.placeholder("float", [None, 100])
output_ph = tf.placeholder("float", [None, 100])

# network
# a. with stack
# hiddens = slim.stack(input_ph, slim.fully_connected, [7, 7, 7],
#                     activation_fn=tf.nn.sigmoid, scope="hidden")
# a2. larget network
hiddens = slim.stack(input_ph, slim.fully_connected, [100, 100, 100],
                     activation_fn=tf.nn.sigmoid, scope="hidden")
# b. without stack
# sgm = tf.nn.sigmoid
# hiddens = slim.fully_connected(input_ph, 7, activation_fn=sgm, scope='hidden1')
# hiddens = slim.fully_connected(hiddens, 7, activation_fn=sgm, scope='hidden2')
# hiddens = slim.fully_connected(hiddens, 7, activation_fn=sgm, scope='hidden3')
# c. without stack, with dropout
# sgm = tf.nn.sigmoid
# hiddens = slim.fully_connected(input_ph, 7, activation_fn=sgm, scope='hidden1')
# drpout = slim.stack(hiddens, fc_drop, [7, 7], scope='hd_fc1')
# hiddens = slim.fully_connected(drpout, 7, activation_fn=sgm, scope='hidden2')
# drpout = slim.stack(hiddens, fc_drop, [7, 7], scope='hd_fc2')
# hiddens = slim.fully_connected(drpout, 7, activation_fn=sgm, scope='hidden3')


# a. without dropout
prediction = slim.fully_connected(
    hiddens, 100, activation_fn=None, scope="output")
# b. with dropout
# drpout = slim.stack(hiddens, fc_drop, [100, 100], scope='fc')
# prediction = slim.fully_connected(
#     drpout, 100, activation_fn=None, scope="output")
loss = tf.contrib.losses.mean_squared_error(prediction, output_ph)

train_op = slim.learning.create_train_op(loss, tf.train.AdamOptimizer(0.001))

init_op = tf.initialize_all_variables()

with tf.Session() as sess:
    coord = tf.train.Coordinator()
    threads = tf.train.start_queue_runners(sess=sess, coord=coord)

    try:
        sess.run(init_op)
        for idx in range(10000):
            # for idx in range(100000):
            inpbt, outbt = sess.run([inputs_batch, output_batch])
            _, t_loss = sess.run(
                [train_op, loss],
                feed_dict={input_ph: inpbt, output_ph: outbt})

            if (idx+1) % 100 == 0:
                print("%d,%f" % (idx+1, t_loss))
                sys.stdout.flush()
    finally:
        coord.request_stop()

    coord.join(threads)
```

### 学習誤差の推移

Jupyter コード

```py:check_result_170215.ipynb
%matplotlib inline

'''
v0.2 Mar, 5, 2017
v0.1 Feb. 16, 2017
'''

import numpy as np
import matplotlib.pyplot as plt

# data1 = np.loadtxt('res.learn.170215', delimiter=',')
# data1 = np.loadtxt('res.learn.N=1000_170216', delimiter=',')
data1 = np.loadtxt('res.withStack_170222', delimiter=',')

# data2 = np.loadtxt('res.learn.N=1000_dropout@output_170218', delimiter=',')
# data2 = np.loadtxt('res.learn.N=1000_dropout@output_longrun_170218', delimiter=',')
# data2 = np.loadtxt('res.withoutStack_170222', delimiter=',')
#data2 = np.loadtxt('res.withoutStack_withDropout_170222', delimiter=',')
data2 = np.loadtxt('res.wordUpsideDown_1000word_170305', delimiter=',')

input1 = data1[:,0]
output1 = data1[:,1]
input2 = data2[:,0]
output2 = data2[:,1]

fig = plt.figure()
ax1 = fig.add_subplot(2,1,1)
ax2 = fig.add_subplot(2,1,2)

ax1.scatter(input1,output1)
ax2.scatter(input2,output2)


ax1.set_xlabel('step')
ax1.set_ylabel('Error')
ax1.set_xlim(0, 10000)
ax1.set_ylim(0, 0.2)
ax1.grid(True)

ax2.set_xlabel('step')
ax2.set_ylabel('Error')
ax2.set_xlim(0, 10000)
#ax2.set_ylim(0, 0.2)
ax2.grid(True)

fig.show()
```

下のグラフが今回の結果。

![qiita.png](https://qiita-image-store.s3.amazonaws.com/0/32870/2bb065c6-3b5b-8e79-ecb5-6eef3e546218.png)

Errorが小さくなっていき、学習が進んでいるようには見えるが、絶対Error値で10程度で学習が停滞している。ランダム画像の場合は0.1で、今回(英単語)では10なので、Errorが高くなった。

## 入力値を255で割ってみた

RGBのB成分は`[0,255]`の値になっている。これを255で割ってみて、`[0,1]`の範囲にするとどうなるか。

prep_data_exec_170304の

```
echo $aword
```

を以下とした。

```
echo "$aword/255" | bc -l
```

### 結果

Jupyterコード。

```py:check_result_170215.ipynb
%matplotlib inline

'''
v0.3 Mar, 5, 2017
v0.2 Mar, 5, 2017
v0.1 Feb. 16, 2017
'''

import numpy as np
import matplotlib.pyplot as plt

# data1 = np.loadtxt('res.learn.170215', delimiter=',')
# data1 = np.loadtxt('res.learn.N=1000_170216', delimiter=',')
#data1 = np.loadtxt('res.withStack_170222', delimiter=',')
data1 = np.loadtxt('res.wordUpsideDown_1000word_170305', delimiter=',')

# data2 = np.loadtxt('res.learn.N=1000_dropout@output_170218', delimiter=',')
# data2 = np.loadtxt('res.learn.N=1000_dropout@output_longrun_170218', delimiter=',')
# data2 = np.loadtxt('res.withoutStack_170222', delimiter=',')
#data2 = np.loadtxt('res.withoutStack_withDropout_170222', delimiter=',')
data2 = np.loadtxt('res.wordUpsideDown_1000word_normalized_170305', delimiter=',')

input1 = data1[:,0]
output1 = data1[:,1]
input2 = data2[:,0]
output2 = data2[:,1]

fig = plt.figure()
ax1 = fig.add_subplot(2,1,1)
ax2 = fig.add_subplot(2,1,2)

ax1.scatter(input1,output1)
ax2.scatter(input2,output2)


ax1.set_xlabel('step')
ax1.set_ylabel('Error')
ax1.set_xlim(0, 10000)
#ax1.set_ylim(0, 0.2)
ax1.grid(True)

ax2.set_xlabel('step')
ax2.set_ylabel('Error')
ax2.set_xlim(0, 10000)
ax2.set_ylim(0, 0.002)
ax2.grid(True)

fig.show()
```

下図において、上のグラフは`[0,255]`の値で学習させた場合。下のグラフは`[0,1]`の値で学習させた場合。

![qiita.png](https://qiita-image-store.s3.amazonaws.com/0/32870/d96ef6b5-ccf5-870a-d0de-08ed45910c2b.png)

下のグラフではError値が最初から小さいが、学習が進んでいる様子が見られない。

ネットワークに与える学習データ扱いの理解が不足している。
http://stats.stackexchange.com/questions/211436/why-do-we-normalize-images-by-subtracting-the-datasets-image-mean-and-not-the-c

hidden layerでsigmoid関数を使っていることが関係あるのかどうか。