JuliaのFlux.jlを使って論理回路を再現する

##動機
- __課題「深層学習をしてみろ」__
    - 処理が重いのは嫌だなあ
    - 論理回路なら素早くできるでしょ
    - せっかくだし前から興味あるJuliaでやろう

##モデル
- 入力：2
- 出力：3（AND, OR, XOR）
- 中間層：2
- その他はコード参照

##環境
```
$ sw_vers 
ProductName:	Mac OS X
ProductVersion:	10.14.2
BuildVersion:	18C54
$ julia --version
julia version 1.0.3
```

##ライブラリの準備
どちらか
- Shell: `julia -e 'using Pkg; Pkg.add("Flux")'`
- Julia: `using Pkg; Pkg.add("Flux")`

##コード
```dl.jl
println("start  load Packages")
using Printf
using Flux: Chain, Dense, relu, mse, SGD, params, train!
println("finish load Packages")

# X：入力  Y：正解出力  DATA：学習用の形式
const X = [0 0 1 1
           0 1 0 1]
const Y = [0 0 0 1  # AND
           0 1 1 1  # OR
           0 1 1 0] # XOR
const DATA = [(X, Y)]

# （エポック数、損失の桁数、損失）と出力チェック
function check(num, model, loss)
    println("$(num)\t$(floor(log10(loss(X,Y).data)))\t$(loss(X,Y).data)")
    for input in [[0,0],[0,1],[1,0],[1,1]]
        out = map(tr -> @sprintf("%.3f",tr.data), model(input))
        println("$(input) -> $(out)")
    end
end

# モデルを作成、epoch数だけ学習、showtime回テスト出力
function learn(epoch::Int=1000, showtime::Int=5)
    # # Chain(layers...)  Dense(入力数, 出力数, 活性化関数)
    model = Chain(Dense(2,3,relu),
                  Dense(3,3,relu),
                  Dense(3,3,relu))
    # loss：損失関数  opt：学習法
    loss(x,y) = mse(model(x),y)  # 自乗誤差 (MSE; Mean Squared Error)
    opt = SGD(params(model))  # 確率的勾配降下法(SGD; Stochastic Gradient Descent)

    check(0,model,loss)
    every = div(epoch,showtime-1)
    for num in 1:epoch
        train!(loss, DATA, opt)
        if num % every == 0
            check(num,model,loss)
        end
    end
end

# パッケージの読み込みに時間がかかるので、繰り返しできるようにする
function main()
    learn()
    while true
        print("""

        [Rr] retry
        [Qq] quit
        """)
        while (s = readline(); length(s) == 0)
        end
        c = s[1]
        if c=='R' || c== 'r'
            learn()
        elseif c=='Q' || c== 'q'
            return
        end
    end
end

main()

```

##結果
- 成功例

```
0	-1.0	0.47293654911172206
[0, 0] -> ["0.000", "0.000", "0.000"]
[0, 1] -> ["0.000", "0.112", "0.000"]
[1, 0] -> ["0.000", "0.000", "0.000"]
[1, 1] -> ["0.002", "0.057", "0.000"]
250	-2.0	0.030001683128810273
[0, 0] -> ["0.111", "0.291", "0.361"]
[0, 1] -> ["0.000", "1.002", "0.730"]
[1, 0] -> ["0.115", "0.810", "0.973"]
[1, 1] -> ["0.906", "1.032", "0.000"]
500	-4.0	0.00027232311165393993
[0, 0] -> ["0.019", "0.035", "0.026"]
[0, 1] -> ["0.000", "1.014", "0.986"]
[1, 0] -> ["0.001", "0.979", "0.999"]
[1, 1] -> ["0.995", "0.990", "0.000"]
750	-6.0	2.0916321222147786e-6
[0, 0] -> ["0.001", "0.004", "0.001"]
[0, 1] -> ["0.000", "1.001", "0.999"]
[1, 0] -> ["0.000", "0.998", "1.000"]
[1, 1] -> ["1.000", "0.999", "0.000"]
1000	-8.0	2.83701935891779e-8
[0, 0] -> ["0.000", "0.001", "0.000"]
[0, 1] -> ["0.000", "1.000", "1.000"]
[1, 0] -> ["0.000", "1.000", "1.000"]
[1, 1] -> ["1.000", "1.000", "0.000"]
```

- 失敗例

```
0	-1.0	0.37417270318834966
[0, 0] -> ["0.000", "0.000", "0.000"]
[0, 1] -> ["0.000", "0.260", "0.151"]
[1, 0] -> ["0.157", "0.000", "0.158"]
[1, 1] -> ["0.000", "0.327", "0.191"]
250	-1.0	0.17150528384173427
[0, 0] -> ["0.000", "0.043", "0.077"]
[0, 1] -> ["0.000", "1.115", "0.926"]
[1, 0] -> ["0.010", "0.000", "0.951"]
[1, 1] -> ["0.000", "0.850", "0.080"]
500	-1.0	0.1667269386356042
[0, 0] -> ["0.000", "0.008", "0.011"]
[0, 1] -> ["0.000", "1.014", "0.996"]
[1, 0] -> ["0.001", "0.000", "0.995"]
[1, 1] -> ["0.000", "0.983", "0.004"]
750	-1.0	0.16666717825853591
[0, 0] -> ["0.000", "0.001", "0.001"]
[0, 1] -> ["0.000", "1.001", "1.000"]
[1, 0] -> ["0.000", "0.000", "1.000"]
[1, 1] -> ["0.000", "0.998", "0.000"]
1000	-1.0	0.16666667087303277
[0, 0] -> ["0.000", "0.000", "0.000"]
[0, 1] -> ["0.000", "1.000", "1.000"]
[1, 0] -> ["0.000", "0.000", "1.000"]
[1, 1] -> ["0.000", "1.000", "0.000"]
```
- モデルを100個作って損失を見る
![gurahu.png](https://qiita-image-store.s3.amazonaws.com/0/186959/1c04e8f2-fe1f-2ae6-b6a2-472278571704.png)

##重みの保存と読み込み
```save.jl
using Flux: Tracker, cpu
using BSON: @save
weights = Tracker.data.(params(cpu(model)))
@save "weights.bson" weights
```

```load.jl
using Flux: loadparams!
using BSON: @load
@load "good.bson" weights
loadparams!(model, weights)
```

##感想
- [FluxML/model-zoo](https://github.com/FluxML/model-zoo/)がとても参考になる
- Juliaはライブラリの読み込みが重くて少々辛い
    - 一度読み込めばその後は快適なので、REPLかなんかでやるべき
- 論理回路でこんなにも結果が踊らされるとは予想外
    - 重みの初期値やモデルの設計はちゃんとしないと駄目っぽい
    - 学習に失敗するケースが多すぎる
    - どうしたら初期乱数に左右されない安定した学習ができるのか？
- 最後の図は`Plots`パッケージを使って用意したかったが、楽をするためにExcelを使用してしまった