ゼロから作るDeep Learning Java編 6.3 Batch Normalization

# 目次
* [第1章 はじめに](https://qiita.com/saka1029/items/8c8b4bd17669e3da614a)
* [第2章 パーセプトロン](https://qiita.com/saka1029/items/bce542064073a2e6b404)
* [第3章 ニューラルネットワーク](https://qiita.com/saka1029/items/dcc40a9868d709fc6510)
* [第4章 ニューラルネットワークの学習](https://qiita.com/saka1029/items/39b13dac9f5fccd5b551)
* [第5章 誤差逆伝播法](https://qiita.com/saka1029/items/00aed8cdb557781d643a)
* 第6章 学習に関するテクニック
    * [6.1 パラメータの更新](https://qiita.com/saka1029/items/996853b91c54e0f49f09)
    * [6.2 重みの初期値](https://qiita.com/saka1029/items/d013d52deb79461855da)
    * [6.3 Batch Normalization](https://qiita.com/saka1029/items/be2645313b7e7f44e1c5)
    * [6.4 正則化](https://qiita.com/saka1029/items/36261c6784cf8ffd3ed2)
    * [6.5 ハイパーパラメータの検証](https://qiita.com/saka1029/items/1c54a5874811908659a4)
* [第7章 畳み込みニューラルネットワーク](https://qiita.com/saka1029/items/2807f76abcade7de61b4)
* [第8章 ディープラーニング](https://qiita.com/saka1029/items/7f3c0d2e503d6ca583cc)
* [最後に](https://qiita.com/saka1029/items/02175486de2213bfd5c2)

## 6.3 Batch Normalizationの評価

[Layer](https://github.com/saka1029/Deep.Learning/blob/master/src/main/java/deep/learning/common/Layer.java)インタフェースに対してtrain_flgを渡せるように拡張した[BatchNormLayer](https://github.com/saka1029/Deep.Learning/blob/master/src/main/java/deep/learning/common/BatchNormLayer.java)を作成しました。

```java:Layer.java
public interface Layer {

    INDArray forward(INDArray x);
    INDArray backward(INDArray x);

}
```

```java:BatchNormLayer.java
public interface BatchNormLayer extends Layer {

    public default INDArray forward(INDArray x) {
        throw new IllegalAccessError();
    }

    INDArray forward(INDArray x, boolean train_flg);

}
```
あとはこのインターフェースの実装である[Dropout](https://github.com/saka1029/Deep.Learning/blob/master/src/main/java/deep/learning/common/Dropout.java)クラスと[BatchNormalization](https://github.com/saka1029/Deep.Learning/blob/master/src/main/java/deep/learning/common/BatchNormalization.java)クラスを実装しました。
さらに[MultiLayerNetExtend](https://github.com/saka1029/Deep.Learning/blob/master/src/main/java/deep/learning/C6/MultiLayerNetExtend.java)クラスを実装し、以下のテストコードを実行します。

```java
INDArray x_train;
INDArray t_train;
int max_epochs = 20;
int train_size;
int batch_size = 100;
double learning_rate = 0.01;
DataSet trainDataSet;

List<List<Double>> __train(String weight_init_std) {
    MultiLayerNetExtend bn_network = new MultiLayerNetExtend(
        784, new int[] {100, 100, 100, 100, 100}, 10,
        /*activation=*/"relu",
        /*weight_init_std=*/ weight_init_std,
        /*weight_decay_lambda=*/ 0,
        /*use_dropout=*/ false,
        /*dropout_ration=*/ 0.5,
        /*use_batchNorm=*/ true);
    MultiLayerNetExtend network = new MultiLayerNetExtend(
        784, new int[] {100, 100, 100, 100, 100}, 10,
        /*activation=*/"relu",
        /*weight_init_std=*/ weight_init_std,
        /*weight_decay_lambda=*/ 0,
        /*use_dropout=*/ false,
        /*dropout_ration=*/ 0.5,
        /*use_batchNorm=*/ false);
    List<MultiLayerNetExtend> networks = Arrays.asList(bn_network, network);
    Optimizer optimizer = new SGD(learning_rate);
    List<Double> train_acc_list = new ArrayList<>();
    List<Double> bn_train_acc_lsit = new ArrayList<>();
    int iter_per_epoch = Math.max(train_size / batch_size, 1);
    int epoch_cnt = 0;
    for (int i = 0; i < 1000000000; ++i) {
        DataSet sample = trainDataSet.sample(batch_size);
        INDArray x_batch = sample.getFeatureMatrix();
        INDArray t_batch = sample.getLabels();
        for (MultiLayerNetExtend _network : networks) {
            Params grads = _network.gradient(x_batch, t_batch);
            optimizer.update(_network.params, grads);
        }
        if (i % iter_per_epoch == 0) {
            double train_acc = network.accuracy(x_train, t_train);
            double bn_train_acc = bn_network.accuracy(x_train, t_train);
            train_acc_list.add(train_acc);
            bn_train_acc_lsit.add(bn_train_acc);
            System.out.println("epoch:" + epoch_cnt + " | " + train_acc + " - " + bn_train_acc);
            ++epoch_cnt;
            if (epoch_cnt >= max_epochs)
                break;
        }
    }
    return Arrays.asList(train_acc_list, bn_train_acc_lsit);
}

@Test
public void C6_3_2_Batch_Normalizationの評価() throws IOException {
    // ch06/batch_norm_test.pyのJava版です。
    MNISTImages train = new MNISTImages(Constants.TrainImages, Constants.TrainLabels);
    x_train = train.normalizedImages();
    t_train = train.oneHotLabels();
    trainDataSet = new DataSet(x_train, t_train);
    train_size = x_train.size(0);

    // グラフの描画
    File dir = Constants.WeightImages;
    if (!dir.exists()) dir.mkdirs();
    String[] names = {"BatchNormalization", "Normal"};
    Color[] colors = {Color.BLUE, Color.RED};
    INDArray weight_scale_list = Functions.logspace(0, -4, 16);
    INDArray x = Functions.arrange(max_epochs);
    for (int i = 0; i < weight_scale_list.length(); ++i) {
        System.out.println( "============== " + (i+1) + "/16" + " ==============");
        double w = weight_scale_list.getDouble(i);
        List<List<Double>> acc_list = __train(String.valueOf(w));
        GraphImage graph = new GraphImage(640, 480, -1, -0.1, 20, 1.0);
        for (int j = 0; j < names.length; ++j) {
            graph.color(colors[j]);
            graph.textInt(names[j] + " : " + w, 20, 20 * j + 20);
            graph.plot(0, acc_list.get(j).get(0));
            for (int k = 1; k < acc_list.get(j).size(); ++k) {
                graph.line(k - 1, acc_list.get(j).get(k - 1), k, acc_list.get(j).get(k));
                graph.plot(k, acc_list.get(j).get(k));
            }
        }
        File file = new File(dir, "BatchNormalization#" + w + ".png");
        graph.writeTo(file);
    }
}
```

結果のグラフは以下のようになりました。

### W=1.0
![BatchNormalization#1.0.png](https://qiita-image-store.s3.amazonaws.com/0/78427/d5b4709a-5791-0e48-0ff5-9a76b2b0ed4f.png)


### W=0.29286444187164307
![BatchNormalization#0.29286444187164307.png](https://qiita-image-store.s3.amazonaws.com/0/78427/24bb4ff8-7208-0301-2103-55a137d768d1.png)

### W=0.00009999999747378752
![BatchNormalization#9.999999747378752E-5.png](https://qiita-image-store.s3.amazonaws.com/0/78427/49376e22-168c-7940-dd53-837e077a38c6.png)

本書とは違って、W（重みの初期値の標準偏差）が小さい程、学習の進み方が速いという結果になりました。