Deep Learningの重みとバイアスの初期値を比較した

# 初期値が気になった
深層学習だとよくハイパーパラメータが大切なんて聞きますが、重みとかの初期値ってどうなんでしょう。相当重要な気がしますが・・・。
調べてみて軽く比較してみました。考察とかはしないので各自で解釈していただければ。（そして教えていただければ・・・）

# 比較対象
- 活性化関数
    - ReLU
    - Sigmoid
- 重み
    - He
    - Xavier
- バイアス
    - np.zeros
    - np.rand
    - np.randn

# 使用するデータ
ゼロから作るDeep Learning2に載ってたやつ
![Figure_1.png](https://qiita-image-store.s3.amazonaws.com/0/305208/e1fdf5f0-3ddd-bc48-d144-f290714dbdae.png)

# 比較結果
###he, zeros
#####relu
![relu_he_zeros.png](https://qiita-image-store.s3.amazonaws.com/0/305208/f3c20307-7ea8-9f46-cb83-12d6b8a72426.png)
![relu_he_zeros_fig.png](https://qiita-image-store.s3.amazonaws.com/0/305208/e6cdc28a-f304-1d14-c0c3-995262a16140.png)

#####sigmoid
![sig_he_zeros.png](https://qiita-image-store.s3.amazonaws.com/0/305208/a1b06394-0516-dacd-30bf-cd2ea4eb9c84.png)
![sig_he_zeros_fig.png](https://qiita-image-store.s3.amazonaws.com/0/305208/7419043e-5f52-f67b-ccd5-c904eb353ec6.png)

###xavier, zeros
#####relu
![relu_xavier_zeros.png](https://qiita-image-store.s3.amazonaws.com/0/305208/9910ba71-bcfc-7620-938a-4c6b85c8b6b4.png)
![relu_xavier_zeros_fig.png](https://qiita-image-store.s3.amazonaws.com/0/305208/786231a6-e349-cd72-3091-045d922f6756.png)

#####sigmoid
![sig_xavier_zeros.png](https://qiita-image-store.s3.amazonaws.com/0/305208/c5c9627a-99f8-824f-23ca-7c35e5452add.png)
![sig_xavier_zeros_fig.png](https://qiita-image-store.s3.amazonaws.com/0/305208/b5b57580-e5d3-0bd7-68c0-82cbc702f893.png)

###he, uni
#####relu
![relu_he_uni.png](https://qiita-image-store.s3.amazonaws.com/0/305208/82a47ff9-1848-ccdd-f5cd-e863b5c8d779.png)
![relu_he_uni_fig.png](https://qiita-image-store.s3.amazonaws.com/0/305208/ca78cc16-a412-2e55-ed51-95b37c56494f.png)

#####sigmoid
![sig_he_uni.png](https://qiita-image-store.s3.amazonaws.com/0/305208/4289d1ae-bbbc-5e1c-e185-d1fc791c41fc.png)
![sig_he_uni_fig.png](https://qiita-image-store.s3.amazonaws.com/0/305208/2033aa6b-dc7a-62a0-a616-14d287b5d4c0.png)

###he, randn
#####relu
![relu_he_nor.png](https://qiita-image-store.s3.amazonaws.com/0/305208/216d73c9-cd6b-3d95-5b39-792bd7420efd.png)
![relu_he_nor_fig.png](https://qiita-image-store.s3.amazonaws.com/0/305208/9073a569-f01c-d110-0f96-26ed7c27bbf9.png)

#####sigmoid
![sig_he_nor.png](https://qiita-image-store.s3.amazonaws.com/0/305208/59c4c791-e7f6-6134-eabd-bfe055b8847d.png)
![sig_he_nor_fig.png](https://qiita-image-store.s3.amazonaws.com/0/305208/987b5322-a273-b0be-e56d-c0006582126e.png)

###he, rand
#####relu
![r.png](https://qiita-image-store.s3.amazonaws.com/0/305208/ace05614-9ffe-f8a1-10f3-24bbcafeec22.png)
![rf.png](https://qiita-image-store.s3.amazonaws.com/0/305208/03d70ec0-4cbf-b303-1cdb-be98314b0158.png)

#####sigmoid
![s.png](https://qiita-image-store.s3.amazonaws.com/0/305208/575e1c5d-e3ed-715c-4775-1bff23a22c15.png)
![sf.png](https://qiita-image-store.s3.amazonaws.com/0/305208/df0d469d-524e-eb8b-1eb4-6144011086c6.png)



###xavier, randn
#####relu
![relu_x_nor.png](https://qiita-image-store.s3.amazonaws.com/0/305208/2136ad90-2f07-b121-aaf8-25f87fa20662.png)
![relu_x_nor_fig.png](https://qiita-image-store.s3.amazonaws.com/0/305208/f65be71f-ece5-0a92-9b09-d698d79836ae.png)

#####sigmoid
![sig_x_nor.png](https://qiita-image-store.s3.amazonaws.com/0/305208/543e7782-f1ad-95fc-635d-7f42523f04fa.png)
![sig_x_nor_fig.png](https://qiita-image-store.s3.amazonaws.com/0/305208/f4d89886-82bf-b8d0-d23a-987d4702bde0.png)

###xavier, rand
#####relu
![relu_x_uni.png](https://qiita-image-store.s3.amazonaws.com/0/305208/bb2cd81f-b453-a2ad-f81e-af9a4ea523dc.png)
![relu_x_uni_fig.png](https://qiita-image-store.s3.amazonaws.com/0/305208/6e2c5558-668c-54aa-6f5b-b8a6250dc6f9.png)

#####sigmoid
![sig_x_uni.png](https://qiita-image-store.s3.amazonaws.com/0/305208/b05b6389-bc53-d73d-b08c-0e76a182c11a.png)
![sig_x_uni_fig.png](https://qiita-image-store.s3.amazonaws.com/0/305208/e3de2a93-831e-6f51-8789-68207e89b543.png)

# 感想
とりあえずhe, zerosにしとけばいい感じですかね。
それにしてもバイアスで結構変わるものですね、正直舐めてました。