【ラビットチャレンジ】【機械学習】非線形回帰モデル

「ラビットチャレンジ」 提出レポート
#１．非線形回帰モデル
複数な非線形構造を内在する現象に対して、非線形回帰モデリングを実施する必要がある。
■　**基底展開法**
　●　回帰関数として、既定関数法と呼ばれる既知の非線形関数とパラメータベクトルの線形結合を使用する。
　●　未知パラメータは、線形回帰モデルと同様に最小二乗法や最尤法による推定する。
　●　よく使われる基底関数：多項式関数、ガウス型基底関数、スプライン関数。
■　**未学習**
　●　学習データに対して、十分小さいな誤差が得られないモデルという。
　●　対策：表現力の高いモデルを利用する。
■　**過学習**
　●　学習データに対して、小さな誤差が得られたが、検証データに対して、誤差が大きくなってしまう。
　●　対策：学習データの数を増やすや正則法を利用する。
#２．正則化法
機械学習では、過学習を防ぐ手段として正則化が行われることが多い。L1正則化とL2正則化をよく用いられる正則化です。
■　**$L1$ 正則化**
　●　ペナルティとして学習モデルのパラメータの絶対値の総和（$L1$ノルム）を用いるものである。
　●　特定のデータの重みを０にする事で、不要なデータを削除するので、次元圧縮のために用いられる。
■　**$L2$ 正則化**
　●　ペナルティとして学習モデルのパラメータの二乗の総和（$L2$ノルム）を用いるものである。
　●　データの大きさに応じて0に近づけて、滑らかなモデルとするので、モデルの過学習を防ぐことで精度を高めるために用いられる。
#３．モデル選択
適切な予測モデルを構築するためには、モデルの汎化性能評価が不可欠です。**ホールドアウト法（Holdout）**と**交差検証法（Cross Validation)**はモデルの性能評価の手法とする。
■　**ホールドアウト法（Holdout）**
　●　データセットを訓練データとテストデータに分割し、訓練データを使って学習モデルを構築し、テストデータを入力した時の出力を答え合わせすることで予測精度を検証する。
　●　問題点：データの分割の仕方によって性能評価に影響が出る可能性がある。
■　**交差検証法（Cross Validation)**
　●　データの分割を何度も繰り返して、複数のモデルを訓練し、一般に最後は複数のモデルの平均値を取って最終的な性能とする。
　●　ホールドアウト法より、評価結果の信頼が高くなる。
#４．実装演習結果
###【実装演習】
■　**必要モジュールインポートとデータ関数設定**
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/552833/1af991ff-7ad4-ce17-bc4c-311e8f0e24af.png)
■　**真の関数からノイズを伴うデータを生成**
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/552833/469a14e4-ec3d-b8f0-1595-20160a1eabdd.png)
■　**線形回帰モデルより予測**
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/552833/4cbb44fb-ffc8-f773-bd1c-5a12fe073827.png)
**【考察】**上記の結果によると、線形回帰を求めるsklearn.linear_modelモジュールからLinearRegressionのクラスを使用すると、正解率が低いです。
■　**KernelRidge**
Kernel Ridge Regression は $L2$ 制約付き最小二乗学習です。
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/552833/4c7d1b08-3d4c-b214-e727-74ceaadcc5b8.png)
■　**RBFカーネルによる最小二乗学習**
**#Ridge**
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/552833/97b219e9-4e71-37f8-3944-c5b07429f407.png)
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/552833/491679ca-8aef-a1ee-9a19-a5766e8075ae.png)
**【考察】**sklearn.linear_modelモジュールからRidgeのクラスを使用すると、パラメータalphaの設定が必要となる。正解率は0.83で、線形モデルより改善された。
**#Lasso**
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/552833/eca01291-96fa-6bd2-f3d3-9f7e9991ebe8.png)
■　**Support Vector Regression**
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/552833/5f1e713b-c9a6-0958-60e3-a9018fa9575a.png)
■　**NonLiner Regressions via DL by ReLU**
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/552833/d44865b2-7473-74f3-d0a1-7af74e95157c.png)
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/552833/91be258f-664e-f2a3-eb13-712db1dcee8f.png)

```html:"Training"
Train on 90 samples, validate on 10 samples
Epoch 1/100
90/90 [==============================] - 1s 9ms/step - loss: 1.5399 - val_loss: 1.0177

Epoch 00001: saving model to ./weights.01-1.02.hdf5
Epoch 2/100
90/90 [==============================] - 0s 2ms/step - loss: 1.3654 - val_loss: 0.7789

Epoch 00002: saving model to ./weights.02-0.78.hdf5
Epoch 3/100
90/90 [==============================] - 0s 2ms/step - loss: 0.8682 - val_loss: 0.8151

Epoch 00003: saving model to ./weights.03-0.82.hdf5
Epoch 4/100
90/90 [==============================] - 0s 2ms/step - loss: 0.9923 - val_loss: 0.6966

Epoch 00004: saving model to ./weights.04-0.70.hdf5
Epoch 5/100
90/90 [==============================] - 0s 3ms/step - loss: 0.8529 - val_loss: 0.5245

Epoch 00005: saving model to ./weights.05-0.52.hdf5
Epoch 6/100
90/90 [==============================] - 0s 2ms/step - loss: 0.7379 - val_loss: 0.4245

Epoch 00006: saving model to ./weights.06-0.42.hdf5
Epoch 7/100
90/90 [==============================] - 0s 2ms/step - loss: 0.6347 - val_loss: 0.4024

Epoch 00007: saving model to ./weights.07-0.40.hdf5
Epoch 8/100
90/90 [==============================] - 0s 2ms/step - loss: 0.7075 - val_loss: 0.4524

Epoch 00008: saving model to ./weights.08-0.45.hdf5
Epoch 9/100
90/90 [==============================] - 0s 2ms/step - loss: 0.6787 - val_loss: 0.3682

Epoch 00009: saving model to ./weights.09-0.37.hdf5
Epoch 10/100
90/90 [==============================] - 0s 2ms/step - loss: 0.5505 - val_loss: 0.4591

Epoch 00010: saving model to ./weights.10-0.46.hdf5
Epoch 11/100
90/90 [==============================] - 0s 3ms/step - loss: 1.0630 - val_loss: 0.5899

Epoch 00011: saving model to ./weights.11-0.59.hdf5
Epoch 12/100
90/90 [==============================] - 0s 2ms/step - loss: 0.6482 - val_loss: 0.3552

Epoch 00012: saving model to ./weights.12-0.36.hdf5
Epoch 13/100
90/90 [==============================] - 0s 3ms/step - loss: 0.9105 - val_loss: 0.4476

Epoch 00013: saving model to ./weights.13-0.45.hdf5
Epoch 14/100
90/90 [==============================] - 0s 2ms/step - loss: 0.6084 - val_loss: 0.5306

Epoch 00014: saving model to ./weights.14-0.53.hdf5
Epoch 15/100
90/90 [==============================] - 0s 2ms/step - loss: 0.7093 - val_loss: 0.5703

Epoch 00015: saving model to ./weights.15-0.57.hdf5
Epoch 16/100
90/90 [==============================] - 0s 3ms/step - loss: 0.5859 - val_loss: 0.2589

Epoch 00016: saving model to ./weights.16-0.26.hdf5
Epoch 17/100
90/90 [==============================] - 0s 2ms/step - loss: 0.5765 - val_loss: 0.3469

Epoch 00017: saving model to ./weights.17-0.35.hdf5
Epoch 18/100
90/90 [==============================] - 0s 2ms/step - loss: 0.5619 - val_loss: 0.4871

Epoch 00018: saving model to ./weights.18-0.49.hdf5
Epoch 19/100
90/90 [==============================] - 0s 2ms/step - loss: 0.6217 - val_loss: 0.7794

Epoch 00019: saving model to ./weights.19-0.78.hdf5
Epoch 20/100
90/90 [==============================] - 0s 2ms/step - loss: 0.8317 - val_loss: 0.3086

Epoch 00020: saving model to ./weights.20-0.31.hdf5
Epoch 21/100
90/90 [==============================] - 0s 2ms/step - loss: 0.6066 - val_loss: 0.3045

Epoch 00021: saving model to ./weights.21-0.30.hdf5
Epoch 22/100
90/90 [==============================] - 0s 2ms/step - loss: 0.5178 - val_loss: 0.3959

Epoch 00022: saving model to ./weights.22-0.40.hdf5
Epoch 23/100
90/90 [==============================] - 0s 2ms/step - loss: 0.5563 - val_loss: 0.3050

Epoch 00023: saving model to ./weights.23-0.30.hdf5
Epoch 24/100
90/90 [==============================] - 0s 3ms/step - loss: 0.4811 - val_loss: 0.3663

Epoch 00024: saving model to ./weights.24-0.37.hdf5
Epoch 25/100
90/90 [==============================] - 0s 3ms/step - loss: 0.4220 - val_loss: 0.2775

Epoch 00025: saving model to ./weights.25-0.28.hdf5
Epoch 26/100
90/90 [==============================] - 0s 3ms/step - loss: 0.4845 - val_loss: 0.3477

Epoch 00026: saving model to ./weights.26-0.35.hdf5
Epoch 27/100
90/90 [==============================] - 0s 3ms/step - loss: 0.4824 - val_loss: 0.4453

Epoch 00027: saving model to ./weights.27-0.45.hdf5
Epoch 28/100
90/90 [==============================] - 0s 2ms/step - loss: 0.4690 - val_loss: 0.2465

Epoch 00028: saving model to ./weights.28-0.25.hdf5
Epoch 29/100
90/90 [==============================] - 0s 2ms/step - loss: 0.4113 - val_loss: 0.2765

Epoch 00029: saving model to ./weights.29-0.28.hdf5
Epoch 30/100
90/90 [==============================] - 0s 2ms/step - loss: 0.3957 - val_loss: 0.2737

Epoch 00030: saving model to ./weights.30-0.27.hdf5
Epoch 31/100
90/90 [==============================] - 0s 3ms/step - loss: 0.3699 - val_loss: 0.2247

Epoch 00031: saving model to ./weights.31-0.22.hdf5
Epoch 32/100
90/90 [==============================] - 0s 3ms/step - loss: 0.3853 - val_loss: 0.3053

Epoch 00032: saving model to ./weights.32-0.31.hdf5
Epoch 33/100
90/90 [==============================] - 0s 2ms/step - loss: 0.5317 - val_loss: 0.9632

Epoch 00033: saving model to ./weights.33-0.96.hdf5
Epoch 34/100
90/90 [==============================] - 0s 2ms/step - loss: 0.5188 - val_loss: 0.4550

Epoch 00034: saving model to ./weights.34-0.46.hdf5
Epoch 35/100
90/90 [==============================] - 0s 2ms/step - loss: 0.4946 - val_loss: 0.3305

Epoch 00035: saving model to ./weights.35-0.33.hdf5
Epoch 36/100
90/90 [==============================] - 0s 3ms/step - loss: 0.4418 - val_loss: 0.2938

Epoch 00036: saving model to ./weights.36-0.29.hdf5
Epoch 37/100
90/90 [==============================] - 0s 3ms/step - loss: 0.4241 - val_loss: 0.3461

Epoch 00037: saving model to ./weights.37-0.35.hdf5
Epoch 38/100
90/90 [==============================] - 0s 2ms/step - loss: 0.3522 - val_loss: 0.2416

Epoch 00038: saving model to ./weights.38-0.24.hdf5
Epoch 39/100
90/90 [==============================] - 0s 2ms/step - loss: 0.3410 - val_loss: 0.2154

Epoch 00039: saving model to ./weights.39-0.22.hdf5
Epoch 40/100
90/90 [==============================] - 0s 2ms/step - loss: 0.3501 - val_loss: 0.4063

Epoch 00040: saving model to ./weights.40-0.41.hdf5
Epoch 41/100
90/90 [==============================] - 0s 2ms/step - loss: 0.3530 - val_loss: 0.2538

Epoch 00041: saving model to ./weights.41-0.25.hdf5
Epoch 42/100
90/90 [==============================] - 0s 3ms/step - loss: 0.3726 - val_loss: 0.1919

Epoch 00042: saving model to ./weights.42-0.19.hdf5
Epoch 43/100
90/90 [==============================] - 0s 2ms/step - loss: 0.3455 - val_loss: 0.1800

Epoch 00043: saving model to ./weights.43-0.18.hdf5
Epoch 44/100
90/90 [==============================] - 0s 2ms/step - loss: 0.3373 - val_loss: 0.2227

Epoch 00044: saving model to ./weights.44-0.22.hdf5
Epoch 45/100
90/90 [==============================] - 0s 2ms/step - loss: 0.3095 - val_loss: 0.2392

Epoch 00045: saving model to ./weights.45-0.24.hdf5
Epoch 46/100
90/90 [==============================] - 0s 3ms/step - loss: 0.3631 - val_loss: 0.1921

Epoch 00046: saving model to ./weights.46-0.19.hdf5
Epoch 47/100
90/90 [==============================] - 0s 3ms/step - loss: 0.3119 - val_loss: 0.2159

Epoch 00047: saving model to ./weights.47-0.22.hdf5
Epoch 48/100
90/90 [==============================] - 0s 2ms/step - loss: 0.3129 - val_loss: 0.1837

Epoch 00048: saving model to ./weights.48-0.18.hdf5
Epoch 49/100
90/90 [==============================] - 0s 2ms/step - loss: 0.2856 - val_loss: 0.2100

Epoch 00049: saving model to ./weights.49-0.21.hdf5
Epoch 50/100
90/90 [==============================] - 0s 2ms/step - loss: 0.2907 - val_loss: 0.1866

Epoch 00050: saving model to ./weights.50-0.19.hdf5
Epoch 51/100
90/90 [==============================] - 0s 2ms/step - loss: 0.3109 - val_loss: 0.2164

Epoch 00051: saving model to ./weights.51-0.22.hdf5
Epoch 52/100
90/90 [==============================] - 0s 3ms/step - loss: 0.3558 - val_loss: 0.2273

Epoch 00052: saving model to ./weights.52-0.23.hdf5
Epoch 53/100
90/90 [==============================] - 0s 3ms/step - loss: 0.3120 - val_loss: 0.2116

Epoch 00053: saving model to ./weights.53-0.21.hdf5
Epoch 54/100
90/90 [==============================] - 0s 2ms/step - loss: 0.3400 - val_loss: 0.2549

Epoch 00054: saving model to ./weights.54-0.25.hdf5
Epoch 55/100
90/90 [==============================] - 0s 2ms/step - loss: 0.3027 - val_loss: 0.1980

Epoch 00055: saving model to ./weights.55-0.20.hdf5
Epoch 56/100
90/90 [==============================] - 0s 2ms/step - loss: 0.3048 - val_loss: 0.1839

Epoch 00056: saving model to ./weights.56-0.18.hdf5
Epoch 57/100
90/90 [==============================] - 0s 2ms/step - loss: 0.3008 - val_loss: 0.2261

Epoch 00057: saving model to ./weights.57-0.23.hdf5
Epoch 58/100
90/90 [==============================] - 0s 2ms/step - loss: 0.2925 - val_loss: 0.1979

Epoch 00058: saving model to ./weights.58-0.20.hdf5
Epoch 59/100
90/90 [==============================] - 0s 2ms/step - loss: 0.3516 - val_loss: 0.2349

Epoch 00059: saving model to ./weights.59-0.23.hdf5
Epoch 60/100
90/90 [==============================] - 0s 2ms/step - loss: 0.3265 - val_loss: 0.2879

Epoch 00060: saving model to ./weights.60-0.29.hdf5
Epoch 61/100
90/90 [==============================] - 0s 2ms/step - loss: 0.3820 - val_loss: 0.2063

Epoch 00061: saving model to ./weights.61-0.21.hdf5
Epoch 62/100
90/90 [==============================] - 0s 2ms/step - loss: 0.4572 - val_loss: 0.2729

Epoch 00062: saving model to ./weights.62-0.27.hdf5
Epoch 63/100
90/90 [==============================] - 0s 2ms/step - loss: 0.3665 - val_loss: 0.1768

Epoch 00063: saving model to ./weights.63-0.18.hdf5
Epoch 64/100
90/90 [==============================] - 0s 2ms/step - loss: 0.2842 - val_loss: 0.2659

Epoch 00064: saving model to ./weights.64-0.27.hdf5
Epoch 65/100
90/90 [==============================] - 0s 3ms/step - loss: 0.2819 - val_loss: 0.3310

Epoch 00065: saving model to ./weights.65-0.33.hdf5
Epoch 66/100
90/90 [==============================] - 0s 3ms/step - loss: 0.4116 - val_loss: 0.1841

Epoch 00066: saving model to ./weights.66-0.18.hdf5
Epoch 67/100
90/90 [==============================] - 0s 2ms/step - loss: 0.3487 - val_loss: 0.3098

Epoch 00067: saving model to ./weights.67-0.31.hdf5
Epoch 68/100
90/90 [==============================] - 0s 3ms/step - loss: 0.3119 - val_loss: 0.2996

Epoch 00068: saving model to ./weights.68-0.30.hdf5
Epoch 69/100
90/90 [==============================] - 0s 2ms/step - loss: 0.3584 - val_loss: 0.2107

Epoch 00069: saving model to ./weights.69-0.21.hdf5
Epoch 70/100
90/90 [==============================] - 0s 2ms/step - loss: 0.3264 - val_loss: 0.2875

Epoch 00070: saving model to ./weights.70-0.29.hdf5
Epoch 71/100
90/90 [==============================] - 0s 3ms/step - loss: 0.3813 - val_loss: 0.2908

Epoch 00071: saving model to ./weights.71-0.29.hdf5
Epoch 72/100
90/90 [==============================] - 0s 2ms/step - loss: 0.2980 - val_loss: 0.3795

Epoch 00072: saving model to ./weights.72-0.38.hdf5
Epoch 73/100
90/90 [==============================] - 0s 2ms/step - loss: 0.3139 - val_loss: 0.1953

Epoch 00073: saving model to ./weights.73-0.20.hdf5
Epoch 74/100
90/90 [==============================] - 0s 2ms/step - loss: 0.3133 - val_loss: 0.2336

Epoch 00074: saving model to ./weights.74-0.23.hdf5
Epoch 75/100
90/90 [==============================] - 0s 2ms/step - loss: 0.3332 - val_loss: 0.2375

Epoch 00075: saving model to ./weights.75-0.24.hdf5
Epoch 76/100
90/90 [==============================] - 0s 3ms/step - loss: 0.3013 - val_loss: 0.2362

Epoch 00076: saving model to ./weights.76-0.24.hdf5
Epoch 77/100
90/90 [==============================] - 0s 2ms/step - loss: 0.2725 - val_loss: 0.1961

Epoch 00077: saving model to ./weights.77-0.20.hdf5
Epoch 78/100
90/90 [==============================] - 0s 3ms/step - loss: 0.3080 - val_loss: 0.2879

Epoch 00078: saving model to ./weights.78-0.29.hdf5
Epoch 79/100
90/90 [==============================] - 0s 2ms/step - loss: 0.3290 - val_loss: 0.2109

Epoch 00079: saving model to ./weights.79-0.21.hdf5
Epoch 80/100
90/90 [==============================] - 0s 2ms/step - loss: 0.3404 - val_loss: 0.3119

Epoch 00080: saving model to ./weights.80-0.31.hdf5
Epoch 81/100
90/90 [==============================] - 0s 2ms/step - loss: 0.3082 - val_loss: 0.2819

Epoch 00081: saving model to ./weights.81-0.28.hdf5
Epoch 82/100
90/90 [==============================] - 0s 2ms/step - loss: 0.3588 - val_loss: 0.2166

Epoch 00082: saving model to ./weights.82-0.22.hdf5
Epoch 83/100
90/90 [==============================] - 0s 2ms/step - loss: 0.3540 - val_loss: 0.3222

Epoch 00083: saving model to ./weights.83-0.32.hdf5
Epoch 84/100
90/90 [==============================] - 0s 2ms/step - loss: 0.3680 - val_loss: 0.2149

Epoch 00084: saving model to ./weights.84-0.21.hdf5
Epoch 85/100
90/90 [==============================] - 0s 2ms/step - loss: 0.2883 - val_loss: 0.1962

Epoch 00085: saving model to ./weights.85-0.20.hdf5
Epoch 86/100
90/90 [==============================] - 0s 2ms/step - loss: 0.2764 - val_loss: 0.2149

Epoch 00086: saving model to ./weights.86-0.21.hdf5
Epoch 87/100
90/90 [==============================] - 0s 2ms/step - loss: 0.3137 - val_loss: 0.2827

Epoch 00087: saving model to ./weights.87-0.28.hdf5
Epoch 88/100
90/90 [==============================] - 0s 2ms/step - loss: 0.3411 - val_loss: 0.2367

Epoch 00088: saving model to ./weights.88-0.24.hdf5
Epoch 89/100
90/90 [==============================] - 0s 2ms/step - loss: 0.3253 - val_loss: 0.3265

Epoch 00089: saving model to ./weights.89-0.33.hdf5
Epoch 90/100
90/90 [==============================] - 0s 2ms/step - loss: 0.3212 - val_loss: 0.2589

Epoch 00090: saving model to ./weights.90-0.26.hdf5
Epoch 91/100
90/90 [==============================] - 0s 2ms/step - loss: 0.3255 - val_loss: 0.2383

Epoch 00091: saving model to ./weights.91-0.24.hdf5
Epoch 92/100
90/90 [==============================] - 0s 2ms/step - loss: 0.2798 - val_loss: 0.3637

Epoch 00092: saving model to ./weights.92-0.36.hdf5
Epoch 93/100
90/90 [==============================] - 0s 2ms/step - loss: 0.3192 - val_loss: 0.2963

Epoch 00093: saving model to ./weights.93-0.30.hdf5
Epoch 94/100
90/90 [==============================] - 0s 2ms/step - loss: 0.3099 - val_loss: 0.2598

Epoch 00094: saving model to ./weights.94-0.26.hdf5
Epoch 95/100
90/90 [==============================] - 0s 2ms/step - loss: 0.3259 - val_loss: 0.2751

Epoch 00095: saving model to ./weights.95-0.28.hdf5
Epoch 96/100
90/90 [==============================] - 0s 3ms/step - loss: 0.2871 - val_loss: 0.2547

Epoch 00096: saving model to ./weights.96-0.25.hdf5
Epoch 97/100
90/90 [==============================] - 0s 2ms/step - loss: 0.3404 - val_loss: 0.2437

Epoch 00097: saving model to ./weights.97-0.24.hdf5
Epoch 98/100
90/90 [==============================] - 0s 2ms/step - loss: 0.2997 - val_loss: 0.2109

Epoch 00098: saving model to ./weights.98-0.21.hdf5
Epoch 99/100
90/90 [==============================] - 0s 2ms/step - loss: 0.3539 - val_loss: 0.2640

Epoch 00099: saving model to ./weights.99-0.26.hdf5
Epoch 100/100
90/90 [==============================] - 0s 2ms/step - loss: 0.3389 - val_loss: 0.1863

Epoch 00100: saving model to ./weights.100-0.19.hdf5
```
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/552833/5ba696f1-f8f8-7b42-2476-5bfba2d84b34.png)

##【機械学習】レポート一覧
[【ラビットチャレンジ】【機械学習】線形回帰モデル](https://qiita.com/HienDT27/items/0cbed5d1771b13d0311c)
[【ラビットチャレンジ】【機械学習】ロジスティク回帰モデル](https://qiita.com/HienDT27/items/cd067099c970a9786a47)
[【ラビットチャレンジ】【機械学習】主成分分析](https://qiita.com/HienDT27/items/bd53678695d33a6f8ee5)
[【ラビットチャレンジ】【機械学習】アルゴリズム](https://qiita.com/HienDT27/items/4731d66f31e0d247caeb)
[【ラビットチャレンジ】【機械学習】サポートベクターマシン](https://qiita.com/HienDT27/items/aa7317a8394212940068)