TabNetを使う（回帰編）

テーブルデータの機械学習モデルとして注目されている TabNet なるものを、私も使ってみようとしました。まずは「回帰編」としてお届けします。

# TabNet インストール


```python
!pip install git+https://github.com/dreamquark-ai/tabnet
```


#  「ワインの品質」データ読み込み

データは <a href="http://archive.ics.uci.edu/ml/index.php" target="_blank">UC Irvine Machine Learning Repository</a> から取得したものを少し改変しました。

* 赤ワイン https://raw.githubusercontent.com/chemo-wakate/tutorial-6th/master/beginner/data/winequality-red.txt

* 白ワイン https://raw.githubusercontent.com/chemo-wakate/tutorial-6th/master/beginner/data/winequality-white.txt


<ol>
<li>fixed acidity : 不揮発酸濃度（ほぼ酒石酸濃度）
<li>volatile acidity : 揮発酸濃度（ほぼ酢酸濃度）
<li>citric acid : クエン酸濃度
<li>residual sugar : 残存糖濃度
<li>chlorides : 塩化物濃度
<li>free sulfur dioxide : 遊離亜硫酸濃度
<li>total sulfur dioxide : 亜硫酸濃度
<li>density : 密度
<li>pH : pH
<li>sulphates : 硫酸塩濃度
<li>alcohol : アルコール度数
<li>quality (score between 0 and 10) : 0-10 の値で示される品質のスコア
</ol>




```python
import pandas as pd

red_wine = pd.read_csv('https://raw.githubusercontent.com/chemo-wakate/tutorial-6th/master/beginner/data/winequality-red.txt', sep='\t', index_col=0) 
red_wine
```




<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>fixed acidity</th>
      <th>volatile acidity</th>
      <th>citric acid</th>
      <th>residual sugar</th>
      <th>chlorides</th>
      <th>free sulfur dioxide</th>
      <th>total sulfur dioxide</th>
      <th>density</th>
      <th>pH</th>
      <th>sulphates</th>
      <th>alcohol</th>
      <th>quality</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>7.4</td>
      <td>0.700</td>
      <td>0.00</td>
      <td>1.9</td>
      <td>0.076</td>
      <td>11.0</td>
      <td>34.0</td>
      <td>0.99780</td>
      <td>3.51</td>
      <td>0.56</td>
      <td>9.4</td>
      <td>5</td>
    </tr>
    <tr>
      <th>1</th>
      <td>7.8</td>
      <td>0.880</td>
      <td>0.00</td>
      <td>2.6</td>
      <td>0.098</td>
      <td>25.0</td>
      <td>67.0</td>
      <td>0.99680</td>
      <td>3.20</td>
      <td>0.68</td>
      <td>9.8</td>
      <td>5</td>
    </tr>
    <tr>
      <th>2</th>
      <td>7.8</td>
      <td>0.760</td>
      <td>0.04</td>
      <td>2.3</td>
      <td>0.092</td>
      <td>15.0</td>
      <td>54.0</td>
      <td>0.99700</td>
      <td>3.26</td>
      <td>0.65</td>
      <td>9.8</td>
      <td>5</td>
    </tr>
    <tr>
      <th>3</th>
      <td>11.2</td>
      <td>0.280</td>
      <td>0.56</td>
      <td>1.9</td>
      <td>0.075</td>
      <td>17.0</td>
      <td>60.0</td>
      <td>0.99800</td>
      <td>3.16</td>
      <td>0.58</td>
      <td>9.8</td>
      <td>6</td>
    </tr>
    <tr>
      <th>4</th>
      <td>7.4</td>
      <td>0.700</td>
      <td>0.00</td>
      <td>1.9</td>
      <td>0.076</td>
      <td>11.0</td>
      <td>34.0</td>
      <td>0.99780</td>
      <td>3.51</td>
      <td>0.56</td>
      <td>9.4</td>
      <td>5</td>
    </tr>
    <tr>
      <th>...</th>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
    </tr>
    <tr>
      <th>1594</th>
      <td>6.2</td>
      <td>0.600</td>
      <td>0.08</td>
      <td>2.0</td>
      <td>0.090</td>
      <td>32.0</td>
      <td>44.0</td>
      <td>0.99490</td>
      <td>3.45</td>
      <td>0.58</td>
      <td>10.5</td>
      <td>5</td>
    </tr>
    <tr>
      <th>1595</th>
      <td>5.9</td>
      <td>0.550</td>
      <td>0.10</td>
      <td>2.2</td>
      <td>0.062</td>
      <td>39.0</td>
      <td>51.0</td>
      <td>0.99512</td>
      <td>3.52</td>
      <td>0.76</td>
      <td>11.2</td>
      <td>6</td>
    </tr>
    <tr>
      <th>1596</th>
      <td>6.3</td>
      <td>0.510</td>
      <td>0.13</td>
      <td>2.3</td>
      <td>0.076</td>
      <td>29.0</td>
      <td>40.0</td>
      <td>0.99574</td>
      <td>3.42</td>
      <td>0.75</td>
      <td>11.0</td>
      <td>6</td>
    </tr>
    <tr>
      <th>1597</th>
      <td>5.9</td>
      <td>0.645</td>
      <td>0.12</td>
      <td>2.0</td>
      <td>0.075</td>
      <td>32.0</td>
      <td>44.0</td>
      <td>0.99547</td>
      <td>3.57</td>
      <td>0.71</td>
      <td>10.2</td>
      <td>5</td>
    </tr>
    <tr>
      <th>1598</th>
      <td>6.0</td>
      <td>0.310</td>
      <td>0.47</td>
      <td>3.6</td>
      <td>0.067</td>
      <td>18.0</td>
      <td>42.0</td>
      <td>0.99549</td>
      <td>3.39</td>
      <td>0.66</td>
      <td>11.0</td>
      <td>6</td>
    </tr>
  </tbody>
</table>
<p>1599 rows × 12 columns</p>
</div>




```python
import pandas as pd

white_wine = pd.read_csv('https://raw.githubusercontent.com/chemo-wakate/tutorial-6th/master/beginner/data/winequality-white.txt', sep='\t', index_col=0) 
white_wine
```




<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>fixed acidity</th>
      <th>volatile acidity</th>
      <th>citric acid</th>
      <th>residual sugar</th>
      <th>chlorides</th>
      <th>free sulfur dioxide</th>
      <th>total sulfur dioxide</th>
      <th>density</th>
      <th>pH</th>
      <th>sulphates</th>
      <th>alcohol</th>
      <th>quality</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>7.0</td>
      <td>0.27</td>
      <td>0.36</td>
      <td>20.7</td>
      <td>0.045</td>
      <td>45.0</td>
      <td>170.0</td>
      <td>1.00100</td>
      <td>3.00</td>
      <td>0.45</td>
      <td>8.8</td>
      <td>6</td>
    </tr>
    <tr>
      <th>1</th>
      <td>6.3</td>
      <td>0.30</td>
      <td>0.34</td>
      <td>1.6</td>
      <td>0.049</td>
      <td>14.0</td>
      <td>132.0</td>
      <td>0.99400</td>
      <td>3.30</td>
      <td>0.49</td>
      <td>9.5</td>
      <td>6</td>
    </tr>
    <tr>
      <th>2</th>
      <td>8.1</td>
      <td>0.28</td>
      <td>0.40</td>
      <td>6.9</td>
      <td>0.050</td>
      <td>30.0</td>
      <td>97.0</td>
      <td>0.99510</td>
      <td>3.26</td>
      <td>0.44</td>
      <td>10.1</td>
      <td>6</td>
    </tr>
    <tr>
      <th>3</th>
      <td>7.2</td>
      <td>0.23</td>
      <td>0.32</td>
      <td>8.5</td>
      <td>0.058</td>
      <td>47.0</td>
      <td>186.0</td>
      <td>0.99560</td>
      <td>3.19</td>
      <td>0.40</td>
      <td>9.9</td>
      <td>6</td>
    </tr>
    <tr>
      <th>4</th>
      <td>7.2</td>
      <td>0.23</td>
      <td>0.32</td>
      <td>8.5</td>
      <td>0.058</td>
      <td>47.0</td>
      <td>186.0</td>
      <td>0.99560</td>
      <td>3.19</td>
      <td>0.40</td>
      <td>9.9</td>
      <td>6</td>
    </tr>
    <tr>
      <th>...</th>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
    </tr>
    <tr>
      <th>4893</th>
      <td>6.2</td>
      <td>0.21</td>
      <td>0.29</td>
      <td>1.6</td>
      <td>0.039</td>
      <td>24.0</td>
      <td>92.0</td>
      <td>0.99114</td>
      <td>3.27</td>
      <td>0.50</td>
      <td>11.2</td>
      <td>6</td>
    </tr>
    <tr>
      <th>4894</th>
      <td>6.6</td>
      <td>0.32</td>
      <td>0.36</td>
      <td>8.0</td>
      <td>0.047</td>
      <td>57.0</td>
      <td>168.0</td>
      <td>0.99490</td>
      <td>3.15</td>
      <td>0.46</td>
      <td>9.6</td>
      <td>5</td>
    </tr>
    <tr>
      <th>4895</th>
      <td>6.5</td>
      <td>0.24</td>
      <td>0.19</td>
      <td>1.2</td>
      <td>0.041</td>
      <td>30.0</td>
      <td>111.0</td>
      <td>0.99254</td>
      <td>2.99</td>
      <td>0.46</td>
      <td>9.4</td>
      <td>6</td>
    </tr>
    <tr>
      <th>4896</th>
      <td>5.5</td>
      <td>0.29</td>
      <td>0.30</td>
      <td>1.1</td>
      <td>0.022</td>
      <td>20.0</td>
      <td>110.0</td>
      <td>0.98869</td>
      <td>3.34</td>
      <td>0.38</td>
      <td>12.8</td>
      <td>7</td>
    </tr>
    <tr>
      <th>4897</th>
      <td>6.0</td>
      <td>0.21</td>
      <td>0.38</td>
      <td>0.8</td>
      <td>0.020</td>
      <td>22.0</td>
      <td>98.0</td>
      <td>0.98941</td>
      <td>3.26</td>
      <td>0.32</td>
      <td>11.8</td>
      <td>6</td>
    </tr>
  </tbody>
</table>
<p>4898 rows × 12 columns</p>
</div>




```python
from sklearn.model_selection import train_test_split

random_state = 0
X_train, X_test, y_train, y_test = train_test_split(red_wine.iloc[:, :-1].values, red_wine.iloc[:, [-1]].values, test_size=.3, random_state=random_state) 
X_train, X_valid, y_train, y_valid = train_test_split(X_train, y_train, test_size=.3, random_state=random_state) 
```


```python
X_train.shape, y_train.shape
```




    ((783, 11), (783, 1))




```python
X_valid.shape, y_valid.shape
```




    ((336, 11), (336, 1))




```python
X_test.shape, y_test.shape
```




    ((480, 11), (480, 1))



以上のように、赤ワインのデータと白ワインのデータがありますが、赤ワインの品質を回帰モデルで予測してみようと思います。このデータを用いるのが適切ではない可能性もありますが、まあそれはおいといて。白ワインのデータは、下記にあるように「事前学習」に使ってみます。


# 事前学習なしでTabNet学習

まずは事前学習なしで、赤ワインのデータのみを用いて回帰モデルを作って性能を見てみます。n_steps という変数は、ニューラルネットワークにおける層の数みたいなもので、どの特徴量を使ってdecision makingをしたかを表すものだと理解しています（この理解が間違ってたらすみません）。n_stepを変えると挙動がだいぶ変わります。


```python
import torch
from pytorch_tabnet.tab_model import TabNetRegressor
# from pytorch_tabnet.tab_model import TabNetClassifier

SEED = 53
N_STEPS = 3
tabnet_params = dict(n_d=8, n_a=8, n_steps=N_STEPS, gamma=1.3,
                     n_independent=2, n_shared=2,
                     seed=SEED, lambda_sparse=1e-3, 
                     optimizer_fn=torch.optim.Adam, 
                     optimizer_params=dict(lr=2e-2),
                     mask_type="entmax",
                     scheduler_params=dict(mode="min",
                                           patience=5,
                                           min_lr=1e-5,
                                           factor=0.9,),
                     scheduler_fn=torch.optim.lr_scheduler.ReduceLROnPlateau,
                     verbose=10
                    )

model = TabNetRegressor(**tabnet_params)
# model = TabNetClassifier() 
model.fit(
  X_train, y_train,
  eval_set=[(X_valid, y_valid)],
  max_epochs=5000,
  patience=100,
  eval_metric={'rmse'} # 'rmsle', 'mse', 'mae'
  # from_unsupervised=pretrainer
)
```

    Device used : cuda
    epoch 0  | loss: 32.64989| val_0_rmse: 11.13777|  0:00:00s
    epoch 10 | loss: 4.0107  | val_0_rmse: 9.12493 |  0:00:00s
    epoch 20 | loss: 0.82725 | val_0_rmse: 2.45153 |  0:00:01s
    epoch 30 | loss: 0.53165 | val_0_rmse: 7.55144 |  0:00:02s
    epoch 40 | loss: 0.44194 | val_0_rmse: 5.80592 |  0:00:02s
    epoch 50 | loss: 0.40315 | val_0_rmse: 4.01958 |  0:00:03s
    epoch 60 | loss: 0.39364 | val_0_rmse: 3.2229  |  0:00:04s
    epoch 70 | loss: 0.35804 | val_0_rmse: 3.16619 |  0:00:04s
    epoch 80 | loss: 0.37049 | val_0_rmse: 2.94055 |  0:00:05s
    epoch 90 | loss: 0.32763 | val_0_rmse: 2.65781 |  0:00:06s
    epoch 100| loss: 0.33686 | val_0_rmse: 2.25962 |  0:00:07s
    epoch 110| loss: 0.33178 | val_0_rmse: 2.03964 |  0:00:07s
    epoch 120| loss: 0.3237  | val_0_rmse: 1.88684 |  0:00:08s
    epoch 130| loss: 0.29202 | val_0_rmse: 1.67104 |  0:00:09s
    epoch 140| loss: 0.3079  | val_0_rmse: 1.55122 |  0:00:10s
    epoch 150| loss: 0.29544 | val_0_rmse: 1.39493 |  0:00:10s
    epoch 160| loss: 0.28853 | val_0_rmse: 1.25286 |  0:00:11s
    epoch 170| loss: 0.29431 | val_0_rmse: 1.19063 |  0:00:12s
    epoch 180| loss: 0.27545 | val_0_rmse: 1.15625 |  0:00:13s
    epoch 190| loss: 0.28683 | val_0_rmse: 1.09202 |  0:00:13s
    epoch 200| loss: 0.27095 | val_0_rmse: 0.99627 |  0:00:14s
    epoch 210| loss: 0.28285 | val_0_rmse: 0.96371 |  0:00:15s
    epoch 220| loss: 0.26719 | val_0_rmse: 0.90612 |  0:00:16s
    epoch 230| loss: 0.27271 | val_0_rmse: 0.88299 |  0:00:16s
    epoch 240| loss: 0.2663  | val_0_rmse: 0.82608 |  0:00:17s
    epoch 250| loss: 0.25915 | val_0_rmse: 0.82236 |  0:00:18s
    epoch 260| loss: 0.24917 | val_0_rmse: 0.80364 |  0:00:19s
    epoch 270| loss: 0.25684 | val_0_rmse: 0.77136 |  0:00:19s
    epoch 280| loss: 0.24275 | val_0_rmse: 0.78264 |  0:00:20s
    epoch 290| loss: 0.23762 | val_0_rmse: 0.77993 |  0:00:21s
    epoch 300| loss: 0.26254 | val_0_rmse: 0.77074 |  0:00:21s
    epoch 310| loss: 0.25207 | val_0_rmse: 0.76689 |  0:00:22s
    epoch 320| loss: 0.23648 | val_0_rmse: 0.74517 |  0:00:23s
    epoch 330| loss: 0.25361 | val_0_rmse: 0.73926 |  0:00:23s
    epoch 340| loss: 0.24606 | val_0_rmse: 0.72841 |  0:00:24s
    epoch 350| loss: 0.24784 | val_0_rmse: 0.72501 |  0:00:25s
    epoch 360| loss: 0.23659 | val_0_rmse: 0.71072 |  0:00:26s
    epoch 370| loss: 0.24355 | val_0_rmse: 0.71563 |  0:00:26s
    epoch 380| loss: 0.23749 | val_0_rmse: 0.7265  |  0:00:27s
    epoch 390| loss: 0.22572 | val_0_rmse: 0.71044 |  0:00:28s
    epoch 400| loss: 0.22856 | val_0_rmse: 0.7037  |  0:00:28s
    epoch 410| loss: 0.22577 | val_0_rmse: 0.70413 |  0:00:29s
    epoch 420| loss: 0.26083 | val_0_rmse: 0.70473 |  0:00:30s
    epoch 430| loss: 0.2384  | val_0_rmse: 0.70011 |  0:00:31s
    epoch 440| loss: 0.23224 | val_0_rmse: 0.70506 |  0:00:31s
    epoch 450| loss: 0.23907 | val_0_rmse: 0.70495 |  0:00:32s
    epoch 460| loss: 0.24532 | val_0_rmse: 0.70434 |  0:00:33s
    epoch 470| loss: 0.24456 | val_0_rmse: 0.70559 |  0:00:33s
    epoch 480| loss: 0.22083 | val_0_rmse: 0.70538 |  0:00:34s
    epoch 490| loss: 0.2365  | val_0_rmse: 0.70484 |  0:00:35s
    epoch 500| loss: 0.24641 | val_0_rmse: 0.70772 |  0:00:35s
    epoch 510| loss: 0.22065 | val_0_rmse: 0.70898 |  0:00:36s
    epoch 520| loss: 0.22796 | val_0_rmse: 0.70955 |  0:00:37s
    epoch 530| loss: 0.21979 | val_0_rmse: 0.711   |  0:00:37s
    
    Early stopping occurred at epoch 530 with best_epoch = 430 and best_val_0_rmse = 0.70011
    Best weights from best epoch are automatically used!


学習が終わったようなので学習の経過を確認しましょう。


```python
import matplotlib.pyplot as plt

for param in ['loss', 'lr', 'val_0_rmse']:
    plt.plot(model.history[param], label=param)
    plt.xlabel('epoch')
    plt.grid()
    plt.legend()
    plt.show()
```


    

    

![TabNet3_12_0.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/211162/8d50632c-9abd-8d1a-cce1-65b0264223f0.png)


    

    

![TabNet3_12_1.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/211162/baebcaf9-f3d4-d02f-c7c4-49419d98d252.png)


    

    
![TabNet3_12_2.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/211162/40c8df9a-1ed6-a9b5-286e-e7b0132023c0.png)


性能評価


```python
from sklearn.metrics import mean_squared_error

test_score = mean_squared_error(model.predict(X_test), y_test)

print(f"BEST VALID SCORE: {model.best_cost}")
print(f"FINAL TEST SCORE: {test_score}")
```

    BEST VALID SCORE: 0.7001122835576945
    FINAL TEST SCORE: 0.5577702945336053


予測結果と真の値とを比較するプロット。今回はワインの品質として整数値を与えるのが本当なので、プロットの形もこんなのになります。


```python
for name, X, y in [["training", X_train, y_train], ["validation", X_valid, y_valid], ["test", X_test, y_test]]:
    plt.scatter(y, model.predict(X), alpha=0.5, label=name)
    plt.plot([3, 8], [3, 8])
    plt.grid()
    plt.legend()
    plt.xlabel("True")
    plt.ylabel("Predicted")
    plt.show()
```

![TabNet3_16_0.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/211162/df677af9-dece-7e5a-4987-87ee4590dede.png)

    
![TabNet3_16_1.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/211162/85d8480f-065e-6692-bbe8-1e96f3bf9ae6.png)

    



    
![TabNet3_16_2.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/211162/45846ed4-4a00-9029-9d97-1227cdb88996.png)

    



    

    


# 自己事前学習ありでTabNet学習

TabNetでは次のようにして「事前学習」を行なえます。果たしてこれで性能が向上するでしょうか？


```python
from pytorch_tabnet.pretraining import TabNetPretrainer

pretrainer = TabNetPretrainer(**tabnet_params)
pretrainer.fit(
    X_train,
    eval_set=[X_valid],
    max_epochs=5000,
    patience=100,
    )
```

    Device used : cuda
    epoch 0  | loss: 84416.57031| val_0_unsup_loss: 33251194.0|  0:00:00s
    epoch 10 | loss: 2395.65454| val_0_unsup_loss: 19814.19141|  0:00:01s
    epoch 20 | loss: 301.95297| val_0_unsup_loss: 5600.98291|  0:00:01s
    epoch 30 | loss: 153.68903| val_0_unsup_loss: 3039.54907|  0:00:02s
    epoch 40 | loss: 51.07078| val_0_unsup_loss: 736.71814|  0:00:03s
    epoch 50 | loss: 69.6201 | val_0_unsup_loss: 442.73096|  0:00:04s
    epoch 60 | loss: 34.41461| val_0_unsup_loss: 233.06084|  0:00:05s
    epoch 70 | loss: 47.174  | val_0_unsup_loss: 247.93704|  0:00:06s
    epoch 80 | loss: 28.79832| val_0_unsup_loss: 293.99844|  0:00:07s
    epoch 90 | loss: 37.22774| val_0_unsup_loss: 248.3813|  0:00:07s
    epoch 100| loss: 22.29504| val_0_unsup_loss: 187.73672|  0:00:08s
    epoch 110| loss: 22.49255| val_0_unsup_loss: 158.42435|  0:00:09s
    epoch 120| loss: 14.54969| val_0_unsup_loss: 144.55991|  0:00:10s
    epoch 130| loss: 12.70208| val_0_unsup_loss: 104.17162|  0:00:11s
    epoch 140| loss: 12.65242| val_0_unsup_loss: 94.375  |  0:00:12s
    epoch 150| loss: 7.38195 | val_0_unsup_loss: 101.70587|  0:00:13s
    epoch 160| loss: 9.93811 | val_0_unsup_loss: 71.79494|  0:00:13s
    epoch 170| loss: 14.58065| val_0_unsup_loss: 41.21515|  0:00:14s
    epoch 180| loss: 12.00856| val_0_unsup_loss: 38.96758|  0:00:15s
    epoch 190| loss: 15.64681| val_0_unsup_loss: 27.69069|  0:00:16s
    epoch 200| loss: 5.2482  | val_0_unsup_loss: 23.3637 |  0:00:17s
    epoch 210| loss: 19.1653 | val_0_unsup_loss: 27.76682|  0:00:18s
    epoch 220| loss: 6.18701 | val_0_unsup_loss: 18.53819|  0:00:18s
    epoch 230| loss: 4.77499 | val_0_unsup_loss: 19.45325|  0:00:19s
    epoch 240| loss: 6.98359 | val_0_unsup_loss: 18.99881|  0:00:20s
    epoch 250| loss: 21.51803| val_0_unsup_loss: 33.55697|  0:00:21s
    epoch 260| loss: 5.4652  | val_0_unsup_loss: 11.66067|  0:00:22s
    epoch 270| loss: 7.04799 | val_0_unsup_loss: 13.46543|  0:00:23s
    epoch 280| loss: 4.69859 | val_0_unsup_loss: 13.60153|  0:00:23s
    epoch 290| loss: 4.7595  | val_0_unsup_loss: 12.37127|  0:00:24s
    epoch 300| loss: 5.02166 | val_0_unsup_loss: 13.52421|  0:00:25s
    epoch 310| loss: 3.72067 | val_0_unsup_loss: 15.02659|  0:00:26s
    epoch 320| loss: 3.20332 | val_0_unsup_loss: 14.02866|  0:00:27s
    epoch 330| loss: 3.21631 | val_0_unsup_loss: 14.31985|  0:00:27s
    epoch 340| loss: 3.85526 | val_0_unsup_loss: 15.13815|  0:00:28s
    epoch 350| loss: 3.31001 | val_0_unsup_loss: 13.70876|  0:00:29s
    epoch 360| loss: 3.56261 | val_0_unsup_loss: 13.16991|  0:00:30s
    epoch 370| loss: 3.35176 | val_0_unsup_loss: 14.11328|  0:00:31s
    epoch 380| loss: 2.76765 | val_0_unsup_loss: 13.73672|  0:00:31s
    
    Early stopping occurred at epoch 382 with best_epoch = 282 and best_val_0_unsup_loss = 10.97403
    Best weights from best epoch are automatically used!


事前学習の学習履歴を表示します。


```python
import matplotlib.pyplot as plt

for param in ['loss', 'lr', 'val_0_unsup_loss']:
    plt.plot(pretrainer.history[param])
    plt.xlabel('epoch')
    plt.ylabel(param)
    plt.grid()
    plt.show()
```

![TabNet3_20_0.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/211162/43e3e0eb-87ec-6953-52d2-f9cb767cfcbb.png)

    
![TabNet3_20_1.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/211162/af88160b-bb26-8476-946a-80ceb5decad5.png)

    



    
![TabNet3_20_2.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/211162/24cd1a63-fb82-cd1e-a31c-606ba97906d9.png)

    



    

    


事前学習した結果を from_unsupervised=pretrainer として利用して学習します。


```python
import torch
from pytorch_tabnet.tab_model import TabNetRegressor
# from pytorch_tabnet.tab_model import TabNetClassifier

model = TabNetRegressor(**tabnet_params)
# model = TabNetClassifier() 
model.fit(
  X_train, y_train,
  eval_set=[(X_valid, y_valid)],
  max_epochs=5000,
  patience=100,
  eval_metric={'rmse'}, # 'rmsle', 'mse', 'mae'
  from_unsupervised=pretrainer
)
```

    Device used : cuda
    Loading weights from unsupervised pretraining
    epoch 0  | loss: 44.76851| val_0_rmse: 6.73764 |  0:00:00s
    epoch 10 | loss: 7.06438 | val_0_rmse: 1.96124 |  0:00:00s
    epoch 20 | loss: 1.31673 | val_0_rmse: 2.24009 |  0:00:01s
    epoch 30 | loss: 0.59232 | val_0_rmse: 0.97692 |  0:00:02s
    epoch 40 | loss: 0.49472 | val_0_rmse: 0.78751 |  0:00:03s
    epoch 50 | loss: 0.43891 | val_0_rmse: 0.93067 |  0:00:03s
    epoch 60 | loss: 0.40727 | val_0_rmse: 0.88366 |  0:00:04s
    epoch 70 | loss: 0.40231 | val_0_rmse: 0.83118 |  0:00:05s
    epoch 80 | loss: 0.39851 | val_0_rmse: 0.81392 |  0:00:05s
    epoch 90 | loss: 0.37589 | val_0_rmse: 0.7614  |  0:00:06s
    epoch 100| loss: 0.37226 | val_0_rmse: 0.74418 |  0:00:07s
    epoch 110| loss: 0.36799 | val_0_rmse: 0.74657 |  0:00:08s
    epoch 120| loss: 0.3517  | val_0_rmse: 0.71815 |  0:00:08s
    epoch 130| loss: 0.34294 | val_0_rmse: 0.70896 |  0:00:09s
    epoch 140| loss: 0.3428  | val_0_rmse: 0.69662 |  0:00:10s
    epoch 150| loss: 0.33453 | val_0_rmse: 0.69915 |  0:00:11s
    epoch 160| loss: 0.33938 | val_0_rmse: 0.70477 |  0:00:11s
    epoch 170| loss: 0.32963 | val_0_rmse: 0.71207 |  0:00:12s
    epoch 180| loss: 0.33767 | val_0_rmse: 0.71087 |  0:00:13s
    epoch 190| loss: 0.33722 | val_0_rmse: 0.70626 |  0:00:13s
    epoch 200| loss: 0.33723 | val_0_rmse: 0.70464 |  0:00:14s
    epoch 210| loss: 0.32091 | val_0_rmse: 0.70531 |  0:00:15s
    epoch 220| loss: 0.30413 | val_0_rmse: 0.7041  |  0:00:15s
    epoch 230| loss: 0.30876 | val_0_rmse: 0.70894 |  0:00:16s
    
    Early stopping occurred at epoch 239 with best_epoch = 139 and best_val_0_rmse = 0.69576
    Best weights from best epoch are automatically used!


その結果。


```python
import matplotlib.pyplot as plt

for param in ['loss', 'lr', 'val_0_rmse']:
    plt.plot(model.history[param], label=param)
    plt.xlabel('epoch')
    plt.grid()
    plt.legend()
    plt.show()
```

![TabNet3_24_0.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/211162/1abbe7e8-907b-1150-3706-7a49ad3ffef0.png)

    
![TabNet3_24_1.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/211162/7ca731c5-ffcf-89de-0b16-784c201c786d.png)

    



    
![TabNet3_24_2.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/211162/93aaffbd-efc6-b327-fffc-ef656bfe9aa2.png)

    



    

    


性能は向上したでしょうか。


```python
from sklearn.metrics import mean_squared_error

test_score = mean_squared_error(model.predict(X_test), y_test)

print(f"BEST VALID SCORE: {model.best_cost}")
print(f"FINAL TEST SCORE: {test_score}")
```

    BEST VALID SCORE: 0.6957614248635677
    FINAL TEST SCORE: 0.4984989548147685


「今回は」性能が向上したようです。私の少ない経験上、train_test_split による分割が変化しても、random seed が変化しても、n_steps が変化しても、性能が向上するどころかむしろ悪化するケースが多いように思います。


```python
for name, X, y in [["training", X_train, y_train], ["validation", X_valid, y_valid], ["test", X_test, y_test]]:
    plt.scatter(y, model.predict(X), alpha=0.5, label=name)
    plt.plot([3, 8], [3, 8])
    plt.grid()
    plt.legend()
    plt.xlabel("True")
    plt.ylabel("Predicted")
    plt.show()
```

![TabNet3_28_0.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/211162/661274ec-1d71-03d7-d744-8ef2a6e12302.png)

    
![TabNet3_28_1.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/211162/b0b68a3c-d34a-e5a0-1d77-985e49a071f1.png)

    



    
![TabNet3_28_2.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/211162/521174d9-3d3c-b8ce-67ea-0a06947776d0.png)

    



    

    


# 転移学習？半教師あり学習？

ここまでは、赤ワインのデータのみを用いて、赤ワインの品質を予測しました。事前学習も、赤ワインのデータのみ（ラベルデータなし）を用いました。次は、試しに白ワインのデータ（ラベルデータなし）を事前学習に用いてみます。これって、転移学習と呼べるのかな？半教師あり学習と呼べるのかな？という疑問が浮かびましたがよく分かりません。


```python
from sklearn.model_selection import train_test_split

random_state = 53
X_train2, X_valid2, y_train2, y_valid2 = train_test_split(white_wine.iloc[:, :-1].values, white_wine.iloc[:, [-1]].values, test_size=.3, random_state=random_state) 
```


```python
X_train2.shape, y_train2.shape
```




    ((3428, 11), (3428, 1))




```python
X_valid2.shape, y_valid2.shape
```




    ((1470, 11), (1470, 1))



事前学習用のデータを置き換えるだけで、それ以外は特に変わりませんね。


```python
from pytorch_tabnet.pretraining import TabNetPretrainer

import numpy as np

pretrainer = TabNetPretrainer(**tabnet_params)
pretrainer.fit(
    np.concatenate([X_train, X_train2]),
    eval_set=[np.concatenate([X_valid, X_valid2])],
    max_epochs=5000,
    patience=100,
    )
```

    Device used : cuda
    epoch 0  | loss: 16660.33889| val_0_unsup_loss: 2446820.75|  0:00:00s
    epoch 10 | loss: 28.30764| val_0_unsup_loss: 613.03979|  0:00:04s
    epoch 20 | loss: 27.08781| val_0_unsup_loss: 283.35773|  0:00:07s
    epoch 30 | loss: 3.24114 | val_0_unsup_loss: 62.06253|  0:00:11s
    epoch 40 | loss: 7.27796 | val_0_unsup_loss: 91.33012|  0:00:15s
    epoch 50 | loss: 13.21588| val_0_unsup_loss: 15.54838|  0:00:18s
    epoch 60 | loss: 20.57722| val_0_unsup_loss: 32.31664|  0:00:22s
    epoch 70 | loss: 3.17627 | val_0_unsup_loss: 11.90612|  0:00:25s
    epoch 80 | loss: 1.67535 | val_0_unsup_loss: 39.38104|  0:00:29s
    epoch 90 | loss: 1.55993 | val_0_unsup_loss: 32.15377|  0:00:32s
    epoch 100| loss: 3.09699 | val_0_unsup_loss: 34.45182|  0:00:36s
    epoch 110| loss: 2.2177  | val_0_unsup_loss: 36.49228|  0:00:39s
    epoch 120| loss: 2.08958 | val_0_unsup_loss: 32.53637|  0:00:43s
    epoch 130| loss: 1.42668 | val_0_unsup_loss: 38.60402|  0:00:46s
    epoch 140| loss: 1.40155 | val_0_unsup_loss: 44.6946 |  0:00:50s
    epoch 150| loss: 1.37884 | val_0_unsup_loss: 40.0892 |  0:00:54s
    epoch 160| loss: 1.3745  | val_0_unsup_loss: 41.22863|  0:00:57s
    epoch 170| loss: 1.56642 | val_0_unsup_loss: 41.14534|  0:01:01s
    
    Early stopping occurred at epoch 170 with best_epoch = 70 and best_val_0_unsup_loss = 11.90612
    Best weights from best epoch are automatically used!



```python
import matplotlib.pyplot as plt

for param in ['loss', 'lr', 'val_0_unsup_loss']:
    plt.plot(pretrainer.history[param])
    plt.xlabel('epoch')
    plt.ylabel(param)
    plt.grid()
    plt.show()
```

![TabNet3_35_0.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/211162/3b6d9d03-d336-0e23-e2cd-615e0fbf2f84.png)

    
![TabNet3_35_1.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/211162/3fb9cfb0-6b3c-435c-464c-58e1b71114b3.png)

    



    
![TabNet3_35_2.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/211162/ab25db83-684b-a2ee-da8f-60f52b8a0e90.png)

    



    

    


事前学習を終えて、本番の学習です。


```python
import torch
from pytorch_tabnet.tab_model import TabNetRegressor
# from pytorch_tabnet.tab_model import TabNetClassifier

model = TabNetRegressor(**tabnet_params)
# model = TabNetClassifier() 
model.fit(
  X_train, y_train,
  eval_set=[(X_valid, y_valid)],
  max_epochs=5000,
  patience=100,
  eval_metric={'rmse'}, # 'rmsle', 'mse', 'mae'
  from_unsupervised=pretrainer
)
```

    Device used : cuda
    Loading weights from unsupervised pretraining
    epoch 0  | loss: 45.92977| val_0_rmse: 7.19955 |  0:00:00s
    epoch 10 | loss: 9.63374 | val_0_rmse: 2.01992 |  0:00:00s
    epoch 20 | loss: 2.10203 | val_0_rmse: 0.92411 |  0:00:01s
    epoch 30 | loss: 0.73812 | val_0_rmse: 0.98633 |  0:00:02s
    epoch 40 | loss: 0.55103 | val_0_rmse: 1.05893 |  0:00:03s
    epoch 50 | loss: 0.43789 | val_0_rmse: 0.9111  |  0:00:03s
    epoch 60 | loss: 0.42148 | val_0_rmse: 0.8362  |  0:00:04s
    epoch 70 | loss: 0.4075  | val_0_rmse: 0.8341  |  0:00:05s
    epoch 80 | loss: 0.42913 | val_0_rmse: 0.78552 |  0:00:05s
    epoch 90 | loss: 0.38573 | val_0_rmse: 0.76019 |  0:00:06s
    epoch 100| loss: 0.38899 | val_0_rmse: 0.74047 |  0:00:07s
    epoch 110| loss: 0.36617 | val_0_rmse: 0.73821 |  0:00:08s
    epoch 120| loss: 0.35307 | val_0_rmse: 0.715   |  0:00:08s
    epoch 130| loss: 0.34752 | val_0_rmse: 0.70871 |  0:00:09s
    epoch 140| loss: 0.33744 | val_0_rmse: 0.69098 |  0:00:10s
    epoch 150| loss: 0.34046 | val_0_rmse: 0.69176 |  0:00:11s
    epoch 160| loss: 0.31546 | val_0_rmse: 0.68321 |  0:00:11s
    epoch 170| loss: 0.32649 | val_0_rmse: 0.67995 |  0:00:12s
    epoch 180| loss: 0.32596 | val_0_rmse: 0.68038 |  0:00:13s
    epoch 190| loss: 0.30284 | val_0_rmse: 0.67161 |  0:00:13s
    epoch 200| loss: 0.31748 | val_0_rmse: 0.67932 |  0:00:14s
    epoch 210| loss: 0.30808 | val_0_rmse: 0.67495 |  0:00:15s
    epoch 220| loss: 0.30112 | val_0_rmse: 0.67347 |  0:00:15s
    epoch 230| loss: 0.29536 | val_0_rmse: 0.67413 |  0:00:16s
    epoch 240| loss: 0.29671 | val_0_rmse: 0.67068 |  0:00:17s
    epoch 250| loss: 0.28007 | val_0_rmse: 0.66754 |  0:00:18s
    epoch 260| loss: 0.2999  | val_0_rmse: 0.66688 |  0:00:18s
    epoch 270| loss: 0.29132 | val_0_rmse: 0.66692 |  0:00:19s
    epoch 280| loss: 0.28029 | val_0_rmse: 0.66665 |  0:00:20s
    epoch 290| loss: 0.278   | val_0_rmse: 0.66944 |  0:00:21s
    epoch 300| loss: 0.29261 | val_0_rmse: 0.66767 |  0:00:21s
    epoch 310| loss: 0.28728 | val_0_rmse: 0.66728 |  0:00:22s
    epoch 320| loss: 0.27489 | val_0_rmse: 0.66367 |  0:00:23s
    epoch 330| loss: 0.26925 | val_0_rmse: 0.66052 |  0:00:23s
    epoch 340| loss: 0.26379 | val_0_rmse: 0.65765 |  0:00:24s
    epoch 350| loss: 0.28442 | val_0_rmse: 0.65711 |  0:00:25s
    epoch 360| loss: 0.28069 | val_0_rmse: 0.65617 |  0:00:26s
    epoch 370| loss: 0.28303 | val_0_rmse: 0.65606 |  0:00:26s
    epoch 380| loss: 0.29653 | val_0_rmse: 0.65791 |  0:00:27s
    epoch 390| loss: 0.27669 | val_0_rmse: 0.65919 |  0:00:28s
    epoch 400| loss: 0.2771  | val_0_rmse: 0.66249 |  0:00:28s
    epoch 410| loss: 0.26686 | val_0_rmse: 0.66416 |  0:00:29s
    epoch 420| loss: 0.26393 | val_0_rmse: 0.66477 |  0:00:30s
    epoch 430| loss: 0.26517 | val_0_rmse: 0.66507 |  0:00:31s
    epoch 440| loss: 0.28231 | val_0_rmse: 0.66406 |  0:00:31s
    epoch 450| loss: 0.26679 | val_0_rmse: 0.66527 |  0:00:32s
    epoch 460| loss: 0.27933 | val_0_rmse: 0.66518 |  0:00:33s
    
    Early stopping occurred at epoch 461 with best_epoch = 361 and best_val_0_rmse = 0.6559
    Best weights from best epoch are automatically used!



```python
import matplotlib.pyplot as plt

for param in ['loss', 'lr', 'val_0_rmse']:
    plt.plot(model.history[param], label=param)
    plt.xlabel('epoch')
    plt.grid()
    plt.legend()
    plt.show()
```

![TabNet3_38_0.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/211162/e9132258-09aa-20a3-a9da-009e12ebfb80.png)

    
![TabNet3_38_1.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/211162/44016449-9409-f54d-8274-248258dc4e7d.png)

    



    
![TabNet3_38_2.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/211162/5976f259-c732-fb29-e365-af15e6f6961e.png)

    



    

    



```python
from sklearn.metrics import mean_squared_error

test_score = mean_squared_error(model.predict(X_test), y_test)

print(f"BEST VALID SCORE: {model.best_cost}")
print(f"FINAL TEST SCORE: {test_score}")
```

    BEST VALID SCORE: 0.6558970337385635
    FINAL TEST SCORE: 0.48485043537972694


おお、向上しましたね。今までで一番良い数字です。ですがこれも、私の少ない経験上、train_test_split による分割が変化しても、random seed が変化しても、n_steps が変化しても、性能が向上するどころかむしろ悪化するケースが多いように思います。


```python
for name, X, y in [["training", X_train, y_train], ["validation", X_valid, y_valid], ["test", X_test, y_test]]:
    plt.scatter(y, model.predict(X), alpha=0.5, label=name)
    plt.plot([3, 8], [3, 8])
    plt.grid()
    plt.legend()
    plt.xlabel("True")
    plt.ylabel("Predicted")
    plt.show()
```

![TabNet3_41_1.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/211162/a18240da-5a20-f286-9e45-c476f62fb646.png)

    
![TabNet3_41_1.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/211162/eca4937c-c0bd-30fa-d646-c6a88215cac2.png)

    



    
![TabNet3_41_2.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/211162/016d461c-dd11-ea7e-5ce9-7d27fb32d1c8.png)

    



    

    


# Feature importance (Global interpretability)

TabNetの特徴として、RandomForestなどと類似した Feature importanceが算出できることが挙げられます。TabNetでは、Global interpretability とも呼ばれます。


```python
plt.barh(list(red_wine.columns[:-1])[::-1], model.feature_importances_[::-1])
plt.grid()
plt.show()
```

![TabNet3_43_0.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/211162/1c2a023f-8072-58fb-e28b-ec90597950e9.png)

    

    


# Mask (Local interpretability)

TabNetではそれに加えて、どの特徴量を使うか decision making するのに用いた mask というのを見ることができます。Local interpretability とも呼ばれます。mask は n_steps の数だけあります。ここでは、予測したデータの先頭 50 個についてのみ図示してみます。


```python
explain_matrix, masks = model.explain(X_test)
```


```python
fig, axs = plt.subplots(N_STEPS, 1, figsize=(21, 3*N_STEPS))

for i in range(N_STEPS):
    axs[i].imshow(masks[i][:50].T)
    axs[i].set_title(f"mask {i}")
    axs[i].set_yticks(range(len(red_wine.columns[:-1])))
    axs[i].set_yticklabels(list(red_wine.columns[:-1]))
```

![TabNet3_46_0.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/211162/28fbddb0-a267-cd81-0600-cbb4b3d7a43f.png)

    

    

# ScikitAllstarsと比較

[ScikitAllstars](https://qiita.com/maskot1977/items/dc82e00b46209e0d527a) は、scikit-learn の主要な教師あり機械学習モデルをひとまとめに oputuna でハイパラチューニングするツールです。上記と同じデータを予測して比較してみましょう。


```python
# Optuna のインストール
!pip install optuna
```


```python
# ScikitAllStars のインストール
!pip install git+https://github.com/maskot1977/scikitallstars.git
```



```python
import pandas as pd

red_wine = pd.read_csv('https://raw.githubusercontent.com/chemo-wakate/tutorial-6th/master/beginner/data/winequality-red.txt', sep='\t', index_col=0) 
red_wine
```




<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>fixed acidity</th>
      <th>volatile acidity</th>
      <th>citric acid</th>
      <th>residual sugar</th>
      <th>chlorides</th>
      <th>free sulfur dioxide</th>
      <th>total sulfur dioxide</th>
      <th>density</th>
      <th>pH</th>
      <th>sulphates</th>
      <th>alcohol</th>
      <th>quality</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>7.4</td>
      <td>0.700</td>
      <td>0.00</td>
      <td>1.9</td>
      <td>0.076</td>
      <td>11.0</td>
      <td>34.0</td>
      <td>0.99780</td>
      <td>3.51</td>
      <td>0.56</td>
      <td>9.4</td>
      <td>5</td>
    </tr>
    <tr>
      <th>1</th>
      <td>7.8</td>
      <td>0.880</td>
      <td>0.00</td>
      <td>2.6</td>
      <td>0.098</td>
      <td>25.0</td>
      <td>67.0</td>
      <td>0.99680</td>
      <td>3.20</td>
      <td>0.68</td>
      <td>9.8</td>
      <td>5</td>
    </tr>
    <tr>
      <th>2</th>
      <td>7.8</td>
      <td>0.760</td>
      <td>0.04</td>
      <td>2.3</td>
      <td>0.092</td>
      <td>15.0</td>
      <td>54.0</td>
      <td>0.99700</td>
      <td>3.26</td>
      <td>0.65</td>
      <td>9.8</td>
      <td>5</td>
    </tr>
    <tr>
      <th>3</th>
      <td>11.2</td>
      <td>0.280</td>
      <td>0.56</td>
      <td>1.9</td>
      <td>0.075</td>
      <td>17.0</td>
      <td>60.0</td>
      <td>0.99800</td>
      <td>3.16</td>
      <td>0.58</td>
      <td>9.8</td>
      <td>6</td>
    </tr>
    <tr>
      <th>4</th>
      <td>7.4</td>
      <td>0.700</td>
      <td>0.00</td>
      <td>1.9</td>
      <td>0.076</td>
      <td>11.0</td>
      <td>34.0</td>
      <td>0.99780</td>
      <td>3.51</td>
      <td>0.56</td>
      <td>9.4</td>
      <td>5</td>
    </tr>
    <tr>
      <th>...</th>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
    </tr>
    <tr>
      <th>1594</th>
      <td>6.2</td>
      <td>0.600</td>
      <td>0.08</td>
      <td>2.0</td>
      <td>0.090</td>
      <td>32.0</td>
      <td>44.0</td>
      <td>0.99490</td>
      <td>3.45</td>
      <td>0.58</td>
      <td>10.5</td>
      <td>5</td>
    </tr>
    <tr>
      <th>1595</th>
      <td>5.9</td>
      <td>0.550</td>
      <td>0.10</td>
      <td>2.2</td>
      <td>0.062</td>
      <td>39.0</td>
      <td>51.0</td>
      <td>0.99512</td>
      <td>3.52</td>
      <td>0.76</td>
      <td>11.2</td>
      <td>6</td>
    </tr>
    <tr>
      <th>1596</th>
      <td>6.3</td>
      <td>0.510</td>
      <td>0.13</td>
      <td>2.3</td>
      <td>0.076</td>
      <td>29.0</td>
      <td>40.0</td>
      <td>0.99574</td>
      <td>3.42</td>
      <td>0.75</td>
      <td>11.0</td>
      <td>6</td>
    </tr>
    <tr>
      <th>1597</th>
      <td>5.9</td>
      <td>0.645</td>
      <td>0.12</td>
      <td>2.0</td>
      <td>0.075</td>
      <td>32.0</td>
      <td>44.0</td>
      <td>0.99547</td>
      <td>3.57</td>
      <td>0.71</td>
      <td>10.2</td>
      <td>5</td>
    </tr>
    <tr>
      <th>1598</th>
      <td>6.0</td>
      <td>0.310</td>
      <td>0.47</td>
      <td>3.6</td>
      <td>0.067</td>
      <td>18.0</td>
      <td>42.0</td>
      <td>0.99549</td>
      <td>3.39</td>
      <td>0.66</td>
      <td>11.0</td>
      <td>6</td>
    </tr>
  </tbody>
</table>
<p>1599 rows × 12 columns</p>
</div>




```python
from sklearn.model_selection import train_test_split

random_state = 0
X_train, X_test, y_train, y_test = train_test_split(red_wine.iloc[:, :-1].values, red_wine.iloc[:, [-1]].values, test_size=.3, random_state=random_state) 
```

# AllstarsModel の学習


```python
from scikitallstars import allstars, depict

allstars_model = allstars.fit(X_train, y_train, timeout=1000, n_trials=100, feature_selection=True)
```

    feature selection: X_train (1119, 11) -> (1119, 3)


    
    {'GradientBoosting': 0.4388699431816158, 'ExtraTrees': 0.49443573617340997, 'RandomForest': 0.4434120482153883, 'AdaBoost': 0.4073180139136409, 'MLP': 0.3670088611377368, 'SVR': 0.40403614655933495, 'kNN': 0.43064459007357375, 'Ridge': 0.39026600919071514, 'Lasso': 0.39172487228440867, 'PLS': 0.39819809406714757, 'LinearRegression': 0.4125666488108234}



```python
from sklearn.metrics import mean_squared_error

test_score = mean_squared_error(allstars_model.predict(X_test), y_test)

print(f"FINAL TEST SCORE: {test_score}")
```

    FINAL TEST SCORE: 0.42362855816765



```python
import matplotlib.pyplot as plt

for name, X, y in [["training", X_train, y_train], ["test", X_test, y_test]]:
    plt.scatter(y, allstars_model.predict(X), alpha=0.5, label=name)
    plt.plot([3, 8], [3, 8])
    plt.grid()
    plt.legend()
    plt.xlabel("True")
    plt.ylabel("Predicted")
    plt.show()
```

![WineQualityScikitAllStars_8_0.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/211162/c026a1a4-78b5-ef6a-6414-315b545a67b9.png)

    
![WineQualityScikitAllStars_8_1.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/211162/50690ae1-8f4d-e138-4f17-f91316b8aa0f.png)

    



    
    


# StackingModel の学習


```python
stacking_model = allstars.get_best_stacking(allstars_model, X_train, y_train, timeout=1000, n_trials=100)
```




```python
from sklearn.metrics import mean_squared_error

test_score = mean_squared_error(stacking_model.predict(X_test), y_test)

print(f"FINAL TEST SCORE: {test_score}")
```

    FINAL TEST SCORE: 0.4301126301454623



```python
import matplotlib.pyplot as plt

for name, X, y in [["training", X_train, y_train], ["test", X_test, y_test]]:
    plt.scatter(y, stacking_model.predict(X), alpha=0.5, label=name)
    plt.plot([3, 8], [3, 8])
    plt.grid()
    plt.legend()
    plt.xlabel("True")
    plt.ylabel("Predicted")
    plt.show()
```

![WineQualityScikitAllStars_12_0.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/211162/48b8127a-a26d-97ff-1faf-446ad5c870bc.png)

    
![WineQualityScikitAllStars_12_1.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/211162/3aedba9b-8eff-0943-7d60-9946018ea5e7.png)

    



    
    


# 結果


あれ？ TabNet 負けちゃった。まあ、どんなデータでも勝てるわけじゃないんでしょうね。あるいは私の TabNet に対する理解が足りなくて、しょぼい使い方しかできてない可能性も。

今回は回帰モデルを作ってみましたが、そのうち分類とかもやります。そのうち。