Feature Selection Datasets

# Feature Selection Datasets

機械学習の勉強や手法のベンチマーク用に集められたと思われるデータセットとして、 Feature Selection Datasets があります。

http://featureselection.asu.edu/datasets.php

非常に多くのデータがあるので、その中身を一覧して、ちょうどいいデータを見つけたいと思って軽く解析してみました。

データを取得してデータ構造を見るだけでなく、 scikit-learn の RandomForestClassifier を使って、分類問題の難易度も見てみました。

# コード

```Python
import os
import timeit
from scipy import io
import pandas as pd
import urllib.request
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split
```

取得データの一覧はこちらです。２つほど、URLが間違っているのを直しました。


```Python
dataset_url = [
        "http://featureselection.asu.edu/files/datasets/BASEHOCK.mat",
        "http://featureselection.asu.edu/files/datasets/PCMAC.mat",
        "http://featureselection.asu.edu/files/datasets/RELATHE.mat",
        "http://featureselection.asu.edu/files/datasets/COIL20.mat",
        "http://featureselection.asu.edu/files/datasets/ORL.mat",
        "http://featureselection.asu.edu/files/datasets/orlraws10P.mat",
        "http://featureselection.asu.edu/files/datasets/pixraw10P.mat",
        "http://featureselection.asu.edu/files/datasets/warpAR10P.mat",
        "http://featureselection.asu.edu/files/datasets/warpPIE10P.mat",
        "http://featureselection.asu.edu/files/datasets/Yale.mat",
        "http://featureselection.asu.edu/files/datasets/USPS.mat",
        "http://featureselection.asu.edu/files/datasets/ALLAML.mat",
        "http://featureselection.asu.edu/files/datasets/Carcinom.mat",
        "http://featureselection.asu.edu/files/datasets/CLL_SUB_111.mat",
        "http://featureselection.asu.edu/files/datasets/colon.mat",
        "http://featureselection.asu.edu/files/datasets/GLI_85.mat",
        "http://featureselection.asu.edu/files/datasets/GLIOMA.mat",
        "http://featureselection.asu.edu/files/datasets/leukemia.mat",
        "http://featureselection.asu.edu/files/datasets/lung.mat",
        "http://featureselection.asu.edu/files/datasets/lung_discrete.mat",
        "http://featureselection.asu.edu/files/datasets/lymphoma.mat",
        "http://featureselection.asu.edu/files/datasets/nci9.mat",
        "http://featureselection.asu.edu/files/datasets/Prostate_GE.mat",
        "http://featureselection.asu.edu/files/datasets/SMK_CAN_187.mat",
        "http://featureselection.asu.edu/files/datasets/TOX_171.mat",
        "http://featureselection.asu.edu/files/datasets/arcene.mat",
        "http://featureselection.asu.edu/files/datasets/gisette.mat",
        "http://featureselection.asu.edu/files/datasets/Isolet.mat",
        "http://featureselection.asu.edu/files/datasets/madelon.mat"
]
```


```Python
result = {
    'dataset':[],
    'byte':[],
    'X.shape':[],
    'X_type':[],
    'y.shape':[],
    'n_class':[],
    'RF_max':[],
    'RF_mean':[],
    'RF_min':[],
    'sec':[],
    }

for url in dataset_url:
    result['dataset'].append(url.split("/")[-1])

    filename = 'dataset.mat'
    urllib.request.urlretrieve(url, filename)
    result['byte'].append(os.path.getsize(filename))

    matdata = io.loadmat(filename, squeeze_me=True)
    X = matdata['X']
    y = matdata['Y'].flatten()
    result['X.shape'].append(X.shape)
    result['X_type'].append(pd.DataFrame(X).nunique()[0])

    result['y.shape'].append(y.shape)
    result['n_class'].append(pd.DataFrame(y).nunique()[0])

    scores = []
    times = []
    for _ in range(10):
        X_train, X_test, y_train, y_test = train_test_split(X, y)
        model = RandomForestClassifier()
        times.append(timeit.timeit(lambda: model.fit(X_train, y_train), number=1))
        scores.append(model.score(X_test,y_test))

    result['RF_max'].append(max(scores))
    result['RF_mean'].append(sum(scores) / len(scores))
    result['RF_min'].append(min(scores))

    result['sec'].append(sum(times) / len(times))
```

# 結果

* dataset : データセットの名前
* byte : データセットのサイズ（byte）
* X.shape : 説明変数の形
* X_type : 説明変数に入っている数値のバリエーション。2なら２種類の値しか入っていない離散値と考えることができる。十分に多ければ実質連続値と考えることができる。
* y.shape : 目的変数の形
* n_class : 目的変数の数値のバリエーション、すなわちクラスの数。
* RF_max, RF_mean, RF_min : ランダムフォレストで分類問題を解いた時の正解率の最大値、平均値、最小値
* sec : 分類問題を解くのに要した時間 (sec) の平均値


```Python
pd.DataFrame(result).sort_values("RF_max")
```




<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>dataset</th>
      <th>byte</th>
      <th>X.shape</th>
      <th>X_type</th>
      <th>y.shape</th>
      <th>n_class</th>
      <th>RF_max</th>
      <th>RF_mean</th>
      <th>RF_min</th>
      <th>sec</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>21</th>
      <td>nci9.mat</td>
      <td>169288</td>
      <td>(60, 9712)</td>
      <td>3</td>
      <td>(60,)</td>
      <td>9</td>
      <td>0.666667</td>
      <td>0.433333</td>
      <td>0.266667</td>
      <td>0.183649</td>
    </tr>
    <tr>
      <th>23</th>
      <td>SMK_CAN_187.mat</td>
      <td>11861244</td>
      <td>(187, 19993)</td>
      <td>171</td>
      <td>(187,)</td>
      <td>2</td>
      <td>0.723404</td>
      <td>0.655319</td>
      <td>0.574468</td>
      <td>0.670948</td>
    </tr>
    <tr>
      <th>28</th>
      <td>madelon.mat</td>
      <td>1496573</td>
      <td>(2600, 500)</td>
      <td>40</td>
      <td>(2600,)</td>
      <td>2</td>
      <td>0.733846</td>
      <td>0.707385</td>
      <td>0.680000</td>
      <td>2.456003</td>
    </tr>
    <tr>
      <th>13</th>
      <td>CLL_SUB_111.mat</td>
      <td>5875157</td>
      <td>(111, 11340)</td>
      <td>111</td>
      <td>(111,)</td>
      <td>3</td>
      <td>0.750000</td>
      <td>0.657143</td>
      <td>0.464286</td>
      <td>0.326307</td>
    </tr>
    <tr>
      <th>24</th>
      <td>TOX_171.mat</td>
      <td>3470586</td>
      <td>(171, 5748)</td>
      <td>169</td>
      <td>(171,)</td>
      <td>4</td>
      <td>0.813953</td>
      <td>0.772093</td>
      <td>0.697674</td>
      <td>0.405085</td>
    </tr>
    <tr>
      <th>16</th>
      <td>GLIOMA.mat</td>
      <td>1462087</td>
      <td>(50, 4434)</td>
      <td>50</td>
      <td>(50,)</td>
      <td>4</td>
      <td>0.846154</td>
      <td>0.669231</td>
      <td>0.538462</td>
      <td>0.154852</td>
    </tr>
    <tr>
      <th>9</th>
      <td>Yale.mat</td>
      <td>161021</td>
      <td>(165, 1024)</td>
      <td>77</td>
      <td>(165,)</td>
      <td>15</td>
      <td>0.857143</td>
      <td>0.769048</td>
      <td>0.595238</td>
      <td>0.306511</td>
    </tr>
    <tr>
      <th>25</th>
      <td>arcene.mat</td>
      <td>1900005</td>
      <td>(200, 10000)</td>
      <td>82</td>
      <td>(200,)</td>
      <td>2</td>
      <td>0.900000</td>
      <td>0.788000</td>
      <td>0.680000</td>
      <td>0.417719</td>
    </tr>
    <tr>
      <th>20</th>
      <td>lymphoma.mat</td>
      <td>110185</td>
      <td>(96, 4026)</td>
      <td>3</td>
      <td>(96,)</td>
      <td>9</td>
      <td>0.916667</td>
      <td>0.829167</td>
      <td>0.708333</td>
      <td>0.169875</td>
    </tr>
    <tr>
      <th>2</th>
      <td>RELATHE.mat</td>
      <td>226918</td>
      <td>(1427, 4322)</td>
      <td>2</td>
      <td>(1427,)</td>
      <td>2</td>
      <td>0.921569</td>
      <td>0.898880</td>
      <td>0.876751</td>
      <td>1.218853</td>
    </tr>
    <tr>
      <th>14</th>
      <td>colon.mat</td>
      <td>36319</td>
      <td>(62, 2000)</td>
      <td>3</td>
      <td>(62,)</td>
      <td>2</td>
      <td>0.937500</td>
      <td>0.768750</td>
      <td>0.687500</td>
      <td>0.135427</td>
    </tr>
    <tr>
      <th>7</th>
      <td>warpAR10P.mat</td>
      <td>279711</td>
      <td>(130, 2400)</td>
      <td>63</td>
      <td>(130,)</td>
      <td>10</td>
      <td>0.939394</td>
      <td>0.851515</td>
      <td>0.757576</td>
      <td>0.274956</td>
    </tr>
    <tr>
      <th>1</th>
      <td>PCMAC.mat</td>
      <td>191131</td>
      <td>(1943, 3289)</td>
      <td>4</td>
      <td>(1943,)</td>
      <td>2</td>
      <td>0.944444</td>
      <td>0.922634</td>
      <td>0.899177</td>
      <td>1.491283</td>
    </tr>
    <tr>
      <th>4</th>
      <td>ORL.mat</td>
      <td>376584</td>
      <td>(400, 1024)</td>
      <td>151</td>
      <td>(400,)</td>
      <td>40</td>
      <td>0.950000</td>
      <td>0.921000</td>
      <td>0.830000</td>
      <td>1.216780</td>
    </tr>
    <tr>
      <th>15</th>
      <td>GLI_85.mat</td>
      <td>8743262</td>
      <td>(85, 22283)</td>
      <td>85</td>
      <td>(85,)</td>
      <td>2</td>
      <td>0.954545</td>
      <td>0.863636</td>
      <td>0.772727</td>
      <td>0.269521</td>
    </tr>
    <tr>
      <th>27</th>
      <td>Isolet.mat</td>
      <td>3652673</td>
      <td>(1560, 617)</td>
      <td>1340</td>
      <td>(1560,)</td>
      <td>26</td>
      <td>0.956410</td>
      <td>0.938205</td>
      <td>0.905128</td>
      <td>2.222803</td>
    </tr>
    <tr>
      <th>18</th>
      <td>lung.mat</td>
      <td>4762671</td>
      <td>(203, 3312)</td>
      <td>203</td>
      <td>(203,)</td>
      <td>5</td>
      <td>0.960784</td>
      <td>0.929412</td>
      <td>0.882353</td>
      <td>0.380843</td>
    </tr>
    <tr>
      <th>22</th>
      <td>Prostate_GE.mat</td>
      <td>1524983</td>
      <td>(102, 5966)</td>
      <td>29</td>
      <td>(102,)</td>
      <td>2</td>
      <td>0.961538</td>
      <td>0.900000</td>
      <td>0.807692</td>
      <td>0.207986</td>
    </tr>
    <tr>
      <th>10</th>
      <td>USPS.mat</td>
      <td>15138167</td>
      <td>(9298, 256)</td>
      <td>1617</td>
      <td>(9298,)</td>
      <td>10</td>
      <td>0.965161</td>
      <td>0.960258</td>
      <td>0.955699</td>
      <td>9.295629</td>
    </tr>
    <tr>
      <th>26</th>
      <td>gisette.mat</td>
      <td>10619742</td>
      <td>(7000, 5000)</td>
      <td>345</td>
      <td>(7000,)</td>
      <td>2</td>
      <td>0.974286</td>
      <td>0.968971</td>
      <td>0.961714</td>
      <td>9.597926</td>
    </tr>
    <tr>
      <th>12</th>
      <td>Carcinom.mat</td>
      <td>6917199</td>
      <td>(174, 9182)</td>
      <td>156</td>
      <td>(174,)</td>
      <td>11</td>
      <td>0.977273</td>
      <td>0.868182</td>
      <td>0.772727</td>
      <td>0.557979</td>
    </tr>
    <tr>
      <th>0</th>
      <td>BASEHOCK.mat</td>
      <td>279059</td>
      <td>(1993, 4862)</td>
      <td>2</td>
      <td>(1993,)</td>
      <td>2</td>
      <td>0.985972</td>
      <td>0.974349</td>
      <td>0.965932</td>
      <td>1.789281</td>
    </tr>
    <tr>
      <th>3</th>
      <td>COIL20.mat</td>
      <td>3024549</td>
      <td>(1440, 1024)</td>
      <td>10</td>
      <td>(1440,)</td>
      <td>20</td>
      <td>1.000000</td>
      <td>0.998889</td>
      <td>0.994444</td>
      <td>1.873450</td>
    </tr>
    <tr>
      <th>11</th>
      <td>ALLAML.mat</td>
      <td>3639219</td>
      <td>(72, 7129)</td>
      <td>66</td>
      <td>(72,)</td>
      <td>2</td>
      <td>1.000000</td>
      <td>0.938889</td>
      <td>0.833333</td>
      <td>0.183536</td>
    </tr>
    <tr>
      <th>6</th>
      <td>pixraw10P.mat</td>
      <td>520463</td>
      <td>(100, 10000)</td>
      <td>11</td>
      <td>(100,)</td>
      <td>10</td>
      <td>1.000000</td>
      <td>0.972000</td>
      <td>0.920000</td>
      <td>0.338596</td>
    </tr>
    <tr>
      <th>17</th>
      <td>leukemia.mat</td>
      <td>154743</td>
      <td>(72, 7070)</td>
      <td>3</td>
      <td>(72,)</td>
      <td>2</td>
      <td>1.000000</td>
      <td>0.950000</td>
      <td>0.777778</td>
      <td>0.155346</td>
    </tr>
    <tr>
      <th>8</th>
      <td>warpPIE10P.mat</td>
      <td>458267</td>
      <td>(210, 2420)</td>
      <td>36</td>
      <td>(210,)</td>
      <td>10</td>
      <td>1.000000</td>
      <td>0.962264</td>
      <td>0.924528</td>
      <td>0.410544</td>
    </tr>
    <tr>
      <th>5</th>
      <td>orlraws10P.mat</td>
      <td>951783</td>
      <td>(100, 10304)</td>
      <td>46</td>
      <td>(100,)</td>
      <td>10</td>
      <td>1.000000</td>
      <td>0.988000</td>
      <td>0.960000</td>
      <td>0.415471</td>
    </tr>
    <tr>
      <th>19</th>
      <td>lung_discrete.mat</td>
      <td>7516</td>
      <td>(73, 325)</td>
      <td>3</td>
      <td>(73,)</td>
      <td>7</td>
      <td>1.000000</td>
      <td>0.800000</td>
      <td>0.526316</td>
      <td>0.131734</td>
    </tr>
  </tbody>
</table>
</div>



簡単すぎる問題を解いてもつまらないと思ったので、 RF_max の降順に並べてみました。

データセット選びの参考になればと。