GridSearchCV HalvingGridSearchCV RandomizedSearchCV HalvingRandomSearchCV を理解する

機械学習でハイパーパラメーターチューニングといえば [Optuna](https://qiita.com/maskot1977/items/ed698a67b091b5516ab4) が柔軟なコーディングができて個人的にはお気に入りなんですが、scikit-learnのGridSearchCVとその兄弟たちの使い方を理解してみたくなったので試してみました。

# データセットとその分割

回帰用データセットとして load_diabetes を用い、train_test_split を用いて教師データとテストデータに分割します。

```python
import sklearn.datasets
from sklearn.model_selection import train_test_split

dataset = sklearn.datasets.load_diabetes()
X_train, X_test, Y_train, Y_test = train_test_split(dataset.data, dataset.target)
```

# HistGradientBoostingRegressor

HistGradientBoostingRegressor は、非深層系機械学習モデルとして最強クラスと噂されている（？）勾配ブースティング回帰の、scikit-learnによる軽量型とのことです。


```python
from sklearn.ensemble import HistGradientBoostingRegressor

model = HistGradientBoostingRegressor()
model.fit(X_train, Y_train)
model.score(X_test, Y_test)
```




    0.3174338258206677


以上が、ハイパラチューニングなしでの結果となりますが、さてさて、GridSearchCV HalvingGridSearchCV RandomizedSearchCV HalvingRandomSearchCV の４つの CV (Cross Validation) 手法を用いてどのくらい改善できるでしょうか？

# GridSearchCV

まずは GridSearchCV を使ってみましょう。GridSearchCV は、与えたパラメーターの全組み合わせに対して学習を行なって、ベストな学習器を発見します。

```python
from sklearn.model_selection import GridSearchCV

parameters = dict(
    {
        "loss": ["squared_error", "absolute_error", "poisson"],
        "learning_rate": [0.2, 0.1, 0.05],
        "max_leaf_nodes": [2, 8, 32],
        "max_depth": [2, 8, 32],
        "min_samples_leaf": [2, 8, 32],
        "max_bins": [64, 128, 255],
    }
)
model = GridSearchCV(HistGradientBoostingRegressor(), parameters, cv=5, n_jobs=-1)
model.fit(X_train, Y_train)
print("R2=", model.score(X_test, Y_test))
model.best_estimator_
```

    R2= 0.33232614171975594


少しは改善しましたね。そのベストモデルのハイパーパラメーターは次の通りです。


    HistGradientBoostingRegressor(learning_rate=0.05, loss='absolute_error',
                                  max_bins=128, max_depth=32, max_leaf_nodes=32,
                                  min_samples_leaf=8)


以上のGridSearchCVの詳細な結果は次のように確認できます。

```python
import pandas as pd

pd.DataFrame(model.cv_results_)
```





  <div id="df-a87daaf1-2b03-4753-acf7-29b8699f37e3">
    <div class="colab-df-container">
      <div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>mean_fit_time</th>
      <th>std_fit_time</th>
      <th>mean_score_time</th>
      <th>std_score_time</th>
      <th>param_learning_rate</th>
      <th>param_loss</th>
      <th>param_max_bins</th>
      <th>param_max_depth</th>
      <th>param_max_leaf_nodes</th>
      <th>param_min_samples_leaf</th>
      <th>params</th>
      <th>split0_test_score</th>
      <th>split1_test_score</th>
      <th>split2_test_score</th>
      <th>split3_test_score</th>
      <th>split4_test_score</th>
      <th>mean_test_score</th>
      <th>std_test_score</th>
      <th>rank_test_score</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>0.046092</td>
      <td>0.001038</td>
      <td>0.003924</td>
      <td>0.000962</td>
      <td>0.2</td>
      <td>squared_error</td>
      <td>64</td>
      <td>2</td>
      <td>2</td>
      <td>2</td>
      <td>{'learning_rate': 0.2, 'loss': 'squared_error'...</td>
      <td>0.510208</td>
      <td>0.403854</td>
      <td>0.419810</td>
      <td>0.316751</td>
      <td>0.504165</td>
      <td>0.430958</td>
      <td>0.071473</td>
      <td>344</td>
    </tr>
    <tr>
      <th>1</th>
      <td>0.047879</td>
      <td>0.004027</td>
      <td>0.003343</td>
      <td>0.000078</td>
      <td>0.2</td>
      <td>squared_error</td>
      <td>64</td>
      <td>2</td>
      <td>2</td>
      <td>8</td>
      <td>{'learning_rate': 0.2, 'loss': 'squared_error'...</td>
      <td>0.534582</td>
      <td>0.414399</td>
      <td>0.412338</td>
      <td>0.323632</td>
      <td>0.506877</td>
      <td>0.438365</td>
      <td>0.075322</td>
      <td>186</td>
    </tr>
    <tr>
      <th>2</th>
      <td>0.044900</td>
      <td>0.001714</td>
      <td>0.003401</td>
      <td>0.000135</td>
      <td>0.2</td>
      <td>squared_error</td>
      <td>64</td>
      <td>2</td>
      <td>2</td>
      <td>32</td>
      <td>{'learning_rate': 0.2, 'loss': 'squared_error'...</td>
      <td>0.554374</td>
      <td>0.445249</td>
      <td>0.369375</td>
      <td>0.315602</td>
      <td>0.467517</td>
      <td>0.430423</td>
      <td>0.082305</td>
      <td>357</td>
    </tr>
    <tr>
      <th>3</th>
      <td>0.080580</td>
      <td>0.007363</td>
      <td>0.004332</td>
      <td>0.001743</td>
      <td>0.2</td>
      <td>squared_error</td>
      <td>64</td>
      <td>2</td>
      <td>8</td>
      <td>2</td>
      <td>{'learning_rate': 0.2, 'loss': 'squared_error'...</td>
      <td>0.425588</td>
      <td>0.450360</td>
      <td>0.362742</td>
      <td>0.284584</td>
      <td>0.443050</td>
      <td>0.393265</td>
      <td>0.062492</td>
      <td>596</td>
    </tr>
    <tr>
      <th>4</th>
      <td>0.069854</td>
      <td>0.004283</td>
      <td>0.003402</td>
      <td>0.000314</td>
      <td>0.2</td>
      <td>squared_error</td>
      <td>64</td>
      <td>2</td>
      <td>8</td>
      <td>8</td>
      <td>{'learning_rate': 0.2, 'loss': 'squared_error'...</td>
      <td>0.495745</td>
      <td>0.401067</td>
      <td>0.347681</td>
      <td>0.262674</td>
      <td>0.469976</td>
      <td>0.395429</td>
      <td>0.084328</td>
      <td>590</td>
    </tr>
    <tr>
      <th>...</th>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
    </tr>
    <tr>
      <th>724</th>
      <td>0.145537</td>
      <td>0.009800</td>
      <td>0.003565</td>
      <td>0.000112</td>
      <td>0.05</td>
      <td>poisson</td>
      <td>255</td>
      <td>32</td>
      <td>8</td>
      <td>8</td>
      <td>{'learning_rate': 0.05, 'loss': 'poisson', 'ma...</td>
      <td>0.536389</td>
      <td>0.477789</td>
      <td>0.396651</td>
      <td>0.323495</td>
      <td>0.397866</td>
      <td>0.426438</td>
      <td>0.073521</td>
      <td>427</td>
    </tr>
    <tr>
      <th>725</th>
      <td>0.102996</td>
      <td>0.001280</td>
      <td>0.003632</td>
      <td>0.000067</td>
      <td>0.05</td>
      <td>poisson</td>
      <td>255</td>
      <td>32</td>
      <td>8</td>
      <td>32</td>
      <td>{'learning_rate': 0.05, 'loss': 'poisson', 'ma...</td>
      <td>0.593431</td>
      <td>0.462807</td>
      <td>0.358350</td>
      <td>0.390233</td>
      <td>0.410602</td>
      <td>0.443085</td>
      <td>0.082488</td>
      <td>120</td>
    </tr>
    <tr>
      <th>726</th>
      <td>0.565122</td>
      <td>0.031652</td>
      <td>0.005548</td>
      <td>0.002976</td>
      <td>0.05</td>
      <td>poisson</td>
      <td>255</td>
      <td>32</td>
      <td>32</td>
      <td>2</td>
      <td>{'learning_rate': 0.05, 'loss': 'poisson', 'ma...</td>
      <td>0.474264</td>
      <td>0.399449</td>
      <td>0.398237</td>
      <td>0.223435</td>
      <td>0.258033</td>
      <td>0.350683</td>
      <td>0.094539</td>
      <td>679</td>
    </tr>
    <tr>
      <th>727</th>
      <td>0.378403</td>
      <td>0.015974</td>
      <td>0.004088</td>
      <td>0.000225</td>
      <td>0.05</td>
      <td>poisson</td>
      <td>255</td>
      <td>32</td>
      <td>32</td>
      <td>8</td>
      <td>{'learning_rate': 0.05, 'loss': 'poisson', 'ma...</td>
      <td>0.486323</td>
      <td>0.485514</td>
      <td>0.333050</td>
      <td>0.337733</td>
      <td>0.327221</td>
      <td>0.393968</td>
      <td>0.075152</td>
      <td>595</td>
    </tr>
    <tr>
      <th>728</th>
      <td>0.099264</td>
      <td>0.013943</td>
      <td>0.003375</td>
      <td>0.000581</td>
      <td>0.05</td>
      <td>poisson</td>
      <td>255</td>
      <td>32</td>
      <td>32</td>
      <td>32</td>
      <td>{'learning_rate': 0.05, 'loss': 'poisson', 'ma...</td>
      <td>0.593431</td>
      <td>0.462807</td>
      <td>0.358350</td>
      <td>0.390233</td>
      <td>0.410602</td>
      <td>0.443085</td>
      <td>0.082488</td>
      <td>120</td>
    </tr>
  </tbody>
</table>
<p>729 rows × 19 columns</p>
</div>

今回、６種類のハイパーパラメータがそれぞれ３個の値を持っていたので `3 * 3 * 3 * 3 * 3 * 3 = 729` ということで、累計 729 個の機械学習モデルを生成したということで計算は合っていますね。

どのようなハイパラを探索していたか確認してみましょう。


```python
pd.DataFrame(model.cv_results_["params"])
```





  <div id="df-7a03b29a-e798-4d4f-8ab1-739d12250b97">
    <div class="colab-df-container">
      <div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>learning_rate</th>
      <th>loss</th>
      <th>max_bins</th>
      <th>max_depth</th>
      <th>max_leaf_nodes</th>
      <th>min_samples_leaf</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>0.20</td>
      <td>squared_error</td>
      <td>64</td>
      <td>2</td>
      <td>2</td>
      <td>2</td>
    </tr>
    <tr>
      <th>1</th>
      <td>0.20</td>
      <td>squared_error</td>
      <td>64</td>
      <td>2</td>
      <td>2</td>
      <td>8</td>
    </tr>
    <tr>
      <th>2</th>
      <td>0.20</td>
      <td>squared_error</td>
      <td>64</td>
      <td>2</td>
      <td>2</td>
      <td>32</td>
    </tr>
    <tr>
      <th>3</th>
      <td>0.20</td>
      <td>squared_error</td>
      <td>64</td>
      <td>2</td>
      <td>8</td>
      <td>2</td>
    </tr>
    <tr>
      <th>4</th>
      <td>0.20</td>
      <td>squared_error</td>
      <td>64</td>
      <td>2</td>
      <td>8</td>
      <td>8</td>
    </tr>
    <tr>
      <th>...</th>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
    </tr>
    <tr>
      <th>724</th>
      <td>0.05</td>
      <td>poisson</td>
      <td>255</td>
      <td>32</td>
      <td>8</td>
      <td>8</td>
    </tr>
    <tr>
      <th>725</th>
      <td>0.05</td>
      <td>poisson</td>
      <td>255</td>
      <td>32</td>
      <td>8</td>
      <td>32</td>
    </tr>
    <tr>
      <th>726</th>
      <td>0.05</td>
      <td>poisson</td>
      <td>255</td>
      <td>32</td>
      <td>32</td>
      <td>2</td>
    </tr>
    <tr>
      <th>727</th>
      <td>0.05</td>
      <td>poisson</td>
      <td>255</td>
      <td>32</td>
      <td>32</td>
      <td>8</td>
    </tr>
    <tr>
      <th>728</th>
      <td>0.05</td>
      <td>poisson</td>
      <td>255</td>
      <td>32</td>
      <td>32</td>
      <td>32</td>
    </tr>
  </tbody>
</table>
<p>729 rows × 6 columns</p>
</div>



次のように結合してみます。

```python
cv_results_df = pd.concat(
    [pd.DataFrame(model.cv_results_), pd.DataFrame(model.cv_results_["params"])], axis=1
)
cv_results_df
```





  <div id="df-68d05265-bb48-44bc-b441-ae813161aa49">
    <div class="colab-df-container">
      <div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>mean_fit_time</th>
      <th>std_fit_time</th>
      <th>mean_score_time</th>
      <th>std_score_time</th>
      <th>param_learning_rate</th>
      <th>param_loss</th>
      <th>param_max_bins</th>
      <th>param_max_depth</th>
      <th>param_max_leaf_nodes</th>
      <th>param_min_samples_leaf</th>
      <th>...</th>
      <th>split4_test_score</th>
      <th>mean_test_score</th>
      <th>std_test_score</th>
      <th>rank_test_score</th>
      <th>learning_rate</th>
      <th>loss</th>
      <th>max_bins</th>
      <th>max_depth</th>
      <th>max_leaf_nodes</th>
      <th>min_samples_leaf</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>0.046092</td>
      <td>0.001038</td>
      <td>0.003924</td>
      <td>0.000962</td>
      <td>0.2</td>
      <td>squared_error</td>
      <td>64</td>
      <td>2</td>
      <td>2</td>
      <td>2</td>
      <td>...</td>
      <td>0.504165</td>
      <td>0.430958</td>
      <td>0.071473</td>
      <td>344</td>
      <td>0.20</td>
      <td>squared_error</td>
      <td>64</td>
      <td>2</td>
      <td>2</td>
      <td>2</td>
    </tr>
    <tr>
      <th>1</th>
      <td>0.047879</td>
      <td>0.004027</td>
      <td>0.003343</td>
      <td>0.000078</td>
      <td>0.2</td>
      <td>squared_error</td>
      <td>64</td>
      <td>2</td>
      <td>2</td>
      <td>8</td>
      <td>...</td>
      <td>0.506877</td>
      <td>0.438365</td>
      <td>0.075322</td>
      <td>186</td>
      <td>0.20</td>
      <td>squared_error</td>
      <td>64</td>
      <td>2</td>
      <td>2</td>
      <td>8</td>
    </tr>
    <tr>
      <th>2</th>
      <td>0.044900</td>
      <td>0.001714</td>
      <td>0.003401</td>
      <td>0.000135</td>
      <td>0.2</td>
      <td>squared_error</td>
      <td>64</td>
      <td>2</td>
      <td>2</td>
      <td>32</td>
      <td>...</td>
      <td>0.467517</td>
      <td>0.430423</td>
      <td>0.082305</td>
      <td>357</td>
      <td>0.20</td>
      <td>squared_error</td>
      <td>64</td>
      <td>2</td>
      <td>2</td>
      <td>32</td>
    </tr>
    <tr>
      <th>3</th>
      <td>0.080580</td>
      <td>0.007363</td>
      <td>0.004332</td>
      <td>0.001743</td>
      <td>0.2</td>
      <td>squared_error</td>
      <td>64</td>
      <td>2</td>
      <td>8</td>
      <td>2</td>
      <td>...</td>
      <td>0.443050</td>
      <td>0.393265</td>
      <td>0.062492</td>
      <td>596</td>
      <td>0.20</td>
      <td>squared_error</td>
      <td>64</td>
      <td>2</td>
      <td>8</td>
      <td>2</td>
    </tr>
    <tr>
      <th>4</th>
      <td>0.069854</td>
      <td>0.004283</td>
      <td>0.003402</td>
      <td>0.000314</td>
      <td>0.2</td>
      <td>squared_error</td>
      <td>64</td>
      <td>2</td>
      <td>8</td>
      <td>8</td>
      <td>...</td>
      <td>0.469976</td>
      <td>0.395429</td>
      <td>0.084328</td>
      <td>590</td>
      <td>0.20</td>
      <td>squared_error</td>
      <td>64</td>
      <td>2</td>
      <td>8</td>
      <td>8</td>
    </tr>
    <tr>
      <th>...</th>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
    </tr>
    <tr>
      <th>724</th>
      <td>0.145537</td>
      <td>0.009800</td>
      <td>0.003565</td>
      <td>0.000112</td>
      <td>0.05</td>
      <td>poisson</td>
      <td>255</td>
      <td>32</td>
      <td>8</td>
      <td>8</td>
      <td>...</td>
      <td>0.397866</td>
      <td>0.426438</td>
      <td>0.073521</td>
      <td>427</td>
      <td>0.05</td>
      <td>poisson</td>
      <td>255</td>
      <td>32</td>
      <td>8</td>
      <td>8</td>
    </tr>
    <tr>
      <th>725</th>
      <td>0.102996</td>
      <td>0.001280</td>
      <td>0.003632</td>
      <td>0.000067</td>
      <td>0.05</td>
      <td>poisson</td>
      <td>255</td>
      <td>32</td>
      <td>8</td>
      <td>32</td>
      <td>...</td>
      <td>0.410602</td>
      <td>0.443085</td>
      <td>0.082488</td>
      <td>120</td>
      <td>0.05</td>
      <td>poisson</td>
      <td>255</td>
      <td>32</td>
      <td>8</td>
      <td>32</td>
    </tr>
    <tr>
      <th>726</th>
      <td>0.565122</td>
      <td>0.031652</td>
      <td>0.005548</td>
      <td>0.002976</td>
      <td>0.05</td>
      <td>poisson</td>
      <td>255</td>
      <td>32</td>
      <td>32</td>
      <td>2</td>
      <td>...</td>
      <td>0.258033</td>
      <td>0.350683</td>
      <td>0.094539</td>
      <td>679</td>
      <td>0.05</td>
      <td>poisson</td>
      <td>255</td>
      <td>32</td>
      <td>32</td>
      <td>2</td>
    </tr>
    <tr>
      <th>727</th>
      <td>0.378403</td>
      <td>0.015974</td>
      <td>0.004088</td>
      <td>0.000225</td>
      <td>0.05</td>
      <td>poisson</td>
      <td>255</td>
      <td>32</td>
      <td>32</td>
      <td>8</td>
      <td>...</td>
      <td>0.327221</td>
      <td>0.393968</td>
      <td>0.075152</td>
      <td>595</td>
      <td>0.05</td>
      <td>poisson</td>
      <td>255</td>
      <td>32</td>
      <td>32</td>
      <td>8</td>
    </tr>
    <tr>
      <th>728</th>
      <td>0.099264</td>
      <td>0.013943</td>
      <td>0.003375</td>
      <td>0.000581</td>
      <td>0.05</td>
      <td>poisson</td>
      <td>255</td>
      <td>32</td>
      <td>32</td>
      <td>32</td>
      <td>...</td>
      <td>0.410602</td>
      <td>0.443085</td>
      <td>0.082488</td>
      <td>120</td>
      <td>0.05</td>
      <td>poisson</td>
      <td>255</td>
      <td>32</td>
      <td>32</td>
      <td>32</td>
    </tr>
  </tbody>
</table>
<p>729 rows × 25 columns</p>
</div>

次のようにすれば、各パラメーターが性能にどのくらい影響したかを確認できます。ただし、ここでいう `mean_test_score` は、ややこしいのですが、`(X_test, Y_test)` に対する`score`ではなくて、`(X_train, Y_train)` を `cv=5` として分割したときの test_score の平均ということになります。


```python
for col in cv_results_df.columns:
    if col[:6] == "param_":
        cv_results_df.plot(col, "mean_test_score", kind="scatter", alpha=0.5, grid=True)
```


    
    
![output_6_0.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/211162/04eaf07c-6616-7deb-8e22-924152453d31.png)



        
![output_6_1.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/211162/becf9aa6-a55e-dc24-c555-3e760287081c.png)



    
    
![output_6_2.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/211162/fdd5a14d-63ad-b1aa-8be2-24f50231f64e.png)



    
    
![output_6_3.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/211162/d8b304bf-cdc9-fb57-c50f-10b8f55e958b.png)



    
    
![output_6_4.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/211162/a63db425-a029-f2a2-db98-3928af6480f1.png)



    
    
![output_6_5.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/211162/f7f4ed48-8c5c-acc3-a1c7-456c49178ef6.png)

# HalvingGridSearchCV

次は HalvingGridSearchCV です。これは、GridSearchCV と同様に全組み合わせを探索した後に、スコアの高かった組み合わせを取り出し、上位のモデルに対してより多い `resource` を与えて再学習を行うという操作を繰り返します。

上位いくつの組み合わせを選ぶかは `factor` というパラメーターで調整できます（たとえば `factor=2` とすれば上位 `1/2` のモデルが選択されます）。

`resource` とは何か、というのは base となる estimator によって変わるらしく、GradientBoosting のようなアンサンブル学習器なら `n_estimators` になるようです。


```python
from sklearn.experimental import enable_halving_search_cv
from sklearn.model_selection import HalvingGridSearchCV

parameters = dict(
    {
        "loss": ["squared_error", "absolute_error", "poisson"],
        "learning_rate": [0.2, 0.1, 0.05],
        "max_leaf_nodes": [2, 8, 32],
        "max_depth": [2, 8, 32],
        "min_samples_leaf": [2, 8, 32],
        "max_bins": [64, 128, 255],
    }
)
model = HalvingGridSearchCV(
    HistGradientBoostingRegressor(), parameters, cv=5, n_jobs=-1
)
model.fit(X_train, Y_train)
print("R2=", model.score(X_test, Y_test))
model.best_estimator_
```



    R2= 0.45168983833467624


お、けっこう性能が向上しましたね。


    HistGradientBoostingRegressor(learning_rate=0.2, loss='absolute_error',
                                  max_bins=64, max_depth=2, max_leaf_nodes=2,
                                  min_samples_leaf=2)


クロスバリデーションの詳細は次の通りです。 `iter=0` では単純な GridSearchCV と同じ `3 * 3 * 3 * 3 * 3 * 3 = 729` 個のモデルが学習され、上位 `1/factor` 個のモデルが選択され、より多い `resource` で再学習されるということを繰り返します。

```python
cv_results_df = pd.concat(
    [pd.DataFrame(model.cv_results_), pd.DataFrame(model.cv_results_["params"])], axis=1
)
cv_results_df
```





  <div id="df-14e074fb-b3b9-40d5-99d5-4c9d5ed5b8ca">
    <div class="colab-df-container">
      <div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>iter</th>
      <th>n_resources</th>
      <th>mean_fit_time</th>
      <th>std_fit_time</th>
      <th>mean_score_time</th>
      <th>std_score_time</th>
      <th>param_learning_rate</th>
      <th>param_loss</th>
      <th>param_max_bins</th>
      <th>param_max_depth</th>
      <th>...</th>
      <th>split3_train_score</th>
      <th>split4_train_score</th>
      <th>mean_train_score</th>
      <th>std_train_score</th>
      <th>learning_rate</th>
      <th>loss</th>
      <th>max_bins</th>
      <th>max_depth</th>
      <th>max_leaf_nodes</th>
      <th>min_samples_leaf</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>0</td>
      <td>10</td>
      <td>0.073146</td>
      <td>0.035112</td>
      <td>0.008809</td>
      <td>0.005226</td>
      <td>0.2</td>
      <td>squared_error</td>
      <td>64</td>
      <td>2</td>
      <td>...</td>
      <td>0.999956</td>
      <td>0.999969</td>
      <td>0.999855</td>
      <td>2.293240e-04</td>
      <td>0.20</td>
      <td>squared_error</td>
      <td>64</td>
      <td>2</td>
      <td>2</td>
      <td>2</td>
    </tr>
    <tr>
      <th>1</th>
      <td>0</td>
      <td>10</td>
      <td>0.036513</td>
      <td>0.015857</td>
      <td>0.007700</td>
      <td>0.005450</td>
      <td>0.2</td>
      <td>squared_error</td>
      <td>64</td>
      <td>2</td>
      <td>...</td>
      <td>0.000000</td>
      <td>0.000000</td>
      <td>0.000000</td>
      <td>0.000000e+00</td>
      <td>0.20</td>
      <td>squared_error</td>
      <td>64</td>
      <td>2</td>
      <td>2</td>
      <td>8</td>
    </tr>
    <tr>
      <th>2</th>
      <td>0</td>
      <td>10</td>
      <td>0.048790</td>
      <td>0.014432</td>
      <td>0.005903</td>
      <td>0.003738</td>
      <td>0.2</td>
      <td>squared_error</td>
      <td>64</td>
      <td>2</td>
      <td>...</td>
      <td>0.000000</td>
      <td>0.000000</td>
      <td>0.000000</td>
      <td>0.000000e+00</td>
      <td>0.20</td>
      <td>squared_error</td>
      <td>64</td>
      <td>2</td>
      <td>2</td>
      <td>32</td>
    </tr>
    <tr>
      <th>3</th>
      <td>0</td>
      <td>10</td>
      <td>0.117461</td>
      <td>0.019852</td>
      <td>0.007106</td>
      <td>0.005961</td>
      <td>0.2</td>
      <td>squared_error</td>
      <td>64</td>
      <td>2</td>
      <td>...</td>
      <td>1.000000</td>
      <td>1.000000</td>
      <td>1.000000</td>
      <td>2.120790e-08</td>
      <td>0.20</td>
      <td>squared_error</td>
      <td>64</td>
      <td>2</td>
      <td>8</td>
      <td>2</td>
    </tr>
    <tr>
      <th>4</th>
      <td>0</td>
      <td>10</td>
      <td>0.052612</td>
      <td>0.004677</td>
      <td>0.004469</td>
      <td>0.002280</td>
      <td>0.2</td>
      <td>squared_error</td>
      <td>64</td>
      <td>2</td>
      <td>...</td>
      <td>0.000000</td>
      <td>0.000000</td>
      <td>0.000000</td>
      <td>0.000000e+00</td>
      <td>0.20</td>
      <td>squared_error</td>
      <td>64</td>
      <td>2</td>
      <td>8</td>
      <td>8</td>
    </tr>
    <tr>
      <th>...</th>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
    </tr>
    <tr>
      <th>1075</th>
      <td>3</td>
      <td>270</td>
      <td>0.093966</td>
      <td>0.012485</td>
      <td>0.007732</td>
      <td>0.006771</td>
      <td>0.2</td>
      <td>poisson</td>
      <td>128</td>
      <td>8</td>
      <td>...</td>
      <td>0.633315</td>
      <td>0.616487</td>
      <td>0.614829</td>
      <td>3.458072e-02</td>
      <td>0.20</td>
      <td>poisson</td>
      <td>128</td>
      <td>8</td>
      <td>2</td>
      <td>32</td>
    </tr>
    <tr>
      <th>1076</th>
      <td>3</td>
      <td>270</td>
      <td>0.136020</td>
      <td>0.012811</td>
      <td>0.003038</td>
      <td>0.000059</td>
      <td>0.2</td>
      <td>poisson</td>
      <td>128</td>
      <td>2</td>
      <td>...</td>
      <td>0.793938</td>
      <td>0.759121</td>
      <td>0.779202</td>
      <td>2.804673e-02</td>
      <td>0.20</td>
      <td>poisson</td>
      <td>128</td>
      <td>2</td>
      <td>32</td>
      <td>32</td>
    </tr>
    <tr>
      <th>1077</th>
      <td>3</td>
      <td>270</td>
      <td>0.145647</td>
      <td>0.047377</td>
      <td>0.005804</td>
      <td>0.003204</td>
      <td>0.2</td>
      <td>poisson</td>
      <td>128</td>
      <td>2</td>
      <td>...</td>
      <td>0.793938</td>
      <td>0.759121</td>
      <td>0.779202</td>
      <td>2.804673e-02</td>
      <td>0.20</td>
      <td>poisson</td>
      <td>128</td>
      <td>2</td>
      <td>8</td>
      <td>32</td>
    </tr>
    <tr>
      <th>1078</th>
      <td>3</td>
      <td>270</td>
      <td>0.129800</td>
      <td>0.026974</td>
      <td>0.003437</td>
      <td>0.000220</td>
      <td>0.2</td>
      <td>poisson</td>
      <td>64</td>
      <td>32</td>
      <td>...</td>
      <td>0.876194</td>
      <td>0.870592</td>
      <td>0.870216</td>
      <td>1.899521e-02</td>
      <td>0.20</td>
      <td>poisson</td>
      <td>64</td>
      <td>32</td>
      <td>32</td>
      <td>32</td>
    </tr>
    <tr>
      <th>1079</th>
      <td>3</td>
      <td>270</td>
      <td>0.129445</td>
      <td>0.042735</td>
      <td>0.004040</td>
      <td>0.001468</td>
      <td>0.05</td>
      <td>poisson</td>
      <td>255</td>
      <td>32</td>
      <td>...</td>
      <td>0.723216</td>
      <td>0.705591</td>
      <td>0.705248</td>
      <td>3.103351e-02</td>
      <td>0.05</td>
      <td>poisson</td>
      <td>255</td>
      <td>32</td>
      <td>32</td>
      <td>32</td>
    </tr>
  </tbody>
</table>
<p>1080 rows × 34 columns</p>
</div>


各パラメータが性能にどのような影響を与えたかは次のようにして見られます。

```python
for col in cv_results_df.columns:
    if col[:6] == "param_":
        cv_results_df.plot(col, "mean_test_score", kind="scatter", alpha=0.5, grid=True)
```


    
    
![output_9_0.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/211162/aadc6ee9-e94b-4744-8823-c913788d0d66.png)



    
    
![output_9_1.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/211162/bda4580e-4144-1f1a-197d-7aad475f0efc.png)



    
    
![output_9_2.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/211162/f02d9a5c-01df-a76a-11f9-9622d5927547.png)



    
    
![output_9_3.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/211162/b09e1d6a-ac75-fbb0-e73e-13b2fe848ea0.png)



    
    
![output_9_4.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/211162/3bc1968a-7dc7-53f1-bfeb-dc38de788492.png)



    
    
![output_9_5.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/211162/004eefeb-bbae-a724-fa2b-9b14d0e75935.png)

より多い `resource` で再学習する分、性能が向上するというのは良いとしても、ハイパラの探索範囲は GridSearchCV と変わらないということですね。

# RandomizedSearchCV

次は RandomizedSearchCV です。これは、名前から想像できる通り、指定したハイパーパラメーターの範囲をランダムにサーチします。反復回数は `n_iter=100` のように指定します。


```python
from scipy.stats import randint, uniform
from sklearn.model_selection import RandomizedSearchCV

parameters = dict(
    {
        "loss": ["squared_error", "absolute_error", "poisson"],
        "learning_rate": uniform(0.2, 0.05),
        "max_leaf_nodes": randint(2, 32),
        "max_depth": randint(2, 32),
        "min_samples_leaf": randint(2, 32),
        "max_bins": randint(64, 255),
    }
)
model = RandomizedSearchCV(
    HistGradientBoostingRegressor(), parameters, cv=5, n_jobs=-1, n_iter=100
)
model.fit(X_train, Y_train)
print("R2=", model.score(X_test, Y_test))
model.best_estimator_
```

    R2= 0.44293642952519996


今回、GridSearchCVよりは改善しましたが、HalvingGridSearchCVには負けた、という形ですね。


    HistGradientBoostingRegressor(learning_rate=0.2401623294668279,
                                  loss='absolute_error', max_bins=100, max_depth=27,
                                  max_leaf_nodes=2, min_samples_leaf=31)


結果の詳細です。

```python
cv_results_df = pd.concat(
    [pd.DataFrame(model.cv_results_), pd.DataFrame(model.cv_results_["params"])], axis=1
)
cv_results_df
```





  <div id="df-e98af16a-c16c-4523-be86-2b38686e974e">
    <div class="colab-df-container">
      <div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>mean_fit_time</th>
      <th>std_fit_time</th>
      <th>mean_score_time</th>
      <th>std_score_time</th>
      <th>param_learning_rate</th>
      <th>param_loss</th>
      <th>param_max_bins</th>
      <th>param_max_depth</th>
      <th>param_max_leaf_nodes</th>
      <th>param_min_samples_leaf</th>
      <th>...</th>
      <th>split4_test_score</th>
      <th>mean_test_score</th>
      <th>std_test_score</th>
      <th>rank_test_score</th>
      <th>learning_rate</th>
      <th>loss</th>
      <th>max_bins</th>
      <th>max_depth</th>
      <th>max_leaf_nodes</th>
      <th>min_samples_leaf</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>0.366715</td>
      <td>0.053425</td>
      <td>0.014998</td>
      <td>0.003574</td>
      <td>0.230097</td>
      <td>poisson</td>
      <td>82</td>
      <td>3</td>
      <td>30</td>
      <td>9</td>
      <td>...</td>
      <td>0.274491</td>
      <td>0.323229</td>
      <td>0.060283</td>
      <td>70</td>
      <td>0.230097</td>
      <td>poisson</td>
      <td>82</td>
      <td>3</td>
      <td>30</td>
      <td>9</td>
    </tr>
    <tr>
      <th>1</th>
      <td>0.506211</td>
      <td>0.144560</td>
      <td>0.011002</td>
      <td>0.006287</td>
      <td>0.226155</td>
      <td>absolute_error</td>
      <td>173</td>
      <td>27</td>
      <td>5</td>
      <td>10</td>
      <td>...</td>
      <td>0.428027</td>
      <td>0.419660</td>
      <td>0.093885</td>
      <td>8</td>
      <td>0.226155</td>
      <td>absolute_error</td>
      <td>173</td>
      <td>27</td>
      <td>5</td>
      <td>10</td>
    </tr>
    <tr>
      <th>2</th>
      <td>0.221872</td>
      <td>0.035509</td>
      <td>0.012181</td>
      <td>0.007027</td>
      <td>0.200513</td>
      <td>absolute_error</td>
      <td>66</td>
      <td>23</td>
      <td>5</td>
      <td>30</td>
      <td>...</td>
      <td>0.410435</td>
      <td>0.419785</td>
      <td>0.091442</td>
      <td>7</td>
      <td>0.200513</td>
      <td>absolute_error</td>
      <td>66</td>
      <td>23</td>
      <td>5</td>
      <td>30</td>
    </tr>
    <tr>
      <th>3</th>
      <td>0.196586</td>
      <td>0.026616</td>
      <td>0.006464</td>
      <td>0.003607</td>
      <td>0.213685</td>
      <td>squared_error</td>
      <td>106</td>
      <td>31</td>
      <td>30</td>
      <td>29</td>
      <td>...</td>
      <td>0.322769</td>
      <td>0.327327</td>
      <td>0.089463</td>
      <td>64</td>
      <td>0.213685</td>
      <td>squared_error</td>
      <td>106</td>
      <td>31</td>
      <td>30</td>
      <td>29</td>
    </tr>
    <tr>
      <th>4</th>
      <td>0.192499</td>
      <td>0.050251</td>
      <td>0.007767</td>
      <td>0.003510</td>
      <td>0.227877</td>
      <td>squared_error</td>
      <td>108</td>
      <td>5</td>
      <td>17</td>
      <td>19</td>
      <td>...</td>
      <td>0.254810</td>
      <td>0.287950</td>
      <td>0.057030</td>
      <td>97</td>
      <td>0.227877</td>
      <td>squared_error</td>
      <td>108</td>
      <td>5</td>
      <td>17</td>
      <td>19</td>
    </tr>
    <tr>
      <th>...</th>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
    </tr>
    <tr>
      <th>95</th>
      <td>0.399913</td>
      <td>0.010536</td>
      <td>0.003984</td>
      <td>0.000305</td>
      <td>0.22602</td>
      <td>absolute_error</td>
      <td>170</td>
      <td>26</td>
      <td>15</td>
      <td>5</td>
      <td>...</td>
      <td>0.436215</td>
      <td>0.409727</td>
      <td>0.062342</td>
      <td>17</td>
      <td>0.226020</td>
      <td>absolute_error</td>
      <td>170</td>
      <td>26</td>
      <td>15</td>
      <td>5</td>
    </tr>
    <tr>
      <th>96</th>
      <td>0.083641</td>
      <td>0.015314</td>
      <td>0.003447</td>
      <td>0.000209</td>
      <td>0.206473</td>
      <td>poisson</td>
      <td>116</td>
      <td>2</td>
      <td>30</td>
      <td>4</td>
      <td>...</td>
      <td>0.376263</td>
      <td>0.337107</td>
      <td>0.027581</td>
      <td>53</td>
      <td>0.206473</td>
      <td>poisson</td>
      <td>116</td>
      <td>2</td>
      <td>30</td>
      <td>4</td>
    </tr>
    <tr>
      <th>97</th>
      <td>0.163723</td>
      <td>0.090995</td>
      <td>0.009293</td>
      <td>0.004980</td>
      <td>0.23693</td>
      <td>poisson</td>
      <td>194</td>
      <td>3</td>
      <td>13</td>
      <td>13</td>
      <td>...</td>
      <td>0.285729</td>
      <td>0.282265</td>
      <td>0.052020</td>
      <td>99</td>
      <td>0.236930</td>
      <td>poisson</td>
      <td>194</td>
      <td>3</td>
      <td>13</td>
      <td>13</td>
    </tr>
    <tr>
      <th>98</th>
      <td>0.551714</td>
      <td>0.115441</td>
      <td>0.004080</td>
      <td>0.000183</td>
      <td>0.233486</td>
      <td>squared_error</td>
      <td>159</td>
      <td>16</td>
      <td>29</td>
      <td>2</td>
      <td>...</td>
      <td>0.315230</td>
      <td>0.323911</td>
      <td>0.061675</td>
      <td>68</td>
      <td>0.233486</td>
      <td>squared_error</td>
      <td>159</td>
      <td>16</td>
      <td>29</td>
      <td>2</td>
    </tr>
    <tr>
      <th>99</th>
      <td>0.181508</td>
      <td>0.039065</td>
      <td>0.003228</td>
      <td>0.000828</td>
      <td>0.221521</td>
      <td>squared_error</td>
      <td>249</td>
      <td>18</td>
      <td>12</td>
      <td>6</td>
      <td>...</td>
      <td>0.363508</td>
      <td>0.371630</td>
      <td>0.068751</td>
      <td>40</td>
      <td>0.221521</td>
      <td>squared_error</td>
      <td>249</td>
      <td>18</td>
      <td>12</td>
      <td>6</td>
    </tr>
  </tbody>
</table>
<p>100 rows × 25 columns</p>
</div>

ランダムなので、最適なハイパラの値がほとんど見当がつかないような時に便利なんだろうと思います。

```python
for col in cv_results_df.columns:
    if col[:6] == "param_":
        cv_results_df.plot(col, "mean_test_score", kind="scatter", alpha=0.5, grid=True)
```


    
    
![output_12_0.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/211162/edf4a913-8c81-9140-9eda-eb43d731db5a.png)



    
    
![output_12_1.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/211162/f7c66341-c420-b112-e6e1-a88c769378be.png)



    
    
![output_12_2.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/211162/ac70fdd0-f44e-0617-9509-81e10490d89e.png)



    
    
![output_12_3.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/211162/d0d33d10-4f72-2af5-d587-36bc805afc9c.png)



    
    
![output_12_4.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/211162/f0005734-81dc-236f-a635-fe9ba7e8c9c5.png)



    
    
![output_12_5.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/211162/0c7e7d43-b41a-b8da-cab5-f9de984a2c88.png)

# HalvingRandomSearchCV

最後に、HalvingRandomSearchCVです。これは、GridSearchCVに対するHalvingGridSearchCVと同様に、RandomSearchCVに対して、上位いくつかのモデルに対してresouceを増やして再計算するというものです。


```python
from scipy.stats import randint, uniform
from sklearn.experimental import enable_halving_search_cv
from sklearn.model_selection import HalvingRandomSearchCV

parameters = dict(
    {
        "loss": ["squared_error", "absolute_error", "poisson"],
        "learning_rate": uniform(0.2, 0.05),
        "max_leaf_nodes": randint(2, 32),
        "max_depth": randint(2, 32),
        "min_samples_leaf": randint(2, 32),
        "max_bins": randint(64, 255),
    }
)
model = HalvingRandomSearchCV(
    HistGradientBoostingRegressor(), parameters, cv=5, n_jobs=-1, factor=1.1
)
model.fit(X_train, Y_train)
print("R2=", model.score(X_test, Y_test))
model.best_estimator_
```




    R2= 0.36263251102607696


残念ながら、あまり良い成績は出ませんでしたね（私が上手に使いこなしていないだけかもしれませんが）。


    HistGradientBoostingRegressor(learning_rate=0.23484210133032557,
                                  loss='absolute_error', max_bins=176, max_depth=19,
                                  max_leaf_nodes=16, min_samples_leaf=25)


詳細な結果は次のとおりです。

```python
cv_results_df = pd.concat(
    [pd.DataFrame(model.cv_results_), pd.DataFrame(model.cv_results_["params"])], axis=1
)
cv_results_df
```





  <div id="df-4729407c-60c5-47a9-8843-c97dbeeb9633">
    <div class="colab-df-container">
      <div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>iter</th>
      <th>n_resources</th>
      <th>mean_fit_time</th>
      <th>std_fit_time</th>
      <th>mean_score_time</th>
      <th>std_score_time</th>
      <th>param_learning_rate</th>
      <th>param_loss</th>
      <th>param_max_bins</th>
      <th>param_max_depth</th>
      <th>...</th>
      <th>split3_train_score</th>
      <th>split4_train_score</th>
      <th>mean_train_score</th>
      <th>std_train_score</th>
      <th>learning_rate</th>
      <th>loss</th>
      <th>max_bins</th>
      <th>max_depth</th>
      <th>max_leaf_nodes</th>
      <th>min_samples_leaf</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>0</td>
      <td>10</td>
      <td>0.025926</td>
      <td>0.002128</td>
      <td>0.003607</td>
      <td>0.001518</td>
      <td>0.213565</td>
      <td>poisson</td>
      <td>155</td>
      <td>30</td>
      <td>...</td>
      <td>0.000000</td>
      <td>0.000000</td>
      <td>-4.440892e-17</td>
      <td>8.881784e-17</td>
      <td>0.213565</td>
      <td>poisson</td>
      <td>155</td>
      <td>30</td>
      <td>5</td>
      <td>25</td>
    </tr>
    <tr>
      <th>1</th>
      <td>0</td>
      <td>10</td>
      <td>0.039488</td>
      <td>0.001897</td>
      <td>0.002757</td>
      <td>0.000126</td>
      <td>0.203592</td>
      <td>absolute_error</td>
      <td>238</td>
      <td>14</td>
      <td>...</td>
      <td>-0.031997</td>
      <td>-0.014562</td>
      <td>-1.044763e-02</td>
      <td>1.188607e-02</td>
      <td>0.203592</td>
      <td>absolute_error</td>
      <td>238</td>
      <td>14</td>
      <td>11</td>
      <td>7</td>
    </tr>
    <tr>
      <th>2</th>
      <td>0</td>
      <td>10</td>
      <td>0.027985</td>
      <td>0.003291</td>
      <td>0.003110</td>
      <td>0.000359</td>
      <td>0.234896</td>
      <td>poisson</td>
      <td>212</td>
      <td>26</td>
      <td>...</td>
      <td>0.000000</td>
      <td>0.000000</td>
      <td>-4.440892e-17</td>
      <td>8.881784e-17</td>
      <td>0.234896</td>
      <td>poisson</td>
      <td>212</td>
      <td>26</td>
      <td>2</td>
      <td>20</td>
    </tr>
    <tr>
      <th>3</th>
      <td>0</td>
      <td>10</td>
      <td>0.027623</td>
      <td>0.006139</td>
      <td>0.004061</td>
      <td>0.002246</td>
      <td>0.222381</td>
      <td>poisson</td>
      <td>213</td>
      <td>25</td>
      <td>...</td>
      <td>0.000000</td>
      <td>0.000000</td>
      <td>-4.440892e-17</td>
      <td>8.881784e-17</td>
      <td>0.222381</td>
      <td>poisson</td>
      <td>213</td>
      <td>25</td>
      <td>18</td>
      <td>5</td>
    </tr>
    <tr>
      <th>4</th>
      <td>0</td>
      <td>10</td>
      <td>0.025690</td>
      <td>0.002067</td>
      <td>0.002841</td>
      <td>0.000024</td>
      <td>0.245578</td>
      <td>poisson</td>
      <td>138</td>
      <td>19</td>
      <td>...</td>
      <td>0.000000</td>
      <td>0.000000</td>
      <td>-4.440892e-17</td>
      <td>8.881784e-17</td>
      <td>0.245578</td>
      <td>poisson</td>
      <td>138</td>
      <td>19</td>
      <td>6</td>
      <td>26</td>
    </tr>
    <tr>
      <th>...</th>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
    </tr>
    <tr>
      <th>523</th>
      <td>36</td>
      <td>309</td>
      <td>0.108649</td>
      <td>0.005772</td>
      <td>0.003502</td>
      <td>0.000146</td>
      <td>0.246763</td>
      <td>poisson</td>
      <td>135</td>
      <td>20</td>
      <td>...</td>
      <td>0.934394</td>
      <td>0.927940</td>
      <td>9.328288e-01</td>
      <td>6.655434e-03</td>
      <td>0.246763</td>
      <td>poisson</td>
      <td>135</td>
      <td>20</td>
      <td>15</td>
      <td>30</td>
    </tr>
    <tr>
      <th>524</th>
      <td>36</td>
      <td>309</td>
      <td>0.252199</td>
      <td>0.070923</td>
      <td>0.005747</td>
      <td>0.003214</td>
      <td>0.234842</td>
      <td>absolute_error</td>
      <td>176</td>
      <td>19</td>
      <td>...</td>
      <td>0.802983</td>
      <td>0.786075</td>
      <td>7.970137e-01</td>
      <td>8.098674e-03</td>
      <td>0.234842</td>
      <td>absolute_error</td>
      <td>176</td>
      <td>19</td>
      <td>16</td>
      <td>25</td>
    </tr>
    <tr>
      <th>525</th>
      <td>36</td>
      <td>309</td>
      <td>0.080884</td>
      <td>0.028918</td>
      <td>0.005850</td>
      <td>0.002895</td>
      <td>0.234896</td>
      <td>poisson</td>
      <td>212</td>
      <td>26</td>
      <td>...</td>
      <td>0.661155</td>
      <td>0.632498</td>
      <td>6.533541e-01</td>
      <td>1.317376e-02</td>
      <td>0.234896</td>
      <td>poisson</td>
      <td>212</td>
      <td>26</td>
      <td>2</td>
      <td>20</td>
    </tr>
    <tr>
      <th>526</th>
      <td>36</td>
      <td>309</td>
      <td>0.090304</td>
      <td>0.029305</td>
      <td>0.007267</td>
      <td>0.003304</td>
      <td>0.219922</td>
      <td>poisson</td>
      <td>205</td>
      <td>30</td>
      <td>...</td>
      <td>0.662939</td>
      <td>0.635429</td>
      <td>6.565455e-01</td>
      <td>1.248445e-02</td>
      <td>0.219922</td>
      <td>poisson</td>
      <td>205</td>
      <td>30</td>
      <td>2</td>
      <td>9</td>
    </tr>
    <tr>
      <th>527</th>
      <td>36</td>
      <td>309</td>
      <td>0.332935</td>
      <td>0.057465</td>
      <td>0.005539</td>
      <td>0.003033</td>
      <td>0.218705</td>
      <td>absolute_error</td>
      <td>142</td>
      <td>11</td>
      <td>...</td>
      <td>0.795818</td>
      <td>0.769511</td>
      <td>7.827088e-01</td>
      <td>1.634937e-02</td>
      <td>0.218705</td>
      <td>absolute_error</td>
      <td>142</td>
      <td>11</td>
      <td>16</td>
      <td>26</td>
    </tr>
  </tbody>
</table>
<p>528 rows × 34 columns</p>
</div>


以下の図を見ればわかりますが、最初の iteration の RandomSearchCV で決めたパラメータを変更することなく resource だけ変更しているようです。ここんとこ、改善の余地がありそうな気がしますね。

```python
for col in cv_results_df.columns:
    if col[:6] == "param_":
        cv_results_df.plot(col, "mean_test_score", kind="scatter", alpha=0.5, grid=True)
```


    
    
![output_15_0.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/211162/964a8c49-ef68-5dd7-66eb-810fc28cd484.png)



    
    
![output_15_1.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/211162/4c139323-044f-64db-8b17-ed117eee4bef.png)



    
    
![output_15_2.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/211162/143d33b4-a5ac-6504-cc04-1a65923ddacb.png)



    
    
![output_15_3.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/211162/99810610-11a1-ea5d-6def-a6abe3c637ff.png)



    
    
![output_15_4.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/211162/2cde6ea4-49db-92e2-8b79-d6a72a786770.png)



    
    
![output_15_5.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/211162/fcb86442-e3fc-bbf0-1346-8f14173506f6.png)

# おわりに

というわけで、scikit-learn の GridSearchCV とその兄弟たち HalvingGridSearchCV RandomizedSearchCV HalvingRandomSearchCV を使ってみました。その挙動は色々と参考になりますが、個人的にはやっぱり[Optuna](https://qiita.com/maskot1977/items/d768b627f102b6d94ce1)が柔軟な使い方ができて良いかなと再認識しました。私が使いこなしてないだけかもしれませんが。