UMAPの教師ありハイパーパラメーターチューニング

[「t-SNEの教師ありハイパーパラメーターチューニング」](https://qiita.com/maskot1977/items/2213e33c31cfc5403bf6)と[「Isomapの教師ありハイパーパラメーターチューニング」](https://qiita.com/maskot1977/items/9096a736a7442fd4fbf1)の続編です。


# UMAP のインストール

UMAP は scikit-learn には実装されていないのでインストールします。


```python
!pip install umap-learn
```

    Collecting umap-learn
      Downloading umap-learn-0.5.2.tar.gz (86 kB)
    [K     |████████████████████████████████| 86 kB 4.3 MB/s 
    [?25hRequirement already satisfied: numpy>=1.17 in /usr/local/lib/python3.7/dist-packages (from umap-learn) (1.21.5)
    Requirement already satisfied: scikit-learn>=0.22 in /usr/local/lib/python3.7/dist-packages (from umap-learn) (1.0.2)
    Requirement already satisfied: scipy>=1.0 in /usr/local/lib/python3.7/dist-packages (from umap-learn) (1.4.1)
    Requirement already satisfied: numba>=0.49 in /usr/local/lib/python3.7/dist-packages (from umap-learn) (0.51.2)
    Collecting pynndescent>=0.5
      Downloading pynndescent-0.5.6.tar.gz (1.1 MB)
    [K     |████████████████████████████████| 1.1 MB 18.1 MB/s 
    [?25hRequirement already satisfied: tqdm in /usr/local/lib/python3.7/dist-packages (from umap-learn) (4.63.0)
    Requirement already satisfied: setuptools in /usr/local/lib/python3.7/dist-packages (from numba>=0.49->umap-learn) (57.4.0)
    Requirement already satisfied: llvmlite<0.35,>=0.34.0.dev0 in /usr/local/lib/python3.7/dist-packages (from numba>=0.49->umap-learn) (0.34.0)
    Requirement already satisfied: joblib>=0.11 in /usr/local/lib/python3.7/dist-packages (from pynndescent>=0.5->umap-learn) (1.1.0)
    Requirement already satisfied: threadpoolctl>=2.0.0 in /usr/local/lib/python3.7/dist-packages (from scikit-learn>=0.22->umap-learn) (3.1.0)
    Building wheels for collected packages: umap-learn, pynndescent
      Building wheel for umap-learn (setup.py) ... [?25l[?25hdone
      Created wheel for umap-learn: filename=umap_learn-0.5.2-py3-none-any.whl size=82708 sha256=2ad8270ad7ff916a8cafad0b4ca1352d32e05d95597ed159180383308e885b42
      Stored in directory: /root/.cache/pip/wheels/84/1b/c6/aaf68a748122632967cef4dffef68224eb16798b6793257d82
      Building wheel for pynndescent (setup.py) ... [?25l[?25hdone
      Created wheel for pynndescent: filename=pynndescent-0.5.6-py3-none-any.whl size=53943 sha256=90c3c97fea10d5a6f03000dd193fd4c848b1fa9825d2660b2ecac3ab98214097
      Stored in directory: /root/.cache/pip/wheels/03/f1/56/f80d72741e400345b5a5b50ec3d929aca581bf45e0225d5c50
    Successfully built umap-learn pynndescent
    Installing collected packages: pynndescent, umap-learn
    Successfully installed pynndescent-0.5.6 umap-learn-0.5.2


# UMAP でパラメータを変化させる

お試し用のデータとして、scikit-learn で取得できる糖尿病データを用いてみます。


```python
from umap import UMAP
import sklearn.datasets
import matplotlib.pyplot as plt

dataset = sklearn.datasets.load_diabetes()

mapper = UMAP()
mapper.fit(dataset.data)
embedding = mapper.transform(dataset.data)
plt.scatter(embedding[:, 0], embedding[:, 1], c=dataset.target, alpha=0.5)
plt.colorbar()
plt.show()
```

![UMAPの教師ありハイパラチューニング_3_1.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/211162/ae04a423-094f-5b75-1892-b89c678d7717.png)




    

    


ここで、点の色は dataset.target にある変数で、この値は UMAP の学習に用いられていないことに注意してください。 UMAP は「教師なし学習」の一種であり、学習に用いた高次元データの構造を可視化する働きをするのですが、その結果として、学習に用いていない変数との関係が見えてくることもあります。

さて、パラメータを変えると、このマッピングがどのくらい変わってしまうのかを見てみましょう。

## n_neighbors


```python
from umap import UMAP
import sklearn.datasets
import matplotlib.pyplot as plt

dataset = sklearn.datasets.load_diabetes()

for n_neighbors in (2, 5, 10, 30, 50, 100):
    mapper = UMAP(n_neighbors=n_neighbors)
    mapper.fit(dataset.data)
    embedding = mapper.transform(dataset.data)
    title='n_neighbors = {0}'.format(n_neighbors)
    plt.title(title)
    plt.scatter(embedding[:, 0], embedding[:, 1], c=dataset.target, alpha=0.5)
    plt.colorbar()
    plt.show()
```


    
![UMAPの教師ありハイパラチューニング_5_0.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/211162/232b0ab6-6ec7-e9e5-edde-1a5026ba8db6.png)

    



    
![UMAPの教師ありハイパラチューニング_5_1.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/211162/9c36d515-a172-658a-ca9b-95b0b7b813e2.png)

    



    
![UMAPの教師ありハイパラチューニング_5_2.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/211162/48ebdc92-c357-914c-9872-5f189565f30a.png)

    



    
![UMAPの教師ありハイパラチューニング_5_3.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/211162/ce8d34e9-7dfa-62c3-c343-6dfe658bed26.png)

    



    
![UMAPの教師ありハイパラチューニング_5_4.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/211162/84a58606-baed-993b-b8a5-c8f18fee1034.png)

    



    
![UMAPの教師ありハイパラチューニング_5_5.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/211162/fcda0078-6801-45bc-e92e-2d1192afd8d2.png)

    


## min_dist


```python
from umap import UMAP
import sklearn.datasets
import matplotlib.pyplot as plt

dataset = sklearn.datasets.load_diabetes()

for min_dist in (0.0, 0.1, 0.25, 0.5, 0.8, 0.99):
    mapper = UMAP(min_dist=min_dist)
    mapper.fit(dataset.data)
    embedding = mapper.transform(dataset.data)
    title='min_dist = {0}'.format(min_dist)
    plt.title(title)
    plt.scatter(embedding[:, 0], embedding[:, 1], c=dataset.target, alpha=0.5)
    plt.colorbar()
    plt.show()
```


    
![UMAPの教師ありハイパラチューニング_7_0.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/211162/e7878e0b-4c3e-0b11-c67e-bbca4c1b7fba.png)

    



    
![UMAPの教師ありハイパラチューニング_7_1.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/211162/408785ed-4536-03f1-e74e-1464907450e0.png)

    



    
![UMAPの教師ありハイパラチューニング_7_2.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/211162/c7a16b19-6e5d-2f8b-8684-6648fce05dec.png)

    



    
![UMAPの教師ありハイパラチューニング_7_3.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/211162/41811805-55ea-6e91-3dd4-3b504de2e56b.png)

    



    
![UMAPの教師ありハイパラチューニング_7_4.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/211162/3d351b42-fabf-467d-cf09-47b900986c03.png)

    



    
![UMAPの教師ありハイパラチューニング_7_5.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/211162/a0851491-9b2c-ad86-79d9-c9ac8fd51cc6.png)

    


## metric


```python
from umap import UMAP
import sklearn.datasets
import matplotlib.pyplot as plt

dataset = sklearn.datasets.load_diabetes()

for metric in ["euclidean", "manhattan", "chebyshev", "minkowski", "canberra", 
               "braycurtis", "mahalanobis", "cosine", "correlation"]:

    mapper = UMAP(metric=metric)
    mapper.fit(dataset.data)
    embedding = mapper.transform(dataset.data)
    title='metric = {0}'.format(metric)
    plt.title(title)
    plt.scatter(embedding[:, 0], embedding[:, 1], c=dataset.target, alpha=0.5)
    plt.colorbar()
    plt.show()
```


    
![UMAPの教師ありハイパラチューニング_9_0.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/211162/aeb67e86-2f7f-9209-7107-e835652e221d.png)

    



    
![UMAPの教師ありハイパラチューニング_9_1.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/211162/dfe2e259-edd8-f0f7-354b-9a9c8fa78d78.png)

    



    
![UMAPの教師ありハイパラチューニング_9_2.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/211162/d8f31807-0c32-f438-4258-9d35976aff35.png)

    



    
![UMAPの教師ありハイパラチューニング_9_3.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/211162/ab1449c8-9997-9b41-4f53-7917366cd431.png)

    



    
![UMAPの教師ありハイパラチューニング_9_4.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/211162/8da60821-fdb7-a72d-d122-b8cd3648b162.png)

    



    
![UMAPの教師ありハイパラチューニング_9_5.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/211162/1702a967-c46d-20d1-9d80-7f0211de4e45.png)

    



    
![UMAPの教師ありハイパラチューニング_9_6.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/211162/e3b4c675-3669-4b8c-d72d-e041369ac68e.png)

    



    
![UMAPの教師ありハイパラチューニング_9_7.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/211162/8ccbff44-3ca0-baa0-2097-4d41a96acdd3.png)

    



    
![UMAPの教師ありハイパラチューニング_9_8.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/211162/361e62d7-f07a-4462-3ecc-19f84cdf6e90.png)

    


# 結局、どのパラメータが良いの？

パラメータを変化させると、可視化結果が変わってくることがよく分かりましたね。それでは結局、どのパラメータが良いのでしょうか？

目的変数がある場合、説明変数の中には、目的変数との関係性の薄い変数が含まれることも多いでしょう。教師なし学習だと、目的変数との関係性を考えずにマッピングしてしまうのですが、要らない変数の影響はできるだけ小さくしてマッピングして欲しいですよね（ね？）。

# 回帰問題用スコア関数

そこで、教師なし学習である UMAP を「教師あり UMAP」にするためのスコア関数を設計しましょう。回帰問題の場合は、たとえば次のように作れると思います。マッピングされた２点に対して、目的変数の差を計算し、それを２点間のユークリッド距離で割った数値の総和です。


```python
def regression_scorer(X, Y):
    sum = 0
    n1 = 0
    for x1, y1 in zip(X, Y):
        n1 += 1
        n2 = 0
        for x2, y2 in zip(X, Y):
            n2 += 1
            if n1 > n2:
                dist = ((x1[0] - x2[0])**2 + (x1[1] - x2[1])**2) + 1e-53
                sum += (y1 - y2)**2 / dist

    return sum / (len(Y) * (len(Y) - 1) / 2)
```

# Optuna で最適化

パラメータの最適化といえば Optuna。Optuna と言えばパラメータの最適化。ということでインストールします。


```python
!pip install optuna
```

    Collecting optuna
      Downloading optuna-2.10.0-py3-none-any.whl (308 kB)
    [K     |████████████████████████████████| 308 kB 9.5 MB/s 
    [?25hCollecting alembic
      Downloading alembic-1.7.7-py3-none-any.whl (210 kB)
    [K     |████████████████████████████████| 210 kB 56.1 MB/s 
    [?25hRequirement already satisfied: numpy in /usr/local/lib/python3.7/dist-packages (from optuna) (1.21.5)
    Requirement already satisfied: sqlalchemy>=1.1.0 in /usr/local/lib/python3.7/dist-packages (from optuna) (1.4.32)
    Collecting cmaes>=0.8.2
      Downloading cmaes-0.8.2-py3-none-any.whl (15 kB)
    Requirement already satisfied: PyYAML in /usr/local/lib/python3.7/dist-packages (from optuna) (3.13)
    Requirement already satisfied: scipy!=1.4.0 in /usr/local/lib/python3.7/dist-packages (from optuna) (1.4.1)
    Requirement already satisfied: tqdm in /usr/local/lib/python3.7/dist-packages (from optuna) (4.63.0)
    Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.7/dist-packages (from optuna) (21.3)
    Collecting cliff
      Downloading cliff-3.10.1-py3-none-any.whl (81 kB)
    [K     |████████████████████████████████| 81 kB 10.0 MB/s 
    [?25hCollecting colorlog
      Downloading colorlog-6.6.0-py2.py3-none-any.whl (11 kB)
    Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.7/dist-packages (from packaging>=20.0->optuna) (3.0.7)
    Requirement already satisfied: importlib-metadata in /usr/local/lib/python3.7/dist-packages (from sqlalchemy>=1.1.0->optuna) (4.11.3)
    Requirement already satisfied: greenlet!=0.4.17 in /usr/local/lib/python3.7/dist-packages (from sqlalchemy>=1.1.0->optuna) (1.1.2)
    Collecting Mako
      Downloading Mako-1.2.0-py3-none-any.whl (78 kB)
    [K     |████████████████████████████████| 78 kB 4.3 MB/s 
    [?25hRequirement already satisfied: importlib-resources in /usr/local/lib/python3.7/dist-packages (from alembic->optuna) (5.4.0)
    Collecting cmd2>=1.0.0
      Downloading cmd2-2.4.0-py3-none-any.whl (150 kB)
    [K     |████████████████████████████████| 150 kB 78.3 MB/s 
    [?25hCollecting pbr!=2.1.0,>=2.0.0
      Downloading pbr-5.8.1-py2.py3-none-any.whl (113 kB)
    [K     |████████████████████████████████| 113 kB 74.5 MB/s 
    [?25hCollecting stevedore>=2.0.1
      Downloading stevedore-3.5.0-py3-none-any.whl (49 kB)
    [K     |████████████████████████████████| 49 kB 6.8 MB/s 
    [?25hCollecting autopage>=0.4.0
      Downloading autopage-0.5.0-py3-none-any.whl (29 kB)
    Requirement already satisfied: PrettyTable>=0.7.2 in /usr/local/lib/python3.7/dist-packages (from cliff->optuna) (3.2.0)
    Requirement already satisfied: typing-extensions in /usr/local/lib/python3.7/dist-packages (from cmd2>=1.0.0->cliff->optuna) (3.10.0.2)
    Requirement already satisfied: attrs>=16.3.0 in /usr/local/lib/python3.7/dist-packages (from cmd2>=1.0.0->cliff->optuna) (21.4.0)
    Collecting pyperclip>=1.6
      Downloading pyperclip-1.8.2.tar.gz (20 kB)
    Requirement already satisfied: wcwidth>=0.1.7 in /usr/local/lib/python3.7/dist-packages (from cmd2>=1.0.0->cliff->optuna) (0.2.5)
    Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.7/dist-packages (from importlib-metadata->sqlalchemy>=1.1.0->optuna) (3.7.0)
    Requirement already satisfied: MarkupSafe>=0.9.2 in /usr/local/lib/python3.7/dist-packages (from Mako->alembic->optuna) (2.0.1)
    Building wheels for collected packages: pyperclip
      Building wheel for pyperclip (setup.py) ... [?25l[?25hdone
      Created wheel for pyperclip: filename=pyperclip-1.8.2-py3-none-any.whl size=11137 sha256=031b37d441d73814df94ef903581ef03bb76ca151064df06b639a6e908740c40
      Stored in directory: /root/.cache/pip/wheels/9f/18/84/8f69f8b08169c7bae2dde6bd7daf0c19fca8c8e500ee620a28
    Successfully built pyperclip
    Installing collected packages: pyperclip, pbr, stevedore, Mako, cmd2, autopage, colorlog, cmaes, cliff, alembic, optuna
    Successfully installed Mako-1.2.0 alembic-1.7.7 autopage-0.5.0 cliff-3.10.1 cmaes-0.8.2 cmd2-2.4.0 colorlog-6.6.0 optuna-2.10.0 pbr-5.8.1 pyperclip-1.8.2 stevedore-3.5.0


# 教師あり UMAP

Optuna を使って、UMAP を最適化するクラスを設計します。


```python
import scipy
class SupervisedUMAP:
    def __init__(self, X, Y, scorer):
        self.X = X
        self.Y = Y
        self.scorer = scorer
        self.best_score = 1e53
        self.best_model = None

    def __call__(self, trial):
        n_neighbors = trial.suggest_int("n_neighbors", 2, len(self.Y))
        min_dist = trial.suggest_uniform("min_dist", 0.0, 0.99)
        metric = trial.suggest_categorical("metric", 
                ["euclidean", "manhattan", "chebyshev", "minkowski", "canberra", 
               "braycurtis", "mahalanobis", "cosine", "correlation"])

        mapper = UMAP(
            n_neighbors=n_neighbors, 
            min_dist=min_dist,
            metric=metric
            )
        mapper.fit(self.X)
        embedding = mapper.transform(self.X)
        score = self.scorer(scipy.stats.zscore(embedding), self.Y)

        if self.best_score > score:
            self.best_score = score
            self.best_model = mapper

            print(self.best_model)
            title='trial={0}, score={1:.3e}'.format(trial.number, score)
            plt.title(title)
            plt.scatter(embedding[:, 0], embedding[:, 1], c=dataset.target, alpha=0.5)
            plt.colorbar()
            plt.show()      

        return score
```

最適化計算の実行例です。いくらか省略しながら結果を示します。


```python
import optuna

objective = SupervisedUMAP(dataset.data, dataset.target, regression_scorer)
study = optuna.create_study(direction="minimize")
study.optimize(objective, n_trials=100)
```




    UMAP(metric='braycurtis', min_dist=0.17087855399688387, n_neighbors=165, tqdm_kwds={'bar_format': '{desc}: {percentage:3.0f}%| {bar} {n_fmt}/{total_fmt} [{elapsed}]', 'desc': 'Epochs completed', 'disable': True})



    
![UMAPの教師ありハイパラチューニング_17_2.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/211162/2d9137f0-a320-61f7-15c0-c5e7aa64b71f.png)

    





    UMAP(metric='manhattan', min_dist=0.2525304699853455, n_neighbors=178, tqdm_kwds={'bar_format': '{desc}: {percentage:3.0f}%| {bar} {n_fmt}/{total_fmt} [{elapsed}]', 'desc': 'Epochs completed', 'disable': True})



    
![UMAPの教師ありハイパラチューニング_17_5.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/211162/0a07617a-c423-3481-12ff-6415b0f0dbd0.png)

    





    UMAP(metric='mahalanobis', min_dist=0.10402066419499499, n_neighbors=140, tqdm_kwds={'bar_format': '{desc}: {percentage:3.0f}%| {bar} {n_fmt}/{total_fmt} [{elapsed}]', 'desc': 'Epochs completed', 'disable': True})



    
![UMAPの教師ありハイパラチューニング_17_8.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/211162/3fc7a47d-c335-47f5-283c-03897ad091cb.png)

   


    UMAP(metric='canberra', min_dist=0.965734984822116, n_neighbors=426, tqdm_kwds={'bar_format': '{desc}: {percentage:3.0f}%| {bar} {n_fmt}/{total_fmt} [{elapsed}]', 'desc': 'Epochs completed', 'disable': True})



    
![UMAPの教師ありハイパラチューニング_17_11.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/211162/8729c73f-c83e-586f-edcf-1b709d38feb5.png)

    





    UMAP(metric='minkowski', min_dist=0.552229974539016, n_neighbors=246, tqdm_kwds={'bar_format': '{desc}: {percentage:3.0f}%| {bar} {n_fmt}/{total_fmt} [{elapsed}]', 'desc': 'Epochs completed', 'disable': True})



    
![UMAPの教師ありハイパラチューニング_17_14.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/211162/d2569ae8-efb0-8c91-12f2-3eef698ddb8c.png)

    




    UMAP(angular_rp_forest=True, metric='cosine', min_dist=0.6024771436878388, n_neighbors=116, tqdm_kwds={'bar_format': '{desc}: {percentage:3.0f}%| {bar} {n_fmt}/{total_fmt} [{elapsed}]', 'desc': 'Epochs completed', 'disable': True})



    
![UMAPの教師ありハイパラチューニング_17_17.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/211162/097d3cf6-d556-865a-ece9-559d8c76f4d0.png)

    





    UMAP(metric='minkowski', min_dist=0.5463771791221853, n_neighbors=147, tqdm_kwds={'bar_format': '{desc}: {percentage:3.0f}%| {bar} {n_fmt}/{total_fmt} [{elapsed}]', 'desc': 'Epochs completed', 'disable': True})



    
![UMAPの教師ありハイパラチューニング_17_20.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/211162/24525e44-8e1d-ac8c-9eed-e384f09014a1.png)

    





    UMAP(metric='chebyshev', min_dist=0.465599002041312, n_neighbors=269, tqdm_kwds={'bar_format': '{desc}: {percentage:3.0f}%| {bar} {n_fmt}/{total_fmt} [{elapsed}]', 'desc': 'Epochs completed', 'disable': True})



    
![UMAPの教師ありハイパラチューニング_17_23.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/211162/a5793357-086c-1c91-53c0-424ab24b6883.png)

    





    UMAP(metric='chebyshev', min_dist=0.8192578183604611, n_neighbors=264, tqdm_kwds={'bar_format': '{desc}: {percentage:3.0f}%| {bar} {n_fmt}/{total_fmt} [{elapsed}]', 'desc': 'Epochs completed', 'disable': True})



    
![UMAPの教師ありハイパラチューニング_17_26.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/211162/48f53c1e-a9e2-772c-ade8-70cd4cb279e4.png)

    




    UMAP(metric='chebyshev', min_dist=0.8053374897725938, n_neighbors=272, tqdm_kwds={'bar_format': '{desc}: {percentage:3.0f}%| {bar} {n_fmt}/{total_fmt} [{elapsed}]', 'desc': 'Epochs completed', 'disable': True})



    
![UMAPの教師ありハイパラチューニング_17_29.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/211162/7d6d0de3-edbe-6d76-60b9-aed537898bbd.png)

    





最適化が進むに従って、目的変数の分離が最もしやすいパラメータが選択できたのではないかなと思います。

# 予測値のマッピング

私が UMAP を好きな理由のひとつが、inverse_transform が使えることです。inverse_transform は、２次元マップ上の座標から、もとのベクトルを復元する方法です。


```python
objective.best_model.inverse_transform([[0, 0]])
```




    array([[ 0.02235418, -0.06212789, -0.06006454, -0.03912916,  0.00229801,
             0.02006099, -0.03019603,  0.03950487,  0.00173235, -0.03239265]],
          dtype=float32)



これを用いて、任意の予測モデルによる予測値をマッピングできます。


```python
import numpy as np
from sklearn.gaussian_process import GaussianProcessRegressor
from sklearn.gaussian_process.kernels import RBF

kernel = 1.0 * RBF(length_scale=1.0, length_scale_bounds=(1e-1, 10.0))
model = GaussianProcessRegressor(kernel=kernel, random_state=0)
model.fit(dataset.data, dataset.target)
```




```python
figsize=(6, 4)
h = 1
alpha = 0.8

embedding = objective.best_model.transform(dataset.data)
x_min = embedding[:, 0].min() - 1
x_max = embedding[:, 0].max() + 1
y_min = embedding[:, 1].min() - 1
y_max = embedding[:, 1].max() + 1
xx, yy = np.meshgrid(np.arange(x_min, x_max, h), np.arange(y_min, y_max, h))
```


```python
plt.figure(figsize=figsize)
Z = model.predict(objective.best_model.inverse_transform(np.c_[xx.ravel(), yy.ravel()]))
Z = Z.reshape(xx.shape)
plt.contourf(xx, yy, Z, alpha=alpha)
plt.colorbar()
plt.scatter(embedding[:, 0], embedding[:, 1], c="black", alpha=0.5)
plt.grid()
plt.show()
```


    
![UMAPの教師ありハイパラチューニング_23_0.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/211162/4dfa8e25-6249-a3a4-e98e-cf5eceaeaa89.png)

    


ガウス過程回帰など、予測値の分散を考慮できる方法を用いれば、予測値の分散をマッピングすることでさらなる考察が可能になりそうですね。[Bagging や Blending で予測の分散を考慮する](https://qiita.com/maskot1977/items/a602cc315e36f5199cff)のも良いかもしれません。


```python
plt.figure(figsize=figsize)
Z, Z_std = model.predict(objective.best_model.inverse_transform(np.c_[xx.ravel(), yy.ravel()]), return_std=True)
Z = Z_std.reshape(xx.shape)
plt.contourf(xx, yy, Z, alpha=alpha)
plt.colorbar()
plt.scatter(embedding[:, 0], embedding[:, 1], c="black", alpha=0.5)
plt.grid()
plt.show()
```


    
![UMAPの教師ありハイパラチューニング_25_0.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/211162/229b3a1a-b6f0-c542-b166-d00f9ed04736.png)

    


# 分類問題用スコア関数

回帰問題用スコア関数を少しだけ改変して、分類問題用スコア関数にします。分類問題と異なり、回帰問題では異なるクラス間の「差」は常に１であるとします。


```python
def classification_scorer(X, Y, alpha=1e-3):
    sum = 0
    n1 = 0
    for x1, y1 in zip(X, Y):
        n1 += 1
        n2 = 0
        for x2, y2 in zip(X, Y):
            n2 += 1
            if n1 > n2:
                dist = ((x1[0] - x2[0])**2 + (x1[1] - x2[1])**2) + 1e-53
                if y1 != y2:
                    sum += 1 / dist
                else:
                    pass
                    
    return sum / (len(Y) * (len(Y) - 1) / 2)
```

それ以外は、回帰問題と同じですね。

# 乳がんデータセット

乳がんデータセットを用いた２値分類の例です。


```python
from sklearn.datasets import load_breast_cancer

dataset = load_breast_cancer()
```


```python
import optuna

objective = SupervisedUMAP(dataset.data, dataset.target, classification_scorer)
study = optuna.create_study(direction="minimize")
study.optimize(objective, n_trials=100)
```




    UMAP(min_dist=0.2172843223845458, n_neighbors=218, tqdm_kwds={'bar_format': '{desc}: {percentage:3.0f}%| {bar} {n_fmt}/{total_fmt} [{elapsed}]', 'desc': 'Epochs completed', 'disable': True})



    
![UMAPの教師ありハイパラチューニング_30_2.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/211162/a4322a77-7b12-549c-af17-d2c8ba96c1a1.png)

    





    UMAP(metric='mahalanobis', min_dist=0.47752441562991854, n_neighbors=293, tqdm_kwds={'bar_format': '{desc}: {percentage:3.0f}%| {bar} {n_fmt}/{total_fmt} [{elapsed}]', 'desc': 'Epochs completed', 'disable': True})



    
![UMAPの教師ありハイパラチューニング_30_5.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/211162/9cbaed40-838f-3175-2473-093307f6cec7.png)

    





    UMAP(metric='mahalanobis', min_dist=0.6024885895334441, n_neighbors=52, tqdm_kwds={'bar_format': '{desc}: {percentage:3.0f}%| {bar} {n_fmt}/{total_fmt} [{elapsed}]', 'desc': 'Epochs completed', 'disable': True})



    
![UMAPの教師ありハイパラチューニング_30_8.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/211162/5894561c-22d9-c7c2-fab8-ecb93010d144.png)

    





    UMAP(metric='canberra', min_dist=0.9858623684125, n_neighbors=108, tqdm_kwds={'bar_format': '{desc}: {percentage:3.0f}%| {bar} {n_fmt}/{total_fmt} [{elapsed}]', 'desc': 'Epochs completed', 'disable': True})



    
![UMAPの教師ありハイパラチューニング_30_11.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/211162/71aa0f11-87a7-e3d0-be6c-0c4e5f400362.png)

    





    UMAP(metric='canberra', min_dist=0.929187495285436, n_neighbors=364, tqdm_kwds={'bar_format': '{desc}: {percentage:3.0f}%| {bar} {n_fmt}/{total_fmt} [{elapsed}]', 'desc': 'Epochs completed', 'disable': True})



    
![UMAPの教師ありハイパラチューニング_30_14.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/211162/487117e8-b2a3-e65f-1371-7811bb2b8b9f.png)

    





    UMAP(metric='canberra', min_dist=0.9718475179229352, n_neighbors=326, tqdm_kwds={'bar_format': '{desc}: {percentage:3.0f}%| {bar} {n_fmt}/{total_fmt} [{elapsed}]', 'desc': 'Epochs completed', 'disable': True})



    
![UMAPの教師ありハイパラチューニング_30_17.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/211162/d84ca7fd-677e-faeb-e8f0-163ac69bb1dd.png)

    





    UMAP(metric='canberra', min_dist=0.7476737707654739, n_neighbors=402, tqdm_kwds={'bar_format': '{desc}: {percentage:3.0f}%| {bar} {n_fmt}/{total_fmt} [{elapsed}]', 'desc': 'Epochs completed', 'disable': True})



    
![UMAPの教師ありハイパラチューニング_30_20.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/211162/8431a7ad-312a-8071-074f-bbe4b606d9ad.png)

    




# predict_proba をマッピング

分類モデルの場合は、predict_proba をマッピングすることでその信頼性を見積もるのも良いかもしれません。


```python
import numpy as np
from sklearn.gaussian_process import GaussianProcessClassifier
from sklearn.gaussian_process.kernels import RBF

kernel = 1.0 * RBF(length_scale=1.0, length_scale_bounds=(1e-1, 10.0))
model = GaussianProcessClassifier(kernel=kernel, random_state=0)
model.fit(dataset.data, dataset.target)
```





```python
figsize=(6, 4)
h = 1
alpha = 0.8

embedding = objective.best_model.transform(dataset.data)
x_min = embedding[:, 0].min() - 1
x_max = embedding[:, 0].max() + 1
y_min = embedding[:, 1].min() - 1
y_max = embedding[:, 1].max() + 1
xx, yy = np.meshgrid(np.arange(x_min, x_max, h), np.arange(y_min, y_max, h))
```


```python
for n in range(2):
    plt.figure(figsize=figsize)
    plt.title("predict {} probability".format(n))
    Z = model.predict_proba(objective.best_model.inverse_transform(np.c_[xx.ravel(), yy.ravel()]))[:, n]
    Z = Z.reshape(xx.shape)
    plt.contourf(xx, yy, Z, alpha=alpha)
    plt.colorbar()
    plt.scatter(embedding[:, 0], embedding[:, 1], c="black", alpha=0.5)
    plt.grid()
    plt.show()
```


    
![UMAPの教師ありハイパラチューニング_34_0.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/211162/1bb5dec0-5dba-7f80-2607-4ab80cea829a.png)

    



    
![UMAPの教師ありハイパラチューニング_34_1.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/211162/1a5b93cb-238b-c078-4f9c-94662f6c2a6f.png)

    


# ワインデータセット

ワインデータセットを用いた例です。このように、ラベルが３種類以上の場合でも実行可能です。


```python
from sklearn.datasets import load_wine

dataset = load_wine()
```


```python
import optuna

objective = SupervisedUMAP(dataset.data, dataset.target, classification_scorer)
study = optuna.create_study(direction="minimize")
study.optimize(objective, n_trials=100)
```




    UMAP(metric='canberra', min_dist=0.7820271577093002, n_neighbors=130, tqdm_kwds={'bar_format': '{desc}: {percentage:3.0f}%| {bar} {n_fmt}/{total_fmt} [{elapsed}]', 'desc': 'Epochs completed', 'disable': True})



    
![UMAPの教師ありハイパラチューニング_37_2.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/211162/d92b04de-d697-4c34-1cf0-270f2c709833.png)

    





    UMAP(metric='canberra', min_dist=0.2664782777245317, n_neighbors=116, tqdm_kwds={'bar_format': '{desc}: {percentage:3.0f}%| {bar} {n_fmt}/{total_fmt} [{elapsed}]', 'desc': 'Epochs completed', 'disable': True})



    
![UMAPの教師ありハイパラチューニング_37_5.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/211162/6d2dbcbc-47da-5b93-207f-bac9f73dbd41.png)

    





    UMAP(metric='canberra', min_dist=0.5972978429297398, n_neighbors=132, tqdm_kwds={'bar_format': '{desc}: {percentage:3.0f}%| {bar} {n_fmt}/{total_fmt} [{elapsed}]', 'desc': 'Epochs completed', 'disable': True})



    
![UMAPの教師ありハイパラチューニング_37_8.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/211162/df0b1c54-0e20-72d4-c934-5813e3cd6fe4.png)

    





    UMAP(metric='canberra', min_dist=0.9498778003225311, n_neighbors=34, tqdm_kwds={'bar_format': '{desc}: {percentage:3.0f}%| {bar} {n_fmt}/{total_fmt} [{elapsed}]', 'desc': 'Epochs completed', 'disable': True})



    
![UMAPの教師ありハイパラチューニング_37_11.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/211162/4ab4f4ce-0ae0-bce4-273b-55267c5a7940.png)

    





```python
import numpy as np
from sklearn.gaussian_process import GaussianProcessClassifier
from sklearn.gaussian_process.kernels import RBF

kernel = 1.0 * RBF(length_scale=1.0, length_scale_bounds=(1e-1, 10.0))
model = GaussianProcessClassifier(kernel=kernel, random_state=0)
model.fit(dataset.data, dataset.target)
```




```python
figsize=(6, 4)
h = 1
alpha = 0.8

embedding = objective.best_model.transform(dataset.data)
x_min = embedding[:, 0].min() - 1
x_max = embedding[:, 0].max() + 1
y_min = embedding[:, 1].min() - 1
y_max = embedding[:, 1].max() + 1
xx, yy = np.meshgrid(np.arange(x_min, x_max, h), np.arange(y_min, y_max, h))
```


```python
for n in range(3):
    plt.figure(figsize=figsize)
    plt.title("predict {} probability".format(n))
    Z = model.predict_proba(objective.best_model.inverse_transform(np.c_[xx.ravel(), yy.ravel()]))[:, n]
    Z = Z.reshape(xx.shape)
    plt.contourf(xx, yy, Z, alpha=alpha)
    plt.colorbar()
    plt.scatter(embedding[:, 0], embedding[:, 1], c="black", alpha=0.5)
    plt.grid()
    plt.show()
```


    
![UMAPの教師ありハイパラチューニング_40_0.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/211162/7bfb6356-592f-8e3e-16bb-41b1caa2ce99.png)

    



    
![UMAPの教師ありハイパラチューニング_40_1.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/211162/887857e4-c813-8bac-41a0-060ef6185dab.png)
    



    
![UMAPの教師ありハイパラチューニング_40_2.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/211162/fa4e6ee4-cc81-5e1a-1a2c-0cd65ade93a5.png)

    


# 手書き数字データセット

手書き数字データセットを用いた例です。この例ではラベルが１０種類あります。


```python
from sklearn.datasets import load_digits

dataset = load_digits()
```


```python
import optuna

objective = SupervisedUMAP(dataset.data, dataset.target, classification_scorer)
study = optuna.create_study(direction="minimize")
study.optimize(objective, n_trials=100)
```




    UMAP(metric='minkowski', min_dist=0.08295894663258277, n_neighbors=845, tqdm_kwds={'bar_format': '{desc}: {percentage:3.0f}%| {bar} {n_fmt}/{total_fmt} [{elapsed}]', 'desc': 'Epochs completed', 'disable': True})



    
![UMAPの教師ありハイパラチューニング_43_2.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/211162/d0983511-c7ab-18a9-634e-731353bce049.png)

    





    UMAP(angular_rp_forest=True, metric='correlation', min_dist=0.8610569729117126, n_neighbors=1512, tqdm_kwds={'bar_format': '{desc}: {percentage:3.0f}%| {bar} {n_fmt}/{total_fmt} [{elapsed}]', 'desc': 'Epochs completed', 'disable': True})



    
![UMAPの教師ありハイパラチューニング_43_5.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/211162/018d796f-fe15-920a-a46b-ee0f2829f7b5.png)

    




    UMAP(metric='braycurtis', min_dist=0.22248833651912708, n_neighbors=594, tqdm_kwds={'bar_format': '{desc}: {percentage:3.0f}%| {bar} {n_fmt}/{total_fmt} [{elapsed}]', 'desc': 'Epochs completed', 'disable': True})



    
![UMAPの教師ありハイパラチューニング_43_8.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/211162/06fe1a79-ec9b-d932-7e03-8bc889d1985a.png)

    





    UMAP(min_dist=0.6227914042692446, n_neighbors=530, tqdm_kwds={'bar_format': '{desc}: {percentage:3.0f}%| {bar} {n_fmt}/{total_fmt} [{elapsed}]', 'desc': 'Epochs completed', 'disable': True})



    
![UMAPの教師ありハイパラチューニング_43_11.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/211162/fd3f71bd-29c1-de9b-8713-34c07a4a2354.png)

    




    UMAP(min_dist=0.6510861250675604, n_neighbors=392, tqdm_kwds={'bar_format': '{desc}: {percentage:3.0f}%| {bar} {n_fmt}/{total_fmt} [{elapsed}]', 'desc': 'Epochs completed', 'disable': True})



    
![UMAPの教師ありハイパラチューニング_43_14.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/211162/1756b33b-2ea3-355a-26dc-f75ebd0bf993.png)

    





    UMAP(metric='manhattan', min_dist=0.520042472164499, n_neighbors=299, tqdm_kwds={'bar_format': '{desc}: {percentage:3.0f}%| {bar} {n_fmt}/{total_fmt} [{elapsed}]', 'desc': 'Epochs completed', 'disable': True})



    
![UMAPの教師ありハイパラチューニング_43_17.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/211162/3e2ea0ed-4882-9e77-69c3-03f4314f05c0.png)

    





    UMAP(min_dist=0.5368016672875904, n_neighbors=468, tqdm_kwds={'bar_format': '{desc}: {percentage:3.0f}%| {bar} {n_fmt}/{total_fmt} [{elapsed}]', 'desc': 'Epochs completed', 'disable': True})



    
![UMAPの教師ありハイパラチューニング_43_20.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/211162/b60170b9-02d1-3713-d459-4f3a920fd1b6.png)

    




    UMAP(min_dist=0.48661142862105233, n_neighbors=336, tqdm_kwds={'bar_format': '{desc}: {percentage:3.0f}%| {bar} {n_fmt}/{total_fmt} [{elapsed}]', 'desc': 'Epochs completed', 'disable': True})



    
![UMAPの教師ありハイパラチューニング_43_23.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/211162/5ef06f06-9de9-7602-8acb-25a9859f4e10.png)

    





    UMAP(min_dist=0.5262945217120664, n_neighbors=464, tqdm_kwds={'bar_format': '{desc}: {percentage:3.0f}%| {bar} {n_fmt}/{total_fmt} [{elapsed}]', 'desc': 'Epochs completed', 'disable': True})



    
![UMAPの教師ありハイパラチューニング_43_26.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/211162/93945ef1-8dcd-d39f-29f6-a8a874b95a0c.png)

    





    UMAP(min_dist=0.33899588494964167, n_neighbors=374, tqdm_kwds={'bar_format': '{desc}: {percentage:3.0f}%| {bar} {n_fmt}/{total_fmt} [{elapsed}]', 'desc': 'Epochs completed', 'disable': True})



    
![UMAPの教師ありハイパラチューニング_43_29.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/211162/dd6baa76-3d2e-8010-b774-3308d997e921.png)

    






```python
import numpy as np
from sklearn.neural_network import MLPClassifier

model = MLPClassifier()
model.fit(dataset.data, dataset.target)
```









```python
figsize=(6, 4)
h = 1
alpha = 0.8

embedding = objective.best_model.transform(dataset.data)
x_min = embedding[:, 0].min() - 1
x_max = embedding[:, 0].max() + 1
y_min = embedding[:, 1].min() - 1
y_max = embedding[:, 1].max() + 1
xx, yy = np.meshgrid(np.arange(x_min, x_max, h), np.arange(y_min, y_max, h))
```


```python
for n in range(10):
    plt.figure(figsize=figsize)
    plt.title("predict {} probability".format(n))
    Z = model.predict_proba(objective.best_model.inverse_transform(np.c_[xx.ravel(), yy.ravel()]))[:, n]
    Z = Z.reshape(xx.shape)
    plt.contourf(xx, yy, Z, alpha=alpha)
    plt.colorbar()
    plt.scatter(embedding[:, 0], embedding[:, 1], c="black", alpha=0.1)
    plt.grid()
    plt.show()
```


    
![UMAPの教師ありハイパラチューニング_46_0.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/211162/f24e64f5-63d5-b672-4814-5df71c153967.png)

    



    
![UMAPの教師ありハイパラチューニング_46_1.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/211162/bc5743a8-e59c-e51c-d45c-0a6535d070e3.png)

    



    
![UMAPの教師ありハイパラチューニング_46_2.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/211162/56272377-a48b-8a91-cdbe-8319abee16a2.png)

    



    
![UMAPの教師ありハイパラチューニング_46_3.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/211162/d9a6c831-fe83-b14e-c1af-346ff11f2f5f.png)

    



    
![UMAPの教師ありハイパラチューニング_46_4.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/211162/6cdc6c3a-e817-f0f4-3f44-a74c16c8efb2.png)

    



    
![UMAPの教師ありハイパラチューニング_46_5.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/211162/993cffae-1185-77a1-96f5-7c89633328f7.png)

    



    
![UMAPの教師ありハイパラチューニング_46_6.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/211162/0ac13288-7a66-12a8-87e0-b76a975edeb2.png)

    



    
![UMAPの教師ありハイパラチューニング_46_7.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/211162/6eb3693d-e863-f1ae-f49b-2585845977d5.png)

    



    
![UMAPの教師ありハイパラチューニング_46_8.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/211162/a6fa0981-939f-2d5f-6f7d-24654d6f47b6.png)

    



    
![UMAPの教師ありハイパラチューニング_46_9.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/211162/e0ba26b5-2b5c-38f1-e36b-ac63930dff89.png)

    


# おわりに

t-SNE や Isomap が好きです。でも UMAP はもっと好きです。