Data Every Day: トップパーソナリティデータセット

# tldr

Kggleの[Top Personality Dataset](https://www.kaggle.com/arslanali4343/top-personality-dataset)を[Predicting Movie Preferences - Data Every Day #049](https://www.youtube.com/watch?v=SFV-zQMuR0o&list=PLs_CQl4mUvI7Z9HnwoeOS5EUu9gyvekjK&index=49)に沿ってやっていきます。

実行環境はGoogle Colaboratorです。

# インポート


```python
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns

import sklearn.preprocessing as sp
from sklearn.model_selection import train_test_split
import sklearn.linear_model as slm
import sklearn.svm as svm
import sklearn.neural_network as snn

import tensorflow as tf
```

# データのダウンロード

Google Driveをマウントします。


```python
from google.colab import drive
drive.mount('/content/drive')
```

    Mounted at /content/drive


KaggleのAPIクライアントを初期化し、認証します。
認証情報はGoogle Drive内（`/content/drive/My Drive/Colab Notebooks/Kaggle`）に`kaggle.json`として置いてあります。


```python
import os
kaggle_path = "/content/drive/My Drive/Colab Notebooks/Kaggle"
os.environ['KAGGLE_CONFIG_DIR'] = kaggle_path

from kaggle.api.kaggle_api_extended import KaggleApi
api = KaggleApi()
api.authenticate() 
```

Kaggle APIを使ってデータをダウンロードします。


```python
dataset_id = 'arslanali4343/top-personality-dataset'
dataset = api.dataset_list_files(dataset_id)
file_name = dataset.files[0].name
file_path = os.path.join(api.get_default_download_dir(), file_name)
file_path
```




    '/content/2018-personality-data.csv'




```python
api.dataset_download_file(dataset_id, file_name, force=True, quiet=False)
```

    100%|██████████| 608k/608k [00:00<00:00, 74.4MB/s]

    Downloading 2018-personality-data.csv to /content
    


    





    True




```python

api.dataset_download_file(dataset_id, file_name, force=True, quiet=False)
```

# データの読み込み


```python
data = pd.read_csv(file_path)
```


```python
data
```




<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>userid</th>
      <th>openness</th>
      <th>agreeableness</th>
      <th>emotional_stability</th>
      <th>conscientiousness</th>
      <th>extraversion</th>
      <th>assigned metric</th>
      <th>assigned condition</th>
      <th>movie_1</th>
      <th>predicted_rating_1</th>
      <th>movie_2</th>
      <th>predicted_rating_2</th>
      <th>movie_3</th>
      <th>predicted_rating_3</th>
      <th>movie_4</th>
      <th>predicted_rating_4</th>
      <th>movie_5</th>
      <th>predicted_rating_5</th>
      <th>movie_6</th>
      <th>predicted_rating_6</th>
      <th>movie_7</th>
      <th>predicted_rating_7</th>
      <th>movie_8</th>
      <th>predicted_rating_8</th>
      <th>movie_9</th>
      <th>predicted_rating_9</th>
      <th>movie_10</th>
      <th>predicted_rating_10</th>
      <th>movie_11</th>
      <th>predicted_rating_11</th>
      <th>movie_12</th>
      <th>predicted_rating_12</th>
      <th>is_personalized</th>
      <th>enjoy_watching</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>8e7cebf9a234c064b75016249f2ac65e</td>
      <td>5.0</td>
      <td>2.0</td>
      <td>3.0</td>
      <td>2.5</td>
      <td>6.5</td>
      <td>serendipity</td>
      <td>high</td>
      <td>77658</td>
      <td>4.410466</td>
      <td>95858</td>
      <td>4.271995</td>
      <td>115713</td>
      <td>4.611922</td>
      <td>26674</td>
      <td>4.459407</td>
      <td>93040</td>
      <td>4.147292</td>
      <td>117533</td>
      <td>4.098206</td>
      <td>108979</td>
      <td>4.064843</td>
      <td>112582</td>
      <td>4.149100</td>
      <td>120138</td>
      <td>4.244817</td>
      <td>121372</td>
      <td>4.396004</td>
      <td>127152</td>
      <td>4.120456</td>
      <td>95311</td>
      <td>4.053847</td>
      <td>4</td>
      <td>4</td>
    </tr>
    <tr>
      <th>1</th>
      <td>77c7d756a093150d4377720abeaeef76</td>
      <td>7.0</td>
      <td>4.0</td>
      <td>6.0</td>
      <td>5.5</td>
      <td>4.0</td>
      <td>all</td>
      <td>default</td>
      <td>94959</td>
      <td>4.207280</td>
      <td>1247</td>
      <td>4.266540</td>
      <td>953</td>
      <td>4.211322</td>
      <td>2010</td>
      <td>4.408341</td>
      <td>1234</td>
      <td>4.090358</td>
      <td>5291</td>
      <td>4.202424</td>
      <td>106642</td>
      <td>4.113912</td>
      <td>1209</td>
      <td>4.094422</td>
      <td>56782</td>
      <td>4.019599</td>
      <td>5618</td>
      <td>3.963953</td>
      <td>969</td>
      <td>4.174188</td>
      <td>1232</td>
      <td>4.334877</td>
      <td>2</td>
      <td>3</td>
    </tr>
    <tr>
      <th>2</th>
      <td>b7e8a92987a530cc368719a0e60e26a3</td>
      <td>4.0</td>
      <td>3.0</td>
      <td>4.5</td>
      <td>2.0</td>
      <td>2.5</td>
      <td>serendipity</td>
      <td>medium</td>
      <td>110501</td>
      <td>4.868064</td>
      <td>77658</td>
      <td>4.710444</td>
      <td>101895</td>
      <td>5.029360</td>
      <td>1260</td>
      <td>4.698602</td>
      <td>5971</td>
      <td>4.660769</td>
      <td>98491</td>
      <td>4.962319</td>
      <td>926</td>
      <td>4.706864</td>
      <td>1204</td>
      <td>4.645191</td>
      <td>2288</td>
      <td>4.823212</td>
      <td>3307</td>
      <td>4.676756</td>
      <td>1172</td>
      <td>4.649281</td>
      <td>1212</td>
      <td>4.744990</td>
      <td>2</td>
      <td>2</td>
    </tr>
    <tr>
      <th>3</th>
      <td>92561f21446e017dd6b68b94b23ad5b7</td>
      <td>5.5</td>
      <td>5.5</td>
      <td>4.0</td>
      <td>4.5</td>
      <td>4.0</td>
      <td>popularity</td>
      <td>medium</td>
      <td>2905</td>
      <td>4.526371</td>
      <td>2843</td>
      <td>4.456451</td>
      <td>3629</td>
      <td>4.668444</td>
      <td>3022</td>
      <td>4.676067</td>
      <td>3307</td>
      <td>4.530360</td>
      <td>1211</td>
      <td>4.292660</td>
      <td>3462</td>
      <td>4.341634</td>
      <td>5291</td>
      <td>4.261166</td>
      <td>3030</td>
      <td>4.425689</td>
      <td>1281</td>
      <td>4.479921</td>
      <td>940</td>
      <td>4.355061</td>
      <td>905</td>
      <td>4.317927</td>
      <td>3</td>
      <td>3</td>
    </tr>
    <tr>
      <th>4</th>
      <td>030001ac2145a938b07e686a35a2d638</td>
      <td>5.5</td>
      <td>5.5</td>
      <td>3.5</td>
      <td>4.5</td>
      <td>2.5</td>
      <td>popularity</td>
      <td>medium</td>
      <td>2905</td>
      <td>4.526371</td>
      <td>2843</td>
      <td>4.456451</td>
      <td>3629</td>
      <td>4.668444</td>
      <td>3022</td>
      <td>4.676067</td>
      <td>3307</td>
      <td>4.530360</td>
      <td>1211</td>
      <td>4.292660</td>
      <td>3462</td>
      <td>4.341634</td>
      <td>5291</td>
      <td>4.261166</td>
      <td>3030</td>
      <td>4.425689</td>
      <td>1281</td>
      <td>4.479921</td>
      <td>940</td>
      <td>4.355061</td>
      <td>905</td>
      <td>4.317927</td>
      <td>2</td>
      <td>3</td>
    </tr>
    <tr>
      <th>...</th>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
    </tr>
    <tr>
      <th>1829</th>
      <td>cff910b71f09b3120289ff6b461a9e03</td>
      <td>5.5</td>
      <td>3.5</td>
      <td>2.5</td>
      <td>4.0</td>
      <td>5.5</td>
      <td>popularity</td>
      <td>low</td>
      <td>108979</td>
      <td>4.246346</td>
      <td>33779</td>
      <td>4.002232</td>
      <td>96728</td>
      <td>3.981009</td>
      <td>77307</td>
      <td>4.059373</td>
      <td>111759</td>
      <td>3.888507</td>
      <td>7762</td>
      <td>4.368495</td>
      <td>26082</td>
      <td>4.554531</td>
      <td>89753</td>
      <td>4.145526</td>
      <td>6643</td>
      <td>4.409108</td>
      <td>115122</td>
      <td>3.960470</td>
      <td>7700</td>
      <td>4.178546</td>
      <td>67997</td>
      <td>4.085300</td>
      <td>3</td>
      <td>3</td>
    </tr>
    <tr>
      <th>1830</th>
      <td>1ab3a4c2921d8da640854819b0f6cfce</td>
      <td>4.0</td>
      <td>3.5</td>
      <td>4.5</td>
      <td>4.0</td>
      <td>2.5</td>
      <td>serendipity</td>
      <td>high</td>
      <td>93040</td>
      <td>4.227140</td>
      <td>1199</td>
      <td>4.069527</td>
      <td>83134</td>
      <td>4.336292</td>
      <td>5971</td>
      <td>4.091813</td>
      <td>68237</td>
      <td>4.217929</td>
      <td>745</td>
      <td>4.101192</td>
      <td>4973</td>
      <td>4.244278</td>
      <td>47099</td>
      <td>4.473696</td>
      <td>5618</td>
      <td>4.149697</td>
      <td>903</td>
      <td>4.116152</td>
      <td>38061</td>
      <td>4.155210</td>
      <td>1197</td>
      <td>4.045751</td>
      <td>3</td>
      <td>4</td>
    </tr>
    <tr>
      <th>1831</th>
      <td>a06386edadf3bc614dadb7044708c46c</td>
      <td>6.0</td>
      <td>3.0</td>
      <td>5.5</td>
      <td>3.5</td>
      <td>6.0</td>
      <td>serendipity</td>
      <td>low</td>
      <td>106173</td>
      <td>3.935297</td>
      <td>126397</td>
      <td>4.006561</td>
      <td>109710</td>
      <td>3.947412</td>
      <td>99171</td>
      <td>4.003978</td>
      <td>119194</td>
      <td>4.002666</td>
      <td>69483</td>
      <td>4.235115</td>
      <td>45691</td>
      <td>4.191760</td>
      <td>58530</td>
      <td>3.965657</td>
      <td>26519</td>
      <td>3.998642</td>
      <td>89707</td>
      <td>4.144870</td>
      <td>2571</td>
      <td>3.860041</td>
      <td>108709</td>
      <td>3.899857</td>
      <td>3</td>
      <td>4</td>
    </tr>
    <tr>
      <th>1832</th>
      <td>bad56d9506832cd79d874a6b66b3d813</td>
      <td>5.0</td>
      <td>3.5</td>
      <td>1.5</td>
      <td>3.5</td>
      <td>2.5</td>
      <td>serendipity</td>
      <td>medium</td>
      <td>6874</td>
      <td>4.241766</td>
      <td>38061</td>
      <td>4.350788</td>
      <td>46578</td>
      <td>4.399071</td>
      <td>4848</td>
      <td>4.749688</td>
      <td>44195</td>
      <td>4.493639</td>
      <td>4979</td>
      <td>4.397887</td>
      <td>7438</td>
      <td>4.207513</td>
      <td>3897</td>
      <td>4.212995</td>
      <td>92259</td>
      <td>4.819710</td>
      <td>32</td>
      <td>4.059369</td>
      <td>3730</td>
      <td>4.427336</td>
      <td>3435</td>
      <td>4.844386</td>
      <td>4</td>
      <td>4</td>
    </tr>
    <tr>
      <th>1833</th>
      <td>721ea658e148fc0f76ddd6e2b0e02422</td>
      <td>6.5</td>
      <td>6.5</td>
      <td>2.5</td>
      <td>6.5</td>
      <td>2.0</td>
      <td>popularity</td>
      <td>high</td>
      <td>140737</td>
      <td>4.456842</td>
      <td>110366</td>
      <td>4.283821</td>
      <td>89315</td>
      <td>4.287975</td>
      <td>112006</td>
      <td>4.292487</td>
      <td>113190</td>
      <td>4.345421</td>
      <td>26614</td>
      <td>4.218514</td>
      <td>50872</td>
      <td>4.256004</td>
      <td>4447</td>
      <td>4.187709</td>
      <td>8360</td>
      <td>4.247956</td>
      <td>8961</td>
      <td>4.321693</td>
      <td>597</td>
      <td>4.278465</td>
      <td>457</td>
      <td>4.352895</td>
      <td>2</td>
      <td>2</td>
    </tr>
  </tbody>
</table>
<p>1834 rows × 34 columns</p>
</div>



# 下準備


```python
data.info()
```

    <class 'pandas.core.frame.DataFrame'>
    RangeIndex: 1834 entries, 0 to 1833
    Data columns (total 34 columns):
     #   Column                Non-Null Count  Dtype  
    ---  ------                --------------  -----  
     0   userid                1834 non-null   object 
     1    openness             1834 non-null   float64
     2    agreeableness        1834 non-null   float64
     3    emotional_stability  1834 non-null   float64
     4    conscientiousness    1834 non-null   float64
     5    extraversion         1834 non-null   float64
     6    assigned metric      1834 non-null   object 
     7    assigned condition   1834 non-null   object 
     8    movie_1              1834 non-null   int64  
     9    predicted_rating_1   1834 non-null   float64
     10   movie_2              1834 non-null   int64  
     11   predicted_rating_2   1834 non-null   float64
     12   movie_3              1834 non-null   int64  
     13   predicted_rating_3   1834 non-null   float64
     14   movie_4              1834 non-null   int64  
     15   predicted_rating_4   1834 non-null   float64
     16   movie_5              1834 non-null   int64  
     17   predicted_rating_5   1834 non-null   float64
     18   movie_6              1834 non-null   int64  
     19   predicted_rating_6   1834 non-null   float64
     20   movie_7              1834 non-null   int64  
     21   predicted_rating_7   1834 non-null   float64
     22   movie_8              1834 non-null   int64  
     23   predicted_rating_8   1834 non-null   float64
     24   movie_9              1834 non-null   int64  
     25   predicted_rating_9   1834 non-null   float64
     26   movie_10             1834 non-null   int64  
     27   predicted_rating_10  1834 non-null   float64
     28   movie_11             1834 non-null   int64  
     29   predicted_rating_11  1834 non-null   float64
     30   movie_12             1834 non-null   int64  
     31   predicted_rating_12  1834 non-null   float64
     32   is_personalized      1834 non-null   int64  
     33   enjoy_watching       1834 non-null   int64  
    dtypes: float64(17), int64(14), object(3)
    memory usage: 487.3+ KB



```python
data.isna().sum()
```




    userid                  0
     openness               0
     agreeableness          0
     emotional_stability    0
     conscientiousness      0
     extraversion           0
     assigned metric        0
     assigned condition     0
     movie_1                0
     predicted_rating_1     0
     movie_2                0
     predicted_rating_2     0
     movie_3                0
     predicted_rating_3     0
     movie_4                0
     predicted_rating_4     0
     movie_5                0
     predicted_rating_5     0
     movie_6                0
     predicted_rating_6     0
     movie_7                0
     predicted_rating_7     0
     movie_8                0
     predicted_rating_8     0
     movie_9                0
     predicted_rating_9     0
     movie_10               0
     predicted_rating_10    0
     movie_11               0
     predicted_rating_11    0
     movie_12               0
     predicted_rating_12    0
     is_personalized        0
     enjoy_watching         0
    dtype: int64




```python
data = data.drop([
                  'userid', 
                  ' movie_1',  ' predicted_rating_1',
                  ' movie_2',  ' predicted_rating_2',
                  ' movie_3',  ' predicted_rating_3',
                  ' movie_4',  ' predicted_rating_4',
                  ' movie_5',  ' predicted_rating_5',
                  ' movie_6',  ' predicted_rating_6',
                  ' movie_7',  ' predicted_rating_7',
                  ' movie_8',  ' predicted_rating_8',
                  ' movie_9',  ' predicted_rating_9',
                  ' movie_10',  ' predicted_rating_10',
                  ' movie_11',  ' predicted_rating_11',
                  ' movie_12',  ' predicted_rating_12'], axis=1)
```

# エンコード


```python
{column: list(data[column].unique()) for column in data.columns if data.dtypes[column] == 'object'}
```




    {' assigned condition': [' high', ' default', ' medium', ' low'],
     ' assigned metric': [' serendipity', ' all', ' popularity', ' diversity']}




```python
condition_ordering = [' low', ' medium', ' default', ' high']
```


```python
def ordinal_encode(df, column, ordering):
    df = df.copy()
    df[column] = df[column].apply(lambda x: ordering.index(x))
    return df

def onehot_encode(df, column, prefix):
    df = df.copy()
    dummies = pd.get_dummies(df[column], prefix=prefix)
    df = pd.concat([df, dummies], axis=1)
    df = df.drop(column, axis=1)
    return df
```


```python
data = ordinal_encode(data, ' assigned condition', condition_ordering)
data = onehot_encode(data, ' assigned metric', 'm')
```

# 分割とスケーリング


```python
y = data[' enjoy_watching ']
X = data.drop([' enjoy_watching '], axis=1)
```


```python
scaler = sp.StandardScaler()
X = scaler.fit_transform(X)
```


```python
X_train, X_test, y_train, y_test = train_test_split(X, y, train_size=0.7)
```

# トレーニング


```python
log_model = slm.LogisticRegression()
svm_model = svm.SVC()
ann_model = snn.MLPClassifier(hidden_layer_sizes=(16, 16))

log_model.fit(X_train, y_train)
svm_model.fit(X_train, y_train)
ann_model.fit(X_train, y_train)

log_acc = log_model.score(X_test, y_test)
svm_acc = svm_model.score(X_test, y_test)
ann_acc = ann_model.score(X_test, y_test)
```

    /usr/local/lib/python3.6/dist-packages/sklearn/neural_network/_multilayer_perceptron.py:571: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (200) reached and the optimization hasn't converged yet.
      % self.max_iter, ConvergenceWarning)



```python
import plotly.express as px

fig = px.bar(
    x=['Logistic Regression', 'SVM', 'NN'],
    y=[log_acc, svm_acc, ann_acc],
    color=['Logistic Regression', 'SVM', 'NN'],
    labels={'x': 'Model', 'y': 'Accuracy'},
    title='Model Accuracy',
)
fig.show()
```