タイタニックの名前データを特徴量化

<a href="https://qiita.com/murasaki1994/items/ed8b77f5d953f46e2ae9">前回</a>タイタニックの欠損値を大真面目に埋めたり削除しましたけど、

https://qiita.com/murasaki1994/items/ed8b77f5d953f46e2ae9

今度はタイタニックの名前データも特徴量にしてみました。
参考↓

https://qiita.com/jun40vn/items/d8a1f71fae680589e05c

# コード
```Python3
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
```

## 訓練データの読み込み
```Python3
df_train = pd.read_csv("train.csv")
df_train
```
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/2782889/a5b1d41c-4df9-8d1e-61cc-5320cd11d774.png)

## 名前の特徴量化
名前ではなく敬称を特徴量にしてみます。
```Python3
df_train["Name"] = df_train["Name"].map(lambda x:x.split(",")[1].split(".")[0])
df_train
```
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/2782889/efc89129-9f43-3aa6-de9c-e03a33fa85b3.png)

## 不必要な特徴量を削除
```Python3
df_train = df_train.drop(["Cabin", "Ticket"], axis=1)
df_train
```
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/2782889/37627294-7ec8-68e4-3fe2-b85d8928dc8c.png)

## 欠損値の補完
### Embarked
```Python3
sns.histplot(df_train["Embarked"])
plt.show()
```
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/2782889/19b98327-1fc5-d4d1-41dc-2ea81b480339.png)
Sで補完します。
```Python3
df_train["Embarked"] = df_train["Embarked"].fillna("S")
```
```Python3
df_train.isnull().sum()
```
```
PassengerId      0
Survived         0
Pclass           0
Name             0
Sex              0
Age            177
SibSp            0
Parch            0
Fare             0
Embarked         0
dtype: int64
```
### ダミー変数化
質的変数が全て補完できたのでダミー変数化します。
```Python3
df_train = pd.get_dummies(df_train)
df_train
```
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/2782889/da10c21f-cca4-a118-fba7-017b209fff46.png)

端のほうが見切れるくらい変数が増えました。
### インデックスを作る
PassengerIdをインデックスにする
```Python3
df_train.index = df_train["PassengerId"]
df_train = df_train.drop("PassengerId", axis=1)
```
### 年齢の補完
年齢は量が多いので確率的な推論ではなくちゃんと機械学習による分析にします。
```Python3
x_train = df_train.dropna().drop("Age", axis=1)
x_test = df_train[df_train["Age"].isnull()].drop("Age", axis=1)
y_train = df_train.dropna()["Age"]

from sklearn.ensemble import GradientBoostingRegressor

model = GradientBoostingRegressor()
model.fit(x_train, y_train)
y_pred = model.predict(x_test)

pred = pd.DataFrame(y_pred)
pred.columns = ["Age"]
pred.index = x_test.index
train = pd.concat([x_train, y_train], axis=1)
test = pd.concat([x_test, pred], axis=1)
df_train = pd.concat([train, test])
df_train
```
Ageが補完されました。
```Python3
df_train.isnull().sum()
```
```
Survived              0
Pclass                0
SibSp                 0
Parch                 0
Fare                  0
Name_ Capt            0
Name_ Col             0
Name_ Don             0
Name_ Dr              0
Name_ Jonkheer        0
Name_ Lady            0
Name_ Major           0
Name_ Master          0
Name_ Miss            0
Name_ Mlle            0
Name_ Mme             0
Name_ Mr              0
Name_ Mrs             0
Name_ Ms              0
Name_ Rev             0
Name_ Sir             0
Name_ the Countess    0
Sex_female            0
Sex_male              0
Embarked_C            0
Embarked_Q            0
Embarked_S            0
Age                   0
dtype: int64
```
訓練データの欠損値が全て補完されました。
# テストデータの読み込み
```Python3
df_test = pd.read_csv("test.csv")
df_test
```
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/2782889/1d8bdeef-5268-022e-ea18-bb71afcf3bd9.png)

## 使わない変数の削除
```Python3
df_test.index = df_test["PassengerId"]
df_test = df_test.drop(["PassengerId", "Ticket", "Cabin"], axis=1)
```
## 名前の特徴量化
```Python3
df_test["Name"] = df_test["Name"].map(lambda x:x.split(",")[1].split(".")[0])
df_test
```
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/2782889/1c0dffdb-a6cc-513a-2838-9a6748608170.png)

## Fareの補完
```Python3
plt.hist(df_test["Fare"], bins=40)
```
```
(array([175.,  90.,  58.,  15.,  21.,  14.,  10.,   3.,   2.,   0.,   6.,
          3.,   2.,   0.,   0.,   0.,   5.,   4.,   0.,   1.,   7.,   0.,
          0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,
          0.,   0.,   0.,   0.,   0.,   0.,   1.]),
 array([  0.     ,  12.80823,  25.61646,  38.42469,  51.23292,  64.04115,
         76.84938,  89.65761, 102.46584, 115.27407, 128.0823 , 140.89053,
        153.69876, 166.50699, 179.31522, 192.12345, 204.93168, 217.73991,
        230.54814, 243.35637, 256.1646 , 268.97283, 281.78106, 294.58929,
        307.39752, 320.20575, 333.01398, 345.82221, 358.63044, 371.43867,
        384.2469 , 397.05513, 409.86336, 422.67159, 435.47982, 448.28805,
        461.09628, 473.90451, 486.71274, 499.52097, 512.3292 ]),
 <BarContainer object of 40 artists>)
```
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/2782889/3fd8b07c-f21c-6f3a-9aa2-dd9e5f1ab8b1.png)

外れ値が大きいので平均値ではなく中央値で補完します(ちなみに某資格試験では平均値の重要性について全体の値を反映していることについて言及していました)。
```Python3
df_test["Fare"] = df_test["Fare"].fillna(df_test["Fare"].median())
```
## ダミー変数化
```Python3
df_test = pd.get_dummies(df_test)
df_test
```
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/2782889/d5db58d0-42ee-04df-c54f-def94c6e94f4.png)

よく見ると訓練データと変数の個数が異なることがわかります。
そしてその要因が名前(敬称)です。
## Ageの補完
訓練データと同様に機械学習で補完します。
```Python3
x_train = df_test.dropna().drop("Age", axis=1)
x_test = df_test[df_test["Age"].isnull()].drop("Age", axis=1)
y_train = df_test.dropna()["Age"]

model = GradientBoostingRegressor()
model.fit(x_train, y_train)
y_pred = model.predict(x_test)

pred = pd.DataFrame(y_pred)
pred.index = x_test.index
pred.columns = ["Age"]
test = pd.concat([x_test, pred], axis=1)
train = pd.concat([x_train, y_train], axis=1)
df_test = pd.concat([train, test])
df_test
```
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/2782889/0d4920f0-d5da-a83d-52f4-d5216116e858.png)

# 変数を合わせる
機械学習で予測するうえでPandasでは変数全部そろえる必要があります。
## 変数の確認
### 訓練データ
```Python3
df_train.columns
```
```
Index(['Survived', 'Pclass', 'SibSp', 'Parch', 'Fare', 'Name_ Capt',
       'Name_ Col', 'Name_ Don', 'Name_ Dr', 'Name_ Jonkheer', 'Name_ Lady',
       'Name_ Major', 'Name_ Master', 'Name_ Miss', 'Name_ Mlle', 'Name_ Mme',
       'Name_ Mr', 'Name_ Mrs', 'Name_ Ms', 'Name_ Rev', 'Name_ Sir',
       'Name_ the Countess', 'Sex_female', 'Sex_male', 'Embarked_C',
       'Embarked_Q', 'Embarked_S', 'Age'],
      dtype='object')
```
### テストデータ
```Python3
df_test.columns
```
```
Index(['Pclass', 'SibSp', 'Parch', 'Fare', 'Name_ Col', 'Name_ Dona',
       'Name_ Dr', 'Name_ Master', 'Name_ Miss', 'Name_ Mr', 'Name_ Mrs',
       'Name_ Ms', 'Name_ Rev', 'Sex_female', 'Sex_male', 'Embarked_C',
       'Embarked_Q', 'Embarked_S', 'Age'],
      dtype='object')
```
### 足りない変数を確認する
```Python3
for col in df_train.columns:
    if col not in df_test.columns:
        print("'"+ col +"',")
```
```
'Survived',
'Name_ Capt',
'Name_ Don',
'Name_ Jonkheer',
'Name_ Lady',
'Name_ Major',
'Name_ Mlle',
'Name_ Mme',
'Name_ Sir',
'Name_ the Countess',
```
### 結合関数を作る
そこに無いということはダミー変数では必ずFalseになるはずなのでそのような値をとる変数を疑似的に作ります。
```Python3
def concat(df, columns):
    for col in columns:
        tmp = []
        for i in range(len(df["Age"])):
            tmp.append(False)
        Capt = pd.DataFrame(tmp)
        Capt.index = df.index
        Capt.columns = [col]
        df = pd.concat([df, Capt], axis=1)
    return df
```
この関数に足りない変数を確認するスクリプトで作った出力結果を利用して関数に入れます。
```Python3
df_test = concat(df_test, ['Name_ Capt',
                            'Name_ Don',
                            'Name_ Jonkheer',
                            'Name_ Lady',
                            'Name_ Major',
                            'Name_ Mlle',
                            'Name_ Mme',
                            'Name_ Sir',
                            'Name_ the Countess'])
df_test
```
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/2782889/99a5c035-75a3-f457-a0e0-9fda91954a32.png)

変数が増えました。
### 訓練データ
```Python3
for col in df_test.columns:
    if col not in df_train.columns:
        print("'"+ col +"'")
```
```
'Name_ Dona'
```
一つだけでした
### 結合
```Python3
df_train = concat(df_train, ['Name_ Dona'])
df_train.columns
```
```
Index(['Survived', 'Pclass', 'SibSp', 'Parch', 'Fare', 'Name_ Capt',
       'Name_ Col', 'Name_ Don', 'Name_ Dr', 'Name_ Jonkheer', 'Name_ Lady',
       'Name_ Major', 'Name_ Master', 'Name_ Miss', 'Name_ Mlle', 'Name_ Mme',
       'Name_ Mr', 'Name_ Mrs', 'Name_ Ms', 'Name_ Rev', 'Name_ Sir',
       'Name_ the Countess', 'Sex_female', 'Sex_male', 'Embarked_C',
       'Embarked_Q', 'Embarked_S', 'Age', 'Name_ Dona'],
      dtype='object')
```
### 変数の順番を揃える
念のため変数の順番を揃えます。
```Python3
df_train = df_train.sort_index(axis=1)
df_test = df_test.sort_index(axis=1)
```
# 生存の予測
```Python3
x = df_train.drop("Survived", axis=1)
y = df_train["Survived"]
x_test = df_test

from sklearn.model_selection import train_test_split as tts
from sklearn.ensemble import RandomForestClassifier as RFC
from lightgbm import LGBMClassifier as LGBMC

models = []
for i in range(200):
    x_train, x_val, y_train, y_val = tts(x, y, random_state=i, test_size=0.2, stratify=y)
    model1 = RFC()
    model1.fit(x_train, y_train)
    models.append([model1, model1.score(x_val, y_val)])
    model2 = LGBMC()
    model2.fit(x_train, y_train)
    models.append([model2, model2.score(x_val, y_val)])
models = sorted(models, key=lambda x:x[1], reverse=True)

from scipy.stats import mode
y_pred1 = models[0][0].predict(x_test)
y_pred2 = models[1][0].predict(x_test)
y_pred3 = models[2][0].predict(x_test)
y_pred = []
for i in range(len(y_pred1)):
    y_pred.append(mode([y_pred1[i], y_pred2[i], y_pred3[i]])[0])
```
400個予測モデルを作って上位3個のモデルで予測を行います。
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/2782889/e829acd9-6cae-e55a-d297-e7edd26042dc.png)


これは期待できるかな・・・
```Python3
pred = pd.DataFrame(y_pred)
pred.columns = ["Survived"]
pred.index = x_test.index
pred.sort_index().to_csv("submit_allval.csv")
```
データフレーム化して提出データを作りました。

精度は74.641%でした。前回より悪いやん・・・

# 参考文献

https://qiita.com/jun40vn/items/d8a1f71fae680589e05c