タイタニックの欠損値を真面目に考えてみた

生のデータにはたいてい欠損値が含まれますが、今回はその欠損値に統計解析と機械学習で補完するアプローチをしてみようと思います。
# タイタニックデータ
```Python3
import pandas as pd

df = pd.read_csv("train.csv")
df.head()
```
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/2782889/eeb71ad0-e293-aea1-d9c6-9febb3a1993b.png)
## 欠損値の確認
```Python3
df.isnull().sum()
```
```
PassengerId      0
Survived         0
Pclass           0
Name             0
Sex              0
Age            177
SibSp            0
Parch            0
Ticket           0
Fare             0
Cabin          687
Embarked         2
dtype: int64
```
データ数見てみるとCabinは補完しようがないのでAgeとEmbarkedを補完します。
```Python3
df.index = df["PassengerId"]
df = df.drop([
            'PassengerId',
            'Name',
            'Cabin',
            'Ticket'
            ],
            axis=1)
```
## Embarkedの補完
```Python3
import matplotlib.pyplot as plt
import seaborn as sns
sns.histplot(df["Embarked"])
```
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/2782889/c7fc0d1f-6ae6-dd4a-1d81-7e1c3e18c4a2.png)
Sが一番多かったためSで補完します。
```Python3
df["Embarked"].fillna("S")
```
## Ageの補完(訓練データ)
```Python3
df = pd.get_dummies(df)
df.head()
```
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/2782889/c2239325-5e41-000c-86d5-ff346e5cf461.png)

```Python3
train = df.copy()
train_train = train.dropna()
train_test = train[train["Age"].isnull()]
from lightgbm import LGBMRegressor
x_train = train_train.drop("Age", axis=1)
x_test = train_test.drop("Age", axis=1)
y_train = train_train["Age"]
model = LGBMRegressor()
model.fit(x_train, y_train)
y_pred = model.predict(x_test)
```
予測値で補完します。
```Python3
pred = pd.DataFrame(y_pred)
pred.index = train_test.index
pred.columns = ["Age"]
train_test["Age"] = pred["Age"]
train_test.isnull().sum()
```
```
Survived      0
Pclass        0
Age           0
SibSp         0
Parch         0
Fare          0
Sex_female    0
Sex_male      0
Embarked_C    0
Embarked_Q    0
Embarked_S    0
dtype: int64
```
欠損値がなくなりました。ここでデータを結合しておきます。
```Python3
train = pd.concat([train_test, train_train])
train.sort_index().head(10)
```

## テストデータの欠損値補完
```Python3
test = pd.read_csv("test.csv")
test.isnull().sum()
```
```
PassengerId      0
Pclass           0
Name             0
Sex              0
Age             86
SibSp            0
Parch            0
Ticket           0
Fare             1
Cabin          327
Embarked         0
dtype: int64
```
ここでもCabinは多すぎるのでAgeとFareを補完します。
## Fareの補完
```Python3
plt.hist(test["Fare"], bins=100)
```
```
(array([  3., 152.,  68.,  16.,  26.,  44.,  11.,   9.,   4.,   5.,   9.,
         10.,   5.,   5.,   6.,   5.,   5.,   1.,   2.,   0.,   1.,   1.,
          0.,   0.,   0.,   0.,   6.,   0.,   1.,   2.,   0.,   0.,   2.,
          0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   5.,   0.,   3.,
          1.,   0.,   0.,   0.,   1.,   0.,   0.,   7.,   0.,   0.,   0.,
          0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,
          0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,
          0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,
          0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,
          1.]),
 array([  0.      ,   5.123292,  10.246584,  15.369876,  20.493168,
         25.61646 ,  30.739752,  35.863044,  40.986336,  46.109628,
         51.23292 ,  56.356212,  61.479504,  66.602796,  71.726088,
         76.84938 ,  81.972672,  87.095964,  92.219256,  97.342548,
        102.46584 , 107.589132, 112.712424, 117.835716, 122.959008,
        128.0823  , 133.205592, 138.328884, 143.452176, 148.575468,
        153.69876 , 158.822052, 163.945344, 169.068636, 174.191928,
        179.31522 , 184.438512, 189.561804, 194.685096, 199.808388,
        204.93168 , 210.054972, 215.178264, 220.301556, 225.424848,
        230.54814 , 235.671432, 240.794724, 245.918016, 251.041308,
        256.1646  , 261.287892, 266.411184, 271.534476, 276.657768,
        281.78106 , 286.904352, 292.027644, 297.150936, 302.274228,
        307.39752 , 312.520812, 317.644104, 322.767396, 327.890688,
        333.01398 , 338.137272, 343.260564, 348.383856, 353.507148,
        358.63044 , 363.753732, 368.877024, 374.000316, 379.123608,
        384.2469  , 389.370192, 394.493484, 399.616776, 404.740068,
        409.86336 , 414.986652, 420.109944, 425.233236, 430.356528,
        435.47982 , 440.603112, 445.726404, 450.849696, 455.972988,
        461.09628 , 466.219572, 471.342864, 476.466156, 481.589448,
        486.71274 , 491.836032, 496.959324, 502.082616, 507.205908,
        512.3292  ]),
 ```
5.123292が一番多いので5.123292で補完します。
```Python3
test["Fare"] = test["Fare"].fillna(5.123292)
test.isnull().sum()
```
```
PassengerId      0
Pclass           0
Name             0
Sex              0
Age             86
SibSp            0
Parch            0
Ticket           0
Fare             0
Cabin          327
Embarked         0
dtype: int64
```
## Ageの補完
```Python3
test.index = test["PassengerId"]
test = test.drop([
            'PassengerId',
            'Name',
            'Cabin',
            'Ticket'
            ],
            axis=1)
test = pd.get_dummies(test)
test = test.drop("Sex_female", axis=1)
test = pd.get_dummies(test)
test
```
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/2782889/f2b7d262-623c-f5bb-87df-3ce47bf567d2.png)

```Python3
test_train = test.dropna()
test_test = test[test["Age"].isnull()]
print(len(test_test))
test_test.isnull().sum()
```
```
86
Pclass         0
Age           86
SibSp          0
Parch          0
Fare           0
Sex_male       0
Embarked_C     0
Embarked_Q     0
Embarked_S     0
dtype: int6
```
```Python3
x_train = test_train.drop("Age", axis=1)
x_test = test_test.drop("Age", axis=1)
y_train = test_train["Age"]
model = LGBMRegressor()
model.fit(x_train, y_train)
y_pred = model.predict(x_test)
```
## テストデータの結合
```Python3
df_test = pd.concat([test_train, test_test])
```
```Python3
age = pd.DataFrame(y_pred)
print(len(age), len(x_test))
age.index = test_test.index
age.columns = ["Age"]
test_test["Age"] = age["Age"]
test_test.isnull().sum()
```
```
Pclass        0
Age           0
SibSp         0
Parch         0
Fare          0
Sex_male      0
Embarked_C    0
Embarked_Q    0
Embarked_S    0
dtype: int64
```
欠損値がなくなりました。
## データの結合
```Python3
df_test = pd.concat([test_train, test_test])
df_test.isnull().sum()
```
```
Pclass        0
Age           0
SibSp         0
Parch         0
Fare          0
Sex_male      0
Embarked_C    0
Embarked_Q    0
Embarked_S    0
dtype: int64
```
# 生存の予測
```Python3
df_train = train.copy()
from sklearn.model_selection import train_test_split
from lightgbm import LGBMClassifier
models = []
x = df_train.drop(["Survived", "Sex_female", "Embarked_C"], axis=1)
y = df_train["Survived"]
for i in range(200):
    x_train, x_val, y_train, y_val = train_test_split(x, y, random_state=i, test_size=0.3)
    model = LGBMClassifier()
    model.fit(x_train, y_train)
    models.append([model, model.score(x_val, y_val)])
models = sorted(models, key=lambda x:x[1], reverse=True)
```
精度の良かったモデルで予測します。
```Python3
df_test = df_test.drop(["Embarked_C"], axis=1)
from scipy.stats import mode
model1 = models[0][0]
model2 = models[1][0]
model3 = models[2][0]
y_pred1 = model1.predict(df_test)
y_pred2 = model2.predict(df_test)
y_pred3 = model3.predict(df_test)
y_pred = []
for i in range(len(y_pred1)):
    y_pred.append(mode([y_pred1[i],y_pred2[i],y_pred3[i]])[0])
```
データフレーム化
```Python3
df_pred = pd.DataFrame(y_pred)
df_pred.columns = ["Survived"]
df_pred.index = df_test.index
df_pred
```
提出用ファイル作成
```Python3
df_pred.sort_index().to_csv("submission9.csv")
```

# まとめ
精度は0.75358となりました。(チクショーめー！75%って低すぎやろ)