Titanic分析をざっとしてみる

## これは何の記事？
Titanic分析の学習ログです

## 環境
Google Colab

## 分析の流れ
<ul>
<li>データの概要把握
<li>データの可視化
<li>前処理・特徴量の生成
<li>モデリング
<li>Submit
</ul>

## データの概要把握
```python
# 使用するライブラリをインポート
import pandas as pd
import numpy as np

# データの読み込み
train_df = pd.read_csv("/content/drive/MyDrive/Kaggle/Titanic/train.csv")
test_df = pd.read_csv("/content/drive/MyDrive/Kaggle/Titanic/test.csv")
submission = pd.read_csv("/content/drive/MyDrive/Kaggle/Titanic/gender_submission.csv")

# データ内容を確認
train_df.info()
```
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/3179561/39d96e8e-0922-e758-8fb5-af46d951a37e.png)
・AgeとCabinは欠損値が多いため注意
・Parch：乗船している親や子供の数

```python
# 先頭の5行を確認
train_df.head()
```
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/3179561/081d00bc-9df5-fdb3-f12b-c86d94a75163.png)
・Surviveは0,1フラグで記載

```python
# 男女比
train_df["Sex"].value_counts()
```
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/3179561/6c88e5ae-d251-edf1-5c39-77fa2c777f70.png)

## データを可視化
#### 女性の方が生存率が高い
```python
# 男女別の生存比
train_df[["Sex","Survived","PassengerId"]].groupby(["Sex","Survived"]).count()
```
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/3179561/82284be6-a3c0-2ed9-0c15-efbba348ec05.png)

#### C（Cherbourg港）から乗船した人は生存率が高い
```python
# 港別の生存比
train_df[["Embarked","Survived","PassengerId"]].groupby(["Embarked","Survived"]).count().unstack()
```
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/3179561/2d38e8e0-1b80-7994-09e3-655c9c8a8151.png)

#### 港別の生存率をグラフ化
```python
# 港別生存比の式を代入
embarked_df = train_df[["Embarked","Survived","PassengerId"]].groupby(["Embarked","Survived"]).count().unstack()

# 港別生存比を棒グラフ化
# stacked=Trueは積み上げの指示
embarked_df.plot.bar(stacked=True)
```
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/3179561/cd8fbfbe-e2d5-3851-f2cf-a207c3756712.png)

#### 男女と港はダミー変数化しておく
```python
# 男女をカテゴリ変数からダミー変数に
# 値は2つしかないので1列でよい、最初の列を削除
train_df_dummies = pd.get_dummies(train_df, columns=["Sex"], drop_first=True)

# 港をカテゴリ変数からダミー変数に
train_df_dummies = pd.get_dummies(train_df_dummies, columns=["Embarked"])

train_df_dummies.head()
```
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/3179561/348db492-d61b-6ab1-8fcb-3c9566b58676.png)

#### 若い方が生存率が高い
```python
# 使用するライブラリをインポート
import matplotlib.pyplot as plt

# 生存別の年齢データを取得
# 年齢が欠損しているレコードは削除
death_age = train_df[train_df["Survived"] == 0]["Age"].dropna()
survive_age = train_df[train_df["Survived"] == 1]["Age"].dropna()

# ヒストグラムを作成、ラベルを指定
plt.hist((death_age, survive_age), histtype="barstacked", bins=8, label=("Death", "Survive"))
plt.legend()
plt.xlabel("Age")
plt.ylabel("Count")
plt.show()
```
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/3179561/6f66ef53-8250-5563-0c35-9e3709ebb794.png)

#### 相関係数を確認
```python
# 相関行列を作成
train_corr = train_df_dummies.corr()

train_corr
```
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/3179561/3aae2e85-e753-9c74-255a-c65571a5c916.png)

#### 年齢はあまり関係がない？（欠損値が多いせい？）
```python
# 使用するライブラリをインポート
import seaborn as sns

# ヒートマップを作成
plt.figure(figsize=(9,9))
sns.heatmap(train_corr, vmax=1, vmin=-1, center=0, annot=True)
```
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/3179561/eda4dd8b-42a9-af3b-5821-23f582aa6313.png)

## 前処理・特徴量の生成
#### 学習データとテストデータを結合
```python
# 学習データとテストデータを結合
all_df = pd.concat([train_df, test_df], sort=False).reset_index(drop=True)

all_df
```
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/3179561/93ae9abe-56f5-a677-d2ec-c3889d198699.png)


#### 欠損値確認
```python
all_df.isnull().sum()
```
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/3179561/1108fe48-198c-0b9b-8b55-e93dcf92be6b.png)


#### 欠損値穴埋め（Fare）
```python
# Pclassごとに平均値算出
Fare_mean = all_df[["Pclass", "Fare"]].groupby("Pclass").mean().reset_index()

# カラム変更
Fare_mean.columns = ["Pclass", "Fare_mean"]

Fare_mean
```
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/3179561/df7fa352-8da1-6ee8-849d-a65e09d268fd.png)
```python
# all_dfにFare_meanを結合
all_df = pd.merge(all_df, Fare_mean, on="Pclass", how="left")

# FareがNULLの場合、Fare_meanの値に置き換え
all_df.loc[(all_df["Fare"].isnull()), "Fare"] = all_df["Fare_mean"]

# Fare_meanの列を削除
all_df = all_df.drop("Fare_mean", axis=1)

all_df
```
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/3179561/df473910-4416-47a2-7d9c-283e848103a7.png)
#### 敬称を取得
```python
# 苗字、敬称、名前で分割したdf作成
name_df = all_df["Name"].str.split("[,.]",2,expand=True)
name_df.columns = ["family_name", "honorific", "name"]

# 前後の空白を削除
name_df["family_name"] = name_df["family_name"].str.strip()
name_df["honorific"] = name_df["honorific"].str.strip()
name_df["name"] = name_df["name"].str.strip()

name_df
```
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/3179561/85943fac-7233-d0bb-1ba7-ad5502b2f5ee.png)
```python
# 敬称別でカウント
name_df["honorific"].value_counts()
```
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/3179561/de277032-89b1-dab3-5422-df35c5633c74.png)
```python
# dfを横に結合
all_df = pd.concat([all_df, name_df], axis=1)
all_df
```
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/3179561/df0e7580-e2b7-c551-72f4-12b296a9f259.png)
```python
# 敬称別の分布
plt.figure(figsize=(18, 5))
sns.boxplot(x="honorific", y="Age", data=all_df)
```
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/3179561/936f8c12-e66a-2115-1e0a-ae0cf54f3d80.png)

#### MissとMirsは生存率が高く、Mrは死亡率が高い
```python
# 元のdfにname_dfを結合
train_df = pd.concat([train_df, name_df[0:len(train_df)].reset_index(drop=True)], axis=1)
test_df = pd.concat([train_df, name_df[len(train_df):].reset_index(drop=True)], axis=1)

# 敬称別で生存比を確認
honorific_df = train_df[["honorific", "Survived", "PassengerId"]].dropna().groupby(["honorific", "Survived"]).count().unstack()
honorific_df.plot.bar(stacked=True)
```
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/3179561/67bf35e4-b15d-d1b8-663f-bb4401df6fb6.png)

#### 年齢の欠損は敬称ごとの平均値で補完
```python
# 敬称ごとの平均値を出す
honorific_age_mean = all_df[["honorific", "Age"]].groupby("honorific").mean().reset_index()
honorific_age_mean.columns = ["honorific", "honorific_age"]

# 敬称ごとの平均値カラムを用意し、NULLの場合代入
all_df = pd.merge(all_df, honorific_age_mean, on="honorific", how="left")
all_df.loc[(all_df["Age"].isnull(), "Age")] = all_df["honorific_age"]

# 不要な列削除
all_df = all_df.drop(["honorific_age"], axis=1)
```

#### 家族人数を追加
```python
# 家族人数カラムを用意
all_df["family_num"] = all_df["Parch"] + all_df["SibSp"]
all_df["family_num"].value_counts()
```
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/3179561/60e84d74-225d-9131-43b4-bc7f2296ae38.png)
```python
# 一人乗船フラグを用意
all_df.loc[all_df["family_num"] == 0, "alone"] = 1
all_df["alone"].fillna(0, inplace=True)

all_df
```
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/3179561/9edd46bf-8dc5-59dd-5159-07dc63673a41.png)

#### カテゴリ変数を数値に変換
```python
# 不使用のカラム削除
all_df = all_df.drop(["PassengerId", "Name", "family_name", "name", "Ticket", "Cabin"], axis=1)
all_df.head()
```
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/3179561/98a64617-89e6-598f-6cfc-a2e9fa6231b0.png)
```python
# カテゴリ変数として管理
categories = all_df.columns[all_df.dtypes == "object"]
print(categories)
```
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/3179561/196fe26e-10f6-6241-0a65-afc252f5009f.png)
```python
# 敬称は5種類に統一
all_df.loc[~((all_df["honorific"] == "Mr") |
             (all_df["honorific"] == "Miss") |
             (all_df["honorific"] == "Mrs") |
             (all_df["honorific"] == "Master")), "honorific"] = "other"

all_df.honorific.value_counts()
```
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/3179561/f34a70bd-d3d6-130f-c10d-73e31a133b83.png)
```python
# 使用するライブラリをインポート
from sklearn.preprocessing import LabelEncoder

# 欠損値をNaNで埋める
all_df["Embarked"].fillna("missing", inplace=True)

# Sexを数値変換
le = LabelEncoder()
le = le.fit(all_df["Sex"])
all_df["Sex"] = le.transform(all_df["Sex"])

# 残りはまとめて
for cat in categories:
  le = LabelEncoder()
  if all_df[cat].dtypes == "object":
    le = le.fit(all_df[cat])
    all_df[cat] = le.transform(all_df[cat])

all_df.head()
```
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/3179561/b7d49bda-b306-ce99-7400-24f5e4322ddf.png)

#### データを学習用とテスト用に戻す
```python
train_X = all_df[~all_df["Survived"].isnull()].drop("Survived", axis=1).reset_index(drop=True)
train_Y = train_df["Survived"]

test_X = all_df[all_df["Survived"].isnull()].drop("Survived", axis=1).reset_index(drop=True)
```

## モデリング
#### LightGBMでトライ
```python
# 使用するライブラリをインポート
import lightgbm as lgb
from sklearn.model_selection import train_test_split
from sklearn.model_selection import KFold

X_train, X_valid, y_train, y_valid = train_test_split(train_X, train_Y, test_size=0.2)

# データセット
categories = ["Embarked", "Pclass", "Sex", "honorific", "alone"]

lgb_train = lgb.Dataset(X_train, y_train, categorical_feature=categories)
lgb_eval = lgb.Dataset(X_valid, y_valid, categorical_feature=categories, reference=lgb_train)

# パラメーター設定
lgbm_params = {
    "objective":"binary",
    "random_seed":1234
}

model_lgb = lgb.train(lgbm_params,
                      lgb_train,
                      valid_sets=lgb_eval,
                      num_boost_round=100,
                      early_stopping_rounds=20,
                      verbose_eval=10)
```
※一部のライブラリが最新バージョンに対応してない模様
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/3179561/a73902b5-b49c-1b7c-1a91-a6f3f6b2f7cb.png)

#### Fare, Ageが重要な変数
```python
# 各変数の重要度
model_lgb.feature_importance()
```
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/3179561/b78adecb-fa9d-907c-02f7-e5ca019daae7.png)
```python
# 可視化
importance = pd.DataFrame(model_lgb.feature_importance(), index=X_train.columns, columns=["importance"]).sort_values(by="importance", ascending=True)
importance.plot.barh()
```
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/3179561/6fda8780-a2bd-d4a1-07e5-f1d3f3996d40.png)
```python
# 最も精度の高い学習モデルを適用
y_pred = model_lgb.predict(X_valid, num_iteration=model_lgb.best_iteration)

# accuracyを計算
from sklearn.metrics import accuracy_score
accuracy_score(y_valid, np.round(y_pred))
```
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/3179561/40413f64-9649-6944-2432-5ea9795c94f3.png)

#### ハイパーパラメータを変更
```python
# パラメーターを再設定
lgbm_params = {
    "objective":"binary",
    "max_bin":331,
    "num_leaves":20,
    "min_data_in_leaf":57,
    "random_seed":1234
}

categories = ["Embarked", "Pclass", "Sex", "honorific", "alone"]
lgb_train = lgb.Dataset(X_train, y_train, categorical_feature=categories)
lgb_eval = lgb.Dataset(X_valid, y_valid, categorical_feature=categories, reference=lgb_train)

model_lgb = lgb.train(lgbm_params,
                      lgb_train,
                      valid_sets=lgb_eval,
                      num_boost_round=100,
                      early_stopping_rounds=20,
                      verbose_eval=10)
```
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/3179561/57ff110b-fe93-3a04-c20e-64a350e17de3.png)
```python
# 可視化
importance = pd.DataFrame(model_lgb.feature_importance(), index=X_train.columns, columns=["importance"]).sort_values(by="importance", ascending=True)
importance.plot.barh()
```
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/3179561/543f1b26-14d7-a962-e948-6a088ca48a92.png)
```python
y_pred = model_lgb.predict(X_valid, num_iteration=model_lgb.best_iteration)

# accuracyを計算
from sklearn.metrics import accuracy_score
accuracy_score(y_valid, np.round(y_pred))
```
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/3179561/6c427304-1286-329f-5e3a-386a0da6d80e.png)

#### クロスバリデーション
```python
# 3分割
folds = 3
kf = KFold(n_splits=folds)

models = []

for train_index, val_index in kf.split(train_X):
  X_train = train_X.iloc[train_index]
  X_valid = train_X.iloc[val_index]
  y_train = train_Y.iloc[train_index]
  y_valid = train_Y.iloc[val_index]

  lgb_train = lgb.Dataset(X_train, y_train, categorical_feature=categories)
  lgb_eval = lgb.Dataset(X_valid, y_valid, categorical_feature=categories, reference=lgb_train)

  model_lgb = lgb.train(lgbm_params,
                        lgb_train,
                        valid_sets=lgb_eval,
                        num_boost_round=100,
                        early_stopping_rounds=20,
                        verbose_eval=10
                        )
  
  y_pred = model_lgb.predict(X_valid, num_iteration=model_lgb.best_iteration)
  print(accuracy_score(y_valid, np.round(y_pred)))

  models.append(model_lgb)
```
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/3179561/d6abfefb-6b54-9fba-0290-f0b90c75e408.png)
```python
# テストデータの結果を予測して格納
preds = []

for model in models:
  pred = model.predict(test_X)
  preds.append(pred)

# 平均
preds_array = np.array(preds)
preds_mean = np.mean(preds_array, axis=0)

# 予測生存確率を0か1に変換、0.5より大きいと1
preds_int = (preds_mean > 0.5).astype(int)

submission["Survived"] = preds_int
submission
```
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/3179561/22bc51d3-2df9-8124-d63b-51b661c9d5f6.png)

#### 結果をcsvに書きだす
```python
# csvに書き出し
submission.to_csv("Titanic_submit01.csv", index=False)
# 内容確認
!cat Titanic_submit01.csv
```
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/3179561/cb73c9f9-28fa-8959-5c11-199cc639924b.png)
```python
# ローカルにダウンロード
from google.colab import files
files.download("Titanic_submit01.csv")
```

##  Submit
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/3179561/d2964918-01ce-72a6-ff94-4c4d2ef797ee.png)