Data Every Day: 成人の国勢調査所得


    
    

# tldr

Kggleの[Adult Census Income](https://www.kaggle.com/uciml/adult-census-income)を[High Performance Income Prediction - Data Every Day #029](https://www.youtube.com/watch?v=fy6LqgOFIlc&list=PLs_CQl4mUvI7Z9HnwoeOS5EUu9gyvekjK&index=29)に沿ってやっていきます。

実行環境はGoogle Colaboratorです。

# インポート


```python
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns

import sklearn.preprocessing as sp
from sklearn.model_selection import train_test_split

from sklearn.linear_model import LogisticRegression

import tensorflow as tf
```

# データのダウンロード

Google Driveをマウントします。


```python
from google.colab import drive
drive.mount('/content/drive')
```

    Mounted at /content/drive


KaggleのAPIクライアントを初期化し、認証します。
認証情報はGoogle Drive内（`/content/drive/My Drive/Colab Notebooks/Kaggle`）に`kaggle.json`として置いてあります。


```python
import os
kaggle_path = "/content/drive/My Drive/Colab Notebooks/Kaggle"
os.environ['KAGGLE_CONFIG_DIR'] = kaggle_path

from kaggle.api.kaggle_api_extended import KaggleApi
api = KaggleApi()
api.authenticate() 
```

Kaggle APIを使ってデータをダウンロードします。


```python
dataset_id = 'uciml/adult-census-income'
dataset = api.dataset_list_files(dataset_id)
file_name = dataset.files[0].name
file_path = os.path.join(api.get_default_download_dir(), file_name)
file_path
```

    Warning: Looks like you're using an outdated API Version, please consider updating (server 1.5.10 / client 1.5.9)





    '/content/adult.csv'




```python
api.dataset_download_file(dataset_id, file_name, force=True, quiet=False)
```

    100%|██████████| 450k/450k [00:00<00:00, 49.3MB/s]

    Downloading adult.csv.zip to /content
    


    





    True




```python
import zipfile

zip_path = file_path + '.zip'
with zipfile.ZipFile(zip_path) as existing_zip:
    existing_zip.extractall('/content')
```

# データの読み込み

Padasを使ってダウンロードしてきたCSVファイルを読み込みます。


```python
data = pd.read_csv(file_path)
```


```python
data
```




<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>age</th>
      <th>workclass</th>
      <th>fnlwgt</th>
      <th>education</th>
      <th>education.num</th>
      <th>marital.status</th>
      <th>occupation</th>
      <th>relationship</th>
      <th>race</th>
      <th>sex</th>
      <th>capital.gain</th>
      <th>capital.loss</th>
      <th>hours.per.week</th>
      <th>native.country</th>
      <th>income</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>90</td>
      <td>?</td>
      <td>77053</td>
      <td>HS-grad</td>
      <td>9</td>
      <td>Widowed</td>
      <td>?</td>
      <td>Not-in-family</td>
      <td>White</td>
      <td>Female</td>
      <td>0</td>
      <td>4356</td>
      <td>40</td>
      <td>United-States</td>
      <td>&lt;=50K</td>
    </tr>
    <tr>
      <th>1</th>
      <td>82</td>
      <td>Private</td>
      <td>132870</td>
      <td>HS-grad</td>
      <td>9</td>
      <td>Widowed</td>
      <td>Exec-managerial</td>
      <td>Not-in-family</td>
      <td>White</td>
      <td>Female</td>
      <td>0</td>
      <td>4356</td>
      <td>18</td>
      <td>United-States</td>
      <td>&lt;=50K</td>
    </tr>
    <tr>
      <th>2</th>
      <td>66</td>
      <td>?</td>
      <td>186061</td>
      <td>Some-college</td>
      <td>10</td>
      <td>Widowed</td>
      <td>?</td>
      <td>Unmarried</td>
      <td>Black</td>
      <td>Female</td>
      <td>0</td>
      <td>4356</td>
      <td>40</td>
      <td>United-States</td>
      <td>&lt;=50K</td>
    </tr>
    <tr>
      <th>3</th>
      <td>54</td>
      <td>Private</td>
      <td>140359</td>
      <td>7th-8th</td>
      <td>4</td>
      <td>Divorced</td>
      <td>Machine-op-inspct</td>
      <td>Unmarried</td>
      <td>White</td>
      <td>Female</td>
      <td>0</td>
      <td>3900</td>
      <td>40</td>
      <td>United-States</td>
      <td>&lt;=50K</td>
    </tr>
    <tr>
      <th>4</th>
      <td>41</td>
      <td>Private</td>
      <td>264663</td>
      <td>Some-college</td>
      <td>10</td>
      <td>Separated</td>
      <td>Prof-specialty</td>
      <td>Own-child</td>
      <td>White</td>
      <td>Female</td>
      <td>0</td>
      <td>3900</td>
      <td>40</td>
      <td>United-States</td>
      <td>&lt;=50K</td>
    </tr>
    <tr>
      <th>...</th>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
    </tr>
    <tr>
      <th>32556</th>
      <td>22</td>
      <td>Private</td>
      <td>310152</td>
      <td>Some-college</td>
      <td>10</td>
      <td>Never-married</td>
      <td>Protective-serv</td>
      <td>Not-in-family</td>
      <td>White</td>
      <td>Male</td>
      <td>0</td>
      <td>0</td>
      <td>40</td>
      <td>United-States</td>
      <td>&lt;=50K</td>
    </tr>
    <tr>
      <th>32557</th>
      <td>27</td>
      <td>Private</td>
      <td>257302</td>
      <td>Assoc-acdm</td>
      <td>12</td>
      <td>Married-civ-spouse</td>
      <td>Tech-support</td>
      <td>Wife</td>
      <td>White</td>
      <td>Female</td>
      <td>0</td>
      <td>0</td>
      <td>38</td>
      <td>United-States</td>
      <td>&lt;=50K</td>
    </tr>
    <tr>
      <th>32558</th>
      <td>40</td>
      <td>Private</td>
      <td>154374</td>
      <td>HS-grad</td>
      <td>9</td>
      <td>Married-civ-spouse</td>
      <td>Machine-op-inspct</td>
      <td>Husband</td>
      <td>White</td>
      <td>Male</td>
      <td>0</td>
      <td>0</td>
      <td>40</td>
      <td>United-States</td>
      <td>&gt;50K</td>
    </tr>
    <tr>
      <th>32559</th>
      <td>58</td>
      <td>Private</td>
      <td>151910</td>
      <td>HS-grad</td>
      <td>9</td>
      <td>Widowed</td>
      <td>Adm-clerical</td>
      <td>Unmarried</td>
      <td>White</td>
      <td>Female</td>
      <td>0</td>
      <td>0</td>
      <td>40</td>
      <td>United-States</td>
      <td>&lt;=50K</td>
    </tr>
    <tr>
      <th>32560</th>
      <td>22</td>
      <td>Private</td>
      <td>201490</td>
      <td>HS-grad</td>
      <td>9</td>
      <td>Never-married</td>
      <td>Adm-clerical</td>
      <td>Own-child</td>
      <td>White</td>
      <td>Male</td>
      <td>0</td>
      <td>0</td>
      <td>20</td>
      <td>United-States</td>
      <td>&lt;=50K</td>
    </tr>
  </tbody>
</table>
<p>32561 rows × 15 columns</p>
</div>



# 下準備

## 欠損値の処理


```python
data.isnull().sum()
```




    age               0
    workclass         0
    fnlwgt            0
    education         0
    education.num     0
    marital.status    0
    occupation        0
    relationship      0
    race              0
    sex               0
    capital.gain      0
    capital.loss      0
    hours.per.week    0
    native.country    0
    income            0
    dtype: int64




```python
値に「?」が入っている数を数えます
```


```python
data.isin(['?']).sum()
```




    age                  0
    workclass         1836
    fnlwgt               0
    education            0
    education.num        0
    marital.status       0
    occupation        1843
    relationship         0
    race                 0
    sex                  0
    capital.gain         0
    capital.loss         0
    hours.per.week       0
    native.country     583
    income               0
    dtype: int64




```python
data = data.replace('?', np.NaN)
```


```python
data.isna().sum()
```




    age                  0
    workclass         1836
    fnlwgt               0
    education            0
    education.num        0
    marital.status       0
    occupation        1843
    relationship         0
    race                 0
    sex                  0
    capital.gain         0
    capital.loss         0
    hours.per.week       0
    native.country     583
    income               0
    dtype: int64



educationとeducation.num列は同じ情報をエンコードしただけなので、educationは削除します。


```python
data.loc[:, ['education', 'education.num']]
```




<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>education</th>
      <th>education.num</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>HS-grad</td>
      <td>9</td>
    </tr>
    <tr>
      <th>1</th>
      <td>HS-grad</td>
      <td>9</td>
    </tr>
    <tr>
      <th>2</th>
      <td>Some-college</td>
      <td>10</td>
    </tr>
    <tr>
      <th>3</th>
      <td>7th-8th</td>
      <td>4</td>
    </tr>
    <tr>
      <th>4</th>
      <td>Some-college</td>
      <td>10</td>
    </tr>
    <tr>
      <th>...</th>
      <td>...</td>
      <td>...</td>
    </tr>
    <tr>
      <th>32556</th>
      <td>Some-college</td>
      <td>10</td>
    </tr>
    <tr>
      <th>32557</th>
      <td>Assoc-acdm</td>
      <td>12</td>
    </tr>
    <tr>
      <th>32558</th>
      <td>HS-grad</td>
      <td>9</td>
    </tr>
    <tr>
      <th>32559</th>
      <td>HS-grad</td>
      <td>9</td>
    </tr>
    <tr>
      <th>32560</th>
      <td>HS-grad</td>
      <td>9</td>
    </tr>
  </tbody>
</table>
<p>32561 rows × 2 columns</p>
</div>




```python
data = data.drop('education', axis=1)
```

## オブジェクト型の処理


```python
data.dtypes
```




    age                int64
    workclass         object
    fnlwgt             int64
    education         object
    education.num      int64
    marital.status    object
    occupation        object
    relationship      object
    race              object
    sex               object
    capital.gain       int64
    capital.loss       int64
    hours.per.week     int64
    native.country    object
    income            object
    dtype: object




```python
categorical_features =  [
    'workclass', 
    'marital.status', 
    'occupation', 
    'relationship', 
    'race', 
    'sex', 
    'native.country']
```


```python
def get_uniques(df, columns):
    uniques = dict()
    for column in columns:
        uniques[column] = list(df[column].unique())
    return uniques
```


```python
get_uniques(data, categorical_features)
```




    {'marital.status': ['Widowed',
      'Divorced',
      'Separated',
      'Never-married',
      'Married-civ-spouse',
      'Married-spouse-absent',
      'Married-AF-spouse'],
     'native.country': ['United-States',
      nan,
      'Mexico',
      'Greece',
      'Vietnam',
      'China',
      'Taiwan',
      'India',
      'Philippines',
      'Trinadad&Tobago',
      'Canada',
      'South',
      'Holand-Netherlands',
      'Puerto-Rico',
      'Poland',
      'Iran',
      'England',
      'Germany',
      'Italy',
      'Japan',
      'Hong',
      'Honduras',
      'Cuba',
      'Ireland',
      'Cambodia',
      'Peru',
      'Nicaragua',
      'Dominican-Republic',
      'Haiti',
      'El-Salvador',
      'Hungary',
      'Columbia',
      'Guatemala',
      'Jamaica',
      'Ecuador',
      'France',
      'Yugoslavia',
      'Scotland',
      'Portugal',
      'Laos',
      'Thailand',
      'Outlying-US(Guam-USVI-etc)'],
     'occupation': [nan,
      'Exec-managerial',
      'Machine-op-inspct',
      'Prof-specialty',
      'Other-service',
      'Adm-clerical',
      'Craft-repair',
      'Transport-moving',
      'Handlers-cleaners',
      'Sales',
      'Farming-fishing',
      'Tech-support',
      'Protective-serv',
      'Armed-Forces',
      'Priv-house-serv'],
     'race': ['White',
      'Black',
      'Asian-Pac-Islander',
      'Other',
      'Amer-Indian-Eskimo'],
     'relationship': ['Not-in-family',
      'Unmarried',
      'Own-child',
      'Other-relative',
      'Husband',
      'Wife'],
     'sex': ['Female', 'Male'],
     'workclass': [nan,
      'Private',
      'State-gov',
      'Federal-gov',
      'Self-emp-not-inc',
      'Self-emp-inc',
      'Local-gov',
      'Without-pay',
      'Never-worked']}



各データの特性を見て

- Labelエンコード
- Onehotエンコード
- Ordinalエンコード

のどれでエンコードするか判断する


```python
binary_features = ['sex']
nominal_features = [
    'workclass', 
    'marital.status', 
    'occupation', 
    'relationship', 
    'race', 
    'native.country',
]
```

Binary エンコード


```python
def binary_encode(df, columns):
    label_encoder = sp.LabelEncoder()
    for column in columns:
        df[column] = label_encoder.fit_transform(df[column])
    return df
```

Onehotエンコード


```python
def onehot_encode(df, columns):
    for column in columns:
        dummies = pd.get_dummies(data[column])
        df = pd.concat([df, dummies], axis=1)
        df = df.drop(column, axis=1)
    return df
```


```python
data = binary_encode(data, binary_features)
data = onehot_encode(data, nominal_features)
```

# スケーリング


```python
y = data['income']
X = data.drop('income', axis=1)
```


```python
label_encoder = sp.LabelEncoder()
y = label_encoder.fit_transform(y)
y_mappings = {index: label for index, label in enumerate(label_encoder.classes_)}
```


```python
scaler = sp.MinMaxScaler()
X = pd.DataFrame(scaler.fit_transform(X), columns=X.columns)
```

# Training

Tensorflowを使ってモデルを構築します。
最後のレイヤーの活性化関数はSimoidを使用します。


```python
X_train, X_test, y_train, y_test = train_test_split(X, y, train_size=0.8)
```


```python
model = tf.keras.Sequential([
    tf.keras.layers.Dense(16, activation='relu', input_shape=(88, )),
    tf.keras.layers.Dense(16, activation='relu'),
    tf.keras.layers.Dense(1, activation='sigmoid'),
])

model.summary()

optimizer = tf.keras.optimizers.Adam(learning_rate=0.001)

metrics = [
    tf.keras.metrics.BinaryAccuracy(name='acc'),
    tf.keras.metrics.AUC(name='auc'),
]

model.compile(
    optimizer=optimizer,
    loss='binary_crossentropy',
    metrics=metrics,
)

batch_size = 32
epochs = 100

history = model.fit(
    X_train,
    y_train,
    validation_split=0.2,
    batch_size=batch_size,
    epochs=epochs,
    verbose=0,
)
```

    Model: "sequential_3"
    _________________________________________________________________
    Layer (type)                 Output Shape              Param #   
    =================================================================
    dense_10 (Dense)             (None, 16)                1424      
    _________________________________________________________________
    dense_11 (Dense)             (None, 16)                272       
    _________________________________________________________________
    dense_12 (Dense)             (None, 1)                 17        
    =================================================================
    Total params: 1,713
    Trainable params: 1,713
    Non-trainable params: 0
    _________________________________________________________________


# 結果


```python
plt.figure(figsize=(14,10))

epochs_range = range(1, epochs + 1)

train_loss = history.history['loss']
val_loss = history.history['val_loss']

plt.plot(epochs_range, train_loss, label='Training Loss')
plt.plot(epochs_range, val_loss, label='Validation Loss')

plt.xlabel('Epoch')
plt.ylabel('Loss')
plt.legend()

plt.show()
```


    
![AV _ Healthcare Analytics II_60_0.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/141539/14f6e6a7-21b2-752d-f7ec-940d83efad4e.png)
    



```python
model.evaluate(X_test, y_test)
```

    204/204 [==============================] - 0s 1ms/step - loss: 0.3338 - acc: 0.8475 - auc: 0.9050





    [0.33379316329956055, 0.8475356698036194, 0.9049904346466064]



過学習になる前のValidation lossが最も小さいときのepochsを割り出します


```python
np.argmin(val_loss)
```




    18



上記のepochsで学習を停止させると最も汎用的なモデルが得られます。