Data Every Day: 河川水中の溶存酸素予測

# tldr

Kggleの[Dissolved oxygen prediction in river water](https://www.kaggle.com/vbmokin/dissolved-oxygen-prediction-in-river-water)を[Predicting Oxygen in Rivers - Data Every Day #036](https://www.youtube.com/watch?v=arfFLTtah_A&list=PLs_CQl4mUvI7Z9HnwoeOS5EUu9gyvekjK&index=36)に沿ってやっていきます。

実行環境はGoogle Colaboratorです。

# インポート


```python
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns

import sklearn.preprocessing as sp
from sklearn.model_selection import train_test_split
import sklearn.linear_model as slm

import tensorflow as tf
```

# データのダウンロード

Google Driveをマウントします。


```python
from google.colab import drive
drive.mount('/content/drive')
```

    Drive already mounted at /content/drive; to attempt to forcibly remount, call drive.mount("/content/drive", force_remount=True).


KaggleのAPIクライアントを初期化し、認証します。
認証情報はGoogle Drive内（`/content/drive/My Drive/Colab Notebooks/Kaggle`）に`kaggle.json`として置いてあります。


```python
import os
kaggle_path = "/content/drive/My Drive/Colab Notebooks/Kaggle"
os.environ['KAGGLE_CONFIG_DIR'] = kaggle_path

from kaggle.api.kaggle_api_extended import KaggleApi
api = KaggleApi()
api.authenticate() 
```

Kaggle APIを使ってデータをダウンロードします。


```python
dataset_id = 'vbmokin/dissolved-oxygen-prediction-in-river-water'
dataset = api.dataset_list_files(dataset_id)
file_name = dataset.files[0].name
file_path = os.path.join(api.get_default_download_dir(), file_name)
file_path
```

    Warning: Looks like you're using an outdated API Version, please consider updating (server 1.5.10 / client 1.5.9)





    '/content/train.csv'




```python
api.dataset_download_file(dataset_id, file_name, force=True, quiet=False)
```

    100%|██████████| 17.6k/17.6k [00:00<00:00, 7.13MB/s]

    Downloading train.csv to /content
    


    





    True



# データの読み込み

Padasを使ってダウンロードしてきたCSVファイルを読み込みます。


```python
data = pd.read_csv(file_path)
```


```python
data
```




<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>Id</th>
      <th>target</th>
      <th>O2_1</th>
      <th>O2_2</th>
      <th>O2_3</th>
      <th>O2_4</th>
      <th>O2_5</th>
      <th>O2_6</th>
      <th>O2_7</th>
      <th>NH4_1</th>
      <th>NH4_2</th>
      <th>NH4_3</th>
      <th>NH4_4</th>
      <th>NH4_5</th>
      <th>NH4_6</th>
      <th>NH4_7</th>
      <th>NO2_1</th>
      <th>NO2_2</th>
      <th>NO2_3</th>
      <th>NO2_4</th>
      <th>NO2_5</th>
      <th>NO2_6</th>
      <th>NO2_7</th>
      <th>NO3_1</th>
      <th>NO3_2</th>
      <th>NO3_3</th>
      <th>NO3_4</th>
      <th>NO3_5</th>
      <th>NO3_6</th>
      <th>NO3_7</th>
      <th>BOD5_1</th>
      <th>BOD5_2</th>
      <th>BOD5_3</th>
      <th>BOD5_4</th>
      <th>BOD5_5</th>
      <th>BOD5_6</th>
      <th>BOD5_7</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>0</td>
      <td>12.58</td>
      <td>9.875</td>
      <td>9.20</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>0.690</td>
      <td>1.040</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>0.0940</td>
      <td>0.0990</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>1.58</td>
      <td>1.825</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>4.80</td>
      <td>5.850</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaN</td>
    </tr>
    <tr>
      <th>1</th>
      <td>3</td>
      <td>9.37</td>
      <td>10.300</td>
      <td>10.75</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>0.710</td>
      <td>0.725</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>0.0585</td>
      <td>0.0515</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>1.21</td>
      <td>0.905</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>5.88</td>
      <td>6.835</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaN</td>
    </tr>
    <tr>
      <th>2</th>
      <td>4</td>
      <td>8.35</td>
      <td>8.290</td>
      <td>7.90</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>2.210</td>
      <td>2.210</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>0.1000</td>
      <td>0.1100</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>1.34</td>
      <td>1.250</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>3.20</td>
      <td>2.700</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaN</td>
    </tr>
    <tr>
      <th>3</th>
      <td>5</td>
      <td>9.57</td>
      <td>8.820</td>
      <td>6.80</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>0.595</td>
      <td>0.675</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>0.0460</td>
      <td>0.0535</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>0.59</td>
      <td>0.790</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>7.70</td>
      <td>7.055</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaN</td>
    </tr>
    <tr>
      <th>4</th>
      <td>6</td>
      <td>6.00</td>
      <td>6.000</td>
      <td>6.50</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>0.600</td>
      <td>0.900</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>0.1800</td>
      <td>0.3400</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>1.36</td>
      <td>1.820</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>5.50</td>
      <td>5.300</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaN</td>
    </tr>
    <tr>
      <th>...</th>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
    </tr>
    <tr>
      <th>142</th>
      <td>208</td>
      <td>6.80</td>
      <td>7.700</td>
      <td>7.50</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>0.380</td>
      <td>1.900</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>0.6200</td>
      <td>0.0640</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>2.80</td>
      <td>3.330</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>5.00</td>
      <td>5.800</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaN</td>
    </tr>
    <tr>
      <th>143</th>
      <td>211</td>
      <td>5.30</td>
      <td>6.300</td>
      <td>5.65</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>0.370</td>
      <td>0.500</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>0.6900</td>
      <td>0.9500</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>4.37</td>
      <td>3.160</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>8.00</td>
      <td>8.000</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaN</td>
    </tr>
    <tr>
      <th>144</th>
      <td>212</td>
      <td>8.60</td>
      <td>8.600</td>
      <td>11.00</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>2.400</td>
      <td>3.600</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>0.1500</td>
      <td>0.1400</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>0.53</td>
      <td>3.000</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>6.80</td>
      <td>7.200</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaN</td>
    </tr>
    <tr>
      <th>145</th>
      <td>213</td>
      <td>9.90</td>
      <td>9.600</td>
      <td>14.10</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>0.310</td>
      <td>0.500</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>0.2100</td>
      <td>0.0800</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>3.10</td>
      <td>3.500</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>5.20</td>
      <td>7.800</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaN</td>
    </tr>
    <tr>
      <th>146</th>
      <td>214</td>
      <td>6.50</td>
      <td>6.500</td>
      <td>7.70</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>0.190</td>
      <td>0.260</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>0.1300</td>
      <td>0.0720</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>0.00</td>
      <td>0.000</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>3.40</td>
      <td>4.100</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaN</td>
    </tr>
  </tbody>
</table>
<p>147 rows × 37 columns</p>
</div>



# 下準備


```python
data.info()
```

    <class 'pandas.core.frame.DataFrame'>
    RangeIndex: 147 entries, 0 to 146
    Data columns (total 37 columns):
     #   Column  Non-Null Count  Dtype  
    ---  ------  --------------  -----  
     0   Id      147 non-null    int64  
     1   target  147 non-null    float64
     2   O2_1    145 non-null    float64
     3   O2_2    145 non-null    float64
     4   O2_3    32 non-null     float64
     5   O2_4    31 non-null     float64
     6   O2_5    33 non-null     float64
     7   O2_6    37 non-null     float64
     8   O2_7    37 non-null     float64
     9   NH4_1   145 non-null    float64
     10  NH4_2   145 non-null    float64
     11  NH4_3   32 non-null     float64
     12  NH4_4   31 non-null     float64
     13  NH4_5   33 non-null     float64
     14  NH4_6   37 non-null     float64
     15  NH4_7   37 non-null     float64
     16  NO2_1   145 non-null    float64
     17  NO2_2   145 non-null    float64
     18  NO2_3   32 non-null     float64
     19  NO2_4   31 non-null     float64
     20  NO2_5   33 non-null     float64
     21  NO2_6   37 non-null     float64
     22  NO2_7   37 non-null     float64
     23  NO3_1   145 non-null    float64
     24  NO3_2   145 non-null    float64
     25  NO3_3   32 non-null     float64
     26  NO3_4   31 non-null     float64
     27  NO3_5   33 non-null     float64
     28  NO3_6   37 non-null     float64
     29  NO3_7   37 non-null     float64
     30  BOD5_1  145 non-null    float64
     31  BOD5_2  145 non-null    float64
     32  BOD5_3  32 non-null     float64
     33  BOD5_4  31 non-null     float64
     34  BOD5_5  33 non-null     float64
     35  BOD5_6  37 non-null     float64
     36  BOD5_7  37 non-null     float64
    dtypes: float64(36), int64(1)
    memory usage: 42.6 KB


## 欠損値の処理


```python
data.isna().sum()
```




    Id          0
    target      0
    O2_1        2
    O2_2        2
    O2_3      115
    O2_4      116
    O2_5      114
    O2_6      110
    O2_7      110
    NH4_1       2
    NH4_2       2
    NH4_3     115
    NH4_4     116
    NH4_5     114
    NH4_6     110
    NH4_7     110
    NO2_1       2
    NO2_2       2
    NO2_3     115
    NO2_4     116
    NO2_5     114
    NO2_6     110
    NO2_7     110
    NO3_1       2
    NO3_2       2
    NO3_3     115
    NO3_4     116
    NO3_5     114
    NO3_6     110
    NO3_7     110
    BOD5_1      2
    BOD5_2      2
    BOD5_3    115
    BOD5_4    116
    BOD5_5    114
    BOD5_6    110
    BOD5_7    110
    dtype: int64



欠損値が100以上ある列は削除します。


```python
null_columns = list(data.columns[data.isna().sum() > 100])
data = data.drop(null_columns, axis=1)
```


```python
data
```




<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>Id</th>
      <th>target</th>
      <th>O2_1</th>
      <th>O2_2</th>
      <th>NH4_1</th>
      <th>NH4_2</th>
      <th>NO2_1</th>
      <th>NO2_2</th>
      <th>NO3_1</th>
      <th>NO3_2</th>
      <th>BOD5_1</th>
      <th>BOD5_2</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>0</td>
      <td>12.58</td>
      <td>9.875</td>
      <td>9.20</td>
      <td>0.690</td>
      <td>1.040</td>
      <td>0.0940</td>
      <td>0.0990</td>
      <td>1.58</td>
      <td>1.825</td>
      <td>4.80</td>
      <td>5.850</td>
    </tr>
    <tr>
      <th>1</th>
      <td>3</td>
      <td>9.37</td>
      <td>10.300</td>
      <td>10.75</td>
      <td>0.710</td>
      <td>0.725</td>
      <td>0.0585</td>
      <td>0.0515</td>
      <td>1.21</td>
      <td>0.905</td>
      <td>5.88</td>
      <td>6.835</td>
    </tr>
    <tr>
      <th>2</th>
      <td>4</td>
      <td>8.35</td>
      <td>8.290</td>
      <td>7.90</td>
      <td>2.210</td>
      <td>2.210</td>
      <td>0.1000</td>
      <td>0.1100</td>
      <td>1.34</td>
      <td>1.250</td>
      <td>3.20</td>
      <td>2.700</td>
    </tr>
    <tr>
      <th>3</th>
      <td>5</td>
      <td>9.57</td>
      <td>8.820</td>
      <td>6.80</td>
      <td>0.595</td>
      <td>0.675</td>
      <td>0.0460</td>
      <td>0.0535</td>
      <td>0.59</td>
      <td>0.790</td>
      <td>7.70</td>
      <td>7.055</td>
    </tr>
    <tr>
      <th>4</th>
      <td>6</td>
      <td>6.00</td>
      <td>6.000</td>
      <td>6.50</td>
      <td>0.600</td>
      <td>0.900</td>
      <td>0.1800</td>
      <td>0.3400</td>
      <td>1.36</td>
      <td>1.820</td>
      <td>5.50</td>
      <td>5.300</td>
    </tr>
    <tr>
      <th>...</th>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
    </tr>
    <tr>
      <th>142</th>
      <td>208</td>
      <td>6.80</td>
      <td>7.700</td>
      <td>7.50</td>
      <td>0.380</td>
      <td>1.900</td>
      <td>0.6200</td>
      <td>0.0640</td>
      <td>2.80</td>
      <td>3.330</td>
      <td>5.00</td>
      <td>5.800</td>
    </tr>
    <tr>
      <th>143</th>
      <td>211</td>
      <td>5.30</td>
      <td>6.300</td>
      <td>5.65</td>
      <td>0.370</td>
      <td>0.500</td>
      <td>0.6900</td>
      <td>0.9500</td>
      <td>4.37</td>
      <td>3.160</td>
      <td>8.00</td>
      <td>8.000</td>
    </tr>
    <tr>
      <th>144</th>
      <td>212</td>
      <td>8.60</td>
      <td>8.600</td>
      <td>11.00</td>
      <td>2.400</td>
      <td>3.600</td>
      <td>0.1500</td>
      <td>0.1400</td>
      <td>0.53</td>
      <td>3.000</td>
      <td>6.80</td>
      <td>7.200</td>
    </tr>
    <tr>
      <th>145</th>
      <td>213</td>
      <td>9.90</td>
      <td>9.600</td>
      <td>14.10</td>
      <td>0.310</td>
      <td>0.500</td>
      <td>0.2100</td>
      <td>0.0800</td>
      <td>3.10</td>
      <td>3.500</td>
      <td>5.20</td>
      <td>7.800</td>
    </tr>
    <tr>
      <th>146</th>
      <td>214</td>
      <td>6.50</td>
      <td>6.500</td>
      <td>7.70</td>
      <td>0.190</td>
      <td>0.260</td>
      <td>0.1300</td>
      <td>0.0720</td>
      <td>0.00</td>
      <td>0.000</td>
      <td>3.40</td>
      <td>4.100</td>
    </tr>
  </tbody>
</table>
<p>147 rows × 12 columns</p>
</div>




```python
(data.isna().sum(axis=1) != 0).sum()
```




    3



欠損値を含む行は3行だけなので、削除します。


```python
data = data.dropna(axis=0)
```

## 不要な列の削除


```python
data = data.drop('Id', axis=1)
```

# X, Yデータの分割


```python
y = data['target']
X = data.drop('target', axis=1)
```

# スケーリング


```python
scaler = sp.StandardScaler()
X = scaler.fit_transform(X)
```

# トレーニング、テストデータの分割


```python
X_train, X_test, y_train, y_test = train_test_split(X, y, train_size=0.7)
```

# トレーニング


```python
model = slm.LinearRegression()
model.fit(X_train, y_train)
model_R2 = model.score(X_test, y_test)
```


```python
print('Model R^2 Score:', model_R2)
```

    Model R^2 Score: -0.6563668417383994


次はyが高いか低いかの2値の予測をしてみます。



```python
y
```




    0      12.58
    1       9.37
    2       8.35
    3       9.57
    4       6.00
           ...  
    142     6.80
    143     5.30
    144     8.60
    145     9.90
    146     6.50
    Name: target, Length: 144, dtype: float64




```python
y_new = pd.qcut(y, q=2, labels=[0,1])
```


```python
X_train, X_test, y_train, y_test = train_test_split(X, y_new, train_size=0.7)
```


```python
model = slm.LogisticRegression()
model.fit(X_train, y_train)
model_acc = model.score(X_test, y_test)
```


```python
print('Model Accuracy:', model_acc)
```

    Model Accuracy: 0.8181818181818182