統計を勉強するためのプログラム

# 1. 2つの変数間の直線的な関係性が大きいと，相関係数は大きくなる

```python
lst = [-1, -0.7, -0.3,  0, 0.3, 0.7, 1]
fig, ax = plt.subplots(1, len(lst), figsize=(10*len(lst), 10))
for idx, corrcoef in enumerate(lst):
    mean = np.array([0, 0])
    cov = np.array([[1, corrcoef], [corrcoef, 1]])
    x, y = np.random.multivariate_normal(mean, cov, 5000).T

    ax[idx].scatter(x, y, color='royalblue')
    ax[idx].set_title(f'corrcoef = {corrcoef:.2f}', size=50)
    ax[idx].tick_params(bottom=False, left=False, labelbottom=False, labelleft=False) 
```
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/463767/e8777662-2e13-45b9-72da-a4bb189531dc.png)

# 2. 直線的な関係になくても，相関係数が大きくなってしまう

```python
x = np.random.randint(1, 1000, 1000)
y = ((x-400)  ** 3 - 100 * (x-200) ** 2 + 100000000) / 1000000 
corr_coef = np.corrcoef(x, y)[0, 1] # 相関行列

fig, ax = plt.subplots()
ax.scatter(x,y,color='royalblue')
ax.set_title(f'corr={corr_coef:.3f}', size=18)
ax.tick_params(bottom=False, left=False, labelbottom=False, labelleft=False) 
```
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/463767/f65fc93b-dff9-2cb9-0566-1396190cb025.png)

# 3. 2次関数の関係にあるけど，直線的な関係性は薄いので，相関係数は低い．

```python
x = np.random.randint(1, 1000, 100)
y = (x - 500 ) ** 2 / 100 + 300
corr_coef = np.corrcoef(x, y)[0, 1]

fig, ax = plt.subplots()
ax.scatter(x,y,color='royalblue')
ax.set_title(f'corr={corr_coef:.3f}', size=18)
ax.tick_params(bottom=False, left=False, labelbottom=False, labelleft=False) 
```
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/463767/73c08e4d-b545-d413-dde7-f66007cd9c56.png)

# 4. データのトリミングを検討する必要がある例.
全体のデータを見る(左図)と，相関係数は低く，
`x = 900`以上のデータという範囲を絞って見る(右図)と，高い相関がある例．

```python
x = np.random.randint(900, 1000, 1000)
noise = np.random.randn(1000)
y = x + 10 * noise
corr_coef = np.corrcoef(x, y)[0, 1]

fig, ax = plt.subplots(1, 2, figsize=(10, 5))
ax[1].scatter(x,y,color='royalblue')
ax[1].set_title(f'corr={corr_coef:.3f}',size=18)
ax[1].tick_params(bottom=False, left=False, labelbottom=False, labelleft=False) 

x2 = list(x) + [600, 700, 800]
y2 = list(y) + [2000, 1800, 1500]
corr_coef = np.corrcoef(x2, y2)[0, 1]

ax[0].scatter(x2,y2,color='royalblue')
ax[0].set_title(f'corr={corr_coef:.3f}',size=18)
ax[0].tick_params(bottom=False, left=False, labelbottom=False, labelleft=False) 
```
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/463767/b628a42e-642b-9443-6979-657e7ee2edd6.png)

# 5. 内挿と外挿（適用範囲を検討しよう）の図
```python
import numpy as np
import matplotlib.pyplot as plt

x = np.array([2.0, 3.5, 4.0, 4.5,  5.0,  5.5])
y = np.array([3.0, 3.2, 3.9, 5.2, 8.4, 10.5])
xp = np.linspace(0, 8, 100)

for val in range(1, 2):
    fx = np.poly1d(np.polyfit(x, y, val))
    fig, ax = plt.subplots()
    ax.plot(xp, fx(xp), '-', color='blue')
    ax.scatter(x, y, color='deepskyblue', s=32)
    ax.text(0.05, 0.8, s=f'y = {fx.coef[0]:.2f} x {fx.coef[1]:.2f}',size='x-large', transform=ax.transAxes)
    ax.axhline([0], color='black')
    ax.set_xlim(0, None)
    ax.set_ylim(-3, 14)
    ax.set_ylabel('Cost [JPY]')
    ax.set_xlabel('Explanatory variables')
```
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/463767/8602ac0f-bd43-5018-3643-eaa2d2f8d18f.png)


```python
import numpy as np
import matplotlib.pyplot as plt

x = np.array([2.0, 3.5, 4.0, 4.5,  5.0,  5.5])
y = np.array([3.0, 3.2, 3.9, 5.2, 8.4, 10.5])
xp = np.linspace(2, 5.5, 100)
xp1 = np.linspace(0, 2, 100)
xp2 = np.linspace(5.5, 8, 100)

for val in range(1, 2):
    fx = np.poly1d(np.polyfit(x, y, val))
    fig, ax = plt.subplots()
    ax.plot(xp, fx(xp), '-', color='blue')
    ax.plot(xp1, fx(xp1), '-', color='red', linestyle='dashed')
    ax.plot(xp2, fx(xp2), '-', color='red', linestyle='dashed')
    ax.scatter(x, y, color='deepskyblue', s=32)
    ax.text(0.05, 0.8, s=f'y = {fx.coef[0]:.2f} x {fx.coef[1]:.2f}',size='x-large', transform=ax.transAxes)
    ax.axhline([0], color='black')
    ax.axvline([2], color='gray', linestyle='dotted')
    ax.axvline([5.5], color='gray', linestyle='dotted')
    ax.set_xlim(0, 8)
    ax.set_ylim(-3, 14)
    ax.set_ylabel('Cost [JPY]')
    ax.set_xlabel('Explanatory variables')
```
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/463767/71c0528a-57da-4152-81f8-095bde9a4a4c.png)

# 6. 過学習の図
次数を上げたり，説明変数の数を増やしていくと，
学習させたデータにはフィットしていくけど，未知のデータ予測の精度は低下してしまう．
図は，次数を上げていった場合．

```python
import numpy as np
import matplotlib.pyplot as plt

x = np.array([2.0, 3.5, 4.0, 4.5,  5.0,  5.5])
y = np.array([3.0, 3.2, 3.9, 5.2, 8.4, 10.5])
xp = np.linspace(0, 8, 100)

for val in range(2, 6):
    fx = np.poly1d(np.polyfit(x, y, val))
    fig, ax = plt.subplots()
    ax.plot(xp, fx(xp), '-', color='blue')
    ax.scatter(x, y, color='deepskyblue', s=32)
    ax.axhline([0], color='black')
    ax.set_xlim(0, None)
    ax.set_ylim(-3, 14)
    ax.set_ylabel('Cost [JPY]')
    ax.set_xlabel('Explanatory variables')
    ax.text(0.75, 0.85, s=f'Digree = {val}',size='x-large', transform=ax.transAxes)
```
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/463767/820702d9-bd47-b3a2-55ae-7d6950d981da.png)
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/463767/0f4ec088-f51b-84c5-453e-28fe502da5c6.png)
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/463767/cf1d0063-6ec0-a1fe-f584-71584042ab3b.png)
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/463767/8f4f11da-1f68-2158-6ed5-b83d63c0cb38.png)