Numpy Pandas チートシート

# Numpy
## 条件指定で抽出
```python
# 配列生成
>>> array1 = np.array([16, 10, 3])
>>> array1.shape
(3,)

# boolリストで条件指定できる
>>> array1[[True, False, True]]
array([16,  3])

# boolリストが返る
>>> print(array1 > 5)
[ True  True False]

# 組み合わせるとこう使える
>>> array1[array1 > 5]
array([16, 10])
```

## bincount
配列の各値の出現回数をカウント

```py
>>> a1 = np.array([0, 1, 1, 0, 1, 0, 0, 0])

>>> np.bincount(a1)
array([5, 3])
```

## arange, linspace
numpy配列作成

```python
>>> np.arange(5)
array([0, 1, 2, 3, 4])

>>> np.linspace(-3, 3, 5)
array([-3. , -1.5,  0. ,  1.5,  3. ])
```

## reshape
配列の次元変換

```python
>>> line = np.linspace(-3, 3, 10)

# 1次元配列
>>> line
array([-3.        , -2.33333333, -1.66666667, -1.        , -0.33333333,
        0.33333333,  1.        ,  1.66666667,  2.33333333,  3.        ])

# 2次元配列に変換（10行1列）
>>> line.reshape(10, 1)
array([[-3.        ],
       [-2.33333333],
       [-1.66666667],
       [-1.        ],
       [-0.33333333],
       [ 0.33333333],
       [ 1.        ],
       [ 1.66666667],
       [ 2.33333333],
       [ 3.        ]])

# どちらかは-1を指定でき、もう片方の次元から推測して変換される
>>> line.reshape(-1, 1)
array([[-3.        ],
       [-2.33333333],
       [-1.66666667],
       [-1.        ],
       [-0.33333333],
       [ 0.33333333],
       [ 1.        ],
       [ 1.66666667],
       [ 2.33333333],
       [ 3.        ]])
```
## Numpy配列の演算
```python
>>> x = np.array([1, 2, 3])
>>> x.mean()
2.0

>>> xc = x - x.mean()
>>> xc
array([-1.,  0.,  1.])
```

## axis指定
https://deepage.net/features/numpy-axis.html

- axis=0: 層
- axis=1: 行
- axis=2: 列

```python
>>> b
array([[[0, 1],
        [2, 3],
        [4, 5]],

       [[0, 1],
        [2, 3],
        [4, 5]]])
>>> b.shape
(2, 3, 2)

>>> b.sum(axis=0)
array([[ 0,  2],
       [ 4,  6],
       [ 8, 10]])

>>> b.sum(axis=1)
array([[6, 9],
       [6, 9]])

>>> b.sum(axis=2)
array([[1, 5, 9],
       [1, 5, 9]])
```

# Pandas
## csv読み込み
### 読み込みと表示
```python
>>> df = pd.read_csv('housing.csv')
>>> df.head()
        x1    x2    x3  x4     x5     x6    x7      x8  x9  x10   x11     x12   x13     y
0  0.00632  18.0  2.31   0  0.538  6.575  65.2  4.0900   1  296  15.3  396.90  4.98  24.0
1  0.02731   0.0  7.07   0  0.469  6.421  78.9  4.9671   2  242  17.8  396.90  9.14  21.6
2  0.02729   0.0  7.07   0  0.469  7.185  61.1  4.9671   2  242  17.8  392.83  4.03  34.7
3  0.03237   0.0  2.18   0  0.458  6.998  45.8  6.0622   3  222  18.7  394.63  2.94  33.4
4  0.06905   0.0  2.18   0  0.458  7.147  54.2  6.0622   3  222  18.7  396.90  5.33  36.2
>>> len(df)
506
>>> df.describe()
               x1          x2          x3          x4          x5     ...             x10         x11         x12         x13           y
count  506.000000  506.000000  506.000000  506.000000  506.000000     ...      506.000000  506.000000  506.000000  506.000000  506.000000
mean     3.613524   11.363636   11.136779    0.069170    0.554695     ...      408.237154   18.455534  356.674032   12.653063   22.532806
std      8.601545   23.322453    6.860353    0.253994    0.115878     ...      168.537116    2.164946   91.294864    7.141062    9.197104
min      0.006320    0.000000    0.460000    0.000000    0.385000     ...      187.000000   12.600000    0.320000    1.730000    5.000000
25%      0.082045    0.000000    5.190000    0.000000    0.449000     ...      279.000000   17.400000  375.377500    6.950000   17.025000
50%      0.256510    0.000000    9.690000    0.000000    0.538000     ...      330.000000   19.050000  391.440000   11.360000   21.200000
75%      3.677082   12.500000   18.100000    0.000000    0.624000     ...      666.000000   20.200000  396.225000   16.955000   25.000000
max     88.976200  100.000000   27.740000    1.000000    0.871000     ...      711.000000   22.000000  396.900000   37.970000   50.000000
>>> df.shape
(506, 14)
>>> df.info()
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 506 entries, 0 to 505
Data columns (total 14 columns):
x1     506 non-null float64
x2     506 non-null float64
x3     506 non-null float64
x4     506 non-null int64
x5     506 non-null float64
x6     506 non-null float64
x7     506 non-null float64
x8     506 non-null float64
x9     506 non-null int64
x10    506 non-null int64
x11    506 non-null float64
x12    506 non-null float64
x13    506 non-null float64
y      506 non-null float64
dtypes: float64(11), int64(3)
memory usage: 55.4 KB
```

## 情報の取り出し
### 列の取り出し
- 1次元指定でSeriesとして取得

```bash
>>> type(df['x1'])
<class 'pandas.core.series.Series'>

>>> df['x1']
0       0.00632
1       0.02731
         ...   
504     0.10959
505     0.04741
Name: x1, Length: 506, dtype: float64
```

- 2次元指定でDataFrameとして取得

```bash
>>> type(df[['x1']])
<class 'pandas.core.frame.DataFrame'>

>>> df[['x1']]
           x1
0     0.00632
1     0.02731
..        ...
504   0.10959
505   0.04741

[506 rows x 1 columns]

# 複数列
>>> df[['x1', 'x2']]
           x1    x2
0     0.00632  18.0
1     0.02731   0.0
..        ...   ...
504   0.10959   0.0
505   0.04741   0.0

[506 rows x 2 columns]
```

### 行の取り出し
```python
>>> df[500:]
          x1   x2     x3  x4     x5     x6    x7      x8  x9  x10   x11     x12    x13     y
500  0.22438  0.0   9.69   0  0.585  6.027  79.7  2.4982   6  391  19.2  396.90  14.33  16.8
501  0.06263  0.0  11.93   0  0.573  6.593  69.1  2.4786   1  273  21.0  391.99   9.67  22.4
502  0.04527  0.0  11.93   0  0.573  6.120  76.7  2.2875   1  273  21.0  396.90   9.08  20.6
503  0.06076  0.0  11.93   0  0.573  6.976  91.0  2.1675   1  273  21.0  396.90   5.64  23.9
504  0.10959  0.0  11.93   0  0.573  6.794  89.3  2.3889   1  273  21.0  393.45   6.48  22.0
505  0.04741  0.0  11.93   0  0.573  6.030  80.8  2.5050   1  273  21.0  396.90   7.88  11.9

>>> df[:5]
        x1    x2    x3  x4     x5     x6    x7      x8  x9  x10   x11     x12   x13     y
0  0.00632  18.0  2.31   0  0.538  6.575  65.2  4.0900   1  296  15.3  396.90  4.98  24.0
1  0.02731   0.0  7.07   0  0.469  6.421  78.9  4.9671   2  242  17.8  396.90  9.14  21.6
2  0.02729   0.0  7.07   0  0.469  7.185  61.1  4.9671   2  242  17.8  392.83  4.03  34.7
3  0.03237   0.0  2.18   0  0.458  6.998  45.8  6.0622   3  222  18.7  394.63  2.94  33.4
4  0.06905   0.0  2.18   0  0.458  7.147  54.2  6.0622   3  222  18.7  396.90  5.33  36.2

>>> df[100:150:10]
          x1   x2     x3  x4     x5     x6    x7      x8  x9  x10   x11     x12    x13     y
100  0.14866  0.0   8.56   0  0.520  6.727  79.9  2.7778   5  384  20.9  394.76   9.42  27.5
110  0.10793  0.0   8.56   0  0.520  6.195  54.4  2.7778   5  384  20.9  393.49  13.00  21.7
120  0.06899  0.0  25.65   0  0.581  5.870  69.7  2.2577   2  188  19.1  389.15  14.37  22.0
130  0.34006  0.0  21.89   0  0.624  6.458  98.9  2.1185   4  437  21.2  395.04  12.60  19.2
140  0.29090  0.0  21.89   0  0.624  6.174  93.6  1.6119   4  437  21.2  388.08  24.16  14.0
```


### 行と列の取り出し
#### ラベル指定(loc)
```python
>>> df.loc[10:15, ['x1', 'x2']]
         x1    x2
10  0.22489  12.5
11  0.11747  12.5
12  0.09378  12.5
13  0.62976   0.0
14  0.63796   0.0
15  0.62739   0.0
```

#### インデックス指定(iloc)
```python
>>> df.iloc[10:15, 0:2]
         x1    x2
10  0.22489  12.5
11  0.11747  12.5
12  0.09378  12.5
13  0.62976   0.0
14  0.63796   0.0

>>> df.iloc[:, :-1]  # 全行と最後の列以外
           x1    x2     x3  x4     x5     x6     x7      x8  x9  x10   x11     x12    x13
0     0.00632  18.0   2.31   0  0.538  6.575   65.2  4.0900   1  296  15.3  396.90   4.98
1     0.02731   0.0   7.07   0  0.469  6.421   78.9  4.9671   2  242  17.8  396.90   9.14
2     0.02729   0.0   7.07   0  0.469  7.185   61.1  4.9671   2  242  17.8  392.83   4.03
..        ...   ...    ...  ..    ...    ...    ...     ...  ..  ...   ...     ...    ...
504   0.10959   0.0  11.93   0  0.573  6.794   89.3  2.3889   1  273  21.0  393.45   6.48
505   0.04741   0.0  11.93   0  0.573  6.030   80.8  2.5050   1  273  21.0  396.90   7.88

[506 rows x 13 columns]
```

### 条件指定で抽出
#### 基本はNumpyと同じ

```python
>>> df['x1'] > 50
0      False
1      False
       ...  
504    False
505    False
Name: x1, Length: 506, dtype: bool

>>> df[df['x1'] > 50]
          x1   x2    x3  x4     x5     x6     x7      x8  x9  x10   x11     x12    x13     y
380  88.9762  0.0  18.1   0  0.671  6.968   91.9  1.4165  24  666  20.2  396.90  17.21  10.4
405  67.9208  0.0  18.1   0  0.693  5.683  100.0  1.4254  24  666  20.2  384.97  22.98   5.0
410  51.1358  0.0  18.1   0  0.597  5.757  100.0  1.4130  24  666  20.2    2.60  10.11  15.0
418  73.5341  0.0  18.1   0  0.679  5.957  100.0  1.8026  24  666  20.2   16.45  20.62   8.8

>>> df[(df['x1'] > 50) & (df['x6'] > 6)]
          x1   x2    x3  x4     x5     x6    x7      x8  x9  x10   x11    x12    x13     y
380  88.9762  0.0  18.1   0  0.671  6.968  91.9  1.4165  24  666  20.2  396.9  17.21  10.4
```

#### ただし、条件にはSeries(1次元)で渡さないとダメ
DataFrame(2次元)で渡すと変になる
##### SeriesとDataFrameの違い

```bash
>>> df.head(3)
         date home_team away_team  home_score  away_score tournament     city   country  neutral
0  1872-11-30  Scotland   England           0           0   Friendly  Glasgow  Scotland    False
1  1873-03-08   England  Scotland           4           2   Friendly   London   England    False
2  1874-03-07  Scotland   England           2           1   Friendly  Glasgow  Scotland    False

# 1次元指定だとSeries, 2次元指定だとDataFrameが返る
>>> type(df['home_team'])
<class 'pandas.core.series.Series'>

>>> type(df[['home_team']])
<class 'pandas.core.frame.DataFrame'>

>>> type(df[['home_team', 'away_team']])
<class 'pandas.core.frame.DataFrame'>


>>> type(df['home_team'] == 'England')
<class 'pandas.core.series.Series'>

>>> type(df[['home_team']] == 'England')
<class 'pandas.core.frame.DataFrame'>

>>> type(df[['home_team', 'away_team']] == 'England')
<class 'pandas.core.frame.DataFrame'>


# 表示の違い
# Series
>>> df['home_team'] == 'England'
0        False
1         True
2        False
         ...  
39005    False
39006    False
39007    False
Name: home_team, Length: 39008, dtype: bool

# DataFrame
>>> df[['home_team']] == 'England'
       home_team
0          False
1           True
2          False
...          ...
39005      False
39006      False
39007      False

[39008 rows x 1 columns]

# DataFrame
>>> df[['home_team', 'away_team']] == 'England'
       home_team  away_team
0          False       True
1           True      False
2          False       True
...          ...        ...
39005      False      False
39006      False      False
39007      False      False

[39008 rows x 2 columns]
```

##### 条件に指定してみる
```bash
# Series（いい感じ）
>>> df[df['home_team'] == 'England']
             date home_team         away_team  home_score  away_score                    tournament            city  country  neutral
1      1873-03-08   England          Scotland           4           2                      Friendly          London  England    False
3      1875-03-06   England          Scotland           2           2                      Friendly          London  England    False
...           ...       ...               ...         ...         ...                           ...             ...      ...      ...
38881  2018-03-27   England             Italy           1           1                      Friendly          London  England    False
38981  2018-06-02   England           Nigeria           2           1                      Friendly          London  England    False

[480 rows x 9 columns]

# DataFrame（行が抽出できないし、一致するところ以外がNaNになってしまう）
>>> df[df[['home_team']] == 'England']
      date home_team away_team  home_score  away_score tournament city country  neutral
0      NaN       NaN       NaN         NaN         NaN        NaN  NaN     NaN      NaN
1      NaN   England       NaN         NaN         NaN        NaN  NaN     NaN      NaN
2      NaN       NaN       NaN         NaN         NaN        NaN  NaN     NaN      NaN
...    ...       ...       ...         ...         ...        ...  ...     ...      ...
39006  NaN       NaN       NaN         NaN         NaN        NaN  NaN     NaN      NaN
39007  NaN       NaN       NaN         NaN         NaN        NaN  NaN     NaN      NaN

[39008 rows x 9 columns]
```

#### any
- DataFrameに対して、どちらかがTrueの場合に抽出というように使える
- anyを使うことでDataFrameがSeriesに変換される

```bash
# anyなし
>>> type(df[['home_team', 'away_team']] == 'England')
<class 'pandas.core.frame.DataFrame'>

>>> df[['home_team', 'away_team']] == 'England'
       home_team  away_team
0          False       True
1           True      False
2          False       True
...          ...        ...
39005      False      False
39006      False      False
39007      False      False

[39008 rows x 2 columns]

# anyあり
>>> type((df[['home_team', 'away_team']] == 'England').any(axis=1))
<class 'pandas.core.series.Series'>

>>> (df[['home_team', 'away_team']] == 'England').any(axis=1)
0         True
1         True
2         True
         ...  
39005    False
39006    False
39007    False
Length: 39008, dtype: bool

# anyで条件抽出
>>> df[(df[['home_team', 'away_team']] == 'England').any(axis=1)]
             date         home_team         away_team  home_score  away_score                    tournament           city      country  neutral
0      1872-11-30          Scotland           England           0           0                      Friendly        Glasgow     Scotland    False
1      1873-03-08           England          Scotland           4           2                      Friendly         London      England    False
...           ...               ...               ...         ...         ...                           ...            ...          ...      ...
38881  2018-03-27           England             Italy           1           1                      Friendly         London      England    False
38981  2018-06-02           England           Nigeria           2           1                      Friendly         London      England    False

[976 rows x 9 columns]

```

#### クエリ検索
```bash
>>> df.query("home_team == 'Japan' | away_team == 'Japan'")
             date             home_team             away_team   ...                 city               country neutral
443    1917-05-07                 Japan           Philippines   ...                Tokyo                 Japan   False
571    1921-05-30                 Japan           Philippines   ...             Shanghai                 China    True
...           ...                   ...                   ...   ...                  ...                   ...     ...
38916  2018-03-27               Ukraine                 Japan   ...                Liège               Belgium    True
38957  2018-05-30                 Japan                 Ghana   ...             Yokohama                 Japan   False

[597 rows x 9 columns]
```

## 基本的な値を取得
### 平均(mean)
```python
>>> df.mean()
x1       3.613524
x2      11.363636
x3      11.136779
x4       0.069170
x5       0.554695
x6       6.284634
x7      68.574901
x8       3.795043
x9       9.549407
x10    408.237154
x11     18.455534
x12    356.674032
x13     12.653063
y       22.532806
dtype: float64

>>> df[['x1', 'x2']].mean()
x1     3.613524
x2    11.363636
dtype: float64
```

### 標準偏差(std)
```python
>>> df.std()
x1       8.601545
x2      23.322453
x3       6.860353
x4       0.253994
x5       0.115878
x6       0.702617
x7      28.148861
x8       2.105710
x9       8.707259
x10    168.537116
x11      2.164946
x12     91.294864
x13      7.141062
y        9.197104
dtype: float64
```

### 最大値(max)
```python
>>> df
    a   b   c   d
0  11  12  13  14
1  21  22  23  24
2  31  32  33  34

>>> df.max()
a    31
b    32
c    33
d    34
dtype: int64

>>> df.max(axis=1)
0    14
1    24
2    34
dtype: int64
```

## データフレームの列同士の演算
```python
>>> df.head(3)
        x1    x2    x3  x4     x5     x6    x7      x8  x9  x10   x11     x12   x13     y
0  0.00632  18.0  2.31   0  0.538  6.575  65.2  4.0900   1  296  15.3  396.90  4.98  24.0
1  0.02731   0.0  7.07   0  0.469  6.421  78.9  4.9671   2  242  17.8  396.90  9.14  21.6
2  0.02729   0.0  7.07   0  0.469  7.185  61.1  4.9671   2  242  17.8  392.83  4.03  34.7

>>> df.mean()
x1       3.613524
x2      11.363636
x3      11.136779
x4       0.069170
x5       0.554695
x6       6.284634
x7      68.574901
x8       3.795043
x9       9.549407
x10    408.237154
x11     18.455534
x12    356.674032
x13     12.653063
y       22.532806

>>> df_c = df - df.mean()
>>> df_c.head(3)
         x1         x2        x3       x4        x5        x6         x7        x8        x9         x10       x11        x12       x13          y
0 -3.607204   6.636364 -8.826779 -0.06917 -0.016695  0.290366  -3.374901  0.294957 -8.549407 -112.237154 -3.155534  40.225968 -7.673063   1.467194
1 -3.586214 -11.363636 -4.066779 -0.06917 -0.085695  0.136366  10.325099  1.172057 -7.549407 -166.237154 -0.655534  40.225968 -3.513063  -0.932806
2 -3.586234 -11.363636 -4.066779 -0.06917 -0.085695  0.900366  -7.474901  1.172057 -7.549407 -166.237154 -0.655534  36.155968 -8.623063  12.167194
```

## 新しい列の追加
```bash
# 追加前
>>> df.head()
         date home_team away_team  home_score  away_score tournament     city   country  neutral
0  1872-11-30  Scotland   England           0           0   Friendly  Glasgow  Scotland    False
1  1873-03-08   England  Scotland           4           2   Friendly   London   England    False
2  1874-03-07  Scotland   England           2           1   Friendly  Glasgow  Scotland    False
3  1875-03-06   England  Scotland           2           2   Friendly   London   England    False
4  1876-03-04  Scotland   England           3           0   Friendly  Glasgow  Scotland    False

# 列の追加
>>> df['year'] = pd.to_numeric([date.split('-')[0] for date in df['date']])
>>> df['month'] = pd.to_numeric([date.split('-')[1] for date in df['date']])
>>> df['day'] = pd.to_numeric([date.split('-')[2] for date in df['date']])

# 確認
>>> df.head()
         date home_team away_team  home_score  away_score tournament     city   country  neutral  year  month  day
0  1872-11-30  Scotland   England           0           0   Friendly  Glasgow  Scotland    False  1872     11   30
1  1873-03-08   England  Scotland           4           2   Friendly   London   England    False  1873      3    8
2  1874-03-07  Scotland   England           2           1   Friendly  Glasgow  Scotland    False  1874      3    7
3  1875-03-06   England  Scotland           2           2   Friendly   London   England    False  1875      3    6
4  1876-03-04  Scotland   England           3           0   Friendly  Glasgow  Scotland    False  1876      3    4
```

## 並び替え
```bash
>>> df.sort_values('home_score', ascending = False).head()
             date  home_team       away_team  home_score  away_score                    tournament           city    country  neutral  year  month  day
23569  2001-04-11  Australia  American Samoa          31           0  FIFA World Cup qualification  Coffs Harbour  Australia    False  2001      4   11
10860  1979-08-30       Fiji        Kiribati          24           0           South Pacific Games        Nausori       Fiji    False  1979      8   30
23566  2001-04-09  Australia           Tonga          22           0  FIFA World Cup qualification  Coffs Harbour  Australia    False  2001      4    9
22344  2000-02-14     Kuwait          Bhutan          20           0   AFC Asian Cup qualification    Kuwait City     Kuwait    False  2000      2   14
22257  2000-01-26      China            Guam          19           0   AFC Asian Cup qualification          Hanoi    Vietnam     True  2000      1   26
```

## 置き換え

```bash
# Trueを1に置き換え
>>> df.replace({True: 1})
             date         home_team         away_team  home_score  away_score ...        country neutral  year month  day
0      1872-11-30          Scotland           England           0           0 ...       Scotland   False  1872    11   30
1      1873-03-08           England          Scotland           4           2 ...        England   False  1873     3    8
...           ...               ...               ...         ...         ... ...            ...     ...   ...   ...  ...
39006  2018-06-04           Armenia           Moldova           0           0 ...        Austria       1  2018     6    4
39007  2018-06-04             India             Kenya           3           0 ...          India   False  2018     6    4

[39008 rows x 12 columns]

```

## 欠損値処理
### 欠損値がひとつでも含まれていたら行を削除(dropna)
```bash
>>> df.dropna()
           x1    x2     x3  x4     x5     x6     x7      x8  x9  x10   x11     x12    x13     y
0     0.00632  18.0   2.31   0  0.538  6.575   65.2  4.0900   1  296  15.3  396.90   4.98  24.0


[506 rows x 14 columns]
```

### 欠損値を埋める(fillna)
```bash
>>> df.fillna(0)
           x1    x2     x3  x4     x5     x6     x7      x8  x9  x10   x11     x12    x13     y
0     0.00632  18.0   2.31   0  0.538  6.575   65.2  4.0900   1  296  15.3  396.90   4.98  24.0

[506 rows x 14 columns]
```

### 欠損値のある行を抽出(isnull)
```bash
>>> df[df['x1'].isnull()]
Empty DataFrame
Columns: [x1, x2, x3, x4, x5, x6, x7, x8, x9, x10, x11, x12, x13, y]
```

## 要素、行、列に関数を適用
### Seriesの各要素に適用(map)
```bash
# DataFrame
>>> df
    a   b   c   d
0  11  12  13  14
1  21  22  23  24
2  31  32  33  34
>>> type(df)
<class 'pandas.core.frame.DataFrame'>

# Series
>>> df['a']
0    11
1    21
2    31
Name: a, dtype: int64
>>> type(df['a'])
<class 'pandas.core.series.Series'>

# mapでSeriesの各要素に関数を適用
>>> df['a'].map(lambda x: '[{}]'.format(x))
0    [11]
1    [21]
2    [31]
Name: a, dtype: object
```

mapに辞書を渡すこともできる(replaceみたいな使い方)

```bash
>>> df['a']
0    11
1    21
2    31
Name: a, dtype: int64

>>> df['a'].map({11:'one',21:'two'})
0    one
1    two
2    NaN
Name: a, dtype: object

```

### DataFrameの各行・各列に適用(apply)
```bash
# DataFrame
>>> df
    a   b   c   d
0  11  12  13  14
1  21  22  23  24
2  31  32  33  34
>>> type(df)
<class 'pandas.core.frame.DataFrame'>

# applyでDataFrameに関数を適用
>>> df.apply(lambda x: max(x))
a    31
b    32
c    33
d    34
dtype: int64

# 結果はSeriesに変換される
>>> type(df.apply(lambda x: max(x)))
<class 'pandas.core.series.Series'>
```