PyTorchをいちから使い始めてみる (torch 1.5.0 CPUのみ)

ChainerからPyTorchへ乗り換えるため、PyTorchを完全にいちから使い始めてみた結果。MNISTの手書き数字の識別を行う学習器をとりあえず作成(NNモデルはConv4層→FC2層)、動作を確認、98%前後の精度(Accuracy)となり、良好。

## PyTorchのインストール試行

成功の模様。

https://pytorch.org/get-started/locally/  
1.5 Stable、Windows、Python3.7、CUDAなし(CPU)の設定のコマンドを実行。


```python
!pip install torch==1.5.0+cpu torchvision==0.6.0+cpu -f https://download.pytorch.org/whl/torch_stable.html
```

    Looking in links: https://download.pytorch.org/whl/torch_stable.html
    Collecting torch==1.5.0+cpu
      Downloading https://download.pytorch.org/whl/cpu/torch-1.5.0%2Bcpu-cp37-cp37m-win_amd64.whl (109.2MB)
    Collecting torchvision==0.6.0+cpu
      Downloading https://download.pytorch.org/whl/cpu/torchvision-0.6.0%2Bcpu-cp37-cp37m-win_amd64.whl (456kB)
    Requirement already satisfied: numpy in ...\wpy64-3741\python-3.7.4.amd64\lib\site-packages (from torch==1.5.0+cpu) (1.16.5+mkl)
    Requirement already satisfied: future in ...\wpy64-3741\python-3.7.4.amd64\lib\site-packages (from torch==1.5.0+cpu) (0.17.0)
    Requirement already satisfied: pillow>=4.1.1 in ...\wpy64-3741\python-3.7.4.amd64\lib\site-packages (from torchvision==0.6.0+cpu) (6.1.0)
    Installing collected packages: torch, torchvision
      Found existing installation: torch 1.3.1+cpu
        Uninstalling torch-1.3.1+cpu:
          Successfully uninstalled torch-1.3.1+cpu
      Found existing installation: torchvision 0.4.2+cpu
        Uninstalling torchvision-0.4.2+cpu:
          Successfully uninstalled torchvision-0.4.2+cpu
    Successfully installed torch-1.5.0+cpu torchvision-0.6.0+cpu
    Note: you may need to restart the kernel to use updated packages.
    

    WARNING: You are using pip version 19.2.3, however version 20.1 is available.
    You should consider upgrading via the 'python -m pip install --upgrade pip' command.
    


```python
!pip show torch
```

    Name: torch
    Version: 1.5.0+cpu
    Summary: Tensors and Dynamic neural networks in Python with strong GPU acceleration
    Home-page: https://pytorch.org/
    Author: PyTorch Team
    Author-email: packages@pytorch.org
    License: BSD-3
    Location: ...\wpy64-3741\python-3.7.4.amd64\lib\site-packages
    Requires: future, numpy
    Required-by: torchvision
    

## Libs


```python
import numpy as np
import matplotlib.pyplot as plt

import torch
import torchvision
```

## PyTorch初歩試用

参考:

* https://qiita.com/miyamotok0105/items/1fd1d5c3532b174720cd  
* https://qiita.com/north_redwing/items/30f9619f0ee727875250  
* https://www.hellocybernetics.tech/entry/2017/10/19/070522


```python
# sec: 行列生成のテスト

torch.rand(5, 3)
```




    tensor([[0.7130, 0.1860, 0.6266],
            [0.9275, 0.5629, 0.7507],
            [0.7500, 0.7341, 0.9597],
            [0.5301, 0.7026, 0.3152],
            [0.2197, 0.3942, 0.8452]])




```python
# sec: 勾配計算のテスト

x = torch.arange(4, dtype=torch.float32).view(2, 2)
print(x)

x.requires_grad_(True)
print(x)

y = x**2 - 2*x + 1
print(y)

y_ave = y.mean()
print(y_ave)

y_ave.backward()
print(x.grad)
print(y.grad)
```

    tensor([[0., 1.],
            [2., 3.]])
    tensor([[0., 1.],
            [2., 3.]], requires_grad=True)
    tensor([[1., 0.],
            [1., 4.]], grad_fn=<AddBackward0>)
    tensor(1.5000, grad_fn=<MeanBackward0>)
    tensor([[-0.5000,  0.0000],
            [ 0.5000,  1.0000]])
    None
    

## torchvision初歩試用


```python
# sec: MNISTデータの初回呼び出し ダウンロードのテスト

ds = torchvision.datasets.MNIST(root="trains/pytorch-mnist", train=True, download=True)
ds
```

    Downloading http://yann.lecun.com/exdb/mnist/train-images-idx3-ubyte.gz to trains/pytorch-mnist\MNIST\raw\train-images-idx3-ubyte.gz
    
    HBox(children=(IntProgress(value=1, bar_style='info', max=1), HTML(value='')))

    Extracting trains/pytorch-mnist\MNIST\raw\train-images-idx3-ubyte.gz to trains/pytorch-mnist\MNIST\raw
    Downloading http://yann.lecun.com/exdb/mnist/train-labels-idx1-ubyte.gz to trains/pytorch-mnist\MNIST\raw\train-labels-idx1-ubyte.gz
    
    HBox(children=(IntProgress(value=1, bar_style='info', max=1), HTML(value='')))

    Extracting trains/pytorch-mnist\MNIST\raw\train-labels-idx1-ubyte.gz to trains/pytorch-mnist\MNIST\raw
    Downloading http://yann.lecun.com/exdb/mnist/t10k-images-idx3-ubyte.gz to trains/pytorch-mnist\MNIST\raw\t10k-images-idx3-ubyte.gz

    HBox(children=(IntProgress(value=1, bar_style='info', max=1), HTML(value='')))

    Extracting trains/pytorch-mnist\MNIST\raw\t10k-images-idx3-ubyte.gz to trains/pytorch-mnist\MNIST\raw
    Downloading http://yann.lecun.com/exdb/mnist/t10k-labels-idx1-ubyte.gz to trains/pytorch-mnist\MNIST\raw\t10k-labels-idx1-ubyte.gz
    
    HBox(children=(IntProgress(value=1, bar_style='info', max=1), HTML(value='')))

    Extracting trains/pytorch-mnist\MNIST\raw\t10k-labels-idx1-ubyte.gz to trains/pytorch-mnist\MNIST\raw
    Processing...

    ..\torch\csrc\utils\tensor_numpy.cpp:141: UserWarning: The given NumPy array is not writeable, and PyTorch does not support non-writeable tensors. This means you can write to the underlying (supposedly non-writeable) NumPy array using the tensor. You may want to copy the array to protect its data or make it writeable before converting it to a tensor. This type of warning will be suppressed for the rest of this program.

    Done!

    Dataset MNIST
        Number of datapoints: 60000
        Root location: trains/pytorch-mnist
        Split: Train




```python
# sec: MNISTデータの2回目呼び出し

ds = torchvision.datasets.MNIST(root="trains/pytorch-mnist", train=True, download=True)
ds
```




    Dataset MNIST
        Number of datapoints: 60000
        Root location: trains/pytorch-mnist
        Split: Train




```python
# sec: MNISTデータのデータ取り出し・表示

img = np.asarray(ds[0][0])
print(img[20:25, 10:15], img.shape)

img = np.asarray(ds[0][0].convert('F'))
print(img[20:25, 10:15], img.shape)

plt.imshow(img, cmap='gray', interpolation='None')
plt.show()
```

    [[ 24 114 221 253 253]
     [213 253 253 253 253]
     [253 253 253 195  80]
     [253 244 133  11   0]
     [132  16   0   0   0]] (28, 28)
    [[ 24. 114. 221. 253. 253.]
     [213. 253. 253. 253. 253.]
     [253. 253. 253. 195.  80.]
     [253. 244. 133.  11.   0.]
     [132.  16.   0.   0.   0.]] (28, 28)
    


![output_11_1.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/606924/65e13058-89de-c4dd-0ba4-df9af9e967ba.png)



```python
# sec: TorchのTensorを出力する

ds = torchvision.datasets.MNIST(root="trains/pytorch-mnist", train=True, download=True, 
    transform=torchvision.transforms.ToTensor())

# sec: MNISTデータのデータ取り出し・表示

img = ds[0][0]
print(img[0, 20:25, 10:15], ds[0][0].shape)

plt.imshow(img[0, :, :], cmap='gray', interpolation='None')
plt.show()
```

    tensor([[0.0941, 0.4471, 0.8667, 0.9922, 0.9922],
            [0.8353, 0.9922, 0.9922, 0.9922, 0.9922],
            [0.9922, 0.9922, 0.9922, 0.7647, 0.3137],
            [0.9922, 0.9569, 0.5216, 0.0431, 0.0000],
            [0.5176, 0.0627, 0.0000, 0.0000, 0.0000]]) torch.Size([1, 28, 28])
    


![output_12_1.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/606924/34749886-c95f-3c9b-0657-81b167227329.png)


## NNモデル


```python
import torch.nn as nn
import torch.nn.functional as F

class MyConvNet1(nn.Module):
    
    def __init__(self):
        super().__init__()
        self.conv1 = nn.Conv2d(1, 5, 3, stride=1, padding=0)
        self.conv2 = nn.Conv2d(5, 10, 3, stride=1, padding=0)
        self.conv3 = nn.Conv2d(10, 15, 3, stride=1, padding=0)
        self.conv4 = nn.Conv2d(15, 20, 3, stride=1, padding=0)
        self.bn1 = nn.BatchNorm2d(5)
        self.bn2 = nn.BatchNorm2d(10)
        self.bn3 = nn.BatchNorm2d(15)
        self.bn4 = nn.BatchNorm2d(20)
        self.pool1 = nn.MaxPool2d(2)
        self.pool2 = nn.MaxPool2d(2)
        self.fc1 = nn.Linear(20*4*4, 20)
        self.fc2 = nn.Linear(20, 10)
        self.drop1 = nn.Dropout(0.5)
        self.drop2 = nn.Dropout(0.5)

    def forward(self, x):
        x = F.relu(self.bn1(self.conv1(x)))
        x = F.relu(self.bn2(self.conv2(x)))
        x = self.pool1(x)
        x = F.relu(self.bn3(self.conv3(x)))
        x = F.relu(self.bn4(self.conv4(x)))
        x = self.pool2(x)
        x = x.view(-1, 20*4*4)
        x = F.relu(self.fc1(self.drop1(x)))
        x = self.fc2(self.drop2(x))
        return x

TargetNet = MyConvNet1
```


```python
# sec: test (順伝播計算 ゼロ値を入力)

model = TargetNet()
y = model(torch.zeros(1, 1, 28, 28))
print(y)

# sec: test (順伝播計算 MNISTデータを入力)

ds = torchvision.datasets.MNIST(root="trains/pytorch-mnist", train=True, download=True, 
    transform=torchvision.transforms.ToTensor())
y = model(ds[0][0].view(1, 1, 28, 28))
print(y)
```

    tensor([[ 0.2101,  0.0317, -0.0457, -0.0219,  0.1032,  0.0080, -0.2413, -0.2003,
              0.0222, -0.0821]], grad_fn=<AddmmBackward>)
    tensor([[-0.1699,  0.5580, -0.1396, -0.6204,  0.3966, -0.0115, -0.0553, -0.0156,
             -0.3096, -0.1288]], grad_fn=<AddmmBackward>)
    

## 学習

参考:

* https://qiita.com/fukuit/items/215ef75113d97560e599  
* https://qiita.com/takurooo/items/e4c91c5d78059f92e76d  
* https://qiita.com/kazetof/items/6a72926b9f8cd44c218e


```python
# sec: データセット

ds_train = torchvision.datasets.MNIST(root="trains/pytorch-mnist", train=True, download=True, 
    transform=torchvision.transforms.ToTensor())
dl_train = torch.utils.data.DataLoader(ds_train, batch_size=128, shuffle=True, num_workers=2)
print(len(ds_train))

# sec: 設定

model = TargetNet()
criterion = nn.CrossEntropyLoss()
# case: SGD
# optimizer = torch.optim.SGD(model.parameters(), lr=0.001, momentum=0.9)
# case: Adam
optimizer = torch.optim.Adam(model.parameters(), lr=0.001)
# case: end
print(model)
```

    60000
    MyConvNet1(
      (conv1): Conv2d(1, 5, kernel_size=(3, 3), stride=(1, 1))
      (conv2): Conv2d(5, 10, kernel_size=(3, 3), stride=(1, 1))
      (conv3): Conv2d(10, 15, kernel_size=(3, 3), stride=(1, 1))
      (conv4): Conv2d(15, 20, kernel_size=(3, 3), stride=(1, 1))
      (bn1): BatchNorm2d(5, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (bn2): BatchNorm2d(10, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (bn3): BatchNorm2d(15, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (bn4): BatchNorm2d(20, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (pool1): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
      (pool2): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
      (fc1): Linear(in_features=320, out_features=20, bias=True)
      (fc2): Linear(in_features=20, out_features=10, bias=True)
      (drop1): Dropout(p=0.5, inplace=False)
      (drop2): Dropout(p=0.5, inplace=False)
    )
    


```python
# sec: 学習ループ

model.train()
for i_ep in range(10): # loop over the dataset multiple times

    loss_sum = 0.0
    for ds_i in dl_train:
        
        inputs, labels = ds_i

        optimizer.zero_grad()
        outputs = model(inputs) # 順伝播
        loss = criterion(outputs, labels)
        loss.backward() # 逆伝播
        optimizer.step()

        loss_sum += loss.item()
    
    # sec: print
    
    print('%dep loss: %.5f' % (i_ep + 1, loss_sum / len(ds_train)))

n_ep = i_ep + 1
print('Finished Training')
```

    1ep loss: 0.00700
    2ep loss: 0.00359
    3ep loss: 0.00310
    4ep loss: 0.00290
    5ep loss: 0.00272
    6ep loss: 0.00258
    7ep loss: 0.00247
    8ep loss: 0.00243
    9ep loss: 0.00237
    10ep loss: 0.00232
    Finished Training
    

## バリデーション


```python
# sec: データセット

ds_vali = torchvision.datasets.MNIST(root="trains/pytorch-mnist", train=True, download=True, 
    transform=torchvision.transforms.ToTensor())
ds_vali, _ = torch.utils.data.random_split(ds_vali, [1000, len(ds_vali) - 1000]) # イマイチな書き方
dl_vali = torch.utils.data.DataLoader(ds_vali, batch_size=128, shuffle=False, num_workers=2)
print(len(ds_vali))
```

    1000
    


```python
# sec: バリデーション例

model.eval() # 必要 trainのままだとdropoutの効果が残り精度劣化 90.0%で低迷
loss_sum = 0
n_correct = 0
for ds_i in dl_vali:
        
    inputs, labels = ds_i

    with torch.no_grad():
        
        outputs = model(inputs) # 順伝播
        loss = criterion(outputs, labels)
        
        loss_sum += loss.item()
        
        _, preds = torch.max(outputs, dim=1) # (max values, arg indices)
        n_correct += (preds == labels).sum()

# sec: print

print('loss: %.2e, accuracy: %.2f' % (loss_sum / len(ds_vali), float(n_correct) / len(ds_vali) * 100))
```

    loss: 2.43e-04, accuracy: 99.03
    

## 1回の評価結果


```python
# sec: 画像を描画

ds_test = torchvision.datasets.MNIST(root="trains/pytorch-mnist", train=False, download=True, 
    transform=torchvision.transforms.ToTensor())

x, label = ds_test[np.random.randint(0, len(ds_test))]
plt.imshow(x[0], cmap='gray', interpolation="none")
plt.show()

# sec: 1回の評価

model.eval()
x = x[None] # 1軸を追加
with torch.no_grad():
    y = model(x)

print("予測ラベル:", y.numpy().argmax(axis=1)[0], "| 真値ラベル:", label)
```


![output_23_0.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/606924/f71be618-2c83-c1af-866b-608dc980867f.png)


    予測ラベル: 8 | 真値ラベル: 8
    

## テスト


```python
# sec: データセット

ds_test = torchvision.datasets.MNIST(root="trains/pytorch-mnist", train=False, download=True, 
    transform=torchvision.transforms.ToTensor())
dl_test = torch.utils.data.DataLoader(ds_test, batch_size=128, shuffle=False, num_workers=2)
print(len(ds_test))
```

    10000
    


```python
# sec: テスト例

model.eval()
loss_sum = 0
list_pred, list_label = [], []
for ds_i in dl_test:
        
    inputs, labels = ds_i

    with torch.no_grad():
        
        outputs = model(inputs) # 順伝播
        loss = criterion(outputs, labels)
        
        loss_sum += loss.item()
        
        _, preds = torch.max(outputs, dim=1) # (max values, arg indices)
        list_pred.extend(preds)
        list_label.extend(labels)

list_pred = np.array(list_pred)
list_label = np.array(list_label)

# sec: print

print('loss: %.2e, accuracy: %.2f' % (loss_sum / len(ds_test), (list_pred == list_label).sum() / len(ds_test) * 100))
```

    loss: 2.84e-04, accuracy: 99.03
    

他のWeb記事のMNISTデータを用いる評価結果で「精度98.13%」程度なので、結果は妥当。
http://torch.classcat.com/2018/07/26/pytorch-040-examples-mnist-cnn/


```python
# sec: precision, recall, fscore, supportを見る

res_prf = sklearn.metrics.precision_recall_fscore_support(list_label, list_pred)

print("precision, recall, fscore, supportを見る:")
print(res_prf)

fig = plt.figure()
ax = fig.add_subplot(111)
ax.bar(np.arange(10) - 0.2, res_prf[0], width=0.4, label="precision")
ax.bar(np.arange(10) + 0.2, res_prf[1], width=0.4, label="recall")
ax.plot(np.arange(10), res_prf[2], 'r_', ms=12, label="f")
ax.legend(fontsize=9, framealpha=0.5, labelspacing=0.2)
ax.set_ylim([0.97, 1.0])
plt.show()
```

    precision, recall, fscore, supportを見る:
    (array([0.99084435, 0.99297012, 0.98653846, 0.99303483, 0.98681542,
           0.9877095 , 0.99267782, 0.98736638, 0.99588054, 0.9889669 ]), array([0.99387755, 0.99559471, 0.99418605, 0.98811881, 0.99083503,
           0.99103139, 0.99060543, 0.98832685, 0.99281314, 0.97720515]), array([0.99235863, 0.99428069, 0.99034749, 0.99057072, 0.98882114,
           0.98936766, 0.99164054, 0.98784638, 0.99434447, 0.98305085]), array([ 980, 1135, 1032, 1010,  982,  892,  958, 1028,  974, 1009],
          dtype=int64))
    


![output_28_1.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/606924/36e41888-4cd7-9f6f-b93b-a9a84fd9e2b0.png)



```python
# sec: 混同行列を見る

sklearn.metrics.confusion_matrix(list_label, list_pred)
```




    array([[ 974,    0,    1,    0,    0,    0,    4,    1,    0,    0],
           [   0, 1130,    4,    0,    0,    0,    0,    1,    0,    0],
           [   1,    0, 1026,    1,    0,    0,    0,    2,    2,    0],
           [   0,    0,    1,  998,    0,    5,    0,    4,    2,    0],
           [   0,    0,    0,    0,  973,    0,    1,    0,    0,    8],
           [   1,    0,    0,    4,    0,  884,    2,    1,    0,    0],
           [   3,    3,    0,    0,    0,    3,  949,    0,    0,    0],
           [   0,    5,    5,    1,    0,    0,    0, 1016,    0,    1],
           [   2,    0,    3,    0,    0,    0,    0,    0,  967,    2],
           [   2,    0,    0,    1,   13,    3,    0,    4,    0,  986]],
          dtype=int64)


## NNモデルパラメータ保存・読込評価

https://pytorch.org/tutorials/beginner/saving_loading_models.html


```python
# sec: save

torch.save(model.state_dict(), "results/pytorch-mnist/model.pkl")
```


```python
# sec: load

model_e = TargetNet()
model_e.load_state_dict(torch.load("results/pytorch-mnist/model.pkl"))
```




    <All keys matched successfully>




```python
model_e.state_dict()
```




    OrderedDict([('conv1.weight', tensor([[[[-0.1193, -0.1967,  0.0337],
                            [-0.2103,  0.2067,  0.3707],
                            [-0.0072,  0.2572, -0.0811]]],
                  
                  
                          [[[ 0.2073,  0.0639, -0.2909],
                            [ 0.2432, -0.2954, -0.1945],
                            [ 0.0062, -0.1348,  0.1976]]],
                  
                  
                          [[[ 0.2244,  0.2394,  0.2029],
                            [ 0.0774, -0.1356, -0.1406],
                            [-0.1890, -0.1105, -0.2486]]],
                  
                  
                          [[[-0.0757, -0.1669, -0.3270],
                            [ 0.2795,  0.2242,  0.1612],
                            [-0.0702,  0.2889,  0.0423]]],
                  
                  
                          [[[-0.0672,  0.1286, -0.2042],
                            [-0.0348, -0.2120, -0.2940],
                            [-0.3711, -0.1960,  0.1867]]]])),
                 ('conv1.bias',
                  tensor([ 0.0740,  0.0062,  0.1578, -0.1330,  0.2352])),
                 ('conv2.weight', tensor([[[[-0.1560, -0.1605, -0.0265],
                            [ 0.2078,  0.0014, -0.1418],
                            [ 0.0111,  0.0764, -0.0443]],
                  
                           [[-0.0406, -0.0301, -0.0977],
                            [-0.0876, -0.1847, -0.2042],
                            [ 0.0244,  0.1091, -0.1029]],
                  
                           ...(略)...
                  
                           [[-1.2396e-01, -4.7989e-02,  6.0982e-03],
                            [ 1.0323e-01,  7.4295e-02,  4.4253e-02],
                            [ 1.0707e-01,  1.2724e-01,  1.9893e-01]],
                  
                           [[-5.0636e-02, -1.1169e-01, -1.8197e-02],
                            [ 8.4612e-02, -4.1535e-02,  5.4119e-03],
                            [-5.3175e-03, -9.3789e-02,  2.2890e-02]],
                  
                           [[ 7.8540e-02, -1.1353e-01, -1.2255e-01],
                            [ 1.1991e-02, -6.8789e-02, -1.7400e-01],
                            [ 9.7441e-05, -8.2435e-02,  4.7667e-03]]]])),
                 ('conv4.bias',
                  tensor([ 0.0580, -0.0452, -0.0356, -0.0077, -0.0707, -0.0301, -0.0416,  0.0149,
                           0.1250,  0.0470,  0.0371, -0.0528, -0.0887, -0.0169,  0.0178,  0.0726,
                          -0.0794,  0.0061, -0.0065,  0.0043])),
                 ('bn1.weight', tensor([0.9750, 1.0013, 0.9576, 0.9376, 1.2138])),
                 ('bn1.bias',
                  tensor([-0.0668, -0.0248,  0.1255,  0.1388,  0.2274])),
                 ('bn1.running_mean',
                  tensor([ 0.1101, -0.0276,  0.1465, -0.0831,  0.0740])),
                 ('bn1.running_var',
                  tensor([0.0280, 0.0244, 0.0326, 0.0298, 0.0951])),
                 ('bn1.num_batches_tracked', tensor(4690)),
                 ('bn2.weight',
                  tensor([1.1731, 0.8839, 0.8436, 1.0764, 1.0680, 1.0759, 0.9479, 0.8768, 1.1014,
                          0.9595])),
                 ('bn2.bias',
                  tensor([-2.8445e-01, -4.5570e-02,  2.4080e-04, -6.8240e-03, -1.9356e-02,
                          -9.8832e-02, -3.8079e-02, -1.0984e-01, -1.7605e-01,  1.1987e-01])),
                 ('bn2.running_mean',
                  tensor([-0.2551, -0.3085,  0.6618,  0.3906, -0.5419, -0.8479,  0.0599, -0.3507,
                          -0.1456, -0.0181])),
                 ('bn2.running_var',
                  tensor([0.2021, 0.7583, 0.4969, 0.4905, 0.2947, 0.1725, 0.2593, 0.2616, 0.2963,
                          0.2185])),
                 ('bn2.num_batches_tracked', tensor(4690)),
                 ('bn3.weight',
                  tensor([1.0696, 0.8160, 0.9060, 1.0633, 1.0389, 0.9895, 0.8915, 0.8183, 1.1316,
                          0.9516, 0.9707, 0.8907, 1.0231, 0.8617, 1.0554])),
                 ('bn3.bias',
                  tensor([-0.2997, -0.0583, -0.2442, -0.0981, -0.2548, -0.2014, -0.1782, -0.2454,
                          -0.3477, -0.0327, -0.2671, -0.3008, -0.3205, -0.5982, -0.1280])),
                 ('bn3.running_mean',
                  tensor([-0.9035,  0.2130,  0.2813, -1.3235, -0.5512,  0.0552, -0.1724,  0.1172,
                          -1.5437, -0.0501, -0.5553, -0.0547, -1.1366,  0.3776, -1.5574])),
                 ('bn3.running_var',
                  tensor([0.6823, 0.9488, 0.7111, 1.3630, 1.0511, 0.8177, 1.0470, 1.0047, 1.1058,
                          1.9996, 1.1460, 1.3993, 1.0603, 0.4562, 1.7725])),
                 ('bn3.num_batches_tracked', tensor(4690)),
                 ('bn4.weight',
                  tensor([1.2947, 1.3268, 1.2583, 1.2611, 1.2759, 1.3348, 1.3097, 1.3204, 1.3608,
                          1.2124, 1.3248, 1.2563, 1.3738, 1.2736, 1.2208, 1.2850, 1.3084, 1.3131,
                          1.2993, 1.2804])),
                 ('bn4.bias',
                  tensor([-0.5450, -0.5422, -0.3926, -0.4930, -0.5669, -0.4537, -0.5399, -0.5409,
                          -0.5684, -0.4720, -0.5032, -0.5454, -0.5353, -0.4670, -0.4292, -0.5145,
                          -0.3628, -0.6174, -0.5858, -0.4986])),
                 ('bn4.running_mean',
                  tensor([ 0.4598,  0.2837, -0.1339,  0.0366, -0.3407,  0.1557,  0.2131, -0.6676,
                           0.0993,  0.9765,  0.7372,  0.2057, -0.3129, -0.2075,  0.2524, -0.2052,
                           0.7763, -0.3153, -0.8190,  0.8294])),
                 ('bn4.running_var',
                  tensor([0.2823, 0.3211, 0.3877, 0.3735, 0.2755, 0.3754, 0.3229, 0.3310, 0.3055,
                          0.2756, 0.4847, 0.3380, 0.3030, 0.3383, 0.3769, 0.2427, 0.5124, 0.2893,
                          0.3855, 0.3065])),
                 ('bn4.num_batches_tracked', tensor(4690)),
                 ('fc1.weight',
                  tensor([[ 0.1167,  0.1003,  0.1363,  ..., -0.0632, -0.1001, -0.1384],
                          [ 0.0219,  0.0128,  0.0546,  ..., -0.0419,  0.0384,  0.1337],
                          [ 0.0895,  0.0410,  0.0538,  ...,  0.0491,  0.0248, -0.1246],
                          ...,
                          [ 0.0377,  0.0342,  0.0202,  ..., -0.0412,  0.0019,  0.0041],
                          [ 0.0659,  0.0783,  0.0530,  ...,  0.0059,  0.0661,  0.0319],
                          [ 0.0553,  0.0511, -0.0325,  ...,  0.0109, -0.0564, -0.1748]])),
                 ('fc1.bias',
                  tensor([-0.0930, -0.0096, -0.0934, -0.0489,  0.1413,  0.0640,  0.0702,  0.0313,
                           0.0491,  0.1809,  0.0214, -0.1256,  0.0015,  0.0511,  0.0580,  0.0494,
                          -0.0177, -0.0066,  0.0417,  0.0478])),
                 ('fc2.weight',
                  tensor([[-0.3481,  0.1943, -0.1167, -0.3848,  0.2522, -0.5797, -0.3552,  0.2499,
                           -0.4902,  0.1876, -0.6010,  0.1155,  0.1638, -0.3284, -0.3364,  0.2158,
                           -0.3198,  0.1905,  0.2642, -0.1496],
                          [-0.4746, -0.5199, -0.3840, -0.2805,  0.2469,  0.2020,  0.2321, -0.3990,
                            0.2874,  0.2119, -0.4917, -0.2028, -0.3293, -0.4795,  0.2659, -0.4735,
                           -0.3607, -0.2774, -0.3733,  0.2061],
                          [ 0.1365, -0.1580, -0.2591, -0.1550, -0.3192, -0.4425,  0.2455, -0.2653,
                           -0.3550, -0.1905,  0.0099,  0.1221,  0.2218, -0.3097,  0.3020, -0.2191,
                           -0.3595,  0.1949, -0.2289, -0.3345],
                          [ 0.1819, -0.2019, -0.5650,  0.2102, -0.2937, -0.3966,  0.1748, -0.4076,
                            0.2246,  0.2166,  0.2953, -0.4493,  0.1728, -0.2740, -0.0933,  0.2476,
                            0.2063, -0.1891, -0.1732, -0.3121],
                          [-0.4458, -0.6537,  0.1651,  0.1682, -0.2137,  0.2296,  0.1956,  0.2499,
                            0.3215, -0.3628,  0.2233,  0.1229, -0.4657,  0.0259, -0.2530, -0.4996,
                            0.1364, -0.0745, -0.2714, -0.0395],
                          [ 0.1569,  0.2018, -0.2388,  0.1967,  0.2599,  0.1737, -0.4559, -0.2330,
                           -0.1793, -0.0934, -0.1260, -0.3630, -0.2875, -0.2901, -0.4317,  0.2435,
                            0.2380, -0.1329,  0.2140,  0.2101],
                          [-0.2263, -0.0485,  0.1833, -0.5683,  0.3253, -0.1906, -0.3670,  0.0325,
                           -0.2744, -0.3122, -0.4359,  0.1418, -0.3495, -0.6060, -0.4037, -0.1927,
                            0.2340,  0.2258, -0.2329,  0.2628],
                          [-0.2519,  0.1679, -0.3764,  0.1616, -0.3912, -0.3314,  0.2476,  0.1835,
                           -0.1459,  0.1911,  0.2524, -0.3639, -0.1660,  0.1537,  0.2604, -0.2612,
                           -0.4262, -0.3990,  0.2768, -0.5895],
                          [ 0.1794,  0.2254,  0.1708, -0.1893, -0.0620, -0.2753, -0.1750, -0.0990,
                           -0.1114, -0.3117,  0.2690, -0.2005,  0.1797,  0.1494, -0.3352, -0.1336,
                           -0.0673,  0.1785, -0.1971,  0.2140],
                          [ 0.1075,  0.0880,  0.1336,  0.1675, -0.3566,  0.1433, -0.1642,  0.2479,
                            0.2795, -0.3182, -0.2066, -0.1676, -0.2736,  0.1719, -0.2884,  0.1781,
                           -0.2009, -0.2304,  0.2601, -0.2920]])),
                 ('fc2.bias',
                  tensor([-0.3426,  0.7404,  0.7441, -0.2981, -0.2907, -0.5408, -0.0061,  0.0115,
                          -0.1341, -0.0474]))])




```python
# sec: データセット

ds_test = torchvision.datasets.MNIST(root="trains/pytorch-mnist", train=False, download=True, 
    transform=torchvision.transforms.ToTensor())
dl_test = torch.utils.data.DataLoader(ds_test, batch_size=128, shuffle=False, num_workers=2)
print(len(ds_test))
```

    10000
    


```python
# sec: 評価

model_e.eval()
loss_sum = 0
n_correct = 0
for ds_i in dl_test:
        
    inputs, labels = ds_i

    with torch.no_grad():
        
        outputs = model_e(inputs) # 順伝播
        loss = criterion(outputs, labels)
        
        loss_sum += loss.item()
        
        _, preds = torch.max(outputs, dim=1) # (max values, arg indices)
        n_correct += (preds == labels).sum()

# sec: print

print('loss: %.2e, accuracy: %.2f' % (loss_sum / len(ds_test), float(n_correct) / len(ds_test) * 100))
```

    loss: 2.43e-04, accuracy: 99.20
    

## 最適計算パラメータ保存・計算再開

https://pytorch.org/tutorials/beginner/saving_loading_models.html


```python
# sec: save

torch.save({
    'epoch': n_ep,
    'model_state_dict': model.state_dict(),
    'optimizer_state_dict': optimizer.state_dict(),
    'loss': loss }, 
    "results/pytorch-mnist/model&opt.pkl")
```

ここでJupyterの実行中のPythonのカーネルをリスタート可。

### Libs


```python
import numpy as np
import matplotlib.pyplot as plt

import torch
import torchvision
```

### NNモデモ


```python
import torch.nn as nn
import torch.nn.functional as F

class MyConvNet1(nn.Module):
    
    def __init__(self):
        super().__init__()
        self.conv1 = nn.Conv2d(1, 5, 3, stride=1, padding=0)
        self.conv2 = nn.Conv2d(5, 10, 3, stride=1, padding=0)
        self.conv3 = nn.Conv2d(10, 15, 3, stride=1, padding=0)
        self.conv4 = nn.Conv2d(15, 20, 3, stride=1, padding=0)
        self.bn1 = nn.BatchNorm2d(5)
        self.bn2 = nn.BatchNorm2d(10)
        self.bn3 = nn.BatchNorm2d(15)
        self.bn4 = nn.BatchNorm2d(20)
        self.pool1 = nn.MaxPool2d(2)
        self.pool2 = nn.MaxPool2d(2)
        self.fc1 = nn.Linear(20*4*4, 20)
        self.fc2 = nn.Linear(20, 10)
        self.drop1 = nn.Dropout(0.5)
        self.drop2 = nn.Dropout(0.5)

    def forward(self, x):
        x = F.relu(self.bn1(self.conv1(x)))
        x = F.relu(self.bn2(self.conv2(x)))
        x = self.pool1(x)
        x = F.relu(self.bn3(self.conv3(x)))
        x = F.relu(self.bn4(self.conv4(x)))
        x = self.pool2(x)
        x = x.view(-1, 20*4*4)
        x = F.relu(self.fc1(self.drop1(x)))
        x = self.fc2(self.drop2(x))
        return x

TargetNet = MyConvNet1
```


```python
# sec: load

model = TargetNet()
optimizer = torch.optim.Adam(model.parameters())

d = torch.load("results/pytorch-mnist/model&opt.pkl")
model.load_state_dict(d['model_state_dict'])
optimizer.load_state_dict(d['optimizer_state_dict'])
n_ep = d['epoch']
loss = d['loss']
```


```python
# sec: 内容を確認

optimizer.state_dict()
```




    {'state': {2202322517592: {'step': 4690,
       'exp_avg': tensor([[[[ 4.9681e-03, -1.3392e-02, -1.1919e-02],
                 [-1.8946e-02, -1.5522e-02,  3.7579e-03],
                 [-2.7534e-02, -1.6930e-02, -3.9333e-05]]],
       
       
               [[[ 8.3556e-03,  8.2333e-03, -4.6917e-03],
                 [ 8.6733e-03,  1.3580e-02,  1.3980e-03],
                 [-1.4740e-02, -1.8268e-02, -2.1180e-02]]],
       
       
               [[[ 2.6981e-02,  2.1422e-02,  8.6973e-03],
                 [ 2.2680e-02,  2.4799e-02,  1.8023e-02],
                 [ 1.1921e-02,  1.9067e-02,  2.0821e-02]]],
       
       
               [[[ 2.6852e-02,  2.1411e-02,  2.4557e-02],
                 [ 2.6102e-02,  1.6432e-02,  1.0814e-02],
                 [ 2.2461e-02,  9.3432e-03, -1.2549e-02]]],
       
       
               [[[ 1.7876e-03,  3.0547e-03,  9.6555e-03],
                 [-2.2322e-03, -2.0686e-03,  2.5197e-03],
                 [-2.8117e-03, -6.4745e-03, -2.8697e-03]]]]),
       'exp_avg_sq': tensor([[[[0.0133, 0.0118, 0.0097],
                 [0.0132, 0.0078, 0.0049],
                 [0.0131, 0.0075, 0.0063]]],
       
       
               [[[0.0202, 0.0155, 0.0092],
                 [0.0155, 0.0083, 0.0045],
                 [0.0224, 0.0151, 0.0118]]],
       
       
               [[[0.0066, 0.0051, 0.0060],
                 [0.0069, 0.0056, 0.0052],
                 [0.0061, 0.0047, 0.0038]]],
       
       
               [[[0.0056, 0.0047, 0.0053],
                 [0.0032, 0.0026, 0.0039],
                 [0.0025, 0.0015, 0.0030]]],
       
       
               [[[0.0048, 0.0020, 0.0013],
                 [0.0007, 0.0001, 0.0004],
                 [0.0001, 0.0003, 0.0017]]]])},
      2202322517672: {'step': 4690,
       'exp_avg': tensor([-1.5162e-06, -1.2697e-06, -1.2059e-07, -4.3583e-06,  6.9085e-07]),
       'exp_avg_sq': tensor([8.3501e-10, 1.5377e-11, 3.3076e-12, 3.5197e-10, 6.3042e-12])},
      2202322517272: {'step': 4690,
       'exp_avg': tensor([[[[-3.1205e-03,  1.0635e-02,  1.7947e-02],
                 [ 3.0011e-02,  4.7265e-02,  5.7637e-02],
                 [ 4.3873e-02,  4.3586e-02,  5.4873e-02]],
       
                [[ 1.4990e-02,  8.2882e-03, -5.3019e-04],
                 [ 6.4136e-03,  1.4983e-03, -3.0395e-03],
                 [ 9.1574e-04, -3.6870e-03, -1.4301e-02]],
       
                [[ 1.5993e-02,  1.2531e-02,  8.5470e-03],
                 [-3.2829e-03, -3.5228e-03, -1.0266e-02],
                 [-2.1787e-03, -4.9055e-03,  8.4147e-03]],
       
                [[ 1.3000e-02,  1.4188e-02,  2.3311e-02],
                 [ 5.2523e-02,  5.6909e-02,  6.5983e-02],
                 [ 7.4201e-02,  6.4654e-02,  5.4315e-02]],
       
                [[-1.1624e-02, -1.2903e-02, -1.5316e-02],
                 [-2.2582e-02, -1.3517e-02, -1.4352e-02],
                 [-1.2509e-02, -1.4768e-02, -2.3715e-02]]],
       
               ...(略)...
               
      2202322530536: {'step': 4690,
       'exp_avg': tensor([ 0.0005, -0.0052,  0.0022,  0.0002,  0.0017,  0.0033, -0.0029,  0.0005,
               -0.0030,  0.0028]),
       'exp_avg_sq': tensor([0.0001, 0.0001, 0.0001, 0.0001, 0.0001, 0.0001, 0.0001, 0.0001, 0.0001,
               0.0001])}},
     'param_groups': [{'lr': 0.001,
       'betas': (0.9, 0.999),
       'eps': 1e-08,
       'weight_decay': 0,
       'amsgrad': False,
       'params': [2202322517592,
        2202322517672,
        2202322517272,
        2202322517192,
        2202322516952,
        2202322516792,
        2202322517032,
        2202322514152,
        2202322489656,
        2202322491896,
        2202322492216,
        2202322492296,
        2202322492616,
        2202322492696,
        2202322493016,
        2202322493096,
        2202322493336,
        2202322530376,
        2202322530456,
        2202322530536]}]}



最適化の内部の状態変数もすべて保存される模様。

### 学習ループ再開


```python
# sec: データセット

ds_train = torchvision.datasets.MNIST(root="trains/pytorch-mnist", train=True, download=True, 
    transform=torchvision.transforms.ToTensor())
dl_train = torch.utils.data.DataLoader(ds_train, batch_size=128, shuffle=True, num_workers=2)
print(len(ds_train))
```


```python
# sec: 学習再開

criterion = nn.CrossEntropyLoss()

model.train()
for i_ep in range(n_ep, n_ep + 10): # loop over the dataset multiple times

    loss_sum = 0.0
    for ds_i in dl_train:
        
        inputs, labels = ds_i

        optimizer.zero_grad()
        outputs = model(inputs) # 順伝播
        loss = criterion(outputs, labels)
        loss.backward() # 逆伝播
        optimizer.step()

        loss_sum += loss.item()
    
    # sec: print
    
    print('%dep loss: %.5f' % (i_ep + 1, loss_sum / len(ds_train)))

n_ep = i_ep + 1
print('Finished Training')
```

    11ep loss: 0.00227
    12ep loss: 0.00225
    13ep loss: 0.00220
    14ep loss: 0.00220
    15ep loss: 0.00215
    16ep loss: 0.00211
    17ep loss: 0.00207
    18ep loss: 0.00207
    19ep loss: 0.00204
    20ep loss: 0.00204
    Finished Training
    

初回学習では「10ep loss: 0.00232」で終わっていた状態。「11ep loss: 0.00227」から始まり、最適化の続きが開始されている模様。


```python
# sec: データセット

ds_test = torchvision.datasets.MNIST(root="trains/pytorch-mnist", train=False, download=True, 
    transform=torchvision.transforms.ToTensor())
dl_test = torch.utils.data.DataLoader(ds_test, batch_size=128, shuffle=False, num_workers=2)
print(len(ds_test))
```

    10000
    


```python
# sec: テスト例

model.eval()
loss_sum = 0
list_pred, list_label = [], []
for ds_i in dl_test:
        
    inputs, labels = ds_i

    with torch.no_grad():
        
        outputs = model(inputs) # 順伝播
        loss = criterion(outputs, labels)
        
        loss_sum += loss.item()
        
        _, preds = torch.max(outputs, dim=1) # (max values, arg indices)
        list_pred.extend(preds)
        list_label.extend(labels)

list_pred = np.array(list_pred)
list_label = np.array(list_label)

# sec: print

print('loss: %.2e, accuracy: %.2f' % (loss_sum / len(ds_test), (list_pred == list_label).sum() / len(ds_test) * 100))
```

    loss: 2.48e-04, accuracy: 99.03
    

他のWeb記事のMNISTデータを用いる評価結果で「精度98.13%」程度なので、結果は妥当。
http://torch.classcat.com/2018/07/26/pytorch-040-examples-mnist-cnn/


```python
# sec: precision, recall, fscore, supportを見る

res_prf = sklearn.metrics.precision_recall_fscore_support(list_label, list_pred)

print("precision, recall, fscore, supportを見る:")
print(res_prf)

fig = plt.figure()
ax = fig.add_subplot(111)
ax.bar(np.arange(10) - 0.2, res_prf[0], width=0.4, label="precision")
ax.bar(np.arange(10) + 0.2, res_prf[1], width=0.4, label="recall")
ax.plot(np.arange(10), res_prf[2], 'r_', ms=12, label="f")
ax.legend(fontsize=9, framealpha=0.5, labelspacing=0.2)
ax.set_ylim([0.97, 1.0])
plt.show()
```

    precision, recall, fscore, supportを見る:
    (array([0.99084435, 0.99297012, 0.98653846, 0.99303483, 0.98681542,
           0.9877095 , 0.99267782, 0.98736638, 0.99588054, 0.9889669 ]), array([0.99387755, 0.99559471, 0.99418605, 0.98811881, 0.99083503,
           0.99103139, 0.99060543, 0.98832685, 0.99281314, 0.97720515]), array([0.99235863, 0.99428069, 0.99034749, 0.99057072, 0.98882114,
           0.98936766, 0.99164054, 0.98784638, 0.99434447, 0.98305085]), array([ 980, 1135, 1032, 1010,  982,  892,  958, 1028,  974, 1009],
          dtype=int64))
    


![output_53_1.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/606924/a9858dc5-8717-5d41-97b3-efe31c4d3e6c.png)



```python
# sec: 混同行列を見る

sklearn.metrics.confusion_matrix(list_label, list_pred)
```




    array([[ 974,    0,    1,    0,    0,    0,    4,    1,    0,    0],
           [   0, 1130,    4,    0,    0,    0,    0,    1,    0,    0],
           [   1,    0, 1026,    1,    0,    0,    0,    2,    2,    0],
           [   0,    0,    1,  998,    0,    5,    0,    4,    2,    0],
           [   0,    0,    0,    0,  973,    0,    1,    0,    0,    8],
           [   1,    0,    0,    4,    0,  884,    2,    1,    0,    0],
           [   3,    3,    0,    0,    0,    3,  949,    0,    0,    0],
           [   0,    5,    5,    1,    0,    0,    0, 1016,    0,    1],
           [   2,    0,    3,    0,    0,    0,    0,    0,  967,    2],
           [   2,    0,    0,    1,   13,    3,    0,    4,    0,  986]],
          dtype=int64)



### 保存・読込


```python
# sec: save

torch.save(model.state_dict(), "results/pytorch-mnist/model2.pkl")
```


```python
# sec: load

model = TargetNet()
model.load_state_dict(torch.load("results/pytorch-mnist/model2.pkl"))
```




    <All keys matched successfully>