PyTorchでの学習モデル表示等 (TensorWatch)

Microsoftが開発した[TensorWatch](https://github.com/microsoft/tensorwatch/blob/master/README.md)を使うと、PyTorchの学習モデル等の監視がスムーズに出来る。ここでは、Google Colabで紹介する。

- [学習モデル図の表示方法](https://github.com/microsoft/tensorwatch/blob/master/notebooks/network_arch.ipynb)
- [画像の推論の方法](https://github.com/microsoft/tensorwatch/blob/master/notebooks/cnn_pred_explain.ipynb)

(2019/5/28にパッケージングのバグ[#9](https://github.com/microsoft/tensorwatch/issues/9)を報告したところ迅速に対応していただき、pip install tensorwatchでGoogle Colabでも動くようになった。)


#環境設定
以下で設定できる。

```
pip install tensorwatch
```
#各機能について
##モデル図等の表示
モデル図の表示は、以下で出来る。言わずもがなだがtorchvisionのmodelsをそのまま表示できる。テンソルのサイズ等のパラメータも表示されているので使いやすい。ただし、現時点ではファイル書き出し機能は開発中のようなので、画面コピー等の対策が必要なのがちょっと面倒である。なおMicrosoftから[Jupyter Notebook](https://github.com/microsoft/tensorwatch/blob/master/notebooks/network_arch.ipynb)も提供されている。

```Python
import tensorwatch as tw
import torchvision.models
alexnet_model = torchvision.models.alexnet()
tw.draw_model(alexnet_model, [1, 3, 224, 224])
```

モデルのパラメータ数等は以下で表示可能である。なお、パラメータ等は、tensorstatを使っている
。このため、tensorstatの実装状況により、未対応が表示される。

```Python
import tensorwatch as tw
import torchvision.models
alexnet_model = torchvision.models.alexnet()
tw.model_stats(alexnet_model, [1, 3, 224, 224])
```
###参考資料
- [microsoft/tensorwatch](https://github.com/microsoft/tensorwatch)
- モデル図周りのソースコード
  - [Swall0w/torchstat](https://github.com/Swall0w/torchstat)
     - [Supported Layers](https://github.com/Swall0w/torchstat/blob/master/detail.md)
  - [waleedka/hiddenlayer)](https://github.com/waleedka/hiddenlayer)
  - [yulongwang12/visual-attribution](https://github.com/yulongwang12/visual-attribution)

```

module name	input shape	output shape	params	memory(MB)	MAdd	Flops	MemRead(B)	MemWrite(B)	duration[%]	MemR+W(B)
0	conv1	3 224 224	64 112 112	9408.0	3.06	235,225,088.0	118,013,952.0	639744.0	3211264.0	2.89%	3851008.0
1	bn1	64 112 112	64 112 112	128.0	3.06	3,211,264.0	1,605,632.0	3211776.0	3211264.0	0.45%	6423040.0
2	relu	64 112 112	64 112 112	0.0	3.06	802,816.0	802,816.0	3211264.0	3211264.0	0.14%	6422528.0
3	maxpool	64 112 112	64 56 56	0.0	0.77	1,605,632.0	802,816.0	3211264.0	802816.0	4.60%	4014080.0
4	layer1.0.conv1	64 56 56	64 56 56	4096.0	0.77	25,489,408.0	12,845,056.0	819200.0	802816.0	1.13%	1622016.0
5	layer1.0.bn1	64 56 56	64 56 56	128.0	0.77	802,816.0	401,408.0	803328.0	802816.0	0.09%	1606144.0
6	layer1.0.conv2	64 56 56	64 56 56	36864.0	0.77	231,010,304.0	115,605,504.0	950272.0	802816.0	2.03%	1753088.0
7	layer1.0.bn2	64 56 56	64 56 56	128.0	0.77	802,816.0	401,408.0	803328.0	802816.0	0.11%	1606144.0
8	layer1.0.conv3	64 56 56	256 56 56	16384.0	3.06	101,957,632.0	51,380,224.0	868352.0	3211264.0	2.38%	4079616.0
9	layer1.0.bn3	256 56 56	256 56 56	512.0	3.06	3,211,264.0	1,605,632.0	3213312.0	3211264.0	0.31%	6424576.0
10	layer1.0.relu	256 56 56	256 56 56	0.0	3.06	802,816.0	802,816.0	3211264.0	3211264.0	0.13%	6422528.0
11	layer1.0.downsample.0	64 56 56	256 56 56	16384.0	3.06	101,957,632.0	51,380,224.0	868352.0	3211264.0	2.26%	4079616.0
12	layer1.0.downsample.1	256 56 56	256 56 56	512.0	3.06	3,211,264.0	1,605,632.0	3213312.0	3211264.0	0.31%	6424576.0
13	layer1.1.conv1	256 56 56	64 56 56	16384.0	0.77	102,559,744.0	51,380,224.0	3276800.0	802816.0	1.32%	4079616.0
14	layer1.1.bn1	64 56 56	64 56 56	128.0	0.77	802,816.0	401,408.0	803328.0	802816.0	0.09%	1606144.0
15	layer1.1.conv2	64 56 56	64 56 56	36864.0	0.77	231,010,304.0	115,605,504.0	950272.0	802816.0	1.51%	1753088.0
16	layer1.1.bn2	64 56 56	64 56 56	128.0	0.77	802,816.0	401,408.0	803328.0	802816.0	0.09%	1606144.0
17	layer1.1.conv3	64 56 56	256 56 56	16384.0	3.06	101,957,632.0	51,380,224.0	868352.0	3211264.0	2.13%	4079616.0
18	layer1.1.bn3	256 56 56	256 56 56	512.0	3.06	3,211,264.0	1,605,632.0	3213312.0	3211264.0	0.30%	6424576.0
19	layer1.1.relu	256 56 56	256 56 56	0.0	3.06	802,816.0	802,816.0	3211264.0	3211264.0	0.12%	6422528.0
20	layer1.2.conv1	256 56 56	64 56 56	16384.0	0.77	102,559,744.0	51,380,224.0	3276800.0	802816.0	1.74%	4079616.0
21	layer1.2.bn1	64 56 56	64 56 56	128.0	0.77	802,816.0	401,408.0	803328.0	802816.0	0.10%	1606144.0
22	layer1.2.conv2	64 56 56	64 56 56	36864.0	0.77	231,010,304.0	115,605,504.0	950272.0	802816.0	1.44%	1753088.0
23	layer1.2.bn2	64 56 56	64 56 56	128.0	0.77	802,816.0	401,408.0	803328.0	802816.0	0.09%	1606144.0
24	layer1.2.conv3	64 56 56	256 56 56	16384.0	3.06	101,957,632.0	51,380,224.0	868352.0	3211264.0	2.26%	4079616.0
25	layer1.2.bn3	256 56 56	256 56 56	512.0	3.06	3,211,264.0	1,605,632.0	3213312.0	3211264.0	0.31%	6424576.0
26	layer1.2.relu	256 56 56	256 56 56	0.0	3.06	802,816.0	802,816.0	3211264.0	3211264.0	0.13%	6422528.0
27	layer2.0.conv1	256 56 56	128 56 56	32768.0	1.53	205,119,488.0	102,760,448.0	3342336.0	1605632.0	2.86%	4947968.0
28	layer2.0.bn1	128 56 56	128 56 56	256.0	1.53	1,605,632.0	802,816.0	1606656.0	1605632.0	0.18%	3212288.0
29	layer2.0.conv2	128 56 56	128 28 28	147456.0	0.38	231,110,656.0	115,605,504.0	2195456.0	401408.0	1.61%	2596864.0
30	layer2.0.bn2	128 28 28	128 28 28	256.0	0.38	401,408.0	200,704.0	402432.0	401408.0	0.05%	803840.0
31	layer2.0.conv3	128 28 28	512 28 28	65536.0	1.53	102,359,040.0	51,380,224.0	663552.0	1605632.0	1.17%	2269184.0
32	layer2.0.bn3	512 28 28	512 28 28	1024.0	1.53	1,605,632.0	802,816.0	1609728.0	1605632.0	0.17%	3215360.0
33	layer2.0.relu	512 28 28	512 28 28	0.0	1.53	401,408.0	401,408.0	1605632.0	1605632.0	0.13%	3211264.0
34	layer2.0.downsample.0	256 56 56	512 28 28	131072.0	1.53	205,119,488.0	102,760,448.0	3735552.0	1605632.0	2.81%	5341184.0
35	layer2.0.downsample.1	512 28 28	512 28 28	1024.0	1.53	1,605,632.0	802,816.0	1609728.0	1605632.0	0.18%	3215360.0
36	layer2.1.conv1	512 28 28	128 28 28	65536.0	0.38	102,660,096.0	51,380,224.0	1867776.0	401408.0	0.96%	2269184.0
37	layer2.1.bn1	128 28 28	128 28 28	256.0	0.38	401,408.0	200,704.0	402432.0	401408.0	0.05%	803840.0
38	layer2.1.conv2	128 28 28	128 28 28	147456.0	0.38	231,110,656.0	115,605,504.0	991232.0	401408.0	1.53%	1392640.0
39	layer2.1.bn2	128 28 28	128 28 28	256.0	0.38	401,408.0	200,704.0	402432.0	401408.0	0.06%	803840.0
40	layer2.1.conv3	128 28 28	512 28 28	65536.0	1.53	102,359,040.0	51,380,224.0	663552.0	1605632.0	0.98%	2269184.0
41	layer2.1.bn3	512 28 28	512 28 28	1024.0	1.53	1,605,632.0	802,816.0	1609728.0	1605632.0	0.17%	3215360.0
42	layer2.1.relu	512 28 28	512 28 28	0.0	1.53	401,408.0	401,408.0	1605632.0	1605632.0	0.10%	3211264.0
43	layer2.2.conv1	512 28 28	128 28 28	65536.0	0.38	102,660,096.0	51,380,224.0	1867776.0	401408.0	1.14%	2269184.0
44	layer2.2.bn1	128 28 28	128 28 28	256.0	0.38	401,408.0	200,704.0	402432.0	401408.0	0.05%	803840.0
45	layer2.2.conv2	128 28 28	128 28 28	147456.0	0.38	231,110,656.0	115,605,504.0	991232.0	401408.0	1.53%	1392640.0
46	layer2.2.bn2	128 28 28	128 28 28	256.0	0.38	401,408.0	200,704.0	402432.0	401408.0	0.05%	803840.0
47	layer2.2.conv3	128 28 28	512 28 28	65536.0	1.53	102,359,040.0	51,380,224.0	663552.0	1605632.0	1.38%	2269184.0
48	layer2.2.bn3	512 28 28	512 28 28	1024.0	1.53	1,605,632.0	802,816.0	1609728.0	1605632.0	0.18%	3215360.0
49	layer2.2.relu	512 28 28	512 28 28	0.0	1.53	401,408.0	401,408.0	1605632.0	1605632.0	0.09%	3211264.0
50	layer2.3.conv1	512 28 28	128 28 28	65536.0	0.38	102,660,096.0	51,380,224.0	1867776.0	401408.0	1.14%	2269184.0
51	layer2.3.bn1	128 28 28	128 28 28	256.0	0.38	401,408.0	200,704.0	402432.0	401408.0	0.05%	803840.0
52	layer2.3.conv2	128 28 28	128 28 28	147456.0	0.38	231,110,656.0	115,605,504.0	991232.0	401408.0	1.45%	1392640.0
53	layer2.3.bn2	128 28 28	128 28 28	256.0	0.38	401,408.0	200,704.0	402432.0	401408.0	0.05%	803840.0
54	layer2.3.conv3	128 28 28	512 28 28	65536.0	1.53	102,359,040.0	51,380,224.0	663552.0	1605632.0	1.23%	2269184.0
55	layer2.3.bn3	512 28 28	512 28 28	1024.0	1.53	1,605,632.0	802,816.0	1609728.0	1605632.0	0.15%	3215360.0
56	layer2.3.relu	512 28 28	512 28 28	0.0	1.53	401,408.0	401,408.0	1605632.0	1605632.0	0.09%	3211264.0
57	layer3.0.conv1	512 28 28	256 28 28	131072.0	0.77	205,320,192.0	102,760,448.0	2129920.0	802816.0	1.94%	2932736.0
58	layer3.0.bn1	256 28 28	256 28 28	512.0	0.77	802,816.0	401,408.0	804864.0	802816.0	0.09%	1607680.0
59	layer3.0.conv2	256 28 28	256 14 14	589824.0	0.19	231,160,832.0	115,605,504.0	3162112.0	200704.0	2.07%	3362816.0
60	layer3.0.bn2	256 14 14	256 14 14	512.0	0.19	200,704.0	100,352.0	202752.0	200704.0	0.03%	403456.0
61	layer3.0.conv3	256 14 14	1024 14 14	262144.0	0.77	102,559,744.0	51,380,224.0	1249280.0	802816.0	0.90%	2052096.0
62	layer3.0.bn3	1024 14 14	1024 14 14	2048.0	0.77	802,816.0	401,408.0	811008.0	802816.0	0.10%	1613824.0
63	layer3.0.relu	1024 14 14	1024 14 14	0.0	0.77	200,704.0	200,704.0	802816.0	802816.0	0.08%	1605632.0
64	layer3.0.downsample.0	512 28 28	1024 14 14	524288.0	0.77	205,320,192.0	102,760,448.0	3702784.0	802816.0	2.35%	4505600.0
65	layer3.0.downsample.1	1024 14 14	1024 14 14	2048.0	0.77	802,816.0	401,408.0	811008.0	802816.0	0.09%	1613824.0
66	layer3.1.conv1	1024 14 14	256 14 14	262144.0	0.19	102,710,272.0	51,380,224.0	1851392.0	200704.0	0.85%	2052096.0
67	layer3.1.bn1	256 14 14	256 14 14	512.0	0.19	200,704.0	100,352.0	202752.0	200704.0	0.04%	403456.0
68	layer3.1.conv2	256 14 14	256 14 14	589824.0	0.19	231,160,832.0	115,605,504.0	2560000.0	200704.0	1.58%	2760704.0
69	layer3.1.bn2	256 14 14	256 14 14	512.0	0.19	200,704.0	100,352.0	202752.0	200704.0	0.04%	403456.0
70	layer3.1.conv3	256 14 14	1024 14 14	262144.0	0.77	102,559,744.0	51,380,224.0	1249280.0	802816.0	0.91%	2052096.0
71	layer3.1.bn3	1024 14 14	1024 14 14	2048.0	0.77	802,816.0	401,408.0	811008.0	802816.0	0.10%	1613824.0
72	layer3.1.relu	1024 14 14	1024 14 14	0.0	0.77	200,704.0	200,704.0	802816.0	802816.0	0.07%	1605632.0
73	layer3.2.conv1	1024 14 14	256 14 14	262144.0	0.19	102,710,272.0	51,380,224.0	1851392.0	200704.0	0.88%	2052096.0
74	layer3.2.bn1	256 14 14	256 14 14	512.0	0.19	200,704.0	100,352.0	202752.0	200704.0	0.04%	403456.0
75	layer3.2.conv2	256 14 14	256 14 14	589824.0	0.19	231,160,832.0	115,605,504.0	2560000.0	200704.0	1.77%	2760704.0
76	layer3.2.bn2	256 14 14	256 14 14	512.0	0.19	200,704.0	100,352.0	202752.0	200704.0	0.04%	403456.0
77	layer3.2.conv3	256 14 14	1024 14 14	262144.0	0.77	102,559,744.0	51,380,224.0	1249280.0	802816.0	1.10%	2052096.0
78	layer3.2.bn3	1024 14 14	1024 14 14	2048.0	0.77	802,816.0	401,408.0	811008.0	802816.0	0.12%	1613824.0
79	layer3.2.relu	1024 14 14	1024 14 14	0.0	0.77	200,704.0	200,704.0	802816.0	802816.0	0.07%	1605632.0
80	layer3.3.conv1	1024 14 14	256 14 14	262144.0	0.19	102,710,272.0	51,380,224.0	1851392.0	200704.0	1.08%	2052096.0
81	layer3.3.bn1	256 14 14	256 14 14	512.0	0.19	200,704.0	100,352.0	202752.0	200704.0	0.03%	403456.0
82	layer3.3.conv2	256 14 14	256 14 14	589824.0	0.19	231,160,832.0	115,605,504.0	2560000.0	200704.0	1.71%	2760704.0
83	layer3.3.bn2	256 14 14	256 14 14	512.0	0.19	200,704.0	100,352.0	202752.0	200704.0	0.03%	403456.0
84	layer3.3.conv3	256 14 14	1024 14 14	262144.0	0.77	102,559,744.0	51,380,224.0	1249280.0	802816.0	1.08%	2052096.0
85	layer3.3.bn3	1024 14 14	1024 14 14	2048.0	0.77	802,816.0	401,408.0	811008.0	802816.0	0.11%	1613824.0
86	layer3.3.relu	1024 14 14	1024 14 14	0.0	0.77	200,704.0	200,704.0	802816.0	802816.0	0.08%	1605632.0
87	layer3.4.conv1	1024 14 14	256 14 14	262144.0	0.19	102,710,272.0	51,380,224.0	1851392.0	200704.0	1.11%	2052096.0
88	layer3.4.bn1	256 14 14	256 14 14	512.0	0.19	200,704.0	100,352.0	202752.0	200704.0	0.04%	403456.0
89	layer3.4.conv2	256 14 14	256 14 14	589824.0	0.19	231,160,832.0	115,605,504.0	2560000.0	200704.0	1.85%	2760704.0
90	layer3.4.bn2	256 14 14	256 14 14	512.0	0.19	200,704.0	100,352.0	202752.0	200704.0	0.03%	403456.0
91	layer3.4.conv3	256 14 14	1024 14 14	262144.0	0.77	102,559,744.0	51,380,224.0	1249280.0	802816.0	0.95%	2052096.0
92	layer3.4.bn3	1024 14 14	1024 14 14	2048.0	0.77	802,816.0	401,408.0	811008.0	802816.0	0.09%	1613824.0
93	layer3.4.relu	1024 14 14	1024 14 14	0.0	0.77	200,704.0	200,704.0	802816.0	802816.0	0.08%	1605632.0
94	layer3.5.conv1	1024 14 14	256 14 14	262144.0	0.19	102,710,272.0	51,380,224.0	1851392.0	200704.0	1.01%	2052096.0
95	layer3.5.bn1	256 14 14	256 14 14	512.0	0.19	200,704.0	100,352.0	202752.0	200704.0	0.03%	403456.0
96	layer3.5.conv2	256 14 14	256 14 14	589824.0	0.19	231,160,832.0	115,605,504.0	2560000.0	200704.0	1.72%	2760704.0
97	layer3.5.bn2	256 14 14	256 14 14	512.0	0.19	200,704.0	100,352.0	202752.0	200704.0	0.03%	403456.0
98	layer3.5.conv3	256 14 14	1024 14 14	262144.0	0.77	102,559,744.0	51,380,224.0	1249280.0	802816.0	0.90%	2052096.0
99	layer3.5.bn3	1024 14 14	1024 14 14	2048.0	0.77	802,816.0	401,408.0	811008.0	802816.0	0.09%	1613824.0
100	layer3.5.relu	1024 14 14	1024 14 14	0.0	0.77	200,704.0	200,704.0	802816.0	802816.0	0.07%	1605632.0
101	layer4.0.conv1	1024 14 14	512 14 14	524288.0	0.38	205,420,544.0	102,760,448.0	2899968.0	401408.0	1.98%	3301376.0
102	layer4.0.bn1	512 14 14	512 14 14	1024.0	0.38	401,408.0	200,704.0	405504.0	401408.0	0.06%	806912.0
103	layer4.0.conv2	512 14 14	512 7 7	2359296.0	0.10	231,185,920.0	115,605,504.0	9838592.0	100352.0	3.46%	9938944.0
104	layer4.0.bn2	512 7 7	512 7 7	1024.0	0.10	100,352.0	50,176.0	104448.0	100352.0	0.03%	204800.0
105	layer4.0.conv3	512 7 7	2048 7 7	1048576.0	0.38	102,660,096.0	51,380,224.0	4294656.0	401408.0	1.22%	4696064.0
106	layer4.0.bn3	2048 7 7	2048 7 7	4096.0	0.38	401,408.0	200,704.0	417792.0	401408.0	0.06%	819200.0
107	layer4.0.relu	2048 7 7	2048 7 7	0.0	0.38	100,352.0	100,352.0	401408.0	401408.0	0.05%	802816.0
108	layer4.0.downsample.0	1024 14 14	2048 7 7	2097152.0	0.38	205,420,544.0	102,760,448.0	9191424.0	401408.0	2.23%	9592832.0
109	layer4.0.downsample.1	2048 7 7	2048 7 7	4096.0	0.38	401,408.0	200,704.0	417792.0	401408.0	0.07%	819200.0
110	layer4.1.conv1	2048 7 7	512 7 7	1048576.0	0.10	102,735,360.0	51,380,224.0	4595712.0	100352.0	1.17%	4696064.0
111	layer4.1.bn1	512 7 7	512 7 7	1024.0	0.10	100,352.0	50,176.0	104448.0	100352.0	0.03%	204800.0
112	layer4.1.conv2	512 7 7	512 7 7	2359296.0	0.10	231,185,920.0	115,605,504.0	9537536.0	100352.0	2.39%	9637888.0
113	layer4.1.bn2	512 7 7	512 7 7	1024.0	0.10	100,352.0	50,176.0	104448.0	100352.0	0.03%	204800.0
114	layer4.1.conv3	512 7 7	2048 7 7	1048576.0	0.38	102,660,096.0	51,380,224.0	4294656.0	401408.0	1.30%	4696064.0
115	layer4.1.bn3	2048 7 7	2048 7 7	4096.0	0.38	401,408.0	200,704.0	417792.0	401408.0	0.07%	819200.0
116	layer4.1.relu	2048 7 7	2048 7 7	0.0	0.38	100,352.0	100,352.0	401408.0	401408.0	0.05%	802816.0
117	layer4.2.conv1	2048 7 7	512 7 7	1048576.0	0.10	102,735,360.0	51,380,224.0	4595712.0	100352.0	1.24%	4696064.0
118	layer4.2.bn1	512 7 7	512 7 7	1024.0	0.10	100,352.0	50,176.0	104448.0	100352.0	0.03%	204800.0
119	layer4.2.conv2	512 7 7	512 7 7	2359296.0	0.10	231,185,920.0	115,605,504.0	9537536.0	100352.0	2.34%	9637888.0
120	layer4.2.bn2	512 7 7	512 7 7	1024.0	0.10	100,352.0	50,176.0	104448.0	100352.0	0.04%	204800.0
121	layer4.2.conv3	512 7 7	2048 7 7	1048576.0	0.38	102,660,096.0	51,380,224.0	4294656.0	401408.0	1.43%	4696064.0
122	layer4.2.bn3	2048 7 7	2048 7 7	4096.0	0.38	401,408.0	200,704.0	417792.0	401408.0	0.07%	819200.0
123	layer4.2.relu	2048 7 7	2048 7 7	0.0	0.38	100,352.0	100,352.0	401408.0	401408.0	0.06%	802816.0
124	avgpool	2048 7 7	2048 1 1	0.0	0.01	0.0	0.0	0.0	0.0	2.72%	0.0
125	fc	2048	1000	2049000.0	0.00	4,095,000.0	2,048,000.0	8204192.0	4000.0	0.42%	8208192.0
total				25557032.0	109.69	8,219,637,272.0	4,118,537,216.0	8204192.0	4000.0	100.00%	332849216.0
```

## 画像の推論
ImageNetの1000クラス分けデータを基に、推論するスクリプトが提供されている。それを基にして説明する。
ここでは、`imagenet_utils`クラスが肝でimagenetの分類がスムーズにできることがわかる。また、saliency(特徴量とでも言うべき?)によるプロットも特徴が見えることがわかると思う。
[Prediction Explainers for Convolutional Neural Networks](https://github.com/microsoft/tensorwatch/blob/master/notebooks/cnn_pred_explain.ipynb)

```
%matplotlib inline
!pip install tensorwatch
# this is just to get data folder
!git clone http://github.com/microsoft/tensorwatch twatch
from tensorwatch.saliency import saliency
from tensorwatch import image_utils, imagenet_utils, pytorch_utils
model = pytorch_utils.get_model('resnet50')
img = image_utils.open_image('twatch/data/test_images/dogs.png', convert_mode='RGB')
image_utils.show_image(img)
probabilities = imagenet_utils.predict(model=model, images=[img])
imagenet_utils.probabilities2classes(probabilities, topk=5)
input_tensor = imagenet_utils.image2batch(img)
prediction_tensor = pytorch_utils.int2tensor(239)
results = saliency.get_image_saliency_results(model, img, input_tensor, prediction_tensor)
figure = saliency.get_image_saliency_plot(results)
```
### 参考資料
- [yulongwang12/visual-attribution](https://github.com/yulongwang12/visual-attribution)