Repository: catalyst-team/catalyst

Issue:
[fix] bin/check_dl.sh CUDA errors
How to reproduce:

Use a system with a GPU, run:

./bin/check_dl.sh 

You will get some errors. Using CPU there are no such errors

```
Requirement already satisfied: tifffile in /home/light/anaconda3/lib/python3.7/site-packages (2019.7.20)
Requirement already satisfied: numpy>=1.11.3 in /home/light/anaconda3/lib/python3.7/site-packages (from tifffile) (1.16.4)
--2019-08-07 23:07:24--  https://www.dropbox.com/s/0rvuae4mj6jn922/isbi.tar.gz
Resolving www.dropbox.com (www.dropbox.com)... 162.125.70.1, 2620:100:6026:1::a27d:4601
Connecting to www.dropbox.com (www.dropbox.com)|162.125.70.1|:443... connected.
HTTP request sent, awaiting response... 301 Moved Permanently
Location: /s/raw/0rvuae4mj6jn922/isbi.tar.gz [following]
--2019-08-07 23:07:24--  https://www.dropbox.com/s/raw/0rvuae4mj6jn922/isbi.tar.gz
Reusing existing connection to www.dropbox.com:443.
HTTP request sent, awaiting response... 302 Found
Location: https://uce9c445554664bbae3228e82f2a.dl.dropboxusercontent.com/cd/0/inline/AmPZ6XZfQQhKLK6zX_F3qOaAdJ95pbJizbRK0EeP2gRgCDx0OsEsK293DEh6IJWO28AG3CNIdccNAEpxPuvKMhhjJFKkDuPyb-bpYLbyRDiDtQ/file# [following]
--2019-08-07 23:07:25--  https://uce9c445554664bbae3228e82f2a.dl.dropboxusercontent.com/cd/0/inline/AmPZ6XZfQQhKLK6zX_F3qOaAdJ95pbJizbRK0EeP2gRgCDx0OsEsK293DEh6IJWO28AG3CNIdccNAEpxPuvKMhhjJFKkDuPyb-bpYLbyRDiDtQ/file
Resolving uce9c445554664bbae3228e82f2a.dl.dropboxusercontent.com (uce9c445554664bbae3228e82f2a.dl.dropboxusercontent.com)... 162.125.70.6, 2620:100:6026:6::a27d:4606
Connecting to uce9c445554664bbae3228e82f2a.dl.dropboxusercontent.com (uce9c445554664bbae3228e82f2a.dl.dropboxusercontent.com)|162.125.70.6|:443... connected.
HTTP request sent, awaiting response... 302 FOUND
Location: /cd/0/inline2/AmN0KQ8ekqu5YoE2OJMwdy_Tmp6Yw8AAzv2usYp8HsAovt1Wnm3VZR4lbkLPLaxpd258EScT4QpRWdBVdoosJJn9wT30nDrTcaHCx0LdJAeI1XtLyKMeQffxeL3eVw2OwUBH0l9aKQwEqPNyIa7oosLYPN9jtK2dpwgzv6eIhQ973kauHagpqMW5otE3d9I0gazGshRPPKfArXsPEGHxvUYxREpe77IDhCSSIAm1a--bwa1LABsFEe1IP1n382-j64sqM1Js3e2rZH1fQpQO69BVtzkEuN9tq_6BeBtAlL5FAajs0Kn8L1vUZKZ1DvHWjK-QbGsgwzRCscq_vLST8sQd/file [following]
--2019-08-07 23:07:26--  https://uce9c445554664bbae3228e82f2a.dl.dropboxusercontent.com/cd/0/inline2/AmN0KQ8ekqu5YoE2OJMwdy_Tmp6Yw8AAzv2usYp8HsAovt1Wnm3VZR4lbkLPLaxpd258EScT4QpRWdBVdoosJJn9wT30nDrTcaHCx0LdJAeI1XtLyKMeQffxeL3eVw2OwUBH0l9aKQwEqPNyIa7oosLYPN9jtK2dpwgzv6eIhQ973kauHagpqMW5otE3d9I0gazGshRPPKfArXsPEGHxvUYxREpe77IDhCSSIAm1a--bwa1LABsFEe1IP1n382-j64sqM1Js3e2rZH1fQpQO69BVtzkEuN9tq_6BeBtAlL5FAajs0Kn8L1vUZKZ1DvHWjK-QbGsgwzRCscq_vLST8sQd/file
Reusing existing connection to uce9c445554664bbae3228e82f2a.dl.dropboxusercontent.com:443.
HTTP request sent, awaiting response... 200 OK
Length: 23616000 (23M) [application/octet-stream]
Saving to: ‘./data/isbi.tar.gz.5’

isbi.tar.gz.5                                                       100%[==================================================================================================================================================================>]  22,52M  22,1MB/s    in 1,0s    

2019-08-07 23:07:28 (22,1 MB/s) - ‘./data/isbi.tar.gz.5’ saved [23616000/23616000]

[2019-08-07 23:07:50,216] 
0/2 * Epoch 0 (train): _base/lr=0.0010 | _base/momentum=0.9000 | _timers/_fps=3678.8027 | _timers/batch_time=0.0590 | _timers/data_time=0.0132 | _timers/model_time=0.0458 | loss=2.2888
0/2 * Epoch 0 (valid): _base/lr=0.0010 | _base/momentum=0.9000 | _timers/_fps=6024.3511 | _timers/batch_time=0.0053 | _timers/data_time=0.0049 | _timers/model_time=0.0004 | loss=2.1449
[2019-08-07 23:07:50,313] 
1/2 * Epoch 1 (train): _base/lr=0.0010 | _base/momentum=0.9000 | _timers/_fps=5256.5596 | _timers/batch_time=0.0065 | _timers/data_time=0.0060 | _timers/model_time=0.0005 | loss=2.0949
1/2 * Epoch 1 (valid): _base/lr=0.0010 | _base/momentum=0.9000 | _timers/_fps=6083.2832 | _timers/batch_time=0.0053 | _timers/data_time=0.0049 | _timers/model_time=0.0004 | loss=1.8466
Top best models:
logs/cifar_simple_notebook_1/checkpoints//train.1.pth	1.8466
logs/cifar_simple_notebook_1/checkpoints//train.0.pth	2.1449
[2019-08-07 23:07:50,417] 
0/2 * Epoch 0 (train): _base/lr=0.0010 | _base/momentum=0.9000 | _timers/_fps=5709.2818 | _timers/batch_time=0.0056 | _timers/data_time=0.0051 | _timers/model_time=0.0005 | loss=2.1756
0/2 * Epoch 0 (valid): _base/lr=0.0010 | _base/momentum=0.9000 | _timers/_fps=6170.8778 | _timers/batch_time=0.0052 | _timers/data_time=0.0048 | _timers/model_time=0.0004 | loss=2.1189
[2019-08-07 23:07:50,521] 
1/2 * Epoch 1 (train): _base/lr=0.0010 | _base/momentum=0.9000 | _timers/_fps=5351.3396 | _timers/batch_time=0.0063 | _timers/data_time=0.0057 | _timers/model_time=0.0005 | loss=1.8193
1/2 * Epoch 1 (valid): _base/lr=0.0010 | _base/momentum=0.9000 | _timers/_fps=5847.9100 | _timers/batch_time=0.0055 | _timers/data_time=0.0051 | _timers/model_time=0.0004 | loss=1.4512
Top best models:
logs/cifar_simple_notebook_2/checkpoints//train.1.pth	1.4512
logs/cifar_simple_notebook_2/checkpoints//train.0.pth	2.1189
[2019-08-07 23:07:50,625] 
0/2 * Epoch 0 (train): _base/lr=0.0010 | _base/momentum=0.9000 | _timers/_fps=5379.7222 | _timers/batch_time=0.0061 | _timers/data_time=0.0056 | _timers/model_time=0.0005 | loss=2.2873
0/2 * Epoch 0 (valid): _base/lr=0.0010 | _base/momentum=0.9000 | _timers/_fps=5893.3593 | _timers/batch_time=0.0055 | _timers/data_time=0.0050 | _timers/model_time=0.0004 | loss=2.1802
[2019-08-07 23:07:50,737] 
1/2 * Epoch 1 (train): _base/lr=0.0010 | _base/momentum=0.9000 | _timers/_fps=5212.7790 | _timers/batch_time=0.0064 | _timers/data_time=0.0059 | _timers/model_time=0.0005 | loss=2.1101
1/2 * Epoch 1 (valid): _base/lr=0.0010 | _base/momentum=0.9000 | _timers/_fps=6131.4554 | _timers/batch_time=0.0052 | _timers/data_time=0.0048 | _timers/model_time=0.0004 | loss=1.8655
Top best models:
logs/cifar_simple_notebook_3/checkpoints//train.1.pth	1.8655
logs/cifar_simple_notebook_3/checkpoints//train.0.pth	2.1802
[2019-08-07 23:07:50,833] 
0/2 * Epoch 0 (train): _base/lr=0.0010 | _base/momentum=0.9000 | _timers/_fps=5694.1712 | _timers/batch_time=0.0056 | _timers/data_time=0.0052 | _timers/model_time=0.0005 | accuracy01=11.7188 | accuracy03=42.9688 | accuracy05=64.8438 | loss=2.2828
0/2 * Epoch 0 (valid): _base/lr=0.0010 | _base/momentum=0.9000 | _timers/_fps=5766.3016 | _timers/batch_time=0.0056 | _timers/data_time=0.0051 | _timers/model_time=0.0004 | accuracy01=33.5938 | accuracy03=55.4688 | accuracy05=74.2188 | loss=2.1173
[2019-08-07 23:07:50,937] 
1/2 * Epoch 1 (train): _base/lr=0.0010 | _base/momentum=0.9000 | _timers/_fps=4760.2633 | _timers/batch_time=0.0069 | _timers/data_time=0.0063 | _timers/model_time=0.0005 | accuracy01=39.0625 | accuracy03=71.0938 | accuracy05=84.3750 | loss=1.9457
1/2 * Epoch 1 (valid): _base/lr=0.0010 | _base/momentum=0.9000 | _timers/_fps=6014.2628 | _timers/batch_time=0.0053 | _timers/data_time=0.0049 | _timers/model_time=0.0004 | accuracy01=60.9375 | accuracy03=75.0000 | accuracy05=78.1250 | loss=1.6803
Top best models:
logs/cifar_simple_notebook_4/checkpoints//train.1.pth	1.6803
logs/cifar_simple_notebook_4/checkpoints//train.0.pth	2.1173
[2019-08-07 23:07:51,041] 
0/2 * Epoch 0 (train): _base/lr=0.0010 | _base/momentum=0.8998 | _timers/_fps=5139.8662 | _timers/batch_time=0.0063 | _timers/data_time=0.0058 | _timers/model_time=0.0005 | accuracy01=19.5312 | accuracy03=38.2812 | accuracy05=54.6875 | loss=2.2490
0/2 * Epoch 0 (valid): _base/lr=0.0010 | _base/momentum=0.8995 | _timers/_fps=6099.3660 | _timers/batch_time=0.0052 | _timers/data_time=0.0048 | _timers/model_time=0.0004 | accuracy01=47.6562 | accuracy03=76.5625 | accuracy05=91.4062 | loss=2.0199
[2019-08-07 23:07:51,173] 
1/2 * Epoch 1 (train): _base/lr=0.0030 | _base/momentum=0.8748 | _timers/_fps=5050.9584 | _timers/batch_time=0.0065 | _timers/data_time=0.0059 | _timers/model_time=0.0005 | accuracy01=46.8750 | accuracy03=69.5312 | accuracy05=87.5000 | loss=1.8066
1/2 * Epoch 1 (valid): _base/lr=0.0030 | _base/momentum=0.8744 | _timers/_fps=6097.5967 | _timers/batch_time=0.0052 | _timers/data_time=0.0048 | _timers/model_time=0.0004 | accuracy01=60.1562 | accuracy03=80.4688 | accuracy05=91.4062 | loss=1.2285
Top best models:
logs/cifar_simple_notebook_5/checkpoints//train.1.pth	1.2285
logs/cifar_simple_notebook_5/checkpoints//train.0.pth	2.0199
[2019-08-07 23:07:51,256] 
0/2 * Epoch 0 (train): _base/lr=0.0010 | _base/momentum=0.9000 | _timers/_fps=5737.8774 | _timers/batch_time=0.0056 | _timers/data_time=0.0051 | _timers/model_time=0.0005 | accuracy01=25.0000 | accuracy03=47.6562 | accuracy05=66.4062 | loss=2.2039
[2019-08-07 23:07:51,352] 
1/2 * Epoch 1 (train): _base/lr=0.0010 | _base/momentum=0.9000 | _timers/_fps=5270.2953 | _timers/batch_time=0.0064 | _timers/data_time=0.0058 | _timers/model_time=0.0005 | accuracy01=57.0312 | accuracy03=79.6875 | accuracy05=87.5000 | loss=1.8034
Top best models:
logs/cifar_simple_notebook_6/checkpoints//train.1.pth	1.8034
logs/cifar_simple_notebook_6/checkpoints//train.0.pth	2.2039
[2019-08-07 23:07:51,457] 
0/2 * Epoch 0 (train): _base/lr=0.0010 | _base/momentum=0.9000 | _timers/_fps=5820.0063 | _timers/batch_time=0.0055 | _timers/data_time=0.0050 | _timers/model_time=0.0005 | loss=2.2627
0/2 * Epoch 0 (valid): _base/lr=0.0010 | _base/momentum=0.9000 | _timers/_fps=6088.4248 | _timers/batch_time=0.0053 | _timers/data_time=0.0048 | _timers/model_time=0.0004 | loss=2.0988
[2019-08-07 23:07:51,561] 
1/2 * Epoch 1 (train): _base/lr=0.0010 | _base/momentum=0.9000 | _timers/_fps=5292.7394 | _timers/batch_time=0.0063 | _timers/data_time=0.0057 | _timers/model_time=0.0005 | loss=1.9728
1/2 * Epoch 1 (valid): _base/lr=0.0010 | _base/momentum=0.9000 | _timers/_fps=6148.9115 | _timers/batch_time=0.0052 | _timers/data_time=0.0048 | _timers/model_time=0.0004 | loss=1.6786
Top best models:
logs/cifar_simple_notebook_7/checkpoints//train.1.pth	1.6786
logs/cifar_simple_notebook_7/checkpoints//train.0.pth	2.0988
[2019-08-07 23:07:51,717] 
0/2 * Epoch 0 (train): _base/lr=0.0010 | _base/momentum=0.9000 | _timers/_fps=5410.2858 | _timers/batch_time=0.0062 | _timers/data_time=0.0057 | _timers/model_time=0.0005 | accuracy01=20.3125 | accuracy03=50.0000 | accuracy05=62.5000 | loss=2.2170
0/2 * Epoch 0 (valid): _base/lr=0.0010 | _base/momentum=0.9000 | _timers/_fps=6164.8603 | _timers/batch_time=0.0052 | _timers/data_time=0.0048 | _timers/model_time=0.0004 | accuracy01=45.3125 | accuracy03=65.6250 | accuracy05=90.6250 | loss=1.9870
[2019-08-07 23:07:51,867] 
1/2 * Epoch 1 (train): _base/lr=0.0010 | _base/momentum=0.9000 | _timers/_fps=5245.8043 | _timers/batch_time=0.0063 | _timers/data_time=0.0058 | _timers/model_time=0.0005 | accuracy01=40.6250 | accuracy03=65.6250 | accuracy05=78.1250 | loss=1.9946
1/2 * Epoch 1 (valid): _base/lr=0.0010 | _base/momentum=0.9000 | _timers/_fps=5992.1501 | _timers/batch_time=0.0053 | _timers/data_time=0.0049 | _timers/model_time=0.0004 | accuracy01=54.6875 | accuracy03=78.9062 | accuracy05=97.6562 | loss=1.7162
Top best models:
logs/cifar_simple_notebook_8/checkpoints//train.1.pth	1.7162
logs/cifar_simple_notebook_8/checkpoints//train.0.pth	1.9870
[2019-08-07 23:07:56,207] 
0/2 * Epoch 3 (train): _base/lr=0.0500 | _base/momentum=0.000000E+00 | _timers/_fps=5648.2374 | _timers/batch_time=0.0057 | _timers/data_time=0.0052 | _timers/model_time=0.0005 | loss=0.3032
0/2 * Epoch 3 (valid): _base/lr=0.0500 | _base/momentum=0.000000E+00 | _timers/_fps=5846.1501 | _timers/batch_time=0.0055 | _timers/data_time=0.0051 | _timers/model_time=0.0004 | loss=0.0995
[2019-08-07 23:08:00,496] 
1/2 * Epoch 4 (train): _base/lr=0.0500 | _base/momentum=0.000000E+00 | _timers/_fps=5666.6164 | _timers/batch_time=0.0057 | _timers/data_time=0.0052 | _timers/model_time=0.0005 | loss=0.0943
1/2 * Epoch 4 (valid): _base/lr=0.0500 | _base/momentum=0.000000E+00 | _timers/_fps=5949.3689 | _timers/batch_time=0.0054 | _timers/data_time=0.0050 | _timers/model_time=0.0004 | loss=0.0566
Top best models:
logs/cifar_simple_notebook_8/checkpoints//train.4.pth	0.0566
logs/cifar_simple_notebook_8/checkpoints//train.3.pth	0.0995
=> loading checkpoint ./logs/cifar_simple_notebook_8/checkpoints/best.pth
loaded checkpoint ./logs/cifar_simple_notebook_8/checkpoints/best.pth (epoch 4)
Top best models:

Traceback (most recent call last):
  File "examples/_tests_scripts/z_classification.py", line 440, in <module>
    runner_out = runner.predict_batch(runner_in)
  File "/home/light/projects/catalyst/catalyst/dl/runner/supervised.py", line 86, in predict_batch
    output = self._process_input(batch)
  File "/home/light/projects/catalyst/catalyst/dl/runner/supervised.py", line 60, in _process_input_str
    output = self.model(batch[self.input_key])
  File "/home/light/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 493, in __call__
    result = self.forward(*input, **kwargs)
  File "examples/_tests_scripts/z_classification.py", line 59, in forward
    x = F.relu(self.conv1(x))
  File "/home/light/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 493, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/light/anaconda3/lib/python3.7/site-packages/torch/nn/modules/conv.py", line 338, in forward
    self.padding, self.dilation, self.groups)
RuntimeError: Input type (torch.FloatTensor) and weight type (torch.cuda.FloatTensor) should be the same
[2019-08-07 23:08:06,821] 
0/3 * Epoch 0 (train): _base/lr=0.0011 | _base/momentum=0.8992 | _timers/_fps=4275.1204 | _timers/batch_time=0.0674 | _timers/data_time=0.0174 | _timers/model_time=0.0499 | accuracy01=15.6250 | accuracy03=40.6250 | accuracy05=64.8438 | loss=2.2523
0/3 * Epoch 0 (valid): _base/lr=0.0012 | _base/momentum=0.8979 | _timers/_fps=6762.5958 | _timers/batch_time=0.0095 | _timers/data_time=0.0090 | _timers/model_time=0.0004 | accuracy01=31.6406 | accuracy03=74.6094 | accuracy05=90.2344 | loss=2.0553
[2019-08-07 23:08:06,957] 
1/3 * Epoch 1 (train): _base/lr=0.0050 | _base/momentum=0.8508 | _timers/_fps=6347.2345 | _timers/batch_time=0.0101 | _timers/data_time=0.0095 | _timers/model_time=0.0005 | accuracy01=46.4844 | accuracy03=80.8594 | accuracy05=92.9688 | loss=2.1746
1/3 * Epoch 1 (valid): _base/lr=0.0049 | _base/momentum=0.8521 | _timers/_fps=6714.0664 | _timers/batch_time=0.0095 | _timers/data_time=0.0091 | _timers/model_time=0.0004 | accuracy01=58.9844 | accuracy03=80.4688 | accuracy05=90.6250 | loss=1.5222
[2019-08-07 23:08:07,087] 
2/3 * Epoch 2 (train): _base/lr=0.0025 | _base/momentum=0.9010 | _timers/_fps=6031.5264 | _timers/batch_time=0.0110 | _timers/data_time=0.0104 | _timers/model_time=0.0006 | accuracy01=51.9531 | accuracy03=76.5625 | accuracy05=85.9375 | loss=1.7330
2/3 * Epoch 2 (valid): _base/lr=0.0024 | _base/momentum=0.9022 | _timers/_fps=6699.3547 | _timers/batch_time=0.0096 | _timers/data_time=0.0091 | _timers/model_time=0.0004 | accuracy01=50.0000 | accuracy03=76.9531 | accuracy05=85.9375 | loss=1.6718
Top best models:
examples/logs/_tests_mnist_stages1/checkpoints//stage1.1.pth	58.9844
examples/logs/_tests_mnist_stages1/checkpoints//stage1.2.pth	50.0000
examples/logs/_tests_mnist_stages1/checkpoints//stage1.0.pth	31.6406
Load config
Import experiment and runner from logdir
Load model state from checkpoints/best.pth
Tracing
Done
[2019-08-07 23:08:13,979] 
0/3 * Epoch 0 (train): _base/lr=0.0010 | _base/momentum=0.9000 | _timers/_fps=4676.5092 | _timers/batch_time=0.0665 | _timers/data_time=0.0175 | _timers/model_time=0.0489 | accuracy01=15.6250 | accuracy03=40.6250 | accuracy05=64.4531 | loss=2.2529
0/3 * Epoch 0 (valid): _base/lr=0.0010 | _base/momentum=0.9000 | _timers/_fps=6678.3284 | _timers/batch_time=0.0096 | _timers/data_time=0.0091 | _timers/model_time=0.0004 | accuracy01=30.0781 | accuracy03=73.0469 | accuracy05=89.0625 | loss=2.0645
[2019-08-07 23:08:14,185] 
1/3 * Epoch 1 (train): _base/lr=9.000000E-05 | _base/momentum=0.9000 | _timers/_fps=5634.8328 | _timers/batch_time=0.0116 | _timers/data_time=0.0110 | _timers/model_time=0.0006 | accuracy01=48.4375 | accuracy03=83.5938 | accuracy05=93.7500 | loss=1.9684
1/3 * Epoch 1 (valid): _base/lr=9.000000E-05 | _base/momentum=0.9000 | _timers/_fps=6101.5961 | _timers/batch_time=0.0107 | _timers/data_time=0.0102 | _timers/model_time=0.0005 | accuracy01=61.7188 | accuracy03=86.3281 | accuracy05=94.9219 | loss=1.9055
[2019-08-07 23:08:14,324] 
2/3 * Epoch 2 (train): _base/lr=0.0003 | _base/momentum=0.9000 | _timers/_fps=6302.1293 | _timers/batch_time=0.0102 | _timers/data_time=0.0095 | _timers/model_time=0.0007 | accuracy01=64.0625 | accuracy03=91.4062 | accuracy05=95.7031 | loss=1.7676
2/3 * Epoch 2 (valid): _base/lr=0.0003 | _base/momentum=0.9000 | _timers/_fps=6557.4463 | _timers/batch_time=0.0098 | _timers/data_time=0.0093 | _timers/model_time=0.0005 | accuracy01=61.7188 | accuracy03=92.9688 | accuracy05=98.0469 | loss=1.6734
Top best models:
examples/logs/_tests_mnist_stages1/checkpoints//stage1.1.pth	61.7188
examples/logs/_tests_mnist_stages1/checkpoints//stage1.2.pth	61.7188
examples/logs/_tests_mnist_stages1/checkpoints//stage1.0.pth	30.0781
=> loading checkpoint ./examples/logs/_tests_mnist_stages1/checkpoints/best.pth
loaded checkpoint ./examples/logs/_tests_mnist_stages1/checkpoints/best.pth (epoch 1)
Top best models:

[2019-08-07 23:08:24,994] 
0/3 * Epoch 0 (train): _base/lr=0.0005 | _base/momentum=0.9000 | _timers/_fps=4094.0868 | _timers/batch_time=0.0598 | _timers/data_time=0.0119 | _timers/model_time=0.0478 | accuracy01=15.6250 | accuracy03=31.2500 | accuracy05=58.5938 | loss=2.2866
0/3 * Epoch 0 (valid): _base/lr=0.0005 | _base/momentum=0.9000 | _timers/_fps=5859.1346 | _timers/batch_time=0.0055 | _timers/data_time=0.0050 | _timers/model_time=0.0004 | accuracy01=15.6250 | accuracy03=45.3125 | accuracy05=63.2812 | loss=2.2147
[2019-08-07 23:08:25,130] 
1/3 * Epoch 1 (train): _base/lr=4.500000E-05 | _base/momentum=0.9000 | _timers/_fps=5520.1767 | _timers/batch_time=0.0058 | _timers/data_time=0.0052 | _timers/model_time=0.0005 | accuracy01=8.5938 | accuracy03=50.7812 | accuracy05=65.6250 | loss=2.2129
1/3 * Epoch 1 (valid): _base/lr=4.500000E-05 | _base/momentum=0.9000 | _timers/_fps=5909.5534 | _timers/batch_time=0.0054 | _timers/data_time=0.0050 | _timers/model_time=0.0004 | accuracy01=19.5312 | accuracy03=45.3125 | accuracy05=72.6562 | loss=2.1596
[2019-08-07 23:08:25,223] 
2/3 * Epoch 2 (train): _base/lr=0.0001 | _base/momentum=0.9000 | _timers/_fps=5156.8659 | _timers/batch_time=0.0066 | _timers/data_time=0.0060 | _timers/model_time=0.0005 | accuracy01=18.7500 | accuracy03=52.3438 | accuracy05=74.2188 | loss=2.1521
2/3 * Epoch 2 (valid): _base/lr=0.0001 | _base/momentum=0.9000 | _timers/_fps=6056.8775 | _timers/batch_time=0.0053 | _timers/data_time=0.0049 | _timers/model_time=0.0004 | accuracy01=34.3750 | accuracy03=60.9375 | accuracy05=78.9062 | loss=2.0960
Top best models:
examples/logs/_tests_mnist_stages1/checkpoints//stage1.2.pth	34.3750
examples/logs/_tests_mnist_stages1/checkpoints//stage1.1.pth	19.5312
examples/logs/_tests_mnist_stages1/checkpoints//stage1.0.pth	15.6250
[2019-08-07 23:08:29,696] 
0/2 * Epoch 0 (train): _base/lr=0.0010 | _base/momentum=0.9000 | _timers/_fps=4191.0048 | _timers/batch_time=0.0682 | _timers/data_time=0.0179 | _timers/model_time=0.0503 | accuracy01=15.6250 | accuracy03=40.6250 | accuracy05=64.4531 | loss=2.2529
0/2 * Epoch 0 (valid): _base/lr=0.0010 | _base/momentum=0.9000 | _timers/_fps=6103.5708 | _timers/batch_time=0.0107 | _timers/data_time=0.0102 | _timers/model_time=0.0005 | accuracy01=30.0781 | accuracy03=73.0469 | accuracy05=89.0625 | loss=2.0645
[2019-08-07 23:08:29,887] 
1/2 * Epoch 1 (train): _base/lr=9.000000E-05 | _base/momentum=0.9000 | _timers/_fps=6375.5347 | _timers/batch_time=0.0100 | _timers/data_time=0.0095 | _timers/model_time=0.0006 | accuracy01=48.4375 | accuracy03=83.5938 | accuracy05=93.7500 | loss=1.9684
1/2 * Epoch 1 (valid): _base/lr=9.000000E-05 | _base/momentum=0.9000 | _timers/_fps=6787.8846 | _timers/batch_time=0.0094 | _timers/data_time=0.0090 | _timers/model_time=0.0004 | accuracy01=61.7188 | accuracy03=86.3281 | accuracy05=94.9219 | loss=1.9055
Top best models:
examples/logs/_tests_mnist_stages1/checkpoints//stage1.1.pth	61.7188
examples/logs/_tests_mnist_stages1/checkpoints//stage1.0.pth	30.0781
Traceback (most recent call last):
  File "catalyst/dl/scripts/run.py", line 79, in <module>
    main(args, unknown_args)
  File "catalyst/dl/scripts/run.py", line 74, in main
    runner.run_experiment(experiment, check=args.check)
  File "/home/light/projects/catalyst/catalyst/dl/core/runner.py", line 196, in run_experiment
    self._run_event("exception")
  File "/home/light/projects/catalyst/catalyst/dl/core/runner.py", line 96, in _run_event
    getattr(self.state, f"on_{event}_post")()
  File "/home/light/projects/catalyst/catalyst/dl/core/state.py", line 172, in on_exception_post
    logger.on_exception(self)
  File "/home/light/projects/catalyst/catalyst/dl/callbacks/logging.py", line 183, in on_exception
    raise exception
  File "/home/light/projects/catalyst/catalyst/dl/core/runner.py", line 193, in run_experiment
    self._run_stage(stage)
  File "/home/light/projects/catalyst/catalyst/dl/core/runner.py", line 175, in _run_stage
    self._run_epoch(loaders)
  File "/home/light/projects/catalyst/catalyst/dl/core/runner.py", line 162, in _run_epoch
    self._run_loader(loader)
  File "/home/light/projects/catalyst/catalyst/dl/core/runner.py", line 131, in _run_loader
    self._run_batch(batch)
  File "/home/light/projects/catalyst/catalyst/dl/core/runner.py", line 113, in _run_batch
    self._run_event("batch_end")
  File "/home/light/projects/catalyst/catalyst/dl/core/runner.py", line 93, in _run_event
    getattr(callback, f"on_{event}")(self.state)
  File "/home/light/projects/catalyst/catalyst/dl/callbacks/optimizer.py", line 106, in on_batch_end
    grad_clip_fn=self.grad_clip_fn
  File "/home/light/projects/catalyst/catalyst/dl/callbacks/optimizer.py", line 46, in grad_step
    optimizer.step()
  File "/home/light/anaconda3/lib/python3.7/site-packages/torch/optim/adam.py", line 93, in step
    exp_avg.mul_(beta1).add_(1 - beta1, grad)
RuntimeError: expected backend CPU and dtype Float but got backend CUDA and dtype Float
[2019-08-07 23:08:39,626] 
0/100 * Epoch 0 (train): _base/lr=0.0015 | _base/momentum=0.9000 | _timers/_fps=5251.8586 | _timers/batch_time=0.0140 | _timers/data_time=0.0118 | _timers/model_time=0.0022 | loss=0.3634
0/100 * Epoch 0 (valid): _base/lr=0.0021 | _base/momentum=0.9000 | _timers/_fps=4396.1822 | _timers/batch_time=0.0149 | _timers/data_time=0.0141 | _timers/model_time=0.0007 | loss=0.1290
[2019-08-07 23:08:43,797] 
1/100 * Epoch 1 (train): _base/lr=0.0030 | _base/momentum=0.9000 | _timers/_fps=5968.3788 | _timers/batch_time=0.0111 | _timers/data_time=0.0105 | _timers/model_time=0.0006 | loss=0.1011
1/100 * Epoch 1 (valid): _base/lr=0.0043 | _base/momentum=0.9000 | _timers/_fps=6411.2549 | _timers/batch_time=0.0102 | _timers/data_time=0.0097 | _timers/model_time=0.0005 | loss=0.1142
[2019-08-07 23:08:48,231] 
2/100 * Epoch 2 (train): _base/lr=0.0063 | _base/momentum=0.9000 | _timers/_fps=5849.0785 | _timers/batch_time=0.0113 | _timers/data_time=0.0107 | _timers/model_time=0.0006 | loss=0.1265
2/100 * Epoch 2 (valid): _base/lr=0.0088 | _base/momentum=0.9000 | _timers/_fps=5848.7103 | _timers/batch_time=0.0113 | _timers/data_time=0.0107 | _timers/model_time=0.0005 | loss=0.1091
[2019-08-07 23:08:52,620] 
3/100 * Epoch 3 (train): _base/lr=0.0130 | _base/momentum=0.9000 | _timers/_fps=5833.1801 | _timers/batch_time=0.0112 | _timers/data_time=0.0105 | _timers/model_time=0.0006 | loss=0.3298
3/100 * Epoch 3 (valid): _base/lr=0.0183 | _base/momentum=0.9000 | _timers/_fps=6030.5078 | _timers/batch_time=0.0108 | _timers/data_time=0.0102 | _timers/model_time=0.0005 | loss=1.6801
[2019-08-07 23:08:57,031] 
4/100 * Epoch 4 (train): _base/lr=0.0270 | _base/momentum=0.9000 | _timers/_fps=5606.3612 | _timers/batch_time=0.0118 | _timers/data_time=0.0112 | _timers/model_time=0.0006 | loss=1.0114
4/100 * Epoch 4 (valid): _base/lr=0.0378 | _base/momentum=0.9000 | _timers/_fps=6113.1093 | _timers/batch_time=0.0107 | _timers/data_time=0.0102 | _timers/model_time=0.0005 | loss=0.3558
[2019-08-07 23:09:04,529] 
0/4 * Epoch 0 (train): _base/lr=0.0011 | _base/momentum=0.8992 | _timers/_fps=4558.1267 | _timers/batch_time=0.0716 | _timers/data_time=0.0173 | _timers/model_time=0.0543 | accuracy01=15.6250 | accuracy03=40.6250 | accuracy05=64.8438 | loss=2.2523
0/4 * Epoch 0 (valid): _base/lr=0.0012 | _base/momentum=0.8979 | _timers/_fps=6259.3646 | _timers/batch_time=0.0103 | _timers/data_time=0.0097 | _timers/model_time=0.0005 | accuracy01=31.6406 | accuracy03=74.6094 | accuracy05=90.2344 | loss=2.0553
[2019-08-07 23:09:04,738] 
1/4 * Epoch 1 (train): _base/lr=0.0050 | _base/momentum=0.8505 | _timers/_fps=5224.7796 | _timers/batch_time=0.0124 | _timers/data_time=0.0118 | _timers/model_time=0.0006 | accuracy01=46.4844 | accuracy03=80.8594 | accuracy05=92.9688 | loss=2.1803
1/4 * Epoch 1 (valid): _base/lr=0.0049 | _base/momentum=0.8514 | _timers/_fps=4746.6068 | _timers/batch_time=0.0135 | _timers/data_time=0.0128 | _timers/model_time=0.0007 | accuracy01=58.5938 | accuracy03=80.8594 | accuracy05=90.6250 | loss=1.5249
[2019-08-07 23:09:04,903] 
2/4 * Epoch 2 (train): _base/lr=0.0033 | _base/momentum=0.8839 | _timers/_fps=4631.8072 | _timers/batch_time=0.0138 | _timers/data_time=0.0131 | _timers/model_time=0.0007 | accuracy01=51.9531 | accuracy03=76.5625 | accuracy05=86.7188 | loss=1.7325
2/4 * Epoch 2 (valid): _base/lr=0.0033 | _base/momentum=0.8848 | _timers/_fps=5180.3675 | _timers/batch_time=0.0125 | _timers/data_time=0.0119 | _timers/model_time=0.0006 | accuracy01=49.6094 | accuracy03=80.0781 | accuracy05=86.7188 | loss=1.6910
[2019-08-07 23:09:05,048] 
3/4 * Epoch 3 (train): _base/lr=0.0017 | _base/momentum=0.9173 | _timers/_fps=5537.7741 | _timers/batch_time=0.0118 | _timers/data_time=0.0112 | _timers/model_time=0.0006 | accuracy01=57.4219 | accuracy03=78.5156 | accuracy05=85.5469 | loss=1.6145
3/4 * Epoch 3 (valid): _base/lr=0.0016 | _base/momentum=0.9182 | _timers/_fps=6305.4909 | _timers/batch_time=0.0102 | _timers/data_time=0.0096 | _timers/model_time=0.0005 | accuracy01=49.6094 | accuracy03=83.9844 | accuracy05=88.2812 | loss=1.4725
Top best models:
examples/logs/_tests_mnist_stages2/checkpoints//stage1.1.pth	58.5938
examples/logs/_tests_mnist_stages2/checkpoints//stage1.2.pth	49.6094
examples/logs/_tests_mnist_stages2/checkpoints//stage1.3.pth	49.6094
./bin/check_dl.sh: line 83:  7753 Terminated              PYTHONPATH=./examples:./catalyst:${PYTHONPATH} python catalyst/dl/scripts/run.py --expdir=./examples/_tests_mnist_stages --config=./examples/_tests_mnist_stages/config_finder.yml --logdir=./examples/logs/_tests_mnist_stages_finder
[2019-08-07 23:09:09,574] 
0/3 * Epoch 0 (train): _base/lr=0.0010 | _base/momentum=0.9000 | _timers/_fps=4245.6602 | _timers/batch_time=0.0684 | _timers/data_time=0.0181 | _timers/model_time=0.0503 | accuracy01=15.6250 | accuracy03=40.6250 | accuracy05=64.4531 | loss=2.2529
0/3 * Epoch 0 (valid): _base/lr=0.0010 | _base/momentum=0.9000 | _timers/_fps=4760.3054 | _timers/batch_time=0.0134 | _timers/data_time=0.0128 | _timers/model_time=0.0006 | accuracy01=30.0781 | accuracy03=73.0469 | accuracy05=89.0625 | loss=2.0645
[2019-08-07 23:09:09,740] 
1/3 * Epoch 1 (train): _base/lr=9.000000E-05 | _base/momentum=0.9000 | _timers/_fps=5544.1420 | _timers/batch_time=0.0119 | _timers/data_time=0.0113 | _timers/model_time=0.0006 | accuracy01=48.4375 | accuracy03=83.5938 | accuracy05=93.7500 | loss=1.9684
1/3 * Epoch 1 (valid): _base/lr=9.000000E-05 | _base/momentum=0.9000 | _timers/_fps=6642.2913 | _timers/batch_time=0.0096 | _timers/data_time=0.0092 | _timers/model_time=0.0004 | accuracy01=61.7188 | accuracy03=86.3281 | accuracy05=94.9219 | loss=1.9055
[2019-08-07 23:09:09,882] 
2/3 * Epoch 2 (train): _base/lr=0.0003 | _base/momentum=0.9000 | _timers/_fps=5106.5044 | _timers/batch_time=0.0127 | _timers/data_time=0.0120 | _timers/model_time=0.0006 | accuracy01=64.0625 | accuracy03=91.4062 | accuracy05=95.7031 | loss=1.7676
2/3 * Epoch 2 (valid): _base/lr=0.0003 | _base/momentum=0.9000 | _timers/_fps=6212.1950 | _timers/batch_time=0.0105 | _timers/data_time=0.0100 | _timers/model_time=0.0004 | accuracy01=61.7188 | accuracy03=92.9688 | accuracy05=98.0469 | loss=1.6734
Top best models:
examples/logs/_tests_mnist_stages2/checkpoints//stage1.1.pth	61.7188
examples/logs/_tests_mnist_stages2/checkpoints//stage1.2.pth	61.7188
examples/logs/_tests_mnist_stages2/checkpoints//stage1.0.pth	30.0781
[2019-08-07 23:09:14,370] 
0/4 * Epoch 0 (train): _base/lr=0.0011 | _base/momentum=0.8992 | _timers/_fps=4885.4741 | _timers/batch_time=0.0669 | _timers/data_time=0.0158 | _timers/model_time=0.0486 | accuracy01=11.7188 | accuracy03=30.0781 | accuracy05=48.0469 | loss=2.2989
0/4 * Epoch 0 (valid): _base/lr=0.0012 | _base/momentum=0.8979 | _timers/_fps=6851.7349 | _timers/batch_time=0.0093 | _timers/data_time=0.0089 | _timers/model_time=0.0004 | accuracy01=25.0000 | accuracy03=65.2344 | accuracy05=87.8906 | loss=2.1521
[2019-08-07 23:09:14,549] 
1/4 * Epoch 1 (train): _base/lr=0.0050 | _base/momentum=0.8505 | _timers/_fps=5496.6315 | _timers/batch_time=0.0119 | _timers/data_time=0.0102 | _timers/model_time=0.0005 | accuracy01=32.4219 | accuracy03=58.2031 | accuracy05=80.4688 | loss=2.0389
1/4 * Epoch 1 (valid): _base/lr=0.0049 | _base/momentum=0.8514 | _timers/_fps=6093.2729 | _timers/batch_time=0.0107 | _timers/data_time=0.0102 | _timers/model_time=0.0005 | accuracy01=30.8594 | accuracy03=77.3438 | accuracy05=94.9219 | loss=2.3390
[2019-08-07 23:09:14,700] 
2/4 * Epoch 2 (train): _base/lr=0.0033 | _base/momentum=0.8839 | _timers/_fps=5360.3541 | _timers/batch_time=0.0123 | _timers/data_time=0.0111 | _timers/model_time=0.0006 | accuracy01=44.5312 | accuracy03=69.1406 | accuracy05=86.7188 | loss=1.8742
2/4 * Epoch 2 (valid): _base/lr=0.0033 | _base/momentum=0.8848 | _timers/_fps=5723.1059 | _timers/batch_time=0.0114 | _timers/data_time=0.0109 | _timers/model_time=0.0005 | accuracy01=55.4688 | accuracy03=83.2031 | accuracy05=93.3594 | loss=1.4724
[2019-08-07 23:09:14,845] 
3/4 * Epoch 3 (train): _base/lr=0.0017 | _base/momentum=0.9173 | _timers/_fps=5016.8011 | _timers/batch_time=0.0129 | _timers/data_time=0.0120 | _timers/model_time=0.0006 | accuracy01=16.7969 | accuracy03=42.9688 | accuracy05=62.8906 | loss=1.6680
3/4 * Epoch 3 (valid): _base/lr=0.0016 | _base/momentum=0.9182 | _timers/_fps=6810.9671 | _timers/batch_time=0.0094 | _timers/data_time=0.0090 | _timers/model_time=0.0004 | accuracy01=72.6562 | accuracy03=87.1094 | accuracy05=94.9219 | loss=1.0720
Top best models:
examples/logs/_tests_mnist_stages2/checkpoints//stage1.3.pth	72.6562
examples/logs/_tests_mnist_stages2/checkpoints//stage1.2.pth	55.4688
examples/logs/_tests_mnist_stages2/checkpoints//stage1.1.pth	30.8594
[2019-08-07 23:09:23,362] 
0/100 * Epoch 0 (train): _base/lr=0.0015 | _base/momentum=0.9000 | _timers/_fps=5592.3807 | _timers/batch_time=0.0130 | _timers/data_time=0.0112 | _timers/model_time=0.0018 | loss=0.3636
0/100 * Epoch 0 (valid): _base/lr=0.0021 | _base/momentum=0.9000 | _timers/_fps=6732.9528 | _timers/batch_time=0.0096 | _timers/data_time=0.0092 | _timers/model_time=0.0005 | loss=0.1281
[2019-08-07 23:09:27,509] 
1/100 * Epoch 1 (train): _base/lr=0.0030 | _base/momentum=0.9000 | _timers/_fps=6232.1797 | _timers/batch_time=0.0105 | _timers/data_time=0.0100 | _timers/model_time=0.0005 | loss=0.0987
1/100 * Epoch 1 (valid): _base/lr=0.0043 | _base/momentum=0.9000 | _timers/_fps=6321.6718 | _timers/batch_time=0.0105 | _timers/data_time=0.0100 | _timers/model_time=0.0005 | loss=0.0575
[2019-08-07 23:09:31,514] 
2/100 * Epoch 2 (train): _base/lr=0.0063 | _base/momentum=0.9000 | _timers/_fps=6187.8612 | _timers/batch_time=0.0108 | _timers/data_time=0.0102 | _timers/model_time=0.0005 | loss=0.1237
2/100 * Epoch 2 (valid): _base/lr=0.0088 | _base/momentum=0.9000 | _timers/_fps=6851.8269 | _timers/batch_time=0.0095 | _timers/data_time=0.0091 | _timers/model_time=0.0004 | loss=0.1878
[2019-08-07 23:09:35,513] 
3/100 * Epoch 3 (train): _base/lr=0.0130 | _base/momentum=0.9000 | _timers/_fps=6185.7266 | _timers/batch_time=0.0107 | _timers/data_time=0.0101 | _timers/model_time=0.0005 | loss=0.2957
3/100 * Epoch 3 (valid): _base/lr=0.0183 | _base/momentum=0.9000 | _timers/_fps=6856.3497 | _timers/batch_time=0.0095 | _timers/data_time=0.0090 | _timers/model_time=0.0004 | loss=1.2074
[2019-08-07 23:09:39,491] 
4/100 * Epoch 4 (train): _base/lr=0.0270 | _base/momentum=0.9000 | _timers/_fps=6241.3461 | _timers/batch_time=0.0106 | _timers/data_time=0.0100 | _timers/model_time=0.0005 | loss=0.8093
4/100 * Epoch 4 (valid): _base/lr=0.0378 | _base/momentum=0.9000 | _timers/_fps=6735.5109 | _timers/batch_time=0.0096 | _timers/data_time=0.0092 | _timers/model_time=0.0004 | loss=0.3285
[2019-08-07 23:09:43,594] 
5/100 * Epoch 5 (train): _base/lr=0.0559 | _base/momentum=0.9000 | _timers/_fps=5932.1159 | _timers/batch_time=0.0112 | _timers/data_time=0.0106 | _timers/model_time=0.0005 | loss=0.9584
5/100 * Epoch 5 (valid): _base/lr=0.0783 | _base/momentum=0.9000 | _timers/_fps=6601.4981 | _timers/batch_time=0.0099 | _timers/data_time=0.0095 | _timers/model_time=0.0004 | loss=1.8217
Traceback (most recent call last):
  File "catalyst/dl/scripts/run.py", line 79, in <module>
    main(args, unknown_args)
  File "catalyst/dl/scripts/run.py", line 74, in main
    runner.run_experiment(experiment, check=args.check)
  File "/home/light/projects/catalyst/catalyst/dl/core/runner.py", line 196, in run_experiment
    self._run_event("exception")
  File "/home/light/projects/catalyst/catalyst/dl/core/runner.py", line 96, in _run_event
    getattr(self.state, f"on_{event}_post")()
  File "/home/light/projects/catalyst/catalyst/dl/core/state.py", line 172, in on_exception_post
    logger.on_exception(self)
  File "/home/light/projects/catalyst/catalyst/dl/callbacks/logging.py", line 183, in on_exception
    raise exception
  File "/home/light/projects/catalyst/catalyst/dl/core/runner.py", line 193, in run_experiment
    self._run_stage(stage)
  File "/home/light/projects/catalyst/catalyst/dl/core/runner.py", line 175, in _run_stage
    self._run_epoch(loaders)
  File "/home/light/projects/catalyst/catalyst/dl/core/runner.py", line 162, in _run_epoch
    self._run_loader(loader)
  File "/home/light/projects/catalyst/catalyst/dl/core/runner.py", line 131, in _run_loader
    self._run_batch(batch)
  File "/home/light/projects/catalyst/catalyst/dl/core/runner.py", line 113, in _run_batch
    self._run_event("batch_end")
  File "/home/light/projects/catalyst/catalyst/dl/core/runner.py", line 93, in _run_event
    getattr(callback, f"on_{event}")(self.state)
  File "/home/light/projects/catalyst/catalyst/dl/callbacks/scheduler.py", line 216, in on_batch_end
    raise NotImplementedError("End of LRFinder")
NotImplementedError: End of LRFinder

```


Discussion:
Ok, I fixed first stack trace. But second one below is more tricky. Working on it
```
Traceback (most recent call last):
  File "catalyst/dl/scripts/run.py", line 79, in <module>
    main(args, unknown_args)
  File "catalyst/dl/scripts/run.py", line 74, in main
    runner.run_experiment(experiment, check=args.check)
  File "/home/light/projects/catalyst/catalyst/dl/core/runner.py", line 196, in run_experiment
    self._run_event("exception")
  File "/home/light/projects/catalyst/catalyst/dl/core/runner.py", line 96, in _run_event
    getattr(self.state, f"on_{event}_post")()
  File "/home/light/projects/catalyst/catalyst/dl/core/state.py", line 172, in on_exception_post
    logger.on_exception(self)
  File "/home/light/projects/catalyst/catalyst/dl/callbacks/logging.py", line 183, in on_exception
    raise exception
  File "/home/light/projects/catalyst/catalyst/dl/core/runner.py", line 193, in run_experiment
    self._run_stage(stage)
  File "/home/light/projects/catalyst/catalyst/dl/core/runner.py", line 175, in _run_stage
    self._run_epoch(loaders)
  File "/home/light/projects/catalyst/catalyst/dl/core/runner.py", line 162, in _run_epoch
    self._run_loader(loader)
  File "/home/light/projects/catalyst/catalyst/dl/core/runner.py", line 131, in _run_loader
    self._run_batch(batch)
  File "/home/light/projects/catalyst/catalyst/dl/core/runner.py", line 113, in _run_batch
    self._run_event("batch_end")
  File "/home/light/projects/catalyst/catalyst/dl/core/runner.py", line 93, in _run_event
    getattr(callback, f"on_{event}")(self.state)
  File "/home/light/projects/catalyst/catalyst/dl/callbacks/optimizer.py", line 106, in on_batch_end
    grad_clip_fn=self.grad_clip_fn
  File "/home/light/projects/catalyst/catalyst/dl/callbacks/optimizer.py", line 46, in grad_step
    optimizer.step()
  File "/home/light/anaconda3/lib/python3.7/site-packages/torch/optim/adam.py", line 93, in step
    exp_avg.mul_(beta1).add_(1 - beta1, grad)
RuntimeError: expected backend CPU and dtype Float but got backend CUDA and dtype Float
```

Resolving patch:
diff --git a/catalyst/dl/core/runner.py b/catalyst/dl/core/runner.py
--- a/catalyst/dl/core/runner.py
+++ b/catalyst/dl/core/runner.py
@@ -99,6 +99,10 @@ def _run_event(self, event: str):
     def predict_batch(self, batch: Mapping[str, Any]) -> Mapping[str, Any]:
         pass
 
+    @abstractmethod
+    def forward(self, batch: Mapping[str, Any]) -> Mapping[str, Any]:
+        pass
+
     def _run_batch(self, batch):
         self.state.step += self.state.batch_size
         batch = self._batch2device(batch, self.device)
@@ -107,7 +111,7 @@ def _run_batch(self, batch):
 
         self._run_event("batch_start")
         self.state.timer.start("_timers/model_time")
-        self.state.output = self.predict_batch(batch)
+        self.state.output = self.forward(batch)
         self.state.timer.stop("_timers/model_time")
         self.state.timer.stop("_timers/batch_time")
         self._run_event("batch_end")
diff --git a/catalyst/dl/experiment/config.py b/catalyst/dl/experiment/config.py
--- a/catalyst/dl/experiment/config.py
+++ b/catalyst/dl/experiment/config.py
@@ -12,6 +12,7 @@
     MODELS, CRITERIONS, OPTIMIZERS, SCHEDULERS, CALLBACKS
 from catalyst.dl import utils
 from catalyst.utils import merge_dicts, get_short_hash, process_model_params
+from catalyst.utils.torch import any2device, get_device
 from catalyst.dl.core import Experiment, Callback
 from catalyst.dl.utils.torch import _Model, _Criterion, _Optimizer, \
     _Scheduler
@@ -147,6 +148,16 @@ def _get_optimizer(self, *, model_params, **params):
             checkpoint_path = f"{self.logdir}/checkpoints/best_full.pth"
             checkpoint = utils.load_checkpoint(checkpoint_path)
             utils.unpack_checkpoint(checkpoint, optimizer=optimizer)
+
+            # move optimizer to device
+            device = get_device()
+            for param in model_params:
+                param = param["params"][0]
+                state = optimizer.state[param]
+                for key, value in state.items():
+                    state[key] = any2device(value, device)
+
+            # update optimizer params
             for key, value in params.items():
                 for pg in optimizer.param_groups:
                     pg[key] = value
diff --git a/catalyst/dl/runner/supervised.py b/catalyst/dl/runner/supervised.py
--- a/catalyst/dl/runner/supervised.py
+++ b/catalyst/dl/runner/supervised.py
@@ -86,11 +86,30 @@ def _process_output_list(self, output: Mapping[str, Any]):
     def _process_output_none(self, output: Mapping[str, Any]):
         return output
 
-    def predict_batch(self, batch: Mapping[str, Any]):
+    def forward(self, batch):
+        """
+        Should not be called directly outside of runner.
+        If your model has specific interface, override this method to use it
+        """
         output = self._process_input(batch)
         output = self._process_output(output)
         return output
 
+    def predict_batch(self, batch: Mapping[str, Any]):
+        """
+        Run model for a batch of elements
+
+        WARN: You should not override this method. If you need specific model
+        call, override forward() method
+        Args:
+            batch: Key-value batch items
+        Returns: model output key-value
+
+        """
+        batch = self._batch2device(batch, self.device)
+        output = self.forward(batch)
+        return output
+
     def train(
         self,
         model: _Model,


Test patch: