How should we use MKLDNN with a pretrained model? getting (RuntimeError: Input type (Mkldnntorch.FloatTensor) and weight type (torch.FloatTensor) should be the same error)

Shisho_Sama: <p>Hello everyone.<br>
I’m trying to use MKLDNN with a pretrained model, however, after loading the weights.<br>
and converting the submodules into MKLDNN,  it failes with this error :</p>
<pre><code class="lang-auto">(base) user@F3S-uf3:/mnt/d/Codes/Pytorch_Retinaface_MKLDN$ cd /mnt/d/Codes/Pytorch_Retinaface_MKLDN ; env PYTHONIOENCODING=UTF-8 PYTHONUNBUFFERED=1 /home/user/anaconda3/bin/python /home/user/.vscode-server/extensions/ms-python.python-2020.1.58038/pythonFiles/ptvsd_launcher.py --default --nodebug --client --host localhost --port 52636 /mnt/d/Codes/Pytorch_Retinaface_MKLDN/test_detect_align_MKLDNN.py 
Loading pretrained model from ./weights/mobilenet0.25_Final.pth
remove prefix 'module.'
input type for img_path: str
Traceback (most recent call last):
  File "/home/user/.vscode-server/extensions/ms-python.python-2020.1.58038/pythonFiles/ptvsd_launcher.py", line 43, in &lt;module&gt;
    main(ptvsdArgs)
  File "/home/user/.vscode-server/extensions/ms-python.python-2020.1.58038/pythonFiles/lib/python/old_ptvsd/ptvsd/__main__.py", line 432, in main
    run()
  File "/home/user/.vscode-server/extensions/ms-python.python-2020.1.58038/pythonFiles/lib/python/old_ptvsd/ptvsd/__main__.py", line 316, in run_file
    runpy.run_path(target, run_name='__main__')
  File "/home/user/anaconda3/lib/python3.7/runpy.py", line 263, in run_path
    pkg_name=pkg_name, script_name=fname)
  File "/home/user/anaconda3/lib/python3.7/runpy.py", line 96, in _run_module_code
    mod_name, mod_spec, pkg_name, script_name)
  File "/home/user/anaconda3/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/mnt/d/Codes/Pytorch_Retinaface_MKLDN/test_detect_align_MKLDNN.py", line 318, in &lt;module&gt;
    model_weights=model.weight )
  File "/mnt/d/Codes/Pytorch_Retinaface_MKLDN/test_detect_align_MKLDNN.py", line 133, in detect
    loc, conf, landms = net(img)  # forward pass
  File "/home/user/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/mnt/d/Codes/Pytorch_Retinaface_MKLDN/models/retinaface.py", line 225, in forward
    fpn = self.fpn(out)
  File "/home/user/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/mnt/d/Codes/Pytorch_Retinaface_MKLDN/models/net.py", line 84, in forward
    output1 = self.output1(input[0])
  File "/home/user/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/user/anaconda3/lib/python3.7/site-packages/torch/nn/modules/container.py", line 100, in forward
    input = module(input)
  File "/home/user/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/user/anaconda3/lib/python3.7/site-packages/torch/nn/modules/conv.py", line 345, in forward
    return self.conv2d_forward(input, self.weight)
  File "/home/user/anaconda3/lib/python3.7/site-packages/torch/nn/modules/conv.py", line 342, in conv2d_forward
    self.padding, self.dilation, self.groups)
RuntimeError: Input type (Mkldnntorch.FloatTensor) and weight type (torch.FloatTensor) should be the same
</code></pre>
<p>Whats the correct steps to use this with a pretrained model? I cant find any documentations on  this.<br>
Any help is greatly appreciated.<br>
Thanks a lot in advance</p>

ptrblck: <p>Could you try:</p>
<pre><code class="lang-python">from torch.utils import mkldnn as mkldnn_utils

net = ...
net = mkldnn_utils.to_mkldnn(net)
data = torch.randn(...)
data = data.to_mkldnn()
net(data)
</code></pre>

Shisho_Sama: <p>If I load the weights normally, then convert my model(net) to mkldnn, and then convert the input to the mkldnn, and then try to feed forward it, it fails with the following message :</p>
<pre><code class="lang-python">(base) user@F3S-Hasanpour:/mnt/d/Codes/Pytorch_Retinaface_MKLDN$ cd /mnt/d/Codes/Pytorch_Retinaface_MKLDN ; env PYTHONIOENCODING=UTF-8 PYTHONUNBUFFERED=1 /home/user/anaconda3/bin/python /home/user/.vscode-server/extensions/ms-python.python-2020.1.58038/pythonFiles/ptvsd_launcher.py --default --nodebug --client --host localhost --port 50776 /mnt/d/Codes/Pytorch_Retinaface_MKLDN/test_detect_align_MKLDNN.py 
Loading pretrained model from ./weights/mobilenet0.25_Final.pth
remove prefix 'module.'
input type for img_path: str
Traceback (most recent call last):
  File "/home/user/.vscode-server/extensions/ms-python.python-2020.1.58038/pythonFiles/ptvsd_launcher.py", line 43, in &lt;module&gt;
    main(ptvsdArgs)
  File "/home/user/.vscode-server/extensions/ms-python.python-2020.1.58038/pythonFiles/lib/python/old_ptvsd/ptvsd/__main__.py", line 432, in main
    run()
  File "/home/user/.vscode-server/extensions/ms-python.python-2020.1.58038/pythonFiles/lib/python/old_ptvsd/ptvsd/__main__.py", line 316, in run_file
    runpy.run_path(target, run_name='__main__')
  File "/home/user/anaconda3/lib/python3.7/runpy.py", line 263, in run_path
    pkg_name=pkg_name, script_name=fname)
  File "/home/user/anaconda3/lib/python3.7/runpy.py", line 96, in _run_module_code
    mod_name, mod_spec, pkg_name, script_name)
  File "/home/user/anaconda3/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/mnt/d/Codes/Pytorch_Retinaface_MKLDN/test_detect_align_MKLDNN.py", line 327, in &lt;module&gt;
    model_weights=model.weight )
  File "/mnt/d/Codes/Pytorch_Retinaface_MKLDN/test_detect_align_MKLDNN.py", line 142, in detect
    loc, conf, landms = net(img)  # forward pass
  File "/home/user/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/mnt/d/Codes/Pytorch_Retinaface_MKLDN/models/retinaface.py", line 225, in forward
    fpn = self.fpn(out)
  File "/home/user/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/mnt/d/Codes/Pytorch_Retinaface_MKLDN/models/net.py", line 84, in forward
    output1 = self.output1(input[0])
  File "/home/user/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/user/anaconda3/lib/python3.7/site-packages/torch/nn/modules/container.py", line 100, in forward
    input = module(input)
  File "/home/user/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/user/anaconda3/lib/python3.7/site-packages/torch/nn/modules/conv.py", line 345, in forward
    return self.conv2d_forward(input, self.weight)
  File "/home/user/anaconda3/lib/python3.7/site-packages/torch/nn/modules/conv.py", line 342, in conv2d_forward
    self.padding, self.dilation, self.groups)
RuntimeError: Input type (Mkldnntorch.FloatTensor) and weight type (torch.FloatTensor) should be the same

</code></pre>
<p>and if I convert the weights into the mkldnn as well I get:</p>
<pre><code class="lang-python">default --nodebug --client --host localhost --port 50734 /mnt/d/Codes/Pytorch_Retinaface_MKLDN/test_detect_align_MKLDNN.py 
source /home/user/anaconda3/bin/activate
conda activate base
Loading pretrained model from ./weights/mobilenet0.25_Final.pth
remove prefix 'module.'
Traceback (most recent call last):
  File "/home/user/.vscode-server/extensions/ms-python.python-2020.1.58038/pythonFiles/ptvsd_launcher.py", line 43, in &lt;module&gt;
    main(ptvsdArgs)
  File "/home/user/.vscode-server/extensions/ms-python.python-2020.1.58038/pythonFiles/lib/python/old_ptvsd/ptvsd/__main__.py", line 432, in main
    run()
  File "/home/user/.vscode-server/extensions/ms-python.python-2020.1.58038/pythonFiles/lib/python/old_ptvsd/ptvsd/__main__.py", line 316, in run_file
    runpy.run_path(target, run_name='__main__')
  File "/home/user/anaconda3/lib/python3.7/runpy.py", line 263, in run_path
    pkg_name=pkg_name, script_name=fname)
  File "/home/user/anaconda3/lib/python3.7/runpy.py", line 96, in _run_module_code
    mod_name, mod_spec, pkg_name, script_name)
  File "/home/user/anaconda3/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/mnt/d/Codes/Pytorch_Retinaface_MKLDN/test_detect_align_MKLDNN.py", line 327, in &lt;module&gt;
    model_weights=model.weight )
  File "/mnt/d/Codes/Pytorch_Retinaface_MKLDN/test_detect_align_MKLDNN.py", line 96, in detect
    net = load_model(net, trained_model, cpu)
  File "/mnt/d/Codes/Pytorch_Retinaface_MKLDN/test_detect_align_MKLDNN.py", line 67, in load_model
    pretrained_dict = to_mkldnn(pretrained_dict)
  File "/mnt/d/Codes/Pytorch_Retinaface_MKLDN/test_detect_align_MKLDNN.py", line 51, in to_mkldnn
    state_dict_new[k] = p.to_mkldnn()
RuntimeError: cpu_tensor.scalar_type() == ScalarType::Float INTERNAL ASSERT FAILED at /pytorch/aten/src/ATen/native/mkldnn/MKLDNNConversions.cpp:28, please report a bug to PyTorch. dense_to_mkldnn expects float tensor input

</code></pre>

ptrblck: <p>Could you post a dummy code snippet to reproduce this issue?</p>

Shisho_Sama: <p>I uploaded a sample project to test this, you can download and test this using : <a href="https://github.com/pytorch/pytorch/files/4132251/test_pytorch_mkldnn.zip" rel="nofollow noopener">https://github.com/pytorch/pytorch/files/4132251/test_pytorch_mkldnn.zip</a></p>

thomas_beznik: <p>Hello,</p>
<p>I am also getting the same error when using this network: <a href="https://github.com/wolny/pytorch-3dunet" rel="nofollow noopener">https://github.com/wolny/pytorch-3dunet</a>. What should I do in order to be able to use MKLDNN with it? Thank you!</p>

ckmufeng: <p>I met the same problem.<br>
It seems that to_mkldnn() cannot trans dtype other than float to mkldnn.</p>

Shisho_Sama: <p>This issue might give you some hints on how to solve yours : <a href="https://github.com/pytorch/pytorch/issues/32697" rel="nofollow noopener">https://github.com/pytorch/pytorch/issues/32697</a> might be worth checking it out.</p>