Trying to run inference with custom model but get Runtime error asking for Cuda

username: Ben_Sturgeon
cooked: <p>I am trying to run inference with a model on a device which only has access to CPU. I am loading the model in with:<br>
gtf = torch.load(model_dir, map_location=torch.device(“cpu”))<br>
But I get a Runtime Error saying:<br>
RuntimeError: module must have its parameters and buffers on device cuda:0 (device_ids[0]) but found one of them on device: cpu.</p>
<p>How do I get it to stop using Cuda for this operation?<br>
My eventual goal is to run this on an android device using chaquopy so I will not be able to use Cuda at all.</p>

username: ptrblck
cooked: <p>I assume the <code>torch.load</code> line of code is raising the error?<br>
If so, did you store the model directly or the <code>state_dict</code> (which is the recommended way)?</p>

username: Ben_Sturgeon
cooked: <p>I am loading just the stored model, rather than the state_dict. If you think this might be the root cause I will look further into that.</p>
<p>The error actually occurs when I try to do the actual inference. When I have full access to the GPU it runs fine, but when I direct it to the CPU it throws the error that it was expecting a Cuda device.</p>
<p>To give some more context, I’m actually trying to use jit.trace to serialize the inference so I can deploy on mobile. Here is the code I have:</p>
<pre><code>model_dir = "/home/benjamin/imagelift/Odom_reader/trained_char_det/signatrix_efficientdet_coco.pth"
gtf = torch.load(model_dir)

example_inputs = torch.rand(1, 3, 512, 512)
odom_detect = torch.jit.script(gtf,example_inputs)
</code></pre>
<p>The error this returns refers to the jit.trace line:</p>
<p>/home/benjamin/.local/lib/python3.6/site-packages/torch/serialization.py:657: SourceChangeWarning: source code of class ‘torch.nn.modules.conv.Conv2d’ has changed. you can retrieve the original source code by accessing the object’s source attribute or set <code>torch.nn.Module.dump_patches = True</code> and use the patch tool to revert the changes.<br>
warnings.warn(msg, SourceChangeWarning)<br>
/home/benjamin/AndroidStudioProjects/Odom_detect/app/src/main/python/src/model.py:251: TracerWarning: Converting a tensor to a Python index might cause the trace to be incorrect. We can’t record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!<br>
if len(inputs) == 2:<br>
/home/benjamin/AndroidStudioProjects/Odom_detect/app/src/main/python/src/utils.py:84: TracerWarning: Converting a tensor to a Python integer might cause the trace to be incorrect. We can’t record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!<br>
image_shape = np.array(image_shape)<br>
/home/benjamin/AndroidStudioProjects/Odom_detect/app/src/main/python/src/utils.py:96: TracerWarning: torch.from_numpy results are registered as constants in the trace. You can safely ignore this warning if you use this function to create tensors out of constant variables that would be the same every time you call this function. In any other case, this might cause the trace to be incorrect.<br>
anchors = torch.from_numpy(all_anchors.astype(np.float32))<br>
/home/benjamin/AndroidStudioProjects/Odom_detect/app/src/main/python/src/model.py:282: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can’t record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!<br>
if scores_over_thresh.sum() == 0:<br>
Traceback (most recent call last):<br>
File “/home/benjamin/.config/JetBrains/PyCharmCE2020.1/scratches/scratch_1.py”, line 137, in <br>
infer_test()<br>
File “/home/benjamin/.config/JetBrains/PyCharmCE2020.1/scratches/scratch_1.py”, line 113, in infer_test<br>
odom_detect = torch.jit.trace(gtf,example_inputs)<br>
File “/home/benjamin/.local/lib/python3.6/site-packages/torch/jit/<strong>init</strong>.py”, line 875, in trace<br>
check_tolerance, _force_outplace, _module_class)<br>
File “/home/benjamin/.local/lib/python3.6/site-packages/torch/jit/<strong>init</strong>.py”, line 1027, in trace_module<br>
module._c._create_method_from_trace(method_name, func, example_inputs, var_lookup_fn, _force_outplace)<br>
RuntimeError: 0 INTERNAL ASSERT FAILED at /pytorch/torch/csrc/jit/ir/alias_analysis.cpp:318, please report a bug to PyTorch. We don’t have an op for aten::to but it isn’t a special case.  Argument types: Tensor, None, int, Device, bool, bool, bool, int,</p>

username: ptrblck
cooked: <p>Try to recreate the model and load the <code>state_dict</code>. As the warning already show, it seems that the source code was modified, such that the restored model might not work properly anymore.</p>
<p>Once these warning are gone, feel free to update this topic in case you encounter further errors.</p>

username: Ben_Sturgeon
cooked: <p>Thank you for the support on this. I’ve managed to get rid of the errors above, and I have succeeded in running the trace on the loaded module. However, I now get an error when trying to save the model.</p>
<p>I tried 2 approaches, one in which I loaded the default model with pretrained weights, and one where I load in the necessary module for my custom trained model. The error is the same with both.</p>
<blockquote>
<blockquote>
<blockquote>
<p>odom_detect.save(“odom_detect_traced.pt”)</p>
</blockquote>
</blockquote>
</blockquote>
<p>Traceback (most recent call last):<br>
File “”, line 1, in <br>
File “/home/benjamin/.local/lib/python3.6/site-packages/torch/jit/<strong>init</strong>.py”, line 1648, in save<br>
return self._c.save(*args, **kwargs)<br>
RuntimeError:<br>
Could not export Python function call ‘SwishImplementation’. Remove calls to Python functions before export. Did you forget add <span class="mention">@script</span> or c annotation? If this is a nn.ModuleList, add it to <strong>constants</strong>:<br>
/home/benjamin/.local/lib/python3.6/site-packages/efficientnet_pytorch/utils.py(52): forward<br>
/home/benjamin/.local/lib/python3.6/site-packages/torch/nn/modules/module.py(534): _slow_forward<br>
/home/benjamin/.local/lib/python3.6/site-packages/torch/nn/modules/module.py(548): <strong>call</strong><br>
/home/benjamin/.local/lib/python3.6/site-packages/efficientnet_pytorch/model.py(78): forward<br>
/home/benjamin/.local/lib/python3.6/site-packages/torch/nn/modules/module.py(534): _slow_forward<br>
/home/benjamin/.local/lib/python3.6/site-packages/torch/nn/modules/module.py(548): <strong>call</strong><br>
/home/benjamin/imagelift/opencv-text-recognition/src/model.py(193): forward<br>
/home/benjamin/.local/lib/python3.6/site-packages/torch/nn/modules/module.py(534): _slow_forward<br>
/home/benjamin/.local/lib/python3.6/site-packages/torch/nn/modules/module.py(548): <strong>call</strong><br>
/home/benjamin/imagelift/opencv-text-recognition/src/model.py(258): forward<br>
/home/benjamin/.local/lib/python3.6/site-packages/torch/nn/modules/module.py(534): _slow_forward<br>
/home/benjamin/.local/lib/python3.6/site-packages/torch/nn/modules/module.py(548): <strong>call</strong><br>
/home/benjamin/.local/lib/python3.6/site-packages/torch/jit/<strong>init</strong>.py(1027): trace_module<br>
/home/benjamin/.local/lib/python3.6/site-packages/torch/jit/<strong>init</strong>.py(875): trace<br>
(1): </p>
<p>I have tried adding script methods “<span class="mention">@torch.jit.script_method</span>” to many of my forward functions, but I am not sure how deep it is possible for me to go with this, and if it will end up working or not.</p>

username: ptrblck
cooked: <p>Could you post the definition of <code>SwichImplementation</code>? It seems that some Python calls are using inside this method, which cannot be exported.<br>
One workaround would be to split the method into a “Python method” and an exportable method using <code>@torch.jit.ignore</code>.</p>