RuntimeError: Mobile optimized model cannot be inferenced on GPU

zshn25: <p>I have used <code>mobile_optimizer.optimize_for_mobile()</code> function to optimize my model and I wanted to benchmark it with other optimization techniques for faster inference but it gives me runtime error when I try to run it on GPU</p>
<pre><code class="lang-auto">---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
&lt;ipython-input-15-c705c21babec&gt; in &lt;module&gt;
      1 optimized_traced_model = mobile_optimizer.optimize_for_mobile(traced_model)
      2 
----&gt; 3 get_ipython().run_line_magic('timeit', 'with torch.no_grad(): optimized_traced_model(example1)')

~/anaconda3/envs/e2r/lib/python3.7/site-packages/IPython/core/interactiveshell.py in run_line_magic(self, magic_name, line, _stack_depth)
   2324                 kwargs['local_ns'] = sys._getframe(stack_depth).f_locals
   2325             with self.builtin_trap:
-&gt; 2326                 result = fn(*args, **kwargs)
   2327             return result
   2328 

&lt;decorator-gen-60&gt; in timeit(self, line, cell, local_ns)

~/anaconda3/envs/e2r/lib/python3.7/site-packages/IPython/core/magic.py in &lt;lambda&gt;(f, *a, **k)
    185     # but it's overkill for just that one bit of state.
    186     def magic_deco(arg):
--&gt; 187         call = lambda f, *a, **k: f(*a, **k)
    188 
    189         if callable(arg):

~/anaconda3/envs/e2r/lib/python3.7/site-packages/IPython/core/magics/execution.py in timeit(self, line, cell, local_ns)
   1161             for index in range(0, 10):
   1162                 number = 10 ** index
-&gt; 1163                 time_number = timer.timeit(number)
   1164                 if time_number &gt;= 0.2:
   1165                     break

~/anaconda3/envs/e2r/lib/python3.7/site-packages/IPython/core/magics/execution.py in timeit(self, number)
    167         gc.disable()
    168         try:
--&gt; 169             timing = self.inner(it, self.timer)
    170         finally:
    171             if gcold:

&lt;magic-timeit&gt; in inner(_it, _timer)

~/anaconda3/envs/e2r/lib/python3.7/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    720             result = self._slow_forward(*input, **kwargs)
    721         else:
--&gt; 722             result = self.forward(*input, **kwargs)
    723         for hook in itertools.chain(
    724                 _global_forward_hooks.values(),

RuntimeError: The following operation failed in the TorchScript interpreter.
Traceback of TorchScript (most recent call last):
    graph(%input, %weight, %bias, %stride:int[], %padding:int[], %dilation:int[], %groups:int):
        %output_min_max : None = prim::Constant()
        %packed_weight_bias = prepacked::conv2d_clamp_prepack(
                              ~~~~~~~~~ &lt;--- HERE
            %weight, %bias, %stride, %padding, %dilation, %groups,
            %output_min_max, %output_min_max)
RuntimeError: Could not run 'prepacked::conv2d_clamp_prepack' with arguments from the 'CUDA' backend. 'prepacked::conv2d_clamp_prepack' is only available for these backends: [CPU].
</code></pre>
<p>Also, when I run it on CPU, the runtime of the optimized model is more than an order magnitude <strong>slower</strong> then the un-optimized model.</p>

ptrblck: <p>I don’t know which optimizations are applied in <code>optimizer_for_mobile</code>, but would assume that you would see a speedup on a mobile platform, i.e. not necessarily for an x86 architecture.<br>
Did you deploy the model and profiled it on a mobile device?</p>

zshn25: <p>No, I am specifically looking for mobile devices with a GPU like NVIDIA Jetson.</p>

ptrblck: <p>For inference use cases on the Jetson platform, you could check out the <a href="https://github.com/dusty-nv/jetson-inference">jetson-inference</a> repository, which provides some utilities and examples.</p>