How can i run a pytorch model via onnx cuda runtime?

Mukesh1729: <p>I have converted the model to onnx and it works on CPU but not able to pass in a cuda array through it. Any suggestions. Thanks</p>
<pre><code class="lang-auto">x_cpu = np.random.rand(1, 3, 256, 192)
x_gpu = cp.asarray(x_cpu)
x_gpu.shape

def to_numpy(tensor):
    return tensor.detach().cpu().numpy() if tensor.requires_grad else tensor.cpu().numpy()

model_path = '/content/transpose_cuda.onnx'

providers = [
    ('CUDAExecutionProvider', {
        'device_id': 0,
        'arena_extend_strategy': 'kNextPowerOfTwo',
        'gpu_mem_limit': 2 * 1024 * 1024 * 1024,
        'cudnn_conv_algo_search': 'EXHAUSTIVE',
        'do_copy_in_default_stream': True,
    }),
    'CPUExecutionProvider',
]
ort_session = ort.InferenceSession(model_path, providers =['CUDAExecutionProvider'])

# compute ONNX Runtime output prediction
ort_inputs = {ort_session.get_inputs()[0].name: x_gpu}#to_numpy(input_tensor)} 
ort_outs = ort_session.run(None, ort_inputs)

#Comparing output tolerance from pytorch model versus onnx 
# np.testing.assert_allclose(pytorch_out, ort_outs[0], rtol=1e-03, atol=1e-05)
ort_outs[0].shape

</code></pre>
<hr>
<p>RuntimeError                              Traceback (most recent call last)</p>
<p> in ()<br>
18 # compute ONNX Runtime output prediction<br>
19 ort_inputs = {ort_session.get_inputs()[0].name: x_gpu}<span class="hashtag">#to_numpy</span>(input_tensor)}<br>
—&gt; 20 ort_outs = ort_session.run(None, ort_inputs)<br>
21<br>
22 <span class="hashtag">#Comparing</span> output tolerance from pytorch model versus onnx</p>
<p>/usr/local/lib/python3.7/dist-packages/onnxruntime/capi/onnxruntime_inference_collection.py in run(self, output_names, input_feed, run_options)<br>
190             output_names = [output.name for output in self._outputs_meta]<br>
191         try:<br>
 → 192             return self._sess.run(output_names, input_feed, run_options)<br>
193         except C.EPFail as err:<br>
194             if self._enable_fallback:</p>
<p>RuntimeError: Input must be a list of dictionaries or a single numpy array for input ‘input’.</p>

tom: <p>I think even with the gpu execution provider, the standard thing is to pass in numpy (i.e. cpu) arrays. (Use <code>assert 'CUDAExecutionProvider' in onnxruntime.get_available_providers()</code> or nvidia-smi to check that you are using the GPU.)</p>
<p>Best regards</p>
<p>Thomas</p>

Mukesh1729: <p>Hey Tom, I am using gpu. I checked with:<br>
import onnxruntime as ort<br>
ort.get_device()</p>
<p>I referred to this page:</p><aside class="onebox allowlistedgeneric" data-onebox-src="https://onnxruntime.ai/docs/execution-providers/CUDA-ExecutionProvider.html">
  <header class="source">
      <img src="https://onnxruntime.ai/favicon.ico" class="site-icon" width="17" height="16">

      <a href="https://onnxruntime.ai/docs/execution-providers/CUDA-ExecutionProvider.html" target="_blank" rel="noopener nofollow ugc">onnxruntime</a>
  </header>

  <article class="onebox-body">
    

<h3><a href="https://onnxruntime.ai/docs/execution-providers/CUDA-ExecutionProvider.html" target="_blank" rel="noopener nofollow ugc">CUDA</a></h3>

  <p>Instructions to execute ONNX Runtime applications with CUDA</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>