Inference in ONNX mixed precision model

Westerby: <p>Hello,</p>
<p>I trained frcnn model with automatic mixed precision and exported it to ONNX. I wonder however how would inference look like programmaticaly to leverage the speed up of mixed precision model, since pytorch uses <code>with autocast():</code>, and I can’t come with an idea how to put it in the inference engine, like onnxruntime.</p>
<p>My specs:<br>
torch==1.6.0+cu101<br>
torchvision==0.7.0+cu101<br>
onnx==1.7.0<br>
onnxruntime-gpu==1.4.0</p>
<p>Model exports just fine:</p>
<pre><code class="lang-auto">torch.onnx.export(model, 
                  x, 
                  "model_16.onnx", 
                  verbose=True, do_constant_folding=True, opset_version=12,
                  input_names=input_names, output_names=output_names)
</code></pre>
<p>But I wonder how to leverage mixed precision speed up here:</p>
<pre><code class="lang-auto">
import onnxruntime as ort
ort_session = ort.InferenceSession('model_16.onnx')
outputs = ort_session.run(None, {'input': x.numpy()})
</code></pre>

ptrblck: <p>I’m not exactly sure how ONNX exports the model, but if tracing is used, the mixed-precision operations might have been already recorded. Do you see any FP16 operations, if you profile the ONNX model?</p>

Westerby: <p>Hello,</p>
<p>thanks for the suggestions. I run the onnx runtime profiler on 16FP model from torch, but I’m not exactly sure how to look for execution of FP16 operations there. I’m attaching the log.<br>
<a href="http://www.mediafire.com/file/rel0ze3y963nohv/onnxruntime_profile__2020-08-28_10-35-51.json/file" class="onebox" target="_blank" rel="nofollow noopener">http://www.mediafire.com/file/rel0ze3y963nohv/onnxruntime_profile__2020-08-28_10-35-51.json/file</a><br>
Here’s the code I used to run the profiler:</p>
<pre><code class="lang-auto">import onnxruntime as ort
options = ort.SessionOptions()
options.enable_profiling = True
ort_session = ort.InferenceSession('model_16.onnx', options)
outputs = ort_session.run(None, {'input': images[0].cpu().numpy()})
prof_file = ort_session.end_profiling()
</code></pre>
<p>Anyway, if I do simple time measurement for inference, I don’t see much difference. To be honest I expected ONNX model to run faster.</p>
<p>1.Pure torch 16FP model:</p>
<pre><code class="lang-auto">from imutils.video import FPS
fps = FPS().start()
for i in range(100):
    images = list(image.to('cuda:0') for image in x)
    with autocast():
        pred = model(images)
    
    fps.update()

fps.stop()
print('Time taken: {:.2f}'.format(fps.elapsed()))
print('~ FPS : {:.2f}'.format(fps.fps()))

Time taken: 2.19
~ FPS : 45.57
</code></pre>
<ol start="2">
<li>Torch-&gt;ONNX 16FP model:</li>
</ol>
<pre><code class="lang-auto">import onnxruntime as ort
ort_session = ort.InferenceSession('model_16.onnx')
fps = FPS().start()

for i in range(100):
    outputs = ort_session.run(None, {'input': images[0].cpu().numpy()})
    fps.update()

fps.stop()
print('Time taken: {:.2f}'.format(fps.elapsed()))
print('~ FPS : {:.2f}'.format(fps.fps()))

Time taken: 2.15
~ FPS : 46.61
</code></pre>

dheeraj_agrawal: <p>Did you find the solutions for converting automatic mixed precision model to onnx model with AMP.</p>