Using optimize_for_inference on Torchscript model causes error

username: LewsTherin
cooked: <p>I fine-tuned the I3D action detection model on a custom dataset, saved it to Torch script and I’m loading it for inference by:</p>
<pre><code class="lang-auto"># the model is
model = torch.hub.load("facebookresearch/pytorchvideo", "i3d_r50", pretrained=True)

# training on custom dataset
[...]

# save the trained model
torch.jit.save(torch.jit.script(model), some_local_path)

# load it back
model = torch.jit.load(some_local_path, map_location=device).to(device)

# run inference
predictions = model(some_input)
</code></pre>
<p>If I just use the model loaded this way everything is ok…however, if I try adding:</p>
<pre><code class="lang-auto">model = torch.jit.load(some_local_path, map_location=device).to(device)
model = torch.jit.optimize_for_inference(model)
predictions = model(some_input)
</code></pre>
<p>I’m getting the error:</p>
<pre><code class="lang-auto">Traceback of TorchScript (most recent call last):

    graph(%input, %weight, %bias, %stride:int[], %padding:int[], %dilation:int[], %groups:int):
        %res = aten::cudnn_convolution_relu(%input, %weight, %bias, %stride, %padding, %dilation, %groups)
               ~~~~ &lt;--- HERE
        return (%res)
RuntimeError: CUDNN_BACKEND_OPERATIONGRAPH_DESCRIPTOR: cudnnFinalize Failed cudnn_status: CUDNN_STATUS_NOT_SUPPORTED
</code></pre>
<p>Is this a problem in the optimization of the specific model?</p>
<p>P.S. I’ve been suggested against the use of Torchscript here, but this is a fast way to have this running before I explore other options <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="LewsTherin" data-post="1" data-topic="196384">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/l/5f9b8f/48.png" class="avatar"> LewsTherin:</div>
<blockquote>
<p>P.S. I’ve been suggested against the use of Torchscript here</p>
</blockquote>
</aside>
<p>Yes, since <code>TorchScript</code> is in maintenance mode and I don’t believe will get major fixes anymore.<br>
My guess is <a href="https://github.com/pytorch/pytorch/blob/99b69e1ffb8da77d77892b43ef70b96f3e5fc958/torch/csrc/jit/passes/frozen_conv_add_relu_fusion.cpp#L27"><code>FuseFrozenConvAddRelu</code></a> fails, but this is just my guess.</p>

username: LewsTherin
cooked: <p>Fun fact, I noticed that if I use <code>optimize_for_inference</code> after casting the model to half precision, it works:</p>
<pre><code class="lang-auto">model = torch.jit.load(some_local_path, map_location=device).to(device)
self.model.half()
model = torch.jit.optimize_for_inference(model)
</code></pre>

username: rubenc
cooked: <aside class="quote no-group" data-username="LewsTherin" data-post="1" data-topic="196384">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/l/5f9b8f/48.png" class="avatar"> LewsTherin:</div>
<blockquote>
<p>P.S. I’ve been suggested against the use of Torchscript here, but this is a fast way to have this running before I explore other options <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
</blockquote>
</aside>
<p>Since <code>TorchScript</code> is  in maintenance mode, what saving format do you suggest as an alternative for running inference in C++.<br>
Thanks in advanced!</p>

username: ptrblck
cooked: <p>I don’t have an alternative to suggest and also don’t know what the roadmap of the code owners is regarding the C++ frontend.</p>

username: coincheung
cooked: <p>Hi,</p>
<p>Do you think using <code>optimize_for_inference</code> will cause inference result different slightly? I found the max value difference would be as large as 1e-4. Is this normal?</p>

username: ptrblck
cooked: <p>The absolute error could be expected due to rounding errors caused by the limited floating point precision. You should double check the relative error as well and could also compare it against a wider <code>dtype</code>, e.g. <code>float64</code>.</p>