Slow Forward on traced graph - on cuda, 2nd iteration

username: michele-arrival
cooked: <p>I have relatively large network fully exported to torchscript.</p>
<p>I reimport the torchscript in python, and then I need to run the forward pass a number of times on a set of frames.</p>
<p>The weird thing is that while the first forward pass runs at an acceptable speed, the 2nd one (and only the second one, <em>after</em> the first pass has returned the results to CPU) seems to be tacking forever, more than 50 seconds.</p>
<p>I’ve even tried to call</p>
<pre><code class="lang-auto">torch.cuda.synchronize(device=self.device)
</code></pre>
<p>after the <code>.to('cpu')</code>.</p>
<p>The rest of the frames are processed at reasonable speed, it’s really only the 2nd one <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=9" title=":confused:" class="emoji" alt=":confused:">  what could it be doing in this 2nd pass?</p>

username: ptrblck
cooked: <p>Are you only seeing the slow 2nd iteration when using the scripted model and/or only the GPU?<br>
Note that the JIT performs some optimizations, but 50s sounds rather excessive, so I doubt it’s the case here.</p>

username: michele-arrival
cooked: <p>Hi, thanks for your reply</p>
<p>It is a traced model, and it only happens on GPU (on cpu things are even).</p>
<p>I’ve tried this on Pytorch 1.8.1 cuda 10, on both Windows 10 and Ubuntu 20.04 (the former on a Quadro P2000 Mobile with driver 466.11 and the latter has a Titan RTX on 450.102.04) and the results are similarly skewed (the 2nd frame takes 30 seconds on the Titan).</p>
<p>If it is of any assistance in understanding what is going on  these are the profiles from the first 3 frames:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/7/77ac603a44a884a9d68cdaa960a475948e8552b6.png" data-download-href="https://discuss.pytorch.org/uploads/default/77ac603a44a884a9d68cdaa960a475948e8552b6" title="Untitled"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/7/7/77ac603a44a884a9d68cdaa960a475948e8552b6_2_690x467.png" alt="Untitled" data-base62-sha1="h4G8dNFPXogcrXucSe3exd6PxAi" width="690" height="467" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/7/7/77ac603a44a884a9d68cdaa960a475948e8552b6_2_690x467.png, https://discuss.pytorch.org/uploads/default/optimized/3X/7/7/77ac603a44a884a9d68cdaa960a475948e8552b6_2_1035x700.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/7/7/77ac603a44a884a9d68cdaa960a475948e8552b6_2_1380x934.png 2x" data-dominant-color="E2DFE5"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Untitled</span><span class="informations">1770×1200 181 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Captured with this block:</p>
<pre><code class="lang-auto">with torch.autograd.profiler.profile(use_cuda=True, with_stack=True) as prof:
    inp = [torch.from_numpy(i).float().to(self.device) for i in inputs]
    with torch.no_grad():
        out = self.sess(*inp)
    result = [o.detach().cpu().numpy() for o in out]
</code></pre>
<p>Btw is there any way to check that pytorch is loading the correct cuda libraries?</p>
<p>Thank you!</p>

username: ptrblck
cooked: <p>Is the slowdown only happening on scripted models or any GPU workload?</p>
<aside class="quote no-group" data-username="michele-arrival" data-post="3" data-topic="118445">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/michele-arrival/48/35371_2.png" class="avatar"> michele-arrival:</div>
<blockquote>
<p>Btw is there any way to check that pytorch is loading the correct cuda libraries?</p>
</blockquote>
</aside>
<p>You could check the used CUDA version via <code>torch.version.cuda</code> and depending on the way you’ve installed it (binaries or source build) it can be either statically or dynamically linked.</p>

username: michele-arrival
cooked: <aside class="quote no-group" data-username="ptrblck" data-post="4" data-topic="118445">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>Is the slowdown only happening on scripted models or any GPU workload?</p>
</blockquote>
</aside>
<p>Apologies, I didn’t catch that the point of your question initially. It is only on scripted models.</p>
<p>Just to check if I understand correctly now:</p>
<p>I have the training function of my model wrapped in some boilerplate code (partly pytorch-lightning, but it should be irrelevant). I have the option to trace this function and then use it for training, or to skip this step and train with all python host code.</p>
<p>My original question was based on what i observed at inference time (forward function has been exported and re-imported) but indeed even in training i observe a delay on the second iteration only when the forward function is traced, whereas with all python host code the times are even.</p>
<aside class="quote no-group quote-modified" data-username="ptrblck" data-post="4" data-topic="118445">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<aside class="quote no-group" data-username="michele-arrival" data-post="3" data-topic="118445">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/michele-arrival/48/35371_2.png" class="avatar"> michele-arrival:</div>
<blockquote>
<p>Btw is there any way to check that pytorch is loading the correct cuda libraries?</p>
</blockquote>
</aside>
<p>You could check the used CUDA version via <code>torch.version.cuda</code> and depending on the way you’ve installed it (binaries or source build) it can be either statically or dynamically linked.</p>
</blockquote>
</aside>
<p><code>cuda = '10.2'</code> in <code>torch/version.py</code><br>
It’s installed from anaconda, so it should be all statically linked i believe?</p>
<p>Thanks!</p>

username: Dobbie
cooked: <p>I met the same question under pytorch 1.7. Any solution about it？</p>

username: michele-arrival
cooked: <p>Hi Dobbie.</p>
<p>The answer is: kind-of. I can give you an update.</p>
<p>Someone opened an issue on gitlab, after encountering what i thought is a superset of this problem. He saw that with varying input sizes, the first 20 iterations would be slow. On the other hand with fixed input size, only the first 2 iterations would be slow (and the second one, unacceptably so, much like the reason for my question here).<br>
link: <a href="https://github.com/pytorch/pytorch/issues/56245" class="inline-onebox" rel="noopener nofollow ugc">The first 20 loops of inference are Extremely Slow on C++ libtorch v1.8.1 · Issue #56245 · pytorch/pytorch · GitHub</a></p>
<p>He was referred to a solution to avoid the 20 slow iterations, which was to to decrease the optimization depth with <code>torch._C._jit_set_bailout_depth(1)</code><br>
link: <a href="https://github.com/pytorch/pytorch/issues/52286" class="inline-onebox" rel="noopener nofollow ugc">JIT makes model run x14 times slower in pytorch nightly · Issue #52286 · pytorch/pytorch · GitHub</a></p>
<p>Now, the problem for me was that the latter solution did not fix the 2nd iteration. However as it turns out, setting the baliout_depth to 0 would solve just that.</p>
<p>I’m not sure if this is an acceptable/recommended solution, and/or if there’s a better fix to be coming in a future version of pytorch (as i see ticket 52286 is still open) however maybe you want to try this.</p>
<p>So in conclusion, try:</p>
<pre><code class="lang-python">torch._C._jit_set_bailout_depth(0)
</code></pre>
<p>or if you are writing C++:</p>
<pre><code class="lang-auto">#include &lt;torch/csrc/jit/runtime/graph_executor.h&gt;
torch::jit::getBailoutDepth() = 0;
</code></pre>
<p>and see if it works for you without too much penalty</p>
<p>M.</p>

username: Dobbie
cooked: <p>Thank you very much! It works for me without measurable performance penalty. It seems there is few document about bailout depth and it’s hard to find the solution for me without the help of you guys.</p>

username: michele-arrival
cooked: <p>yes, that’s what they mention on <a href="https://github.com/pytorch/pytorch/issues/52286" class="inline-onebox" rel="noopener nofollow ugc">JIT makes model run x14 times slower in pytorch nightly · Issue #52286 · pytorch/pytorch · GitHub</a>, that <code>_jit_set_bailout_depth</code> is undocumented.</p>
<p>I would expect it to be an option that is not exactly supposed to be accessible by the user, as both <code>_C</code> module and function are coded as “private”. That’s why I wonder if it’s the right solution.</p>