Inference on GPU slows after a few iterations

jrmullaney: <p>I’ve run into a bit of a problem when trying to utilise a pre-trained model. I’ve written the model myself based around a UNet architecture.</p>
<p>When iterating inference on a GPU (Nvidia GeForce GTX 750 Ti), I find that it infers incredibly quickly for the first ~25 iterations, then slows dramatically for the remaining iterations.</p>
<p>For example, running the following:</p>
<pre><code class="lang-auto">from convDiff_model import convDiff
import torch
import time

device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")

net = convDiff()
net.to(device)

ran = torch.rand(1, 2, 256, 256).to(device)

with torch.no_grad():
    tic = time.process_time()
    for i in range(48):
        net(ran.float())
        if i % 5 == 4:
            toc = time.process_time()
            print('Iter. %2d to %2d: Mean time: %.3f' % (i-4, i+1, (toc - tic) / 5.) )
            tic = time.process_time()
</code></pre>
<p>produces the output:</p>
<pre><code class="lang-auto">Iter.  0 to  5: Mean time: 0.004
Iter.  5 to 10: Mean time: 0.002
Iter. 10 to 15: Mean time: 0.002
Iter. 15 to 20: Mean time: 0.002
Iter. 20 to 25: Mean time: 0.084
Iter. 25 to 30: Mean time: 0.442
Iter. 30 to 35: Mean time: 0.442
Iter. 35 to 40: Mean time: 0.442
Iter. 40 to 45: Mean time: 0.444
</code></pre>
<p>so you can see that there’s a factor of ~200x slowdown between the first twenty iterations and the last twenty. Oddly, it seems that including/removing <code>with torch.no_grad():</code> makes no difference. Inference remains fast if I <code>empty_cache()</code> every, say, 15 iterations, but that in itself uses 0.45 second, which negates it as a working solution.</p>
<p>[To clarify, in the above code snippet, I’ve removed the loading of the trained model, as this particular behaviour doesn’t seem to depend on the weightings.]</p>
<p>The full model architecture is available at this GitHub repo: <a href="https://github.com/jrmullaney/ConvDiff/tree/master/code" rel="nofollow noopener">https://github.com/jrmullaney/ConvDiff/tree/master/code</a></p>
<p>Is this behaviour normal? I’d have thought each iteration would be independent.</p>
<p>Apologies if this has already been addressed in another post; I had a good look round but couldn’t find any solutions.</p>
<p>Thanks for your help.</p>

ptrblck: <p>CUDA operations are asynchronous, so you would need to synchronize the code before starting and stopping the timer via <code>torch.cuda.synchronize()</code>.<br>
Currently the first iterations are most likely just profiling the kernel launches and will then block in the <code>net(ran)</code> line of code after a few iterations.</p>

jrmullaney: <p>Thanks, <a class="mention" href="/u/ptrblck">@ptrblck</a>, for the very quick response.</p>
<p>Indeed, when I add <code>torch.cuda.synchronize()</code> in the following location:</p>
<pre><code class="lang-auto">with torch.no_grad():
    tic = time.process_time()
    for i in range(48):
        net(ran.float())
        if i % 5 == 4:
            toc = time.process_time()
            print('Iter. %2d to %2d: Mean time: %.3f' % (i-4, i+1, (toc - tic) / 5.) )
            torch.cuda.synchronize()
            tic = time.process_time()
</code></pre>
<p>I get:</p>
<pre><code class="lang-auto">Iter.  0 to  5: Mean time: 0.005
Iter.  5 to 10: Mean time: 0.002
Iter. 10 to 15: Mean time: 0.002
Iter. 15 to 20: Mean time: 0.003
Iter. 20 to 25: Mean time: 0.002
Iter. 25 to 30: Mean time: 0.002
Iter. 30 to 35: Mean time: 0.002
Iter. 35 to 40: Mean time: 0.002
Iter. 40 to 45: Mean time: 0.002
</code></pre>
<p>but the wall clock time between each of those lines is a good 6-8 second. So, I guess each iteration is getting pushed through the GPU quickly, but then a backlog forms after a few iterations (if I interpreted your reply correctly; I’m new to PyTorch and CUDA).</p>
<p>Thanks for clearing that up. Presumably I’ll have to tweak my architecture if I want major speedups?</p>

ptrblck: <p>The <code>toc</code> counter isn’t synchronized, so while the GPU is busy in <code>net(ran.float())</code>, you are immediately stopping the time, print the profiling result, synchronize (and thus wait until the GPU is finished), start the time and stop it immediately after a few iterations.<br>
Try to add another <code>torch.cuda.synchronize()</code> before the <code>toc</code> call and rerun the code.</p>

jrmullaney: <p>Thanks again,</p>
<p>With the following:</p>
<pre><code class="lang-auto">with torch.no_grad():
    tic = time.process_time()
    for i in range(48):
        net(ran.float())
        if i % 5 == 4:
            torch.cuda.synchronize()
            toc = time.process_time()
            print('Iter. %2d to %2d: Mean time: %.3f' % (i-4, i+1, (toc - tic) / 5.) )
            torch.cuda.synchronize()
            tic = time.process_time()
</code></pre>
<p>[is that what you meant?], I get:</p>
<pre><code class="lang-auto">Iter.  0 to  5: Mean time: 1.782
Iter.  5 to 10: Mean time: 1.778
Iter. 10 to 15: Mean time: 1.781
Iter. 15 to 20: Mean time: 1.780
Iter. 20 to 25: Mean time: 1.783
Iter. 25 to 30: Mean time: 1.782
Iter. 30 to 35: Mean time: 1.782
Iter. 35 to 40: Mean time: 1.779
Iter. 40 to 45: Mean time: 1.779
</code></pre>
<p>So the 48 iterations take about 80-90 seconds (wall time) in total.</p>
<p>Is that what you’d expect?</p>
<p>Thanks,</p>

jrmullaney: <p>Oh - and I should say. I’m not hung up about the tics and tocs - those were just for demonstration purposes to profile how the time per iteration changes throughout the loop. The important thing is the total wall time to process a bunch of images.</p>
<p>In production, I’d like this to infer around 100 512x512 images at a time. If this took sub-1s total (i.e., 100*0.002s as in the first few iterations in my first post), this would be great, and even sub-10s would be fine, but 45s is not really viable.</p>

ptrblck: <p>The new timing should reflect the time for 5 iterations.<br>
This should fit the processing time for the complete dataset, if you scale the number up.</p>
<aside class="quote no-group" data-username="jrmullaney" data-post="5" data-topic="84605">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jrmullaney/48/25263_2.png" class="avatar"> jrmullaney:</div>
<blockquote>
<p>So the 48 iterations take about 80-90 seconds (wall time) in total.</p>
</blockquote>
</aside>
<p>I might misunderstand your profiling, but you are accumulating the time for 5 iterations, which would finish in <code>1.78/5*48~=17s</code>.</p>

jrmullaney: <p>Ah. No - that’s not the accumulation of 5, it’s the average over five. With the addition of the <code>synchronise()</code> step, it now corresponds to:<br>
( t0 + t1 + t2 + t3 + t4 + synchronise() ) / 5.<br>
where t0 etc is the time it takes to do the first iteration of net() etc.</p>
<p>If I instead do:</p>
<pre><code class="lang-auto">with torch.no_grad():

    for i in range(48):
        tic = time.process_time()
        net(ran.float())
        toc = time.process_time()
        print('Iter. %2d : time: %.3f' % (i, (toc - tic)) )
</code></pre>
<p>I get:</p>
<pre><code class="lang-auto">Iter.  0 : time: 0.019
Iter.  1 : time: 0.002
Iter.  2 : time: 0.002
Iter.  3 : time: 0.002
.....
Iter. 21 : time: 0.002
Iter. 22 : time: 0.002
Iter. 23 : time: 0.094
Iter. 24 : time: 1.777
Iter. 25 : time: 1.743
Iter. 26 : time: 1.781
.....
Iter. 44 : time: 1.750
Iter. 45 : time: 1.775
Iter. 46 : time: 1.779
Iter. 47 : time: 1.774
</code></pre>
<p>where the <code>.....</code> correspond to lines showing virtually the same times (I’ve removed the <code>synchronised()</code> to give a truer indication of the time per inference).</p>
<p>With those times, the total compute time for 100 images will be:<br>
0.002 x 24 + 1.77 x 76 = 135s</p>

ptrblck: <p>Thanks for the correction. I missed the average in the calculation.</p>
<p>You could set <code>torch.backends.cudnn.benchmark = True</code> at the beginning of your script to use the cudnn heuristics in order to find the fastest kernels for your current workload.</p>
<p>Additionally, you could check out <a href="https://pytorch.org/docs/master/amp.html">automatic-mixed precision</a> (available in master and the nightly binaries), especially if you are using a newer GPU with TensorCores (which unfortunately shouldn’t help much with the GTX 750).</p>

jrmullaney: <p>OK - I’ll give those a try.</p>
<p>Thanks for your help and speedy replies! Very much appreciated.</p>