Model forward suddenly slows down

MingSun-Tse: <p>Hi, I run the following code for testing my model on a single GPU:</p>
<pre><code class="lang-auto">model.eval()
for i, x in enumerate(test_inputs):
    t0 = time.time()
    with torch.no_grad():
        y = model(x)
    print(f'forward time: {time.time() - t0:.4f}s')
</code></pre>
<p>The model is a simple MLP with 11 layers, <span class="hashtag">#neurons</span> in each layer is 1024. Inputs is a tensor of shape [50, 1024].</p>
<p>My problem is: For the first 23 loops, the forward time is like 0.0015s. While after the <span class="hashtag">#23</span> loop, the forward time suddenly jumps to ~0.28s and stay at this level afterwards. I am wondering why the forward speed suddenly slows down at some point while the previous loops work really fast? I feel this is related to pytorch cuda cache. What would you suggest to resolve this problem? Thanks!</p>
<p>My environment:<br>
OS: Ubuntu20.04<br>
CUDA: 11.2<br>
GPU: Tesla V100 16GB<br>
Python: 3.9.5<br>
Pytorch1.9.0</p>

zetyquickly: <p>Hi <a class="mention" href="/u/mingsun-tse">@MingSun-Tse</a></p>
<p>Is it possible that the <code>test_inputs</code> is a dataloader and the loading of the last Xs just slower then the first ones?</p>

MingSun-Tse: <p>Hi <a class="mention" href="/u/zetyquickly">@zetyquickly</a> , thanks for your reply! <code>test_inputs</code> is not a dataloader. It is simply a tensor. For each forward, the <code>x</code> is exactly of the same shape.</p>

zetyquickly: <p>May you write the same test on colab, and see is there an issue? Btw send the link to that colab notebook</p>

InnovArul: <p>Apart from software (pytorch, python, etc) related problems, you may also suspect hardware issues.</p>
<p>For instance, below is an observation that I had come across. I am not sure if this is your case too.<br>
If you try noticing the temperature of GPU in <code>nvidia-smi</code> and if the temperature &gt;= 90C, GPU might enforce software/hardware throttling that would slowdown the GPU.<br>
In my case, for the first few loops, the temperature was less, so it was fast. However, during training due to the rise in temperature, throttle came into effect.</p>

ptrblck: <p>CUDA operations are executed asynchronously, so to get a valid timing you would have to manually synchronize the code before starting and stopping the timer via <code>torch.cuda.synchronize()</code>.<br>
In your current code you might ptofile the kernel launches and the first blocking operation would accumulate the timings of other kernels.</p>

MingSun-Tse: <p>Hi <a class="mention" href="/u/zetyquickly">@zetyquickly</a> , thank you! Sorry it is a private project, so I cannot share the exact testing code on colab.</p>
<p>Hi <a class="mention" href="/u/innovarul">@InnovArul</a> , thank you! I checked the temperature. Idle GPU temperature is 34°C, during inference it is 53°C, so it is not about the temperature.</p>

MingSun-Tse: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> , thank you! I tried to add <code>torch.cuda.synchronize()</code> at the end of each for loop. Indeed, it works! The time of each model forward now is consistently ~0.0015s. The only problem is, <code>torch.cuda.synchronize()</code> itself takes ~0.27s, so the complete testing takes nearly the same total time as before. I guess this points out where is the problem. I also note this issue was reported <a href="https://github.com/NVIDIA-AI-IOT/torch2trt/issues/471" rel="noopener nofollow ugc">before</a>. Now I am wondering, if this can be resolved somehow. Any clue?</p>
<p>Thank you ALL!</p>

ptrblck: <p>It’s not an “issue”, but just how asynchronous execution works.<br>
The wrong timings are created since the CPU can launch the CUDA kernels and continue executing the host code until a synchronization is needed. In the meantime the GPU is working in the background.<br>
If you thus start the timer, launch the kernel, and directly stop the timer afterwards, the GPU might stil be busy and you would not profile the actual GPU workload.<br>
As already described, you need to synchronize the code to be able to profile the kernel timings on the GPU or the better way would be to use <code>torch.utils.benchmark</code>, which was created for use cases like these.</p>

MingSun-Tse: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a>, I see. Great explanation! Thanks!</p>