Increase computational cost while repeating use the trained CNN models

yuwei_geng: <p>Hi,I have some problems of the neural nework models.<br>
There is a trained CNN models and i need to repeated use this model. Here is the code.<br>
self.CNN.eval()<br>
with torch.no_grad():<br>
for i in range(self.TestSteps):<br>
aa = time.time()<br>
initial = self.CNN(initial)<br>
bb = time.time()<br>
print(bb - aa)<br>
The computational cost for each loop is 0.002s but after a few loops, the computational cost increase to 0.3s.<br>
0.0029914379119873047<br>
0.0009975433349609375<br>
0.0019941329956054688<br>
0.0025336742401123047<br>
0.0010161399841308594<br>
0.001994609832763672<br>
0.0009975433349609375<br>
0.003017902374267578<br>
0.001968860626220703<br>
0.0015769004821777344<br>
0.001043081283569336<br>
0.002002239227294922<br>
0.17526578903198242<br>
0.3638777732849121<br>
0.35142970085144043<br>
0.3363661766052246<br>
0.3505854606628418<br>
0.3499774932861328<br>
0.41155457496643066<br>
0.35362887382507324<br>
0.3807251453399658<br>
Could anyone can help me to figure out this problem, I really appreciate your help.</p>

ptrblck: <p>If you are using your GPU for this model execution you would need to synchronize the code before starting and stopping the host timers since CUDA kernels are executed asynchronously.</p>

yuwei_geng: <p>Is that correct if i rewrite the code as this:<br>
self.CNN.eval()<br>
with torch.no_grad():<br>
for i in range(self.TestSteps):<br>
torch.cuda.synchronize()<br>
aa = time.time()<br>
initial = self.CNN(initial)<br>
torch.cuda.synchronize()<br>
bb = time.time()<br>
print(bb - aa)<br>
but the computation time always be 0.3s with torch.cuda.synchronize().<br>
0.34552931785583496<br>
0.35210347175598145<br>
0.37857532501220703<br>
0.3551297187805176<br>
0.3482997417449951<br>
0.34366416931152344<br>
0.3459787368774414<br>
0.3441014289855957</p>

ptrblck: <p>Your code is not properly formatted, but looks correct from what I can see.</p>
<aside class="quote no-group" data-username="yuwei_geng" data-post="3" data-topic="219353">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/yuwei_geng/48/76521_2.png" class="avatar"> yuwei_geng:</div>
<blockquote>
<p>but the computation time always be 0.3s with torch.cuda.synchronize().</p>
</blockquote>
</aside>
<p>Yes, your previous profiling was invalid and was not profiling the kernel execution time.</p>

yuwei_geng: <p>Thanks for the explanation. Just to confirm — does this mean the actual GPU runtime for each CNN model inference is around 0.3 seconds, rather than the 0.002 seconds I initially observed? I’m curious why it can be that fast initially but cannot consistently maintain that speed. Since I’m working on improving efficiency, I’m hoping to achieve a meaningful speedup by leveraging the trained CNN model.</p>
<p>Additionally, I noticed that when I manually release CUDA memory using <code>torch.cuda.empty_cache()</code>, the inference time remains consistently around 0.002 seconds — but only when I don’t call <code>torch.cuda.synchronize()</code></p>
<p>self.CNN.eval()<br>
with torch.no_grad():<br>
for i in range(self.TestSteps):<br>
aa = time.time()<br>
initial = self.CNN(initial)<br>
bb = time.time()<br>
print(bb - aa)<br>
torch.cuda.empty_cache()<br>
However, if I include <code>torch.cuda.synchronize()</code>, the measured time still stays around 0.3 seconds.</p>

ptrblck: <aside class="quote no-group" data-username="yuwei_geng" data-post="5" data-topic="219353">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/yuwei_geng/48/76521_2.png" class="avatar"> yuwei_geng:</div>
<blockquote>
<p>Just to confirm — does this mean the actual GPU runtime for each CNN model inference is around 0.3 seconds, rather than the 0.002 seconds I initially observed?</p>
</blockquote>
</aside>
<p>Yes, since your initial profiling is wrong and is not measuring the GPU kernel execution time.</p>
<aside class="quote no-group" data-username="yuwei_geng" data-post="5" data-topic="219353">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/yuwei_geng/48/76521_2.png" class="avatar"> yuwei_geng:</div>
<blockquote>
<p>I’m curious why it can be that fast initially but cannot consistently maintain that speed.</p>
</blockquote>
</aside>
<p>The kernels were never as fast as your profile showed since you were not waiting for the GPU to complete its execution. Instead you were measuring the Python, C++, backend, etc. overhead to launch the kernel(s) as well as implicit synchronizations.<br>
As mentioned before: CUDA kernels are executed asynchronously so you would need to synchronize host timers before starting and stopping them or you could use CUDA events to measure the execution time on the device.</p>
<aside class="quote no-group" data-username="yuwei_geng" data-post="5" data-topic="219353">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/yuwei_geng/48/76521_2.png" class="avatar"> yuwei_geng:</div>
<blockquote>
<p>but only when I don’t call <code>torch.cuda.synchronize()</code></p>
</blockquote>
</aside>
<p>If you want to measure the GPU kernel execution time with host timers you need to synchronize the code. Otherwise your profiling is wrong and misleading.</p>

yuwei_geng: <p>Thank you so much for the detailed explanation — that makes perfect sense now.</p>