First call to torch.rand consistently slower than subsequent calls

minimario: <p>I did a simple experiment shown below (1.13.1+cu117). The first call to torch.rand is consistently slower than the subsequent calls. I benchmarked on both CUDA (top) and CPU (bottom).</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/3/63a2cb05b3f8cb039ebceabce513f4936c883998.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/63a2cb05b3f8cb039ebceabce513f4936c883998" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/6/3/63a2cb05b3f8cb039ebceabce513f4936c883998_2_690x378.jpeg" alt="image" data-base62-sha1="edq3edZZj1xt50C2Ry3pECe8hVC" width="690" height="378" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/6/3/63a2cb05b3f8cb039ebceabce513f4936c883998_2_690x378.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/6/3/63a2cb05b3f8cb039ebceabce513f4936c883998_2_1035x567.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/6/3/63a2cb05b3f8cb039ebceabce513f4936c883998.jpeg 2x" data-dominant-color="444243"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1063×583 209 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>On CPU, the effect disappears, though the first torch.sum is consistently slower than the rest.<br>
I am profiling like so, in case there is something wrong with my profiling procedure.</p>
<pre><code class="lang-auto">CUDA_LAUNCH_BLOCKING=1 kernprof -lv test.py &amp;&amp; python -m line_profiler test.py.lprof
</code></pre>
<p>Could someone explain what goes on internally that makes this first call so much slower than the second on CUDA but not CPU? And, why on CPU the first call to torch.sum is still much slower than the subsequent calls?</p>
<p>I am told the effect disappears on PyTorch 2.0. Why is this the case? <a title="I also asked on Twitter here">(I also asked on Twitter here, there is some discussion but not a definite answer)</a></p>

ptrblck: <p>I don’t know how <code>kernprof</code> profiles the workload but CUDA is initialized lazily and an overhead in the first calls is expected. This is due to:</p>
<ul>
<li>the initialization of the CUDA driver and the CUDA context</li>
<li>starting with CUDA 11.7 lazy module loading is enabled by default, which will load the needed CUDA kernel into the CUDA context in its first call to save GPU memory</li>
<li>memory allocations via <code>cudaMalloc</code> are slow and synchronizing, which is why PyTorch uses a caching mechanism to reuse memory if possible.</li>
</ul>

minimario: <p>Cool, thanks a lot! Just a few follow-up questions and clarifications:</p>
<ul>
<li>The initialization of the CUDA driver + context should be included in the torch.cuda.init() line, so per the first image this accounts for 0.26s of the slowness, is that right?</li>
<li>For the second point, basically you mean that the kernels needed to create a random 10k x 10k matrix are loaded only in the first call but not the rest. Can you explain how that saves GPU memory?</li>
<li>For your third point, this only happens on CUDA and not on CPU?</li>
</ul>
<p>Is there a reasonable way to profile the magnitude of the effects of the second and third points (not necessarily with kernprof), e.g. seeing how much time is spent the correct CUDA kernel into the CUDA context?</p>

ptrblck: <p>I don’t know how to interpret your profiler outputs and since CUDA ops are executed asynchronously, I would be careful.<br>
The proper way would be to use e.g. Nsight Systems as described <a href="https://dev-discuss.pytorch.org/t/using-nsight-systems-to-profile-gpu-workload/59">here</a> and profile the entire workload which would show the overhead in the first calls.</p>