A simple print function could hugely can reduce the significant time delay

username: Jiabao
cooked: <p>I recently found a very weird thing, a simple <code>print</code> could cause significant speedup, which is unexpected.</p>
<p>first you should install the package via:<br>
<code>pip install git+https://github.com/Karbo123/pytorch_hash.git</code></p>
<p>then you can try the following codes to reproduce:</p>
<pre><code class="lang-auto">import torch
from pytorch_hash import pytorch_hash

torch.manual_seed(0)
xyz = torch.randn([10, 3], device="cuda")
depth = torch.randn([10], device="cuda")
uv = xyz[:, :2] / depth.unsqueeze(1)

# print(depth) # magic print function
ind_slot = pytorch_hash(uv.contiguous())
print(ind_slot)
print(depth) # without the magic print, it takes very long time
print("all done!")
</code></pre>
<p>you may observe <code>print(depth)</code> takes very long time to execute, but if you add the <code>print(depth)</code> before pytorch_hash executes, it prints very fast. <strong>I really wonder what actually happen inside the print function?</strong></p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="Jiabao" data-post="1" data-topic="153344">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/j/9fc29f/48.png" class="avatar"> Jiabao:</div>
<blockquote>
<p>you may observe <code>print(depth)</code> takes very long time to execute, but if you add the <code>print(depth)</code> before pytorch_hash executes, it prints very fast.</p>
</blockquote>
</aside>
<p>Could you explain what “it prints very fast” means here?<br>
<code>print</code> statements are synchronizing the device with the host. If you want to profile specific methods use the <code>torch.utils.benchmark</code> util. or synchronize the code via <code>torch.cuda.synchronize()</code>.</p>

username: Jiabao
cooked: <p>Hi, I really appreciate for the quick reply. The real problem here is that, the <code>print()</code> statement causes significant time delay compared to usual <code>print()</code>. We all know generally, printing a tensor should takes very short time (e.g. no more than 1sec).</p>
<p>If you run the codes I provided, you may reproduce the problem, you will find that the simple <code>print(depth)</code> statement will take very long time (e.g. more than 30sec) to print out to the terminal, but actually the tensor <code>depth</code> is quite small (i.e. shape=(10,)), and the tensor size shouldn’t account for the problem, because I think copying the cuda memory to the host memory should be super fast as the memory size is very small.</p>
<p>However, very surprisedly, if you additionally add the <code>print(depth)</code> statement before <code>ind_slot = pytorch_hash(uv.contiguous())</code>, you will find the time delay of the two <code>print(depth)</code> statements will no longer be a problem, they run very quickly as usual (e.g. no more than 1sec).</p>
<p>Also, very interestingly, you can replace the <code>print(depth)</code> to just print one element <code>print(depth[-1])</code>, and it also takes the same effect. But if <code>print(depth)</code> becomes <code>depth[-1]</code>, <code>depth</code>, <code>depth.cpu()</code> or <code>print(depth.cpu())</code>, it just cannot reduce the time delay.</p>
<p>I really want to know why the simple <code>print()</code> can hugely reduce the unexpected time delay?</p>
<p>By the way, I use <code>torch.__version__==1.8.0+cu111</code>, and <code>NVCC==V11.1.74</code>.</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="Jiabao" data-post="3" data-topic="153344">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/j/9fc29f/48.png" class="avatar"> Jiabao:</div>
<blockquote>
<p>We all know generally, printing a tensor should takes very short time (e.g. no more than 1sec).</p>
</blockquote>
</aside>
<p>That’s not entirely true, as it depends where the data comes from.<br>
CUDA operations are executed asynchronously so if you run a code snippet such as:</p>
<pre data-code-wrap="python"><code class="lang-python">out = torch.matmul(x, x)
</code></pre>
<p>the actual matmul kernel will be executed in the background while the CPU can execute the next line of code.<br>
However, since a <code>print</code> statement needs to see the data, transfer it to the host (which will thus synchronize the code), the actual runtime depends on the GPU execution time.<br>
I.e. the <code>print</code> statement in:</p>
<pre data-code-wrap="python"><code class="lang-python">out = torch.malmul(x, x)
print(out)
</code></pre>
<p>has to wait for the GPU to finish its execution (assuming <code>x</code> is on the GPU), move <code>out</code> to the host, and finally print it.</p>
<aside class="quote no-group" data-username="Jiabao" data-post="3" data-topic="153344">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/j/9fc29f/48.png" class="avatar"> Jiabao:</div>
<blockquote>
<p>I really want to know why the simple <code>print()</code> can hugely reduce the unexpected time delay?</p>
</blockquote>
</aside>
<p>Because you are implicitly synchronizing the code and thus get different timings.<br>
Use either the bottleneck util. to profile code or synchronize explicitly via <code>torch.cuda.synchronize()</code> to properly profile the code and get valid times.</p>

username: Jiabao
cooked: <p>Hi, thanks for the detailed reply! I do understand the cuda’s implicit synchronization mechanism. And to check whether explicit synchronization could solve the problem, I try the following codes (only show the core part).</p>
<pre><code class="lang-auto"># print(depth) # magic print function
torch.cuda.synchronize()
ind_slot = pytorch_hash(uv.contiguous())
torch.cuda.synchronize()
print(ind_slot)
torch.cuda.synchronize()
print(depth) # without the magic print, it takes very long time
torch.cuda.synchronize()
print("all done!")
</code></pre>
<p>You can still to see that, after <code>print(ind_slot)</code> is finished (print to the terminial), it takes very very long time to see the next print result (i.e. <code>print(depth)</code>). By the way, I’ve also tried to run the code with <code>CUDA_LAUNCH_BLOCKING=1 python test.py</code>, but it doesn’t help.</p>
<p>One noteworthy thing is that, <strong>the variable <code>depth</code> needn’t to be computed, we just declare it using <code>depth = torch.randn([10], device="cuda")</code>.</strong></p>
<p>But if you additionally add <code>print(depth)</code> to the beginning, the second <code>print(depth)</code> runs very fast as expected. This is very strange.</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="Jiabao" data-post="5" data-topic="153344">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/j/9fc29f/48.png" class="avatar"> Jiabao:</div>
<blockquote>
<p>I do understand the cuda’s implicit synchronization mechanism.</p>
</blockquote>
</aside>
<p>CUDA is not implicitly synchronizing for you, but the <code>print</code> statement will add a sync.</p>
<p>I cannot reproduce any change in the reported time by using or removing the “magic” print statement and get ~3s (2.9-3.1s) on multiple runs.<br>
Maybe creating a profile (e.g. via Nsight Systems) might help you see the execution and check the kernel times.</p>

username: Jiabao
cooked: <p>Thanks, update the pytorch to <code>1.10.2+cu111</code> solves the problem.</p>