Bizzare extra time consumption in Pytorch (GPU) 1.1.0 / 1.2.0?

username: ZhengdiYu
cooked: <p>Hi, everyone,</p>
<p>I have been struggling with an extra and confusing time problem which I encountered for twice:</p>
<p><strong>First time:</strong><br>
I have two model running simultaneously with a joint loss but it is quite slow(up to 0.4s~0.5s+) for each step. So I was trying to improve my efficiency through checking some time spot using <code>time.time()</code>. And I found out The first small model is just fine(like 0.002s). However when I was checking the second model. At the beginning of my second model’s forwarding, Something weird just happen:</p>
<pre><code class="lang-auto">def forward(feature, coords, permutation):
    """
    feature : [B, d, N, 1]
    coords: [B, N ,3]
    """
    t1 = time.time()
    coords: [:, permutation]  
    t2 =  time.time()
    feature = feature[:, :, permutation]   
    t3 = time.time()
    print(t2 - t1) # running for 0.24s
    print(t3 - t2)  # 0.00...  very short anyway
</code></pre>
<pre><code class="lang-auto">def forward(feature, coords, permutation):
    """
    feature : [B, d, N, 1]
    coords: [B, N ,3]
    """
    t1 = time.time()
    feature = feature[:, :, permutation]   
    t2 =  time.time()
    coords: [:, permutation]  
    t3 = time.time()
    print(t2 - t1) # running for 0.24s
    print(t3 - t2)  # 0.00...  very small anyway
</code></pre>
<p>It seems that the what really counts is its sequence instead of what the code truly does.<br>
<strong>Second:</strong></p>
<p>I was debugging my code using <code>time.time()</code> like before, and the time consumption was becoming even weirder. In my second model, I found a very time consuming line :<code> a[b]</code> , especially when there is an action that put b.to(device), it is just a simple slice operation but it took 0.4s+, <strong>95%+ of the whole model</strong>. So I change a way to get the slice, removing the <code>.to(device)</code> part it decreased to several <strong>ms</strong> like it is solved. However, unfortunately, I found the total time still remain unchanged , what is more, I found <strong>another line</strong> of code <strong>became suddenly very time consuming</strong> with no reason just like a ghost with absolutely no changes done to it. So I did rewrite the sentence again. But it is vain, because the “time consuming” part just moved to next part like a virus.</p>
<p>However I found a common feature between them. Every “ghost time consuming” part will be related to (to(device) / device / … etc.) , and when you fix one of them, another one will just become the “ghost time consuming” part again.</p>
<p><strong>Is there a preparation stage of GPU like it can’t be removed ? If yes, why dose every step needs this ? Literally every step in my iteration has a “ghost” time spot.</strong></p>
<p>Yours sincerely,<br>
ZD</p>

username: ptrblck
cooked: <p>CUDA operations are executed asynchronously, so that you would need to synchronize the code before starting and stopping the timers via <code>torch.cuda.synchronize()</code>.<br>
Since you are not synchronizing the code at the moment, the returned times will accumulate all previously enqueued kernels until a synchronization is forced.</p>
<p>Could you add the synchronizations and rerun your profiling, please?</p>

username: ZhengdiYu
cooked: <p>Thanks for your reply,</p>
<p>I tried using this before .time() like this</p>
<pre><code class="lang-auto">torch.cuda.synchronize()
t1 = time.time()
...some operations here...
torch.cuda.synchronize()
t2 = time.time()
print("Time consumption:", )
</code></pre>
<p>I add <code>torch.cuda.synchronize()</code> before every <code>time.time()</code> and I have got quite a different result from the situation using only <code>time.time()</code> without <code>torch.cuda.synchronize()</code> . I finally localized the time consuming part and found the code below actually takes over 0.4s+. From what you told me, could I assume that <code>torch.cuda.synchronize()</code> could help me get absolute time of these operations ?<br>
If is, then my question became:</p>
<pre><code class="lang-auto"># inputs shape: [1, 128, 120, 120]
torch.max(inputs.view(b, -1), dim=1)[0]  # very time consuming ~0.4s +
torch.max(inputs) # very quick ~ 0.0003s
</code></pre>
<p>Why is there the difference ? I couldn’t see the mechanism behind it.  From what I see it shouldn’t be like this</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="ZhengdiYu" data-post="3" data-topic="87843">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/z/3da27b/48.png" class="avatar"> ZhengdiYu:</div>
<blockquote>
<p>From what you told me, could I assume that <code>torch.cuda.synchronize()</code> could help me get absolute time of these operations ?</p>
</blockquote>
</aside>
<p>Synchronizations will make sure the timers grab the runtime of all operations between the sync points.<br>
Otherwise your profiling would run into the “ghost” operations you’ve mentioned, i.e. you would most of the time profile the kernel launch only and would accumulate the complete runtime of all scheduled kernels into the next blocking operation.</p>
<aside class="quote no-group" data-username="ZhengdiYu" data-post="3" data-topic="87843">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/z/3da27b/48.png" class="avatar"> ZhengdiYu:</div>
<blockquote>
<p>Why is there the difference ?</p>
</blockquote>
</aside>
<p>The first operation will return the max values in <code>dim=1</code>, so the output would contain the <code>values</code> and <code>indices</code> in the shape <code>[b]</code>, while the second call treats the <code>inputs</code> tensor as a flat tensor and will only return the max. value as a scalar.</p>
<p>That being said, I cannot reproduce the large timing difference using this code:</p>
<pre data-code-wrap="python"><code class="lang-python">x = torch.randn(1024, 1024*128, device='cuda')

nb_iters = 1000

torch.cuda.synchronize()
t0 = time.time()
for _ in range(nb_iters):
    out = torch.max(x, dim=1)
torch.cuda.synchronize()
t1 = time.time()
print((t1 - t0)/nb_iters)
&gt; 0.012184044361114503

torch.cuda.synchronize()
t0 = time.time()
for _ in range(nb_iters):
    out = torch.max(x)
torch.cuda.synchronize()
t1 = time.time()
print((t1 - t0)/nb_iters)
&gt; 0.012206459522247314
</code></pre>

username: ZhengdiYu
cooked: <p>Thank you for your answer! It’s much clearer now.</p>
<p>After I captured the “ghost time” by using <code>torch.cuda.synchronize()</code> namely the code I showed above, I just change<code> torch.max(torch.max(inputs.view(b, -1), dim=1)[0])</code> into <code>torch.max(inputs)</code> and the “ghost” time just vanished from my whole project after a long time of debugging and putting that off to next stage. However I don’t understand the reason behind it. Maybe it’s due to the computation graph or something else to which I didn’t pay much attention. Because when I do these two separately outside my project in a new file, I got the same result as you produced.</p>
<p>Anyway, the problem has gone even though I haven’t figured out why it is that way.</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="ZhengdiYu" data-post="5" data-topic="87843">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/z/3da27b/48.png" class="avatar"> ZhengdiYu:</div>
<blockquote>
<p>Anyway, the problem has gone even though I haven’t figured out why it is that way.</p>
</blockquote>
</aside>
<p>If you don’t synchronize the code before starting and stopping the timer, you won’t capture the real runtime of the CUDA operations, since they are executed asynchronously in the background.<br>
Basically your Python script will call the CUDA operation, which will be scheduled on the GPU and executed, while the Python script can just execute the next lines of code.<br>
E.g. in this code snippet:</p>
<pre data-code-wrap="python"><code class="lang-python">t0 = time.time()
y = x.sum() # x is a CUDATensor
t1 = time.time()
</code></pre>
<p>you would only profile the Python overhead of calling the summation CUDA kernel, not the execution itself.</p>
<p>The code would block automatically, e.g. if you are trying to use a result tensor:</p>
<pre data-code-wrap="python"><code class="lang-python"># async execution
output = model(x)
loss = criterion(output, target)
loss.backward()
# blocking
print(loss.cpu().item())
</code></pre>
<p>These synchronization points might be in different parts of your code, so that your profilng result would accumulate all previous CUDA operation into the first blocking operation, which will yield wrong results.</p>