Tensor.detach() takes seconds

Alexis: <p>Hello! This code:</p>
<pre><code class="lang-auto">...some model
with torch.no_grad():
                result = self.model(tensor)
                t0 = time.time()
                print(result.shape)
                result = result.cpu()
                print(f"Took seconds {time.time() - t0}")
</code></pre>
<p>has output</p>
<pre><code class="lang-auto">torch.Size([1, 2, 1033, 1033])
Took seconds 4.281269311904907
torch.Size([1, 2, 833, 833])
Took seconds 2.7177305221557617
torch.Size([1, 2, 673, 673])
Took seconds 1.7987253665924072
</code></pre>
<p>I did to know that it is detach() operation. detach() operation takes seconds for a tensor. Why?</p>

KFrank: <p>Hi Alexis!</p>
<aside class="quote no-group" data-username="Alexis" data-post="1" data-topic="169079" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/838e76/48.png" class="avatar"> Alexis:</div>
<blockquote>
<p>detach() operation takes seconds for a tensor. Why?</p>
</blockquote>
</aside>
<p>Calls to cuda operations can return asynchronously, but then a subsequent<br>
call that attempts to use the resulting tensor will block until the cuda call<br>
actually finishes.</p>
<p>Because of this you need to use <code>torch.cuda.synchronize()</code> in order to<br>
get meaningful timings.  Try this:</p>
<pre data-code-wrap="python"><code class="lang-python">with torch.no_grad():
    result = self.model(tensor)
    torch.cuda.synchronize()   # wait for self.model() to actually finish
    t0 = time.time()
    print(result.shape)
    result = result.cpu()
    torch.cuda.synchronize()
    print(f"Took seconds {time.time() - t0}")   # only times result.shape and result.cpu()
</code></pre>
<p>Best.</p>
<p>K. Frank</p>

Alexis: <p>Thank you for your answer!<br>
This code</p>
<pre><code class="lang-auto">torch.cuda.synchronize()
</code></pre>
<p>now takes 4 seconds. Why? Is it time used to calculate model output? I don’t get it.</p>

KFrank: <p>Hi Alexis!</p>
<aside class="quote no-group" data-username="Alexis" data-post="3" data-topic="169079" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/838e76/48.png" class="avatar"> Alexis:</div>
<blockquote>
<p>Is it time used to calculate model output?</p>
</blockquote>
</aside>
<p>Yes, you are most likely seeing the time it takes for your model to run<br>
as <code>synchronize()</code> waits for the gpu to finish what it was doing.  It could<br>
also possibly include time for something the gpu was still doing before<br>
you called <code>result = self.model(tensor)</code>.</p>
<p>Try something like this:</p>
<pre data-code-wrap="python"><code class="lang-python">with torch.no_grad():
    t0 = time.time()
    torch.cuda.synchronize()
    print(f"First synchronize(): {time.time() - t0}")
    t0 = time.time()
    result = self.model(tensor)
    torch.cuda.synchronize()
    print(f"Time for model (tensor): {time.time() - t0}")
    t0 = time.time()
    result = result.cpu()
    torch.cuda.synchronize()
    print(f"Time for result.cpu() {time.time() - t0}")
</code></pre>
<p>The point is that hypothetically your gpu might have still been doing<br>
something when you called <code>model (tensor)</code>.  The <em>first</em> <code>synchronize()</code><br>
waits for the gpu to finish whatever that might have been (if anything).<br>
The time for the first <code>synchronize()</code> tells you how long this whatever<br>
(if anything) took.</p>
<p>This example then calls <code>model (tensor)</code> after which he <em>second</em><br>
<code>synchronize()</code> waits for <code>model (tensor)</code> to finish.  This way the<br>
second timing is therefore just (except for some <em>de minimis</em> <code>time()</code><br>
and <code>print()</code> calls) the amount of time <code>model (tensor)</code> took.</p>
<p>(And then this example times how long it takes to move <code>result</code> to<br>
the cpu, which could also be an interesting number.)</p>
<p>Best.</p>
<p>K. Frank</p>

Alexis: <p>Looks like it waits model to finish calculations.</p>
<p>But why do we have still to wait for operation to complete while we already have result tensor?</p>

KFrank: <p>Hi Aexis!</p>
<aside class="quote no-group" data-username="Alexis" data-post="5" data-topic="169079" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/838e76/48.png" class="avatar"> Alexis:</div>
<blockquote>
<p>But why do we have still to wait for operation to complete while we already have result tensor?</p>
</blockquote>
</aside>
<p>A somewhat simplified technical explanation:</p>
<p>At its outermost level, a pytorch tensor is a data structure in <em>cpu</em> memory.<br>
This data structure contains a “handle” to the underlying data which may<br>
live in either cpu or gpu memory.  (This tensor data structure also contains<br>
a bunch of other useful information.)</p>
<p>When you execute <code>result = self.model (tensor)</code> you set the python<br>
reference <code>result</code> to refer to a pytorch-tensor data structure (on the cpu).<br>
At this point, the outermost shell of this tensor exists (on the cpu) and it<br>
wraps gpu data which may or may not be ready yet.</p>
<p>But your python script can continue executing – for example, it could do<br>
some cpu computation or retrieve some data from a different gpu –  as<br>
long  as it doesn’t attempt to use the not-yet-ready gpu tensor.</p>
<p>To say it another way, you do already have the <em>shell</em> of the <code>result</code> tensor,<br>
but you don’t yet have the full <code>result</code> tensor – in particular, you don’t have<br>
its actual data.</p>
<p>Pytorch has a lot of heavy-duty machinery under the hood, and this is<br>
part of it.  This scheme gives you the convenience of programming in<br>
python (which runs on the cpu) and use python to manipulate gpu tensors<br>
nearly transparently.</p>
<p>Best.</p>
<p>K. Frank</p>