GPU operations seem not asynchronous

username: unbreading
cooked: <p>Question: GPU operations are not asynchronous in my case.</p>
<p>Description:<br>
I run something like<br>
<code>t = time.time()</code><br>
<code>loss = model(x)</code><br>
<code>loss.backward()</code><br>
<code>cost = time.time() - t</code><br>
but I got almost the same result with/without <code>torch.cuda.synchronize()</code>.<br>
I have called <code>.cuda()</code> for model.(the model is on gpu)<br>
There should be no gpu-cpu transfer(i.e. <code>.cpu()</code> or <code>.gpu()</code>) in model’s <code>forward()</code> method</p>
<p>It seems that GPU operations are not asynchronous in my case.<br>
Why?<br>
Or how can I check if I mistakely sync during model’s <code>forward()</code> method?</p>

username: ptrblck
cooked: <p>Some operations like <code>.item()</code> will add a synchronization point in your code.<br>
Could you post the model definition, so that we could have a look for unwanted sync points?</p>
<p>Also, how large is your workload?</p>

username: SimonW
cooked: <p>IIRC, backward is a synchronization point in pytorch.</p>

username: unbreading
cooked: <p>I check all parts in my model by printing out their execution time without  <code>torch.cuda.synchronize()</code>.<br>
One part with GRU and LayerNorm has a 100x more time cost than other part.</p>
<p>Code in this part is sth like:</p>
<pre><code class="lang-auto">v1 = self._gru(self._ln1(v1 + v0))
v2 = self._gru(self._ln2(v2 + v0))
v3 = self._gru(self._ln3(v3 + v0))
</code></pre>
<p>Here <code>self._ln1</code> and <code>self._ln2</code> and <code>self._ln3</code> are instances of <code>nn.LayerNorm</code><br>
And <code>self._gru</code> is a Residual-GRU  with code</p>
<pre><code class="lang-auto">class ResidualGRU(nn.Module):
    def __init__(self, hidden_size, dropout, num_layers):
        super(ResidualGRU, self).__init__()
        self.enc_layer = nn.GRU(input_size=hidden_size, hidden_size=hidden_size // 2, num_layers=num_layers,
                                batch_first=True, dropout=dropout, bidirectional=True)
        self.enc_ln = nn.LayerNorm(hidden_size)

    def forward(self, input):
        output, _ = self.enc_layer(input)
        return self.enc_ln(output + input)
</code></pre>
<p>May I ask if GRU will cause sync? Or what’s wrong with these code.</p>

username: unbreading
cooked: <p>But I still got same result even if I separately get execution time for <code>forward</code> and <code>backward</code>.</p>
<p>And it seems that <code>backward </code> should be asynchronous since it is a gpu operation?</p>

username: ptrblck
cooked: <p>You cannot time separate modules without synchronization, as the modules will just accumulate the time when a sync was added, e.g. before the next iteration.<br>
This is highly misleading and is thus not recommended.</p>
<p>Anyway, since your execution time is apparently the same, I would like to check the model.<br>
Could you post the model definition and the shapes for the tensors so that we could run it?</p>

username: unbreading
cooked: <p>The problem does not exist any more.<br>
I did not “control variable” when compare the time cost between with/without synchronize. So sth like CPU load, samples order etc may affect the result.<img src="https://discuss.pytorch.org/images/emoji/apple/pensive.png?v=9" title=":pensive:" class="emoji" alt=":pensive:"><br>
When I run code with/without <code>CUDA_LAUNCH_BLOCKING=1</code> at the same time with the same random seed, the time cost are different.</p>
<p>Anyway, thank you very much.  <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>