Escaping if statement synchronization

sbelharbi: <p>**-- how to escape <code>if</code> synchronization. it is a huge problem because the evaluation of a simple <code>if</code> ONLY takes .33 sec while the entire forward in large network takes .0001s. here is a way that leads to wrong results because of synch gpu-to-cpu with non-blocking=True <a href="https://discuss.pytorch.org/t/should-we-set-non-blocking-to-true/38234/18">issue</a> !!! WARNING: NEVER use gpu-to-cpu transfer with non-blocking=True. it is NOT safe. see the previous post. **</p>
<p>hi,<br>
it seems that for a cuda tensors <code>a</code> and <code>b</code>, even with 1 element each, the if control will create a synchronization point such as in this example:</p>
<pre><code class="lang-python">if a == b:
    return 0
</code></pre>
<p>it is not the condition <code>a == b</code> that creates the synchronization, it is the <code>if</code>. <code>if</code> seems to trigger <code>a transfer</code> of the result of the comparison to cpu in a blocking way making it similar to call <code>torch.cuda.synchronize()</code> before <code>if</code>.</p>
<p>pointers: <a href="https://discuss.pytorch.org/t/performance-of-if-statements-in-pytorch/101003">here</a> and <a href="https://discuss.pytorch.org/t/execution-time-slowed-done-when-using-an-if-statement/66081/2">here</a>.</p>
<p>we will see here 2 examples: when it is <code>if</code> that evaluates the expression, and second example where we prepare the boolean value ourselves.</p>
<p>the example below is taken from my ‘real’ code. creating a snippet example wont show this issue.</p>
<p>i tried a solution using <code>non_blocking=True</code> but it GIVES THE WRONG RESULTS. this happens because the non-blcoking=True transfer. tensors are created in cpu filled with 0 but the transfer didnt finished yet while the cpu has already evaluated the condition. the doc does not seem to mention this possibility nor the transfer from gpu-2-cpu: <code>Returns a Tensor with the specified device and (optional) dtype. If dtype is None it is inferred to be self.dtype. When non_blocking, tries to convert asynchronously with respect to the host if possible, e.g., converting a CPU Tensor with pinned memory to a CUDA Tensor.</code></p>
<p>this issue of transfering data from gpu to cpu will happen implicitly, but in safe mode == non-blocking=False,  when you run a python operation , i.e.cpu, over cuda tensors, such as control statement, print, casting (int, …), calling cuda_tensor.item(), …</p>
<h2>
<a name="1-if-evaluates-the-condition-1" class="anchor" href="#1-if-evaluates-the-condition-1"></a>1. ‘if’ evaluates the condition:</h2>
<p>example:</p>
<pre><code class="lang-python">        x = x.detach()  # no gradient. x device: cuda.
        tx = time.perf_counter()
        min_x = x.min()
        max_x = x.max()
        print('time min max {}'.format(time.perf_counter() - tx))

        tx = time.perf_counter()
        if min_x == max_x:
            return min_x
        print('simple test Z {}'.format(time.perf_counter() - tx))
</code></pre>
<p>Output:</p>
<pre><code class="lang-auto">time min max 0.0002352111041545868
simple test Z 0.3392159678041935
</code></pre>
<h2>
<a name="2-we-evaluate-the-condition-and-provide-it-to-if-2" class="anchor" href="#2-we-evaluate-the-condition-and-provide-it-to-if-2"></a>2. we evaluate the condition and provide it to ‘if’:</h2>
<p>in this way, we avoid the synchronization by performing the transfer from gpu to cpu with  <code>non_blocking=True</code>. the <a href="https://pytorch.org/docs/stable/generated/torch.Tensor.to.html" rel="noopener nofollow ugc">doc</a> says, that when this variable is true, the transfer is done asynchronously <strong>if possible</strong>.<br>
the worst scenario is that you will endup doing a synch if resulting condition tensor is not ready.<br>
but the issue, tensors on cpu are more likely to be wrong… because the copy is not finished yet while the cpu has already performed the evaluation…</p>
<p>example:</p>
<pre><code class="lang-python">        tx = time.perf_counter()
        min_x = x.min()
        max_x = x.max()
        print('time min max {}'.format(time.perf_counter() - tx))

        tx = time.perf_counter()
        z = ((min_x - max_x) == 0).to(torch.device("cpu"), non_blocking=True)
        print('compute Z {}'.format(time.perf_counter() - tx))

        if z:
            return min_x
        print('simple test Z {}'.format(time.perf_counter() - tx))
</code></pre>
<p>Output:</p>
<pre><code class="lang-auto">time min max 0.00020601600408554077
compute Z 0.0007294714450836182
simple test Z 1.7356127500534058e-05
</code></pre>
<p>we get back to square one if we set <code>non_blocking</code> to false:</p>
<pre><code class="lang-python">        tx = time.perf_counter()
        min_x = x.min()
        max_x = x.max()
        print('time min max {}'.format(time.perf_counter() - tx))

        if ((min_x - max_x) == 0).to(torch.device("cpu"), non_blocking=False):
            return min_x
        print('simple test Z {}'.format(time.perf_counter() - tx))
</code></pre>
<p>Output:</p>
<pre><code class="lang-auto">otsu time min max 0.00021892786026000977
otsu simple test Z 0.3317955397069454
</code></pre>
<h3>
<a name="3-you-wont-see-this-behavior-in-this-snippet-3" class="anchor" href="#3-you-wont-see-this-behavior-in-this-snippet-3"></a>3. you wont see this behavior in this snippet:</h3>
<pre><code class="lang-python">import time

import torch

if __name__ == '__main__':
    seed = 0
    torch.manual_seed(seed)
    torch.cuda.manual_seed(seed)
    torch.cuda.manual_seed_all(seed)

    device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
    a = torch.rand(200, 200).to(device)

    min_ = a.min()
    max_ = a.max()


    t = time.perf_counter()
    if min_ == max_:
        pass
    print('time {}'.format(time.perf_counter() - t))
</code></pre>
<p>thanks</p>

gcramer23: <p>Hi why won’t the described behavior be seen in 3.</p>
<p>It seems as though this a python problem with the if statement because of the CPU.</p>
<p>I am unsure if this will help since I can’t reproduce the problem, but I will torch.equal work for 1? It outputs a bool <a href="https://github.com/pytorch/pytorch/blob/1be1c901aabd3ddcf55af3ee869e611b7f3f43b6/aten/src/ATen/native/native_functions.yaml#L7391" class="inline-onebox" rel="noopener nofollow ugc">pytorch/native_functions.yaml at 1be1c901aabd3ddcf55af3ee869e611b7f3f43b6 · pytorch/pytorch · GitHub</a>. If it does for your if conditions you can write a GPU kernels.</p>

sbelharbi: <aside class="quote no-group" data-username="gcramer23" data-post="2" data-topic="130263">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/gcramer23/48/37599_2.png" class="avatar"> gcramer23:</div>
<blockquote>
<p>Hi why won’t the described behavior be seen in 3.</p>
</blockquote>
</aside>
<p>i think it has something to do with  the kernels.<br>
in my ‘real’ code, <code>x</code> is computed using some forward operations, kernels could take way more time to get them + the realtime load on the gpu at that moment. in the snippet code, i use <code>torch.rand</code>, i assume it is an easy task that we can get the results fast enough that subsequent operations wouldnt be blocked.</p>
<aside class="quote no-group" data-username="gcramer23" data-post="2" data-topic="130263">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/gcramer23/48/37599_2.png" class="avatar"> gcramer23:</div>
<blockquote>
<p>It seems as though this a python problem with the if statement because of the CPU.</p>
</blockquote>
</aside>
<p>yes, the condition is evalauted on gpu and results in a boolean tensor. but in order for the cpu to access it, it ahs to transfer it to cpu.</p>
<aside class="quote no-group" data-username="gcramer23" data-post="2" data-topic="130263">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/gcramer23/48/37599_2.png" class="avatar"> gcramer23:</div>
<blockquote>
<p>I am unsure if this will help since I can’t reproduce the problem, but I will torch.equal work for 1?</p>
</blockquote>
</aside>
<p>i tried almost every imaginable way to get the value of the cuda boolean tensor without causing the blocking but it didnt work. this include torch.equal, indexing, evaluating the condition beforehand, numel(), … all this results in a blocking (synchronization) in order to make sure the cpu evaluates the right value of the tensor. if it is not the case, and the access to the tensor is not synchronized, cpu will produce wrong results.</p>
<p>it doesnt seem there is a way to escape this. lazy transfer from gpu-2-cpu is unsafe and leads to wrong results as mentioned in header parag.</p>

ptrblck: <p>Just skimmed through the topic and I would claim the synchronization for Python data-dependent control flow is expected, since the CPU has to read the values in order to decide which code path to take.<br>
Since the operation might not have finished yet, the host needs to synchronize there.<br>
The same would apply e.g. for <code>print(data)</code>, as the host cannot (or rather should not) print the values of a CUDATensor until the result was created.</p>

udi: <p><a class="mention" href="/u/sbelharbi">@sbelharbi</a><br>
Skimmed through your question. Seems to me that everything is working properly, and the problem is that you don’t (yet) have a correct mental model of what’s going on under the hood.</p>
<p>Let’s start with the bad news - I am quite certain that your forward pass takes much more time than you think. Probably as much time as you think the ‘if’ statement consumes. This time is just being (correctly) hidden from you by the CUDA system. What you have measured as your forward pass timing is just the time required to queue the work onto the CUDA driver and return to your CPU-based program execution.</p>
<p>The good news is that nothing weird is happening and everything you are trying to do is perfectly achievable. With extra work, you can also implement fine grained synchronization avoiding brute force <code>torch.cuda.synchronize()</code>, which is the concurrency equivalent of using a nuke in the battlefield.</p>
<p>Of course you will have to deal with the real run-time of the computation.</p>
<p>Unfortunately the Torch documentation is extremely sparse in regard to synchronization issues and mostly assumes that users are either:</p>
<ul>
<li>naive, sequential CPU-dwellers who transfer all data to the GPU and then do all computation on the GPU from there on</li>
<li>experienced, scarred CUDA programmers who know everything there is to know about asynchronous kernel execution.</li>
</ul>
<p>Your case seems to fall in between these categories so you have been seeing what seems like concurrency voodoo.</p>
<p>The issue of synchronization and async execution is much too big for a post here so I will just give a few basic principles, hopefully these and what I’ve written above will give enough information to seek a solution:</p>
<ul>
<li>
<p><strong>Every time</strong> you execute a torch operation on the GPU, it is “non-blocking”; in the sense that GPU is told to do some work in the future, and execution resumes on the Python CPU thread almost immediately. This frees up your Python program to keep generating future GPU work, or to prepare new data while the GPU is doing its thing.</p>
</li>
<li>
<p>The reason this asynchronous operation doesn’t cause immediate mayhem is that work is <em>queued</em> on something called a CUDA Stream; on the stream everything proceeds in the order you have queued it. The reason most torch users are not aware of this is that there is a “default stream”, and unless the user requests something else, all GPU operations are queued on it, making everything look sequential and calm, as long as your tensors remain in CUDA land.</p>
</li>
<li>
<p>All of this Stream business is a CUDA-only thing. Therefore it applies only to stuff happening on the ‘cuda’ device. On the CPU there are is no such thing as streams and everything is simply synchronous with your Python program, just as most users expect.</p>
</li>
<li>
<p>So we have the GPU kingdom where everything is sequential because it’s on the default stream, and the CPU realm where everything is sequential because it is Python program synchronous. Each of these two domains maintains its own order, and they are largely independent of each other. This of course leaves the border between them as the problem area.</p>
</li>
<li>
<p>By default when you send a tensor across the boundary (nonblocking=False), the operation is <strong>synchronous on both sides</strong>, i.e. the copy is queued on the default CUDA stream AND your Python program is blocked until the tensor is completely moved to the other domain. This is nice and safe. And wastes precious time. This is why torch came up with nonblocking=True.</p>
</li>
<li>
<p>When you use nonblocking transfers, the copy work is queued on the default CUDA stream but the CPU is free to proceed. If you are doing cpu-&gt;GPU transfers, then most likely what happens next is that your Python program will queue further GPU operations for the tensor you have just uploaded. This is ok because the compute work is queued on the same stream with the copy work, and placed after it, so everything will end up blissfully fine.</p>
</li>
<li>
<p>However, when you move data from the GPU to the CPU with nonblocking=True, there is nothing there to protect you implicitly. It will queue the copy on the CUDA default stream, but you Python program is free to proceed. Quite likely your GPU is still busy doing computations that you have previously queued, and hasn’t even “heard” of the CPU-side tensor you are expecting to do further compute on. Therefore that tensor is going to contain zeros or garbage when your CPU starts crunching it.</p>
</li>
</ul>
<p>To properly use a GPU result on the CPU side with a non-blocking transfer, you will need to make sure that the copy to the CPU has completed before you start consuming the data.</p>
<p>Before discussing how this can be accomplished, you have to ask yourself <strong>whether this is at all useful to you</strong>. In many (simple) cases there is nothing useful for the CPU to do until the compute result from the GPU is available. If this is the case, the easiest way to ensure the copy is complete is to do a blocking transfer (set nonblocking=False). Since the copy work is queued on the default CUDA stream, the copy will begin after all computation on the data has finished. Since Python is blocked for it, you CPU computation will only proceed after the copy has finished. This is also when you need to stop the timer if you want to measure how much time the GPU computation really took.</p>
<p>If you decide that you need more advanced asynchronicity in you program, I advise you to study some more about CUDA kernel launching. Keep in mind that every torch operation on Tensors on the GPU is equivalent to a sequence of one or more CUDA kernel launches in succession. You will need to be familiar with the following CUDA concepts and their torch interfaces: Stream, default stream, Event, overlapped computation and data transfer.</p>
<p>I hope this helps.</p>

udi: <p>Just adding something important I realized I haven’t said explicitly:</p>
<p>The reason why you “tried almost every imaginable way to get the value of the cuda boolean tensor without causing the blocking but it didnt work” is that the work for computing the <code>min</code> and <code>max</code> and possibly the comparison between them hasn’t completed yet. You NEED to wait for the GPU to finish computing before you can have a <strong>useful and correct</strong> result.</p>
<p>Yes, with <code>nonblocking=False</code> you can free up your CPU to proceed before the GPU is done computing, but you can’t expect the CPU to have a correct result to use, before the GPU produces it.</p>
<p>In other words, it’s not that  the synchronization “wastes” time, it’s the GPU computation that <em>takes</em> time. The synchronization simply “cures your blindness” to this time.</p>

wangchengtao: <p>As you said,</p>
<aside class="quote no-group" data-username="udi" data-post="5" data-topic="130263">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/u/8dc957/48.png" class="avatar"> udi:</div>
<blockquote>
<p>By default when you send a tensor across the boundary (nonblocking=False), the operation is <strong>synchronous on both sides</strong>, i.e. the copy is queued on the default CUDA stream AND your Python program is blocked until the tensor is completely moved to the other domain. This is nice and safe. And wastes precious time. This is why torch came up with nonblocking=True.</p>
</blockquote>
</aside>
<p>And I also found that in Pytorch docs, " As an exception, several functions such as <a href="https://pytorch.org/docs/stable/generated/torch.Tensor.to.html#torch.Tensor.to" rel="noopener nofollow ugc"><code>to()</code></a> and <a href="https://pytorch.org/docs/stable/generated/torch.Tensor.copy_.html#torch.Tensor.copy_" rel="noopener nofollow ugc"><code>copy_()</code></a> admit an explicit <code>non_blocking</code> argument, which lets the caller bypass synchronization when it is unnecessary. Another exception is CUDA streams, explained below.".<br>
So if we set nonblocking=True, also as you said,</p>
<aside class="quote no-group" data-username="udi" data-post="5" data-topic="130263">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/u/8dc957/48.png" class="avatar"> udi:</div>
<blockquote>
<p>However, when you move data from the GPU to the CPU with nonblocking=True, there is nothing there to protect you implicitly. It will queue the copy on the CUDA default stream, but you Python program is free to proceed. Quite likely your GPU is still busy doing computations that you have previously queued, and hasn’t even “heard” of the CPU-side tensor you are expecting to do further compute on. Therefore that tensor is going to contain zeros or garbage when your CPU starts crunching it.</p>
</blockquote>
</aside>
<p>I wonder that since another exception is CUDA streams, if we use a non-default stream for data transfer, even if we set nonblocking=False, is Python program also free to proceed?</p>

udi: <aside class="quote no-group" data-username="wangchengtao" data-post="7" data-topic="130263">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/wangchengtao/48/63465_2.png" class="avatar"> wangchengtao:</div>
<blockquote>
<p>I wonder that since another exception is CUDA streams, if we use a non-default stream for data transfer, even if we set nonblocking=False, is Python program also free to proceed?</p>
</blockquote>
</aside>
<p>Off the top of my head, based on intuition rather than knowledge, I would say no, with <code>nonblocking=False</code> Python should not be allowed to proceed.</p>
<p>When an operation is said to be “blocking” (= not <code>nonblocking</code>), any sane programmer’s expectation would be that it will <strong>block</strong> progress for the program until completed, regardless of the details of the operation. In this case, regardless what CUDA stream it was launched on.<br>
So I would expect the PyTorch implementation to synchronize with whatever the <em>current</em> CUDA stream is, whether it is the default one or not.<br>
If it is being indiscriminating, PyTorch may <em>also</em> wait on the default stream, or all streams in the process, but I think that’s unlikely and wasteful.</p>
<p>More generally, if you are using non-default streams, it means that you are (or want to be) a bit of a concurrency ninja.<br>
In that case I would recommend not relying on <em>any kind</em> of implicit synchronization like <code>nonblocking=False</code>.<br>
Instead, you should use explicit synchronization, by recording CUDA events on your CUDA streams and explicitly synchronizing to those events in your CPU (Python) program.<br>
<a href="https://pytorch.org/docs/stable/generated/torch.cuda.Event.html#torch.cuda.Event" rel="noopener nofollow ugc">See the related documentation here</a>.</p>

wangchengtao: <p>I understand.<br>
Thanks very much!</p>