How to overlap h2d and training?

lzypku: <p>I am trying to train neural networks on gpu devices by a large dataset, and the h2d time for the mini-batch is mostly the same as the training time. So I wonder if there is any way for pytorch to overlap the mini-batch time and the training time, maybe something like multi-processing?</p>

lzypku: <p>The h2d means: copy the data from cpu to gpu. Like this operation: <code>data = data.cuda()</code><br>
The h2d time for the minibatch means: the time for copy the data of a minibatch from cpu to gpu.</p>
<p>And the training time means: the time for training the model by the data of a minibatch. The training operation always like this: <code>training_result = model(input_data)</code></p>
<p>Firstly, we should load our training data of a minibatch to the gpu device by <code>data.cuda()</code> , which cost some time, we just name it <code>h2d_time</code> . And then, we will train our model by <code>model(data)</code> , which cost some time, we juest name it <code>training_time</code> . So, the whole time will be <code>h2d_time + training_time</code> for training one minibatch.</p>
<p>However, I wonder if there is any way to overlap the <code>training_time</code> and the <code>h2d_time</code> . For example, we can load the next minibatch data when the last minibatch data is still training.</p>

albanD: <p>Hi,</p>
<p>Thanks for the clear description.</p>
<p>The CUDA is asynchronous for execution by default. So it is already very likely that a lot of overlap is happening.<br>
Note that you should use <code>torch.cuda.synchronize()</code> if you record times on the CPU to make sure you measure the proper time!</p>

lzypku: <p>You mean that our users do not need do anything for the overlap? Previously, I guess maybe something like multiprocessing would be useful <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>By the way, could you please give more details or codes for the use of <code>torch.cuda,synchronize()</code> to record times? Thanks very much.</p>

albanD: <blockquote>
<p>You mean that our users do not need do anything for the overlap?</p>
</blockquote>
<p>No! The CPU will only wait for the GPU to finish computing things when the values are actually needed on the CPU side (when you try to print a Tensor, send it to cpu wit <code>.cpu()</code> or get the content as a python number with <code>.item()</code>, etc).</p>
<p>You can play with this by creating a cuda Tensor and apply simple operations on it and use <code>time.time()</code> to see what takes time (I did not run the code as I don’t have a GPU on this machine, so there might be typo):</p>
<pre data-code-wrap="python"><code class="lang-python">import torch
from torch.cuda import synchronize
from time import time

small_tensor = torch.rand(10, device="cuda")
medium_tensor = torch.rand(1000, device="cuda")
# I used random values for the size here, there might need to be bigger for you
# to see a significant difference depending on your GPU
large_tensor = torch.rand(100000, device="cuda")
# Make sure all is properly finished on the GPU side
synchronize()

start = time()
# Do an inplace op to avoid any output allocation overhead
small_tensor.add_(1)
print("Small no sync ", time() - start)

start = time()
# Do an inplace op to avoid any output allocation overhead
medium_tensor.add_(1)
print("Medium no sync ", time() - start)

start = time()
# Do an inplace op to avoid any output allocation overhead
large_tensor.add_(1)
print("Large no sync ", time() - start)
print("All of these should take more or less the same time to run")
print("even if you increase the size a LOT")
print("This basically just measure how long it takes to queue the task on the GPU")


synchronize()
start = time()
# Do an inplace op to avoid any output allocation overhead
small_tensor.add_(1)
synchronize()
print("Small no sync ", time() - start)

synchronize()
start = time()
# Do an inplace op to avoid any output allocation overhead
medium_tensor.add_(1)
synchronize()
print("Medium no sync ", time() - start)

synchronize()
start = time()
# Do an inplace op to avoid any output allocation overhead
large_tensor.add_(1)
synchronize()
print("Large no sync ", time() - start)
print("In this one, you should see the runtime actually going up with the size!")
</code></pre>
<p>As a bonus, you can run that with the env variable “CUDA_LAUNCH_BLOCKING=1”. It will force the whole cuda api to be synchronous. And so if you set this, you should see similar results both with and without synchronization.</p>

lzypku: <p><img src="https://discuss.pytorch.org/uploads/default/original/3X/4/e/4eb0131cfe4944c470013a436e69219e0790e212.png" alt="image" data-base62-sha1="be6tODVmF7ccelN23v5XIPWtDzA" width="551" height="156"></p>
<p>I just run it. But it seems like the run time go down with the size.</p>

albanD: <p>Ok maybe you want to increase the Tensor sizes A LOT <img src="https://discuss.pytorch.org/images/emoji/apple/stuck_out_tongue.png?v=9" title=":stuck_out_tongue:" class="emoji" alt=":stuck_out_tongue:"><br>
Thinking back at it, they are very very small.<br>
Maybe do something like a regular image: <code>1, 3, 240, 240</code> for the medium and something even bigger for the large</p>

mstniy: <p>I was testing this, and came up with the following script, which does block on the kernels’ execution, evidenced by the large delay between the prints surrounding the loop:</p>
<pre><code class="lang-auto">import torch
from torch.cuda import synchronize

tensor1 = torch.rand((10000,10000), device="cuda")
tensor2 = torch.rand((10000,10000), device="cuda")
tensorres= torch.empty_like(tensor1, device="cuda")
N=10000
print('Waiting for the GPU')
synchronize()
print('Done')

print('Queueing up kernels to the GPU')
for i in range(N):
    tensorres = tensor1*tensor2
print('Done')
</code></pre>
<p>The CPU never actually uses the values of the GPU tensors. Any idea why this might be happening?</p>

ptrblck: <aside class="quote no-group" data-username="mstniy" data-post="8" data-topic="93635">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mstniy/48/44773_2.png" class="avatar"> mstniy:</div>
<blockquote>
<p>which does block on the kernels’ execution</p>
</blockquote>
</aside>
<p>How did you verify that the launches are blocking?<br>
Profiling your code shows that the CPU can run ahead and add the kernel launches to the queue while the GPU is busy with their execution.<br>
I added a final sync to avoid the script exiting while the GPU isn’t done yet.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/6/a67d6344f5edf03230d00d0c7b56cabfdcbee5ff.png" data-download-href="https://discuss.pytorch.org/uploads/default/a67d6344f5edf03230d00d0c7b56cabfdcbee5ff" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/a/6/a67d6344f5edf03230d00d0c7b56cabfdcbee5ff_2_690x91.png" alt="image" data-base62-sha1="nKPTVSMO5QQo6oqvexujJvzVohV" width="690" height="91" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/a/6/a67d6344f5edf03230d00d0c7b56cabfdcbee5ff_2_690x91.png, https://discuss.pytorch.org/uploads/default/optimized/3X/a/6/a67d6344f5edf03230d00d0c7b56cabfdcbee5ff_2_1035x136.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/a/6/a67d6344f5edf03230d00d0c7b56cabfdcbee5ff_2_1380x182.png 2x" data-dominant-color="D0D9E2"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1876×248 28.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

mstniy: <p>I didn’t use the profiler, actually, and was judging off the time it takes to queue the kernels.<br>
Fiddling around a bit more, and using the nsight profiler, I get this interesting behaviour:<br>
With N=1000, queuing up the kernels takes negligible time (~0.005s on my setup)<br>
With N=2000, however, the duration jumps to ~2.3 seconds, so something is going on.<br>
Here are the corresponding nsight timelines, where the top one is for N=1000, and the bottom one is for N=2000 :</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/f/cf494acd110e3b90ab5e9b863b8f6784d3c80294.png" data-download-href="https://discuss.pytorch.org/uploads/default/cf494acd110e3b90ab5e9b863b8f6784d3c80294" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/c/f/cf494acd110e3b90ab5e9b863b8f6784d3c80294_2_690x380.png" alt="image" data-base62-sha1="tzJRzU5ZQfSHPUsaqojL9xZLjyk" width="690" height="380" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/c/f/cf494acd110e3b90ab5e9b863b8f6784d3c80294_2_690x380.png, https://discuss.pytorch.org/uploads/default/optimized/3X/c/f/cf494acd110e3b90ab5e9b863b8f6784d3c80294_2_1035x570.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/c/f/cf494acd110e3b90ab5e9b863b8f6784d3c80294_2_1380x760.png 2x" data-dominant-color="EBE9EB"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1553×856 69.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Note that the initialization takes ~3 seconds in both, but after that N=2000 takes significantly more time to queue the kernels, much more than 2x.</p>
<p>Thought that was interesting. It might not be of practical importance to queue thousands of kernels.</p>

ptrblck: <p>I’ve used your previously posted code of <code>N=10000</code> and didn’t see this behavior, so I don’t know what’s causing the change in your code for <code>N=2000</code>.</p>

mstniy: <p>That’s odd. So on your setup queuing the kernels takes negligible time even for N=10000?</p>