How could I copy data from cpu to gpu asynchronically?

username: coincheung
cooked: <p>Hi,</p>
<p>My test code is like this:</p>
<pre><code class="lang-auto">a = torch.randn(16, 3, 16, 16)
a1 = a.cuda(1)
a2 = a.cuda(2)
a3 = a.cuda(3)
a4 = a.cuda(4)
torch.cuda.synchronize()

a = torch.randn(16, 3, 1204, 1024)
t1 = time.time()
a1 = a.cuda(1)
a2 = a.cuda(2)
a3 = a.cuda(3)
a4 = a.cuda(4)
t2 = time.time()
torch.cuda.synchronize()
print(t2 - t1)

a = torch.randn(16, 3, 1204, 1024)
t3 = time.time()
a1 = a.cuda(1)
t4 = time.time()
torch.cuda.synchronize()
print(t4 - t3)
</code></pre>
<p>The results is:</p>
<pre><code class="lang-auto">0.18216228485107422
0.03782033920288086
</code></pre>
<p>Which means that copying one cpu tensor to one gpu tensor is faster than copying same tensor to multi-gpu, the process seems to be a one-by-one operation. How could I make the copy in parallel please?</p>

username: ptrblck
cooked: <p>You would have to synchronize the code before starting and stopping the timer, while your current code stops the timers already and synchronizes afterwards.</p>

username: coincheung
cooked: <aside class="quote no-group" data-username="coincheung" data-post="1" data-topic="116018">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/c/ad7895/48.png" class="avatar"> coincheung:</div>
<blockquote>
<p><code>torch.cuda.synchronize()</code></p>
</blockquote>
</aside>
<p>Hi,</p>
<p>Will this line work to synchronize ?</p>

username: ptrblck
cooked: <p>Yes, this will synchronize the default device.</p>

username: coincheung
cooked: <p>So what is the most efficient way to copy one tensor to multi-gpus in parallel ?</p>

username: ptrblck
cooked: <p>The copy code looks alright, but your profiling code is still wrong, since you are stopping the timer without a previous synchronization. Did you create new profiles and checked the time?</p>

username: coincheung
cooked: <p>HI,</p>
<p>I changed my code like this:</p>
<pre><code class="lang-auto">from concurrent.futures import ThreadPoolExecutor

executor = ThreadPoolExecutor(max_workers=8)


a = torch.randn(16, 3, 16, 16)
a1 = a.cuda(1)
a2 = a.cuda(2)
a3 = a.cuda(3)
a4 = a.cuda(4)

a = torch.randn(16, 3, 1204, 1024)
def move(device_id):
    data = a.cuda(device_id)
    torch.cuda.synchronize(device_id)
    return data

torch.cuda.synchronize(1)
torch.cuda.synchronize(2)
torch.cuda.synchronize(3)
torch.cuda.synchronize(4)
t1 = time.time()
#  a1 = a.cuda(1)
#  a2 = a1.cuda(2)
#  a3 = a1.cuda(3)
#  a4 = a1.cuda(4)
#  a1 = a.cuda(1, non_blocking=True)
#  a2 = a.cuda(2, non_blocking=True)
#  a3 = a.cuda(3, non_blocking=True)
#  a4 = a.cuda(4, non_blocking=True)
a1, a2, a3, a4 = list(executor.map(move, [1, 2, 3, 4]))
torch.cuda.synchronize(1)
torch.cuda.synchronize(2)
torch.cuda.synchronize(3)
torch.cuda.synchronize(4)
t2 = time.time()
print(t2 - t1)

print(a1.device)
print(a2.device)
print(a3.device)
print(a4.device)

a = torch.randn(16, 3, 1204, 1024)
t3 = time.time()
a1 = a.cuda(1)
torch.cuda.synchronize(1)
t4 = time.time()
print(t4 - t3)
</code></pre>
<p>And what printed is like this:</p>
<pre><code class="lang-auto">0.06666159629821777
cuda:1
cuda:2
cuda:3
cuda:4
0.02723860740661621
</code></pre>
<p>It seems that the operation is not in parallel, how could I refine this please ?</p>