Slow CPU<=>GPU transfer

maralm: <p>Hi,</p>
<p>I have issues with cpu&lt;=&gt;gpu transfer which is extremely slow in pytorch using V100.<br>
I have a cpu to gpu transfer test script. When pinning memory and just transfering, the transfer time is normal:</p>
<pre><code class="lang-auto">import torch
import numpy
import torch.nn as nn
from torch.autograd import Variable
import time
import sys

rank = 0


N = 6000
H = 4000
K = 10

gpu_device = torch.device("cuda:"+str(rank))

w1 = torch.randn(N, H, device=torch.device("cpu"))
w2 = torch.randn(N, H, device=torch.device("cpu"))
w3 = torch.randn(N, H, device=torch.device("cpu"))
w4 = torch.randn(N, H, device=torch.device("cpu"))
w5 = torch.randn(N, H, device=torch.device("cpu"))
w6 = torch.randn(N, H, device=torch.device("cpu"))
w7 = torch.randn(N, H, device=torch.device("cpu"))
w8 = torch.randn(N, H, device=torch.device("cpu"))

ww1 = torch.nn.Parameter(w1).pin_memory()
ww2 = torch.nn.Parameter(w2).pin_memory()
ww3 = torch.nn.Parameter(w3).pin_memory()
ww4 = torch.nn.Parameter(w4).pin_memory()
ww5 = torch.nn.Parameter(w5).pin_memory()
ww6 = torch.nn.Parameter(w6).pin_memory()
ww7 = torch.nn.Parameter(w7).pin_memory()
ww8 = torch.nn.Parameter(w8).pin_memory()

x1 = torch.randn(N, H, dtype=torch.float32, device=gpu_device)
x2 = torch.randn(N, H, dtype=torch.float32, device=gpu_device)
x3 = torch.randn(N, H, dtype=torch.float32, device=gpu_device)
x4 = torch.randn(N, H, dtype=torch.float32, device=gpu_device)
x5 = torch.randn(N, H, dtype=torch.float32, device=gpu_device)
x6 = torch.randn(N, H, dtype=torch.float32, device=gpu_device)
x7 = torch.randn(N, H, dtype=torch.float32, device=gpu_device)
x8 = torch.randn(N, H, dtype=torch.float32, device=gpu_device)

# ww1 = ww1+1
# ww2 = ww2+1
# ww3 = ww3+1
# ww4 = ww4+1
# ww5 = ww5+1
# ww6 = ww6+1
# ww7 = ww7+1
# ww8 = ww8+1

torch.cuda.synchronize()
t_start = time.time()

for i in range(K):

     x1.copy_(ww1.data, non_blocking=True)
     x2.copy_(ww2.data, non_blocking=True)
     x3.copy_(ww3.data, non_blocking=True)
     x4.copy_(ww4.data, non_blocking=True)
     x5.copy_(ww5.data, non_blocking=True)
     x6.copy_(ww6.data, non_blocking=True)
     x7.copy_(ww7.data, non_blocking=True)
     x8.copy_(ww8.data, non_blocking=True)

torch.cuda.synchronize()
t_end = time.time()

t_time = (t_end - t_start)*1000
t_size = K*N*H*4*8/1024/1024
t_bw = t_size/(t_end-t_start)


print('size of transfer in both directions  ', t_size, 'MB')	
print('time taken by transfer  ', t_time, 'mSec')	
print('Effective bandwidth ', t_bw, 'MBps')	

</code></pre>
<p>I get 11GB/sec bandwidth here. But when I change the tensors before transfer, it becomes extremely slow:</p>
<pre><code class="lang-auto">import torch
import numpy
import torch.nn as nn
from torch.autograd import Variable
import time
import sys

rank = 0


N = 6000
H = 4000
K = 10

gpu_device = torch.device("cuda:"+str(rank))

w1 = torch.randn(N, H, device=torch.device("cpu"))
w2 = torch.randn(N, H, device=torch.device("cpu"))
w3 = torch.randn(N, H, device=torch.device("cpu"))
w4 = torch.randn(N, H, device=torch.device("cpu"))
w5 = torch.randn(N, H, device=torch.device("cpu"))
w6 = torch.randn(N, H, device=torch.device("cpu"))
w7 = torch.randn(N, H, device=torch.device("cpu"))
w8 = torch.randn(N, H, device=torch.device("cpu"))

ww1 = torch.nn.Parameter(w1).pin_memory()
ww2 = torch.nn.Parameter(w2).pin_memory()
ww3 = torch.nn.Parameter(w3).pin_memory()
ww4 = torch.nn.Parameter(w4).pin_memory()
ww5 = torch.nn.Parameter(w5).pin_memory()
ww6 = torch.nn.Parameter(w6).pin_memory()
ww7 = torch.nn.Parameter(w7).pin_memory()
ww8 = torch.nn.Parameter(w8).pin_memory()

x1 = torch.randn(N, H, dtype=torch.float32, device=gpu_device)
x2 = torch.randn(N, H, dtype=torch.float32, device=gpu_device)
x3 = torch.randn(N, H, dtype=torch.float32, device=gpu_device)
x4 = torch.randn(N, H, dtype=torch.float32, device=gpu_device)
x5 = torch.randn(N, H, dtype=torch.float32, device=gpu_device)
x6 = torch.randn(N, H, dtype=torch.float32, device=gpu_device)
x7 = torch.randn(N, H, dtype=torch.float32, device=gpu_device)
x8 = torch.randn(N, H, dtype=torch.float32, device=gpu_device)

ww1 = ww1+1
ww2 = ww2+1
ww3 = ww3+1
ww4 = ww4+1
ww5 = ww5+1
ww6 = ww6+1
ww7 = ww7+1
ww8 = ww8+1

torch.cuda.synchronize()
t_start = time.time()

for i in range(K):

     x1.copy_(ww1.data, non_blocking=True)
     x2.copy_(ww2.data, non_blocking=True)
     x3.copy_(ww3.data, non_blocking=True)
     x4.copy_(ww4.data, non_blocking=True)
     x5.copy_(ww5.data, non_blocking=True)
     x6.copy_(ww6.data, non_blocking=True)
     x7.copy_(ww7.data, non_blocking=True)
     x8.copy_(ww8.data, non_blocking=True)

torch.cuda.synchronize()
t_end = time.time()

t_time = (t_end - t_start)*1000
t_size = K*N*H*4*8/1024/1024
t_bw = t_size/(t_end-t_start)


print('size of transfer in both directions  ', t_size, 'MB')	
print('time taken by transfer  ', t_time, 'mSec')	
print('Effective bandwidth ', t_bw, 'MBps')	

</code></pre>
<p>Here bandwidth is dropped to 3-4GB/sec!<br>
Is that expected? How can I optimize the transfer time between gpu and cpu?</p>
<p>PS: I tested pure cuda transfers and I get 12GB/s which is expected. Also, without using pin_memory(), I get 3-4GB/sec regardless of changing the tensor or not.</p>

ptrblck: <p>Your manipulation <code>ww1 = ww1 + 1</code> will create a new tensor, which is not pinned anymore.<br>
You can check it via <code>print(ww1.is_pinned())</code>, which should return <code>False</code> after the creation of the new tensor.</p>

maralm: <p>I see. So is there any way to transfer more efficiently after changing the values? Let’s say I want to transfer model weights during training (which requires changing the values). This low bandwidth is hurting that.</p>

ptrblck: <p>You could use inplace methods (e.g. <code>ww1.add_(1.)</code>) to keep the memory block alive.<br>
Would that work for you or are you using more complicated (and possibly not implemented) methods?</p>

maralm: <p>I am running forward and backward on gpu and transfer gradients to cpu, then run optimizer on cpu and transfer back the updated weights to gpu. I see different performance on different machines and all are much lower than expected.</p>

ptrblck: <p>The optimizers should update the parameter inplace as shown here:</p>
<pre><code class="lang-python">p = nn.Parameter(torch.ones(10).pin_memory())
print(p.is_pinned())
optimizer = torch.optim.Adam([p], lr=1.)
p.grad = torch.ones_like(p)
print(p.is_pinned())
optimizer.step()
print(p.is_pinned())
</code></pre>
<p>What is your use case to update the parameters on the CPU?</p>

maralm: <p>This is part of a multi-process parallel algorithm we are using which requires this.</p>

ptrblck: <p>Could you check the <code>is_pinned()</code> state before and after executing the <code>step()</code> method?<br>
If you are somehow creating new tensors (and thus reallocate new memory), you would need to pin them again.</p>

maralm: <p>I checked it and is_pinned() is true before and after optimizer. But I don’t see any difference in performance with and without pin_memory().</p>

yousiyu: <p>I found this thread interesting. On a related note:</p>
<pre><code class="lang-auto">cuda(non_blocking=True)
</code></pre>
<p>has a quite significant delay (~50% of a blocking call)<br>
Is there anyway to reduce the delay of the non-blocking call?<br>
This is critical to the data transfer from CPU to multiple GPUs. Ideally we’d like to do data transfers in parallel, e.g.</p>
<pre><code class="lang-auto">x = torch.randn(1000, 1000, 1000)
xs = [x.cuda(i, non_blocking=True) for i in range(8)]
</code></pre>