Introducing SpeedTorch: 4x speed CPU->GPU transfer, 110x GPU->CPU transfer

Santosh-Gupta: <p>                    <a href="https://i.imgur.com/wr4VaUV.png" target="_blank" class="onebox" rel="nofollow noopener">
            <img src="https://i.imgur.com/wr4VaUV.png" width="589" height="500">
          </a>

</p>
<aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://github.githubassets.com/favicon.ico" class="site-icon" width="16" height="16">
      <a href="https://github.com/Santosh-Gupta/SpeedTorch" target="_blank" rel="nofollow noopener">GitHub</a>
  </header>
  <article class="onebox-body">
    <img src="https://avatars2.githubusercontent.com/u/5524261?s=400&amp;v=4" class="thumbnail onebox-avatar" width="60" height="60">

<h3><a href="https://github.com/Santosh-Gupta/SpeedTorch" target="_blank" rel="nofollow noopener">Santosh-Gupta/SpeedTorch</a></h3>

<p>Library for fastest pinned CPU &lt;-&gt; GPU transfer . Contribute to Santosh-Gupta/SpeedTorch development by creating an account on GitHub.</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>This is my submission to the Pytorch Hackathon. It’s a library I made for Pytorch, for fast transfer between pinned CPU tensors and GPU pytorch variables. The inspiration came from needed to train large number of embeddings, which don’t all fit on GPU ram at a desired embedding size, so I needed a faster CPU &lt;-&gt; GPU transfer method. This also allows using any optimizer for sparse training, since every embedding contained in the Pytorch embedding variable receives an update, previously only Pytorch’s SGD, Adagrad, and SparseAdam were suitable for such training.</p>
<p>In addition to augmenting parameter sizes, you can use to increase the speed of which data on your CPU is transferred to Pytorch Cuda variables.</p>
<p>Also, SpeedTorch’s GPU tensors are also overall faster then Pytorch cuda tensors, when taking into account both transferring two and from (overall 2.6x faster). For just transfering to a Pytorch Cuda, Pytorch is still faster, but significantly slower when transfering from a Pytorch Cuda variable.</p>
<p>I have personally used this to nearly double the embedding size of embeddings in two other projects, by holding half the parameters on CPU. The training speed is decent thanks to the fast CPU&lt;-&gt;GPU exchange.</p>
<aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://github.githubassets.com/favicon.ico" class="site-icon" width="16" height="16">
      <a href="https://github.com/Santosh-Gupta/Research2Vec2" target="_blank" rel="nofollow noopener">GitHub</a>
  </header>
  <article class="onebox-body">
    <img src="https://avatars2.githubusercontent.com/u/5524261?s=400&amp;v=4" class="thumbnail onebox-avatar" width="60" height="60">

<h3><a href="https://github.com/Santosh-Gupta/Research2Vec2" target="_blank" rel="nofollow noopener">Santosh-Gupta/Research2Vec2</a></h3>

<p>Updated from the original Research2vec project. Contribute to Santosh-Gupta/Research2Vec2 development by creating an account on GitHub.</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://github.githubassets.com/favicon.ico" class="site-icon" width="16" height="16">
      <a href="https://github.com/Santosh-Gupta/lit2vec2" target="_blank" rel="nofollow noopener">GitHub</a>
  </header>
  <article class="onebox-body">
    <img src="https://avatars2.githubusercontent.com/u/5524261?s=400&amp;v=4" class="thumbnail onebox-avatar" width="60" height="60">

<h3><a href="https://github.com/Santosh-Gupta/lit2vec2" target="_blank" rel="nofollow noopener">Santosh-Gupta/Lit2Vec2</a></h3>

<p>Contribute to Santosh-Gupta/Lit2Vec2 development by creating an account on GitHub.</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>There’s a bit of a learning curve for the very first time getting started with it, so as soon as you run into any sort of friction, feel free to ask a question on the project gitter</p>
<p><a href="https://gitter.im/SpeedTorch" class="onebox" target="_blank" rel="nofollow noopener">https://gitter.im/SpeedTorch</a></p>
<p>And I’ll answer them.</p>
<p>                    <a href="https://i.imgur.com/6o8C1BP.gif" target="_blank" class="onebox" rel="nofollow noopener">
            <img src="https://i.imgur.com/6o8C1BP.gif" width="498" height="498">
          </a>

</p>

colesbury: <p>Interesting work. This can speed-up operations like:</p>
<pre><code class="lang-auto">cpu_tensor[idx] = gpu_tensor.cpu()
</code></pre>
<p>and</p>
<pre><code class="lang-auto">gpu_tensor = cpu_tensor[idx].cuda()
</code></pre>
<p>The library does this by masquerading the pinned-memory CPU tensor as a cupy GPU tensor and using the cupy GPU indexing kernels. It’s not really speeding up the CPU-GPU copy part, it’s avoiding the overhead of a separate CPU indexing operation. (The GitHub page mentions “memmaps”, but it’s not using memory mapping at all).</p>
<p>For CPU &lt;-&gt; GPU copies, the relative speed-up depends on how slow the PyTorch CPU indexing operation is. For the (131072, 128) size, this can be as fast as 2 ms on a multi-core machine or ~25 ms on collab notebook with only one or two virtual CPUs.  For comparison, the CPU-GPU copy from pinned memory is consistent at ~5-6 ms. (Again, it’s not the CPU-GPU copy that’s sped up; it’s the separate CPU indexing operation that’s avoided)</p>
<p>There’s also a significant indexing performance bug in PyTorch 1.1 and 1.2 (fixed in the nightly builds) that makes the indexing much slower. The apparent GPU &lt;-&gt; GPU indexing speed-ups are entirely due to this bug. You could either use the nightly builds or use <code>index_select</code> / <code>index_copy_</code> instead of a[idx] notation in 1.1/1.2 to avoid that slow down in vanilla PyTorch.  (Bug is described in <a href="https://github.com/pytorch/pytorch/pull/24083" rel="nofollow noopener">#24083</a> for reference)</p>
<p>The library is missing some synchronization. Particularly, when copying from GPU to pinned memory (masquerading as GPU via cupy), you need to synchronize before accessing the CPU data; otherwise it may not be consistent.</p>
<p>There’s a few bugs in the benchmark code, mostly minor:</p>
<ol>
<li>
<p><code>sampl = np.random.uniform(low=-1.0, high=1.0, size=(1000000, 128))</code>. This is float64; it should be float32 to match everything else:<br>
<code>sampl = np.random.uniform(low=-1.0, high=1.0, size=(1000000, 128)).astype(np.float32)</code></p>
</li>
<li>
<p>The timing code should have a torch.cuda.synchronize() at the end. Instead of:</p>
</li>
</ol>
<pre><code class="lang-auto">torch.cuda.synchronize()
cupy.cuda.Device().synchronize()

runningTime=0
for i in range(numSamples):
    start = time.time()

    with torch.no_grad():

        gadgetCPU.insertData(u_embeddings.weight.data, indexess )

    end = time.time()
    runningTime = runningTime + end - start

print('set corpus. cupy live pinned')
print(runningTime/numSamples)
</code></pre>
<p>It should be written as:</p>
<pre><code class="lang-auto">torch.cuda.synchronize()
cupy.cuda.Device().synchronize()

start = time.time()
for i in range(numSamples):
    with torch.no_grad():
        gadgetCPU.insertData(u_embeddings.weight.data, indexess )

torch.cuda.synchronize()
end = time.time()
runningTime = end - start

print('set corpus. cupy live pinned')
print(runningTime/numSamples)
</code></pre>

ptrblck: <p>Thanks for the detailed analysis.<br>
I had a quick look at the implementations and besides the missing synchronizations, I’m still trying to figure out, if the CUPY arrays can hold any CPU data, or if it’s restricted to the device only?<br>
E.g. while this line of code</p>
<pre><code class="lang-python">gadgetCPU.insertData(u_embeddings.weight.data,  indexess )
</code></pre>
<p>seems to copy the CUDATensor from <code>u_embeddings</code> onto the CPU, it seems as if the underlying data never left the GPU:</p>
<pre><code class="lang-python">gadgetCPU.CUPYcorpus.device
Out[4]: &lt;CUDA Device 0&gt;
</code></pre>
<p>I’m not really experienced in CUPY so I might miss something.</p>

colesbury: <p>As far as I can tell, CuPy is only intended to hold CUDA data, but in this case it’s actually holding CPU data (pinned memory). You can check with something like:</p>
<pre><code class="lang-auto">cupy.cuda.runtime.pointerGetAttributes(gadgetCPU.CUPYcorpus.data.ptr).memoryType
</code></pre>
<p>This will print 1 (= cudaMemoryTypeHost). On gadgetGPU it’ll print 2 (=cudaMemoryTypeDevice).  (<a href="https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__TYPES.html#group__CUDART__TYPES_1g13de56a8fe75569530ecc3a3106e9b6d" rel="nofollow noopener">cudaMemoryType  reference</a>)</p>
<p>You can do something similar in PyTorch  from the C++ API using torch::from_blob. Here’s an <a href="https://gist.github.com/colesbury/11b3308539f5162b178cc229a2aac9c5" rel="nofollow noopener">example</a>. Note there’s a <a href="https://github.com/pytorch/pytorch/blob/caed485873a750fa698c402e7d64f9b1c386cca9/aten/src/ATen/templates/Functions.h#L32-L38" rel="nofollow noopener">check</a> in from_blob that tries to prevent this sort of thing, but the check is broken (what luck!).</p>

Santosh-Gupta: <p>Thanks for the detailed analysis! I was very curious about what was going on.</p>
<blockquote>
<p>The GitHub page mentions “memmaps”, but it’s not using memory mapping at all</p>
</blockquote>
<p>Ahhh that’s a vestige from my initial approach to using memmaps, I overlooked changing that, it’s fixed now.</p>
<p>I updated the benchmarking code, and I’ll link this analysis to the ‘how it works’ section.</p>
<p>I’ll also include a section for when to use SpeedTorch.</p>
<blockquote>
<p>you need to synchronize before accessing the CPU data; otherwise it may not be consistent.</p>
</blockquote>
<p>So adding <code>torch.cuda.synchronize()</code> and <code>cupy.cuda.Device().synchronize()</code> before the transfer? By consistency, do you mean in terms transfer times?</p>