Multiple GPU parallel computation

username: Kevinzhu
cooked: <p>I have four batch dataset and I would like to know if I can process them in parallel with four GPU. I have looked at the tutorial of DataParallel function. However, it is a bit different for me because I’m not actually running a model or network. I simply run an objective function and compute the gradient of it. So I’m simply constructing my code follows the computation flow, and then package it as a function. Is there anyway that I can make it parallel with several GPUs?</p>
<p>Kevin</p>

username: eqy
cooked: <p>As a first step you might want to see if explicitly assignment tensors to different devices e.g., <code>cuda:0</code> and <code>cuda:1</code> and running the computation yields any speedup, as the CUDA operations should be asynchronous and be parallelizable on different GPUs. However, if your batch dimension is 4, then there may be bottlenecks due to underutilization depending on how much computation the objective function contains. Additionally, you might want to investigate packaging your objective function into an nn.Module even if it doesn’t follow a typical architecture as it would allow you to use <code>DistributedDataParallel</code> which reduces overheads caused by a single-process setup.</p>

username: Kevinzhu
cooked: <p>Thanks for your suggestions. I may not be very clear about my setting. Basically I have four datasets, and each of them consists of same size of training data. So I think you can view it as four independent batch, and each batch has a N dimension. That is why I think I should have four GPU running four datasets. Currently I’m running these four datasets with a for loop, so I expect to see parallel GPU computation would speed up the whole process.</p>
<p>By the way, is there any useful document that I can read to package my function into a module? Or what keyword should I use to search for it?</p>
<p>Kevin</p>

username: eqy
cooked: <p>You might want to read some canonical training scripts e.g., the ImageNet example here:</p><aside class="onebox githubfolder" data-onebox-src="https://github.com/pytorch/examples/tree/master/imagenet">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="32" height="32">

      <a href="https://github.com/pytorch/examples/tree/master/imagenet" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h3><a href="https://github.com/pytorch/examples/tree/master/imagenet" target="_blank" rel="noopener nofollow ugc">examples/imagenet at master · pytorch/examples</a></h3>

  <p><a href="https://github.com/pytorch/examples/tree/master/imagenet" target="_blank" rel="noopener nofollow ugc">master/imagenet</a></p>

  <p><span class="label1">A set of examples around pytorch in Vision, Text, Reinforcement Learning, etc.</span></p>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>One place to start might be to see if you could package your objective function in the forward function of an nn.Module:<br>
<a href="https://pytorch.org/docs/stable/generated/torch.nn.Module.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/generated/torch.nn.Module.html</a></p>
<p>My original suggestion is that if you have four datasets, you could explicitly specify different devices for the <code>.cuda()</code> calls for your different datasets: <a href="https://pytorch.org/docs/stable/generated/torch.Tensor.cuda.html" class="inline-onebox" rel="noopener nofollow ugc">torch.Tensor.cuda — PyTorch 1.10 documentation</a>, which should be able to automatically parallelize things for simple workflows that don’t call <code>torch.cuda.synchronize</code> somewhere.</p>