Parallel training All_Reduce on single GPU

username: Yi_Zhang
cooked: <p>Hi,</p>
<p>I want to use data parallel to train my model on single GPU. I followed the example of Pytorch DISTRIBUTED DATA PARALLEL, and pass the same device_id to 4 processes. With all-reduce sync method, it runs even slower than using a single process. The interesting thing is by disabling all_reduce sync-up for gradients, there is a great speed up of training itself. So I think the GPU has extra compute capacity for multiple training process, but the bottleneck is all-reduce method. Any one know the reason for this bottleneck? Is there any other way to sync param gradients without using all_reduce? Thanks.</p>

username: mrshenli
cooked: <blockquote>
<p>I want to use data parallel to train my model on single GPU. I followed the example of Pytorch DISTRIBUTED DATA PARALLEL, and pass the same device_id to 4 processes. With all-reduce sync method, it runs even slower than using a single process.</p>
</blockquote>
<p>The <code>all_reduce</code> op expects each process to exclusively work on a different GPU. If the same GPU is shared across processes, it does not guarantee to work.</p>
<blockquote>
<p>The interesting thing is by disabling all_reduce sync-up for gradients, there is a great speed up of training itself.</p>
</blockquote>
<p>How did you disable that in DDP?</p>
<blockquote>
<p>So I think the GPU has extra compute capacity for multiple training process, but the bottleneck is all-reduce method.</p>
</blockquote>
<p>Looks like your model is small enough such that each op will just occupy a subset of resources in the GPU.</p>
<blockquote>
<p>Is there any other way to sync param gradients without using all_reduce?</p>
</blockquote>
<p>Since you just use one GPU, you can try using <a href="https://pytorch.org/docs/stable/notes/multiprocessing.html" rel="noopener nofollow ugc">multi-processing</a>. Say launch a main process and use <code>torch.multiprocessing</code> to spawn 4 subprocesses. Use <code>torch.multiprocessing.SimpleQueue</code> to pass grad tensors from sub-processes back to the main process, let the main process accumulate them, and then pass the result back to all subprocesses.</p>
<p>The test below can serve as a example for <code>SimpleQueue</code>:</p>
<aside class="onebox githubblob" data-onebox-src="https://github.com/pytorch/pytorch/blob/2c4b4aa81bc8dba8272e9c7190edcaa3e114ec15/test/test_multiprocessing.py#L580-L600">
  <header class="source">

      <a href="https://github.com/pytorch/pytorch/blob/2c4b4aa81bc8dba8272e9c7190edcaa3e114ec15/test/test_multiprocessing.py#L580-L600" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/2c4b4aa81bc8dba8272e9c7190edcaa3e114ec15/test/test_multiprocessing.py#L580-L600" target="_blank" rel="noopener nofollow ugc">pytorch/pytorch/blob/2c4b4aa81bc8dba8272e9c7190edcaa3e114ec15/test/test_multiprocessing.py#L580-L600</a></h4>



    <pre class="onebox"><code class="lang-py">
      <ol class="start lines" start="580" style="counter-reset: li-counter 579 ;">
          <li>def test_event_multiprocess(self):</li>
          <li>    event = torch.cuda.Event(enable_timing=False, interprocess=True)</li>
          <li>    self.assertTrue(event.query())</li>
          <li></li>
          <li>    ctx = mp.get_context('spawn')</li>
          <li>    p2c = ctx.SimpleQueue()</li>
          <li>    c2p = ctx.SimpleQueue()</li>
          <li>    p = ctx.Process(</li>
          <li>        target=TestMultiprocessing._test_event_multiprocess_child,</li>
          <li>        args=(event, p2c, c2p))</li>
          <li>    p.start()</li>
          <li></li>
          <li>    c2p.get()  # wait for until child process is ready</li>
          <li>    torch.cuda._sleep(50000000)  # spin for about 50 ms</li>
          <li>    event.record()</li>
          <li>    p2c.put(0)  # notify child event is recorded</li>
          <li></li>
          <li>    self.assertFalse(event.query())</li>
          <li>    c2p.get()  # wait for synchronization in child</li>
          <li>    self.assertTrue(event.query())</li>
          <li>    p.join()</li>
      </ol>
    </code></pre>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>


username: Yi_Zhang
cooked: <p>Hi <a class="mention" href="/u/mrshenli">@mrshenli</a>, thanks for reply. I was following this example to use a average_gradients function, which calls all_reduce:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/3/f/3fe9f4f6d83102462e3ab9f1332d8e4dca502782.png" alt="截屏2020-09-15 下午1.49.16" data-base62-sha1="97pkweD1rFzjN7PZDZfA7JtduTM" width="546" height="132"><br>
So to disable all_reduce, I just didn’t call this function when doing the training. And I notice there are two different examples, the other one uses DDP model and the sync-up step is done in backward computation, but I want to have a customized sync-up method, that’s why I choose the way with a separate average_gradients function.</p>
<p>I have tried using SimpleQueue to pass some large data, but it seems slow when calling .get(), I’ll try to use it for passing grad tensors and see how it performs.</p>