Torch.nn.parallel.data_parallel for distributed training: backward pass model update

username: hotcheetos_puff
cooked: <p>When using torch.nn.parallel.data_parallel for distributed training, models are copied onto multiple GPU’s and can complete a forward pass without the copied models effecting each other. However, how do the copied models interact during the backward pass? How are the model weights updated on each GPU?</p>
<p>When reading the documentation [1] I see the explanation:<br>
“gradients from each replica are summed into the original module”</p>
<p>but I’m not sure how to interpret this statement. My best guess is that each GPU’s model is updated with the sum of all GPU’s model’s gradients, which I would then interpret that there is locking across GPU’s so they each start training on a new mini-batch only after they all finish processing their current mini-batch.</p>
<p>[1] <a href="https://pytorch.org/docs/master/nn.html?highlight=dataparallel#dataparallel-layers-multi-gpu-distributed" rel="nofollow noopener">https://pytorch.org/docs/master/nn.html?highlight=dataparallel#dataparallel-layers-multi-gpu-distributed</a></p>

username: albanD
cooked: <p>Hi,</p>
<p>Each gpu compute the gradients for it’s part of the batch. Then they are accumulated on the “main” model where the weight update is done. Then this “main” model shares it’s weight to all the other gpus so that all models have the same weight.</p>

username: hotcheetos_puff
cooked: <p>Hi Alban,</p>
<p>Thanks for clarifying.  Does this mean that this parallelization utilizes locking, ensuring that each GPU model updates its weights from the “main” model before moving on to the next mini-batch?</p>

username: albanD
cooked: <p>Yes the locking is builtin and the weights will properly be updated before they are used.</p>

username: sanchit2843
cooked: <p>Hey, I am facing somme issue with data parallel. I am training on 4 v100 with a batchsize of 1. The time of forward pass seems to scale but the time of backward pass is taking 4* times in comparison to 1 V100. So there is no significant boost in speed when using 4 gpus. I guess the backward pass is taking place on single gpu. I am using nn.parallel.DataParallel, is there any solution to this problem? I can share more details if you want.</p>

username: mrshenli
cooked: <p>Hey <a class="mention" href="/u/sanchit2843">@sanchit2843</a></p>
<p>When using <code>batch_size == 1</code>, <code>DataParallel</code> won’t be able to parallelize the computation, as the input cannot be chunked by the <code>scatter</code> linked below.</p>
<aside class="onebox githubblob" data-onebox-src="https://github.com/pytorch/pytorch/blob/df8d6eeb19423848b20cd727bc4a728337b73829/torch/nn/parallel/data_parallel.py#L151">
  <header class="source">

      <a href="https://github.com/pytorch/pytorch/blob/df8d6eeb19423848b20cd727bc4a728337b73829/torch/nn/parallel/data_parallel.py#L151" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/df8d6eeb19423848b20cd727bc4a728337b73829/torch/nn/parallel/data_parallel.py#L151" target="_blank" rel="noopener nofollow ugc">pytorch/pytorch/blob/df8d6eeb19423848b20cd727bc4a728337b73829/torch/nn/parallel/data_parallel.py#L151</a></h4>



    <pre class="onebox"><code class="lang-py">
      <ol class="start lines" start="141" style="counter-reset: li-counter 140 ;">
          <li>def forward(self, *inputs, **kwargs):</li>
          <li>    if not self.device_ids:</li>
          <li>        return self.module(*inputs, **kwargs)</li>
          <li></li>
          <li>    for t in chain(self.module.parameters(), self.module.buffers()):</li>
          <li>        if t.device != self.src_device_obj:</li>
          <li>            raise RuntimeError("module must have its parameters and buffers "</li>
          <li>                               "on device {} (device_ids[0]) but found one of "</li>
          <li>                               "them on device: {}".format(self.src_device_obj, t.device))</li>
          <li></li>
          <li class="selected">    inputs, kwargs = self.scatter(inputs, kwargs, self.device_ids)</li>
          <li>    if len(self.device_ids) == 1:</li>
          <li>        return self.module(*inputs[0], **kwargs[0])</li>
          <li>    replicas = self.replicate(self.module, self.device_ids[:len(inputs)])</li>
          <li>    outputs = self.parallel_apply(replicas, inputs, kwargs)</li>
          <li>    return self.gather(outputs, self.output_device)</li>
          <li></li>
          <li>def replicate(self, module, device_ids):</li>
          <li>    return replicate(module, device_ids, not torch.is_grad_enabled())</li>
          <li></li>
          <li>def scatter(self, inputs, kwargs, device_ids):</li>
      </ol>
    </code></pre>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<blockquote>
<p>but the time of backward pass is taking 4* times in comparison to 1 V100</p>
</blockquote>
<p>Can you share a minimum repro of this? In general, the 4X slowdown is possible due to Python GIL contention (which you can avoid by using <a href="https://pytorch.org/docs/master/notes/ddp.html" rel="noopener nofollow ugc"><code>DistributedDataParallel</code></a>). But I feel it this should <strong>not</strong> be applicable here as  each batch only contains one sample, and the replicated models are not really involved in forward and backward.</p>

username: sanchit2843
cooked: <p>Hey thanks for the answer, actually I think it is because of all 4 backpropogation going with one gpu only. Is there any solution with which I can run it with one batch size. And what is the meaning of repro? Thanks in advance.</p>

username: sanchit2843
cooked: <p>I tried with batchsize of 2 as well. The epoch time with single gpu is 2.5 hours and with four gpu is 3.3 hrs. Any solution will be helpful.</p>