Manually reduce gradients from multiple GPUs encounter unexpected crash

username: Xiaopeng_Li
cooked: <p>Hi, I’m trying to implement a customized DataParallel, where I want to manually reduce gradients from replicas in multiple GPUs. The reason I’m doing it is that I want each GPU to accumulate gradients for several iterations before doing one reduce gradients operation across multi-GPUs, hence reducing the communication overhead. When using 2 GPUs it seems to work. However, when using 4 GPUs after some iteration the program simply crashes without any error message. I think it’s some lower level C code crashes. Do you have any idea why? Here is my code:<br>
`class DataParallelAccumulation(DataParallel):<br>
def <strong>init</strong>(self, module, device_ids=None, output_device=None, dim=0):<br>
super().<strong>init</strong>(module, device_ids=device_ids, output_device=output_device, dim=dim)<br>
if len(self.device_ids) &gt; 1:<br>
self.replicas = self.replicate(self.module, self.device_ids, detach=True)</p>
<pre><code>def forward(self, *inputs, **kwargs):
    if not self.device_ids:
        return self.module(*inputs, **kwargs)
    inputs, kwargs = self.scatter(inputs, kwargs, self.device_ids)
    if len(self.device_ids) == 1:
        return self.module(*inputs[0], **kwargs[0])
    outputs = self.parallel_apply(self.replicas[:len(inputs)], inputs, kwargs)
    return outputs

def reduce_grads(self):
    if len(self.device_ids) &gt; 1:
        for parameters in zip(self.module.parameters(), *[r.parameters() for r in self.replicas]):
            destination_device = parameters[0].get_device()
            parameters[0].grad = (comm.reduce_add([p.grad for p in parameters[1:]],
                destination=destination_device))

def synchronize(self):
    if len(self.device_ids) &gt; 1:
        self.replicas = self.replicate(self.module, self.device_ids, detach=True)

def replicate(self, module, device_ids, detach=False):
    replicas = replicate(module, device_ids, detach=detach)
    return replicas
</code></pre>
<p>`</p>

username: mrshenli
cooked: <p>Hey <a class="mention" href="/u/xiaopeng_li">@Xiaopeng_Li</a>, which version of PyTorch are you using? After <a href="https://github.com/pytorch/pytorch/pull/33907" rel="nofollow noopener">https://github.com/pytorch/pytorch/pull/33907</a>, the <code>replicate</code> method is no longer supposed to be used this way. It will only replicate non-leaf parameters, and as a result, <code>replicated_model.parameters()</code> will be empty. Can you double check if this is the case in your dev env?</p>
<p>If you really need this, you can try to access the <code>_former_parameters</code> attribute in replicated models. See the code below. But there is no guarantee on how long this attribute can stay.</p>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/blob/0edbe6b063d2525ceaf89f0c603f6e35b3118686/torch/nn/parallel/distributed.py#L347-L357" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/0edbe6b063d2525ceaf89f0c603f6e35b3118686/torch/nn/parallel/distributed.py#L347-L357" target="_blank" rel="nofollow noopener">pytorch/pytorch/blob/0edbe6b063d2525ceaf89f0c603f6e35b3118686/torch/nn/parallel/distributed.py#L347-L357</a></h4>
<pre class="onebox"><code class="lang-py"><ol class="start lines" start="347" style="counter-reset: li-counter 346 ;">
<li>def parameters(m, recurse=True):</li>
<li>    def model_parameters(m):</li>
<li>        ps = m._former_parameters.values() \</li>
<li>            if hasattr(m, "_former_parameters") \</li>
<li>            else m.parameters(recurse=False)</li>
<li>        for p in ps:</li>
<li>            yield p</li>
<li>
</li><li>    for m in m.modules() if recurse else [m]:</li>
<li>        for p in model_parameters(m):</li>
<li>            yield p</li>
</ol></code></pre>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>


username: Xiaopeng_Li
cooked: <p><a class="mention" href="/u/mrshenli">@mrshenli</a> Thanks for the reply. I’m using 1.4.0. In this version the replicated_model.parameters() do exist, and I understand that in 1.5.0 it no longer exist.<br>
As I mentioned, it works for several iteration of training, but crashes at some point. Every time it crashes at different iterations, pretty weired. Am I doing <code>reduce_grads </code> correctly?</p>

username: Xiaopeng_Li
cooked: <p>Close this issue because similar issue is already discussed in <a href="https://github.com/pytorch/pytorch/pull/19577" rel="nofollow noopener">pull-19577</a> and <a href="https://github.com/pytorch/pytorch/pull/21736" rel="nofollow noopener">pull-21736</a>, and the PyTorch team have worked out the solution, that is using <code>no_sync()</code> context manager.</p>