DataParallel model stuck on loss.backward()

semihcanturk: <p>This is a separate issue potentially related to: <a href="https://discuss.pytorch.org/t/sending-a-tensor-to-multiple-gpus/49390" class="inline-onebox">Sending a tensor to multiple GPUs</a></p>
<p>I am training a DataParallel module on two GPUs. The training works as it should when:<br>
a) Training on a single GPU, where the model is not wrapped by the DataParallel Module, regardless of batch size.<br>
b) Training with both GPUs available, but with batch size = 1 so the data is sent to only one GPU.</p>
<p>However, when I increase the batch size to &gt;1 <em>and</em> utilize both GPUs, the program gets stuck on the backward pass, <code>loss.backward()</code>. The rest of the training code (somewhat streamlined) can be found below.</p>
<p>Is this a hardware issue (as suggested in a comment here: <a href="https://discuss.pytorch.org/t/sending-a-tensor-to-multiple-gpus/49390" class="inline-onebox">Sending a tensor to multiple GPUs</a>), or something to do with autograd (all variables have gradients, so don’t think that is the issue)?</p>
<pre><code class="lang-auto"> for batch_idx, (x, y, graph, subject) in enumerate(self.train_loader):

            if model.module.subject != subject:

                model.module.subject = subject
                model.module.graph = graph

            output = model(x.to(self.device))
            target = torch.argmax(y, dim=1)

            optimizer.zero_grad()
            loss = F.nll_loss(output, target, weight=self.w)
            loss.backward()
            optimizer.step()
</code></pre>

Nasi_Kaashki: <aside class="quote no-group" data-username="semihcanturk" data-post="1" data-topic="49478">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/semihcanturk/48/13340_2.png" class="avatar"> semihcanturk:</div>
<blockquote>
<p>aining code (somewhat streamlined) can</p>
</blockquote>
</aside>
<p>I have a similar problem which I wrote <a href="https://discuss.pytorch.org/t/runtimeerror-function-catbackward-returned-an-invalid-gradient-at-index-1-expected-device-cuda-1-but-got-cuda-0/49469">here</a>. However, I think my problem has nothing to do with hardware because I am running <a href="https://github.com/WangYueFt/dgcnn/tree/master/pytorch" rel="noopener nofollow ugc">this code</a> along two GPUs without any problem.</p>

Vijay_Viswanath: <p>Facing the same problem</p>

mrshenli: <p>Hey <a class="mention" href="/u/vijay_viswanath">@Vijay_Viswanath</a> could you please share a repro? BTW, which version of PyTorch are you using?</p>

nirandiw: <p>I am facing the same issues. Do we have a solution for this?</p>

Sangameshkodge: <p>Facing the same problem with DataParallel on muti-gpu with A40 GPUS.<br>
The code runs fine with these settings on 1080Ti, 2080 Ti and TitanXp with the identical environment setup. <a class="mention" href="/u/semihcanturk">@semihcanturk</a> Did you figure out a fix?</p>

winchest: <p>Having the same problem. Do we have a solution?</p>

winchest: <p>For someone might have the same problem, U can see this post:</p><aside class="quote" data-post="10" data-topic="174554">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/w/ecae2f/48.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/training-on-the-multi-gpus-but-stuck-in-loss-backward/174554/10">Training on the multi-GPUs but stuck in loss.backward()</a> 
  </div>
  <blockquote>
    OMGGG！Finally I find that the os.environ code part seems to be conflict with torch.nn.DataParallel.So when I delete the os.environ code It worked!!
  </blockquote>
</aside>
<p>
Hope it will be helpful!</p>