DataParallel Multi-GPU stuck on backward pass

semihcanturk: <p>Hello! I’m dealing with an issue on PyTorch 1.1 where I am training a model on time series data. The data is held in a list of Tensors, where each tensor can be split into multiple batches for parallelization.</p>
<p>When the batch size is 1, only one of two GPUs are utilized (which is expected), and the model trains smoothly. However, when batch size &gt; 1 (this is not the mini_batch size variable, but the number of batches that the model is allowed to use for parallelization) the program gets stuck at backpropagation (line loss.backward()). When debugging, the loss variable is identical (except the float value itself) in its properties both in multi-batch and single-batch cases (screenshot attached below), so I can’t figure out what is going wrong.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/c/cb5edda5386f368eaf8823c8bf21fd1e8d6e2c82.png" data-download-href="https://discuss.pytorch.org/uploads/default/cb5edda5386f368eaf8823c8bf21fd1e8d6e2c82" title="36%20PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/c/cb5edda5386f368eaf8823c8bf21fd1e8d6e2c82_2_304x250.png" alt="36%20PM" data-base62-sha1="t16aSNlsD5CT54VK29FTboHyvxE" width="304" height="250" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/c/cb5edda5386f368eaf8823c8bf21fd1e8d6e2c82_2_304x250.png, https://discuss.pytorch.org/uploads/default/optimized/2X/c/cb5edda5386f368eaf8823c8bf21fd1e8d6e2c82_2_456x375.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/2X/c/cb5edda5386f368eaf8823c8bf21fd1e8d6e2c82_2_608x500.png 2x" data-dominant-color="F4F3F3"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">36%20PM</span><span class="informations">1024×842 112 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I’ve found out that PyTorch used to have problems with Tesla K80 GPUs, but I also tried out with Tesla P100s and the problem persists, so it doesn’t seem to be the issue. I’ve also looked at pretty much all related posts in here, but none seem to address/solve the issue I am currently having. My training code is attached below for reference.</p>
<p>Any help is much appreciated. Thank you!</p>
<pre><code class="lang-auto">    def train_minibatch(self, model, optimizer, mini_batch=10):
        model.train()

        train_loss_value = 0
        predictions = []
        targets = []

        for batch_idx, ts in enumerate(self.train_loader):

            batch_loss_value = 0
            batch_predictions = []
            batch_targets = []

            optimizer.zero_grad()

            for i in range(len(ts)):

                output = model(ts[i].to(self.device))
                target = torch.argmax(cues[:, i], dim=1)
                prediction = output.max(1, keepdim=True)

                torch.cuda.synchronize()

                loss = sum([F.nll_loss(output[j, :].unsqueeze(0), target[j].unsqueeze(-1), weight=self.w) for j in range(len(target))])
                loss = loss / mini_batch
                loss.backward()

                if i &gt; 0 and i % mini_batch == 0:
                    optimizer.step()
                    optimizer.zero_grad()

                train_loss_value += loss.item()
                predictions.extend(prediction.tolist())
                targets.extend(target.tolist())

                batch_loss_value += loss.item()
                batch_predictions.extend(prediction.tolist())
                batch_targets.extend(target.tolist())

            self.print_eval(batch_loss_value, batch_predictions, batch_targets, idx=batch_idx, header='batch idx:')
</code></pre>

harryhan618: <p>Hello! Have you solve the problem? My code also stucks with DistributedDataParallel during backward. And it always stucks at the same iteration. I don’t know how to diving into backward debugging. Hope you can provide the solution. Thanks!</p>

nomadlx: <aside class="quote no-group" data-username="semihcanturk" data-post="1" data-topic="48624">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/semihcanturk/48/13340_2.png" class="avatar"> semihcanturk:</div>
<blockquote>
<p><code>if i &gt; 0 and i % mini_batch == 0:</code></p>
</blockquote>
</aside>
<p>I guess this piece of code may have caused the bug</p>