Data Parallelization meet RuntimeError: copy_if failed to synchronize: cudaErrorAssert: device-side assert triggered

Kaiwkh: <p>Hi,</p>
<p>I am trying to parallelize my loss module:</p>
<pre><code class="lang-auto">class Supervised_EmbeddingLoss(nn.Module):
    def __init__(self, n_cluster, cos_margin, push_balance_weight, pull_momentum, push_momentum):
        super(Supervised_EmbeddingLoss, self).__init__()
        ...
    def forward(self, pred, lbl, centroids, push_minibatch, pad, ignore=255):
        N, D, H, W = pred.size()
        pred = pred[:, :, pad:H - pad, pad:W - pad]
        lbl = lbl[:, pad:H - pad, pad:W - pad]

        pred = pred.transpose(1, 2).transpose(2, 3).contiguous().view(-1, D)
        lbl = lbl.contiguous().view(-1)
        pred = pred[lbl != ignore]
        lbl = lbl[lbl != ignore]

        pull_loss = F.cosine_embedding_loss(pred, centroids[lbl], margin=self.cos_margin,
                                            target=torch.ones_like(lbl, dtype=torch.float), reduction='mean')
        push_loss = 0.
        for i in range(self.n_cluster):
            _pred = pred[lbl != i]
            if push_minibatch != -1:
                r = torch.rand(size=[_pred.size()[0]])
                rinds = torch.argsort(r)[:push_minibatch]
                _pred = _pred[rinds]
            push_loss += F.cosine_embedding_loss(_pred, centroids[i].view(1, D), margin=self.cos_margin,
                                                 target=-1 * torch.ones((_pred.size()[0]), dtype=torch.float).cuda(),
                                                 reduction='mean')
        embed_loss = pull_loss + self.push_balance_weight * push_loss
        return {'embed_loss': embed_loss, 'pull_loss': pull_loss, 'push_loss': push_loss}
</code></pre>
<p>When I was training my model along with this loss function on one GPU it works fine. However, when I wrap my loss module using torch.nn.DataParallel() and trying to parallelize it on two GPUs,I got an error like:</p>
<pre><code class="lang-auto">...
...
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:60: lambda [](int)-&gt;auto::operator()(int)-&gt;auto: block: [460,0,0], thread: [12,0,0] Assertion `index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:60: lambda [](int)-&gt;auto::operator()(int)-&gt;auto: block: [460,0,0], thread: [13,0,0] Assertion `index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:60: lambda [](int)-&gt;auto::operator()(int)-&gt;auto: block: [460,0,0], thread: [14,0,0] Assertion `index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:60: lambda [](int)-&gt;auto::operator()(int)-&gt;auto: block: [460,0,0], thread: [15,0,0] Assertion `index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:60: lambda [](int)-&gt;auto::operator()(int)-&gt;auto: block: [460,0,0], thread: [16,0,0] Assertion `index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:60: lambda [](int)-&gt;auto::operator()(int)-&gt;auto: block: [460,0,0], thread: [17,0,0] Assertion `index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:60: lambda [](int)-&gt;auto::operator()(int)-&gt;auto: block: [460,0,0], thread: [18,0,0] Assertion `index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:60: lambda [](int)-&gt;auto::operator()(int)-&gt;auto: block: [460,0,0], thread: [19,0,0] Assertion `index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:60: lambda [](int)-&gt;auto::operator()(int)-&gt;auto: block: [460,0,0], thread: [20,0,0] Assertion `index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:60: lambda [](int)-&gt;auto::operator()(int)-&gt;auto: block: [460,0,0], thread: [21,0,0] Assertion `index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:60: lambda [](int)-&gt;auto::operator()(int)-&gt;auto: block: [460,0,0], thread: [22,0,0] Assertion `index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:60: lambda [](int)-&gt;auto::operator()(int)-&gt;auto: block: [460,0,0], thread: [23,0,0] Assertion `index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:60: lambda [](int)-&gt;auto::operator()(int)-&gt;auto: block: [460,0,0], thread: [24,0,0] Assertion `index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:60: lambda [](int)-&gt;auto::operator()(int)-&gt;auto: block: [460,0,0], thread: [25,0,0] Assertion `index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:60: lambda [](int)-&gt;auto::operator()(int)-&gt;auto: block: [460,0,0], thread: [26,0,0] Assertion `index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:60: lambda [](int)-&gt;auto::operator()(int)-&gt;auto: block: [460,0,0], thread: [27,0,0] Assertion `index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:60: lambda [](int)-&gt;auto::operator()(int)-&gt;auto: block: [460,0,0], thread: [28,0,0] Assertion `index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:60: lambda [](int)-&gt;auto::operator()(int)-&gt;auto: block: [460,0,0], thread: [29,0,0] Assertion `index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:60: lambda [](int)-&gt;auto::operator()(int)-&gt;auto: block: [460,0,0], thread: [30,0,0] Assertion `index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:60: lambda [](int)-&gt;auto::operator()(int)-&gt;auto: block: [460,0,0], thread: [31,0,0] Assertion `index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"` failed.

Traceback (most recent call last):                                              
  File "train_3/train_0_0_thresh_0_6_mo_1_batch_2_test.py", line 199, in &lt;module&gt;
    trainer.train()
  File "/projectnb/cs585/kai/mt_clust/trainer/mt_trainer.py", line 253, in train
    self._train_epoch()
  File "/projectnb/cs585/kai/mt_clust/trainer/mt_trainer.py", line 102, in _train_epoch
    src_embed_loss = self.supervised_loss(src_pred_embed, source_lbl, centroids, self.cfg.push_minibatch, self.cfg.loss_pad)
  File "/share/pkg.7/pytorch/1.3/install/3.6/lib/python3.6/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/share/pkg.7/pytorch/1.3/install/3.6/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py", line 152, in forward
    outputs = self.parallel_apply(replicas, inputs, kwargs)
  File "/share/pkg.7/pytorch/1.3/install/3.6/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py", line 162, in parallel_apply
    return parallel_apply(replicas, inputs, kwargs, self.device_ids[:len(replicas)])
  File "/share/pkg.7/pytorch/1.3/install/3.6/lib/python3.6/site-packages/torch/nn/parallel/parallel_apply.py", line 85, in parallel_apply
    output.reraise()
  File "/share/pkg.7/pytorch/1.3/install/3.6/lib/python3.6/site-packages/torch/_utils.py", line 385, in reraise
    raise self.exc_type(msg)
RuntimeError: Caught RuntimeError in replica 0 on device 0.
Original Traceback (most recent call last):
  File "/share/pkg.7/pytorch/1.3/install/3.6/lib/python3.6/site-packages/torch/nn/parallel/parallel_apply.py", line 60, in _worker
    output = module(*input, **kwargs)
  File "/share/pkg.7/pytorch/1.3/install/3.6/lib/python3.6/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/projectnb/cs585/kai/mt_clust/model/losses/loss_embed.py", line 34, in forward
    _pred = pred[lbl != i]
RuntimeError: copy_if failed to synchronize: cudaErrorAssert: device-side assert triggered

</code></pre>
<p>It seems like it’s a problem on the indexing line: <strong>_pred = pred[lbl != i]</strong></p>
<p>Can someone give me some clue about this weird problem?</p>

ptrblck: <p>Could you print the shape of <code>pred</code> as well as the result of <code>lbl != i</code> for the failing operation?<br>
This should give you an idea, why the index is out of bounds.</p>

Kaiwkh: <p>I print their size out and their size is:</p>
<blockquote>
<p>torch.Size([210226, 128])</p>
<p>torch.Size([235481, 128])</p>
<p>torch.Size([210226])</p>
<p>torch.Size([235481])</p>
</blockquote>
<p>noting that the computation is parallelized on two GPUs, so the pred and lbl size on each GPU is:<br>
torch.Size([210226, 128])<br>
torch.Size([210226])</p>
<p>and</p>
<p>torch.Size([235481, 128])<br>
torch.Size([235481])</p>
<p>The most confusing thing is when I cancel the parallelization and run it on a single GPU, it works perfectly.</p>

ptrblck: <p>Thanks for the shapes. Could you check, if <code>pred</code> might be empty after the</p>
<pre><code class="lang-python">pred = pred[lbl != ignore]
</code></pre>
<p>operation?<br>
Based on your description I guess you might run into a scenario, where one device only gets <code>ingore</code> indices, and this indexing <code>pred</code> again might yield this error.<br>
You could check the <code>pred.size(0)</code> for a <code>0</code> after the first indexing.</p>

Kaiwkh: <p>I ran the code and it returned:</p>
<pre><code class="lang-auto">236914


235627
</code></pre>
<p>for two tensors. So they are not empty.</p>
<p>Thanks</p>

nandu_mahesh: <p><a class="mention" href="/u/kaiwkh">@Kaiwkh</a>, were you able to fix this issue? I am facing a similar issue where the loss work perfectly in single GPU , but with multiple GPUs (single node) facing the same issue as mentioned</p>