Error using DataParallel with Multiple GPUS

username: mujtabaasif
cooked: <p>I am getting the following error when trying to use Multiple GPUs with DataParallel. Please note the implementation works perfectly fine on a single GPU.<br>
Here is the traceback:</p>
<p>Traceback (most recent call last):<br>
File “train.py”, line 247, in<br>
train_loss = train_xe(model, dataloader_train, optim, text_field)<br>
File “train.py”, line 77, in train_xe<br>
out = model(detections, captions)<br>
File “/opt/conda/envs/m2release/lib/python3.6/site-packages/torch/nn/modules/module.py”, line 547, in  <strong>call</strong><br>
result = self.forward(*input, **kwargs)<br>
File “/opt/conda/envs/m2release/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py”, line 151, in forward<br>
replicas = self.replicate(self.module, self.device_ids[:len(inputs)])<br>
File “/opt/conda/envs/m2release/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py”, line 156, in replicate<br>
return replicate(module, device_ids, not torch.is_grad_enabled())<br>
File “/opt/conda/envs/m2release/lib/python3.6/site-packages/torch/nn/parallel/replicate.py”, line 111, in replicate<br>
buffer_copies_not_rg = _broadcast_coalesced_reshape(buffers_not_rg, devices, detach=True)<br>
File “/opt/conda/envs/m2release/lib/python3.6/site-packages/torch/nn/parallel/replicate.py”, line 75, in _broadcast_coalesced_reshape<br>
return comm.broadcast_coalesced(tensors, devices)<br>
File “/opt/conda/envs/m2release/lib/python3.6/site-packages/torch/cuda/comm.py”, line 39, in broadcast_coalesced<br>
return torch._C._broadcast_coalesced(tensors, devices, buffer_size)<br>
RuntimeError: tensors.size() == order.size() INTERNAL ASSERT FAILED at /pytorch/torch/csrc/utils/tensor_flatten.cpp:66, please report a bug to PyTorch. (reorder_tensors_like at /pytorch/torch/csrc/utils/tensor_flatten.cpp:66)<br>
frame <span class="hashtag">#0:</span> c10::Error::Error(c10::SourceLocation, std::string const&amp;) + 0x33 (0x7f71080e7273 in /opt/conda/envs/m2release/lib/python3.6/site-packages/torch/lib/libc10.so)<br>
frame <a href="https://github.com/pytorch/pytorch/issues/1" rel="nofollow noopener">#1</a>: torch::utils::reorder_tensors_like(std::vector&lt;at::Tensor, std::allocatorat::Tensor &gt;&amp;, c10::ArrayRefat::Tensor) + 0x139f (0x7f710c34c9cf in /opt/conda/envs/m2release/lib/python3.6/site-packages/torch/lib/libtorch.so)<br>
frame <a href="https://github.com/pytorch/pytorch/issues/2" rel="nofollow noopener">#2</a>: torch::cuda::broadcast_coalesced(c10::ArrayRefat::Tensor, c10::ArrayRef, unsigned long) + 0x1d96 (0x7f710c834d76 in /opt/conda/envs/m2release/lib/python3.6/site-packages/torch/lib/libtorch.so)<br>
frame <a href="https://github.com/pytorch/pytorch/issues/3" rel="nofollow noopener">#3</a>: + 0x5f422c (0x7f71528fd22c in /opt/conda/envs/m2release/lib/python3.6/site-packages/torch/lib/libtorch_python.so)<br>
frame <a href="https://github.com/pytorch/pytorch/pull/4" rel="nofollow noopener">#4</a>: + 0x1d3ef4 (0x7f71524dcef4 in /opt/conda/envs/m2release/lib/python3.6/site-packages/torch/lib/libtorch_python.so)</p>
<p>frame <a href="https://github.com/pytorch/pytorch/issues/48" rel="nofollow noopener">#48</a>: __libc_start_main + 0xf0 (0x7f7160f72830 in /lib/x86_64-linux-gnu/libc.so.6)</p>
<p>Thanks <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: mrshenli
cooked: <p>Looks like <code>DataParallel</code> failed to replicate your model to multiple GPUs. Could you please share a minimum repro?</p>

username: mujtabaasif
cooked: <p>I am trying to parallelize this network.<br>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/aimagelab/meshed-memory-transformer/blob/master/train.py#L182" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/aimagelab/meshed-memory-transformer/blob/master/train.py#L182" target="_blank" rel="nofollow noopener">aimagelab/meshed-memory-transformer/blob/master/train.py#L182</a></h4>
<pre class="onebox"><code class="lang-py"><ol class="start lines" start="172" style="counter-reset: li-counter 171 ;">
<li>    print("Building vocabulary")</li>
<li>    text_field.build_vocab(train_dataset, val_dataset, min_freq=5)</li>
<li>    pickle.dump(text_field.vocab, open('vocab_%s.pkl' % args.exp_name, 'wb'))</li>
<li>else:</li>
<li>    text_field.vocab = pickle.load(open('vocab_%s.pkl' % args.exp_name, 'rb'))</li>
<li>
</li>
<li># Model and dataloaders</li>
<li>encoder = MemoryAugmentedEncoder(3, 0, attention_module=ScaledDotProductAttentionMemory,</li>
<li>                                 attention_module_kwargs={'m': args.m})</li>
<li>decoder = MeshedDecoder(len(text_field.vocab), 54, 3, text_field.vocab.stoi['&lt;pad&gt;'])</li>
<li class="selected">model = Transformer(text_field.vocab.stoi['&lt;bos&gt;'], encoder, decoder).to(device)</li>
<li>
</li>
<li>dict_dataset_train = train_dataset.image_dictionary({'image': image_field, 'text': RawField()})</li>
<li>ref_caps_train = list(train_dataset.text)</li>
<li>cider_train = Cider(PTBTokenizer.tokenize(ref_caps_train))</li>
<li>dict_dataset_val = val_dataset.image_dictionary({'image': image_field, 'text': RawField()})</li>
<li>dict_dataset_test = test_dataset.image_dictionary({'image': image_field, 'text': RawField()})</li>
<li>
</li>
<li>
</li>
<li>def lambda_lr(s):</li>
<li>    warm_up = args.warmup</li>
</ol></code></pre>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>
<p>After line 182 I just added<br>
model = torch.nn.DataParallel(model)</p>
<p>I have tried Pytorch 1.1.0, 1.2.0, 1.4.0, and 1.5. (from source)</p>