Multiple Models in DataParallel: RuntimeError: arguments are located on different GPUs

username: jerinphilip
cooked: <p>The following is a traceback, and I think at the moment that the error is due to having to models - generator and discriminator applied with <code>DataParallel</code> separately leading to the following issue at D(G(z)).</p>
<p>What’s the way to fix this?</p>
<pre><code class="lang-auto">  File "/home/jerin/code/fairseq/fairseq/models/lstm.py", line 207, in forward
    x = self.embed_tokens(src_tokens)
  File "/home/jerin/.local/lib/python3.5/site-packages/torch/nn/modules/module.py", line 477, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/jerin/.local/lib/python3.5/site-packages/torch/nn/modules/sparse.py", line 110, in forward
    self.norm_type, self.scale_grad_by_freq, self.sparse)
  File "/home/jerin/.local/lib/python3.5/site-packages/torch/nn/functional.py", line 1110, in embedding
    return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
RuntimeError: arguments are located on different GPUs at /pytorch/aten/src/THC/generic/THCTensorIndex.cu:513
Exception ignored in: &lt;bound method tqdm.__del__ of | epoch 000:   0%|                                                                                                                                                                                                                                                                                 | 0/3750 [00:08&lt;?, ?it/s]&gt;
</code></pre>

username: a57553ad19ca6baa05ac
cooked: <p>I met the same problems, have you fixed it?</p>

username: jerinphilip
cooked: <p>I was trying to switch models between GPUs. I solved this by putting all models in one module inheriting <code>nn.Module</code> and applying DataParallel on that. In the forward method definitions, I called/accessed the individual models (generator and discriminator).</p>

username: a57553ad19ca6baa05ac
cooked: <p>OK, I only have one model. But with the same embedding Error. Thank you anyway.</p>

username: Khuong_Nguyen_Duy
cooked: <p>I have the same error when I trying to use multiple embeddings</p>

username: yyHaker
cooked: <p>I also use the same one embedding and one model, but still the error, do you fix the problem?<br>
I load the embedding from numpy array, but still has the error:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/4/42e61ae27fc15cfec5c8c4519fb0530c071709db.png" data-download-href="https://discuss.pytorch.org/uploads/default/42e61ae27fc15cfec5c8c4519fb0530c071709db" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/2X/4/42e61ae27fc15cfec5c8c4519fb0530c071709db.png" alt="image" data-base62-sha1="9xOvFqLV1IffezF1lw9xCX5jX2r" width="690" height="366" data-dominant-color="1A353E"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">860×457 20.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: alibabadoufu
cooked: <p>When I apply Dataparallel, I have the same problem with multiple model passing embedding weights across each other.</p>