While using nn.DataParallel only accessing one GPU

Tony_Gracious: <p>I have 2gpus in my system. I am using Dataparallel module over my model and I have made my both of the gpu visible using os.environ[“CUDA_VISIBLE_DEVICES”] = “0,1”. But, while running only 0 is selected if zero is the first in visible devises entry or 1 if  it is the first in entry.  What could be reasons why both of gpus wasn’t used together?</p>
<p>Then I tried manually creating replicas. But, I am getting error<br>
RuntimeError: torch/csrc/autograd/variable.cpp:115: get_grad_fn: Assertion output_nr == 0 failed.<br>
This happens at my LSTM part of my code</p>
<p>replicas = t.nn.parallel.replicate( model , device_ids)<br>
inputs = t.nn.parallel.scatter(( dropout, encoder_word_input, encoder_character_input,<br>
decoder_word_input, decoder_character_input,<br>
None), device_ids)<br>
replicas = replicas[:len(inputs)]<br>
outputs = t.nn.parallel.parallel_apply(replicas, inputs)<br>
out = t.nn.parallel.gather(outputs, output_device)</p>

moskomule: <p>Did you specify <code>device_ids</code>?</p>

Tony_Gracious: <p>Yes, I have specified it. It is not working<br>
In my forward function of my model, I am taking multiple inputs. One of the inputs is an floating point, 4 of them are Variables of dimension minibatch x feature size, other is a latent vector input.<br>
Is that could be reason for using only using 1 GPU?</p>

moskomule: <p>Hmm… Can you show us the lines you use <code>DataParallel</code> in your codes?</p>

Tony_Gracious: <p>rvae = RVAE(parameters)<br>
device_ids = [i for i in range(t.cuda.device_count())]<br>
rvae = t.nn.DataParallel(rvae, device_ids)<br>
rvae = rvae.cuda()</p>
<p>forward function is the following</p>
<p>rvae(0.,  encoder_word_input, encoder_character_input,<br>
decoder_word_input, decoder_character_input,<br>
z=None)</p>

moskomule: <p>Just remove <code>rvae = rvae.cuda()</code>. I think this is the wrong part.</p>

Tony_Gracious: <p>That gave me an error. Expected object of type torch.LongTensor but found type torch.cuda.LongTensor for argument <span class="hashtag">#3</span> ‘index’</p>

ptrblck: <p>How do you know only one GPU is working? Is the other one completely empty?<br>
Could you add a <code>print</code> statement in your <code>forward</code> method, showing the current device of the <code>tensor</code>?<br>
You can find a small example <a href="https://pytorch.org/tutorials/beginner/blitz/data_parallel_tutorial.html#simple-model">here</a>. You would have to change it to print the device instead of the shape.</p>

Tony_Gracious: <p>I am using watch  nvidia-smi to see the memory usage. Only one GPU is used and the other is not used</p>

Tony_Gracious: <p>I checked what you told. By using get_device() in forward function. Only one GPU is used.</p>

Tony_Gracious: <p>I tried manually creating replicas.</p>
<p>replicas = t.nn.parallel.replicate(rvae (my model), device_ids)<br>
inputs = t.nn.parallel.scatter(( dropout, encoder_word_input, encoder_character_input, <br>
decoder_word_input, decoder_character_input,<br>
None), device_ids)<br>
replicas = replicas[:len(inputs)]<br>
outputs = t.nn.parallel.parallel_apply(replicas, inputs)<br>
out = t.nn.parallel.gather(outputs, output_device)</p>
<p>But, I am getting error<br>
RuntimeError: torch/csrc/autograd/variable.cpp:115: get_grad_fn: Assertion <code>output_nr == 0</code> failed.</p>
<p>This happens at my LSTM part of my code</p>

moskomule: <p>Sorry, you’re right. You need to send DataParallel to GPU.</p>

justusschock: <p>What is your batchsize? You need a batch_size &gt; 1 to use both GPUs.</p>

akurniawan: <p>I am facing the same problem and my batch_size is &gt; 1. I am using 2 NVIDIA P100 in google kubernetes engine.<br>
<a class="mention" href="/u/tony_gracious">@Tony_Gracious</a>, may I know if you have solved this problem?</p>

Tony_Gracious: <p>No, I didn’t solved it</p>

akurniawan: <p><a class="mention" href="/u/tony_gracious">@Tony_Gracious</a> in my case, it was because I was initially train the model using nn.DataParallel with one GPU, then once I reload the model, it seems DataParallel still store the previous device_ids, hence the single GPU. Now I manage to solve it by every time I load the model, I will re-wrap my model with nn.DataParallel</p>
<pre><code class="lang-auto">model = _load_model()
model = nn.DataParallel(model.module)
</code></pre>
<p>You also need to be careful for your data size. I need to modify <code>model = nn.DataParallel(model.module)</code> into <code>model = nn.DataParallel(model.module, dim=1)</code> since I am using <code>batch_first=False</code></p>

assyl_coop: <p>device = torch.device(‘cuda :0,1’ if torch.cuda.is_available() else ‘cpu’)<br>
rvae = RVAE(parameters)<br>
rvae = nn.DataParallel(rvae)<br>
rvae.to(device)</p>

ToughMind: <p>Anyone solved this？ I met the same situation。</p>

huahuanZ: <p>I think I have solved this problem.<br>
For your model=DataParallel(model) at forward() step, if you pass arguments into forward(), according to pytorch document:</p>
<blockquote>
<p>Arbitrary positional and keyword inputs are allowed to be passed into DataParallel EXCEPT Tensors. All tensors will be scattered on dim specified (default 0). Primitive types will be broadcasted, but all other types will be a shallow copy and can be corrupted if written to in the model’s forward pass.</p>
</blockquote>
<p>which means if the input argument type is tensor then it would be split by dim=0 (which is the batch dimension). For other types like python list/dict/str，DataParallel.forward() automatically copies it to N replicas ( N equals to your GPU number).<br>
The key is that, if you pass an argument like this<br>
<code>[torch.tensor]</code><br>
or<br>
<code>{"example":torch.tensor}</code><br>
Even though they are python list/dict, but DataParallel.forward() is not able to deal with these type of argument (And it won’t raise an error). So the fix is just to simply convert those argument (and all their elements) to python types.</p>

Vanjoy: <p>What do you mean by <code>python types</code>?</p>