Error When Using Multiple GPus

manish: <p>My code works fine when using just 1 GPU using <code>torch.cuda.set_device(0)</code> but it takes a lot of time to train in single GPU.</p>
<p>I tried various ways to Parallelize it, but nothing seems to work.<br>
Currently Iam trying :</p>
<pre><code class="lang-auto">gpu_ids = [0,1,2,3]
model = torch.nn.DataParallel(model,device_ids=gpu_ids)
model.cuda()
</code></pre>
<p>But when i try to access some model methods, like <code>model.encoder.eval()</code> it throws error saying <code>'DataParallel' object has no attribute 'encoder'</code></p>
<p>What am I missing ?</p>

ptrblck: <p>After wrapping the model in <code>DataParallel</code> the <code>Module</code>s are stored in <code>model.module</code>.<br>
Try <code>model.module.encoder</code>.</p>

manish: <p>Hey it seems like solving my problem, but do i have to replace every instance of model.“abc” with model.module.abc</p>
<p>like :<code>model.train() to model.module.train()</code><br>
<code>model.eval()</code> to <code>model.module.eval()</code> and so on</p>
<p>Please let me know it there are some exceptions to this ?</p>

manish: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Now it says <code>cuda runtime error (2) : out of memory at /pytorch/torch/lib/THC/generic/THCStorage.cu:66</code>  and Uses only <strong>GPU- 0</strong></p>
<p>Does DataParallel executes batches parallely and merges them together ?</p>

ptrblck: <p>No, just call <code>.eval</code> on the <code>DataParallel</code> instance.</p>
<p><code>DataParallel</code> uses a bit more memory on the default GPU, which is GPU0 by default. If you are using this GPU for other processes, e.g. your desktop, you could change the order of device ids like: <code>device_ids=[1, 0]</code>.</p>

manish: <p>Previously I was doing like : <code>model.encode.eval()</code> and now doing <code>model.module.encode.eval()</code> is the right way rt ?</p>
<p>But I see that only 1 GPU being used in nvidia-smi . Any possible reason for same ?</p>

manish: <p><a href="https://ideone.com/gJVwSk" rel="nofollow noopener">https://ideone.com/gJVwSk</a> code works fine for 1 GPU. Can you have a look and suggest me what changes to make ?<br>
<a href="https://ideone.com/YyBOa0" rel="nofollow noopener">https://ideone.com/YyBOa0</a> one version is what i changed after you suggested me… Please look where i did wrong…</p>
<p>Please Help me out… Its urgent</p>

ptrblck: <p>Remove <code>torch.cuda.set_device(gpu)</code> and try to use <code>DataParallel</code> again.<br>
Also, could you delete <code>loaded_model</code>? It seems to use some GPU memory without being used.</p>

manish: <p>Yeah i runned it removing<code>torch.cude.set_device(gpu)</code> . It runs but uses only 1 GPU. What i would like to do is say run batch1 in 1 GPU , second batch in another GPU and merge them together. Am i manually supposed to do that ?</p>
<p>Even training 8 epochs took me 24 hours with 1 GPU. I would like to speed this up .</p>

ptrblck: <p>This is done by <code>DataParallel</code> automatically. The batch is split onto the different GPUs.<br>
I still don’t see, why your code only uses one GPU.</p>

manish: <p>No Idea… that’s why <strong>i shared code</strong> here in case i am missing some minute detail.</p>

ptrblck: <p>Since I cannot run the code on my machine, could you create a small code snippet with random input etc. so that I could debug the code?</p>

manish: <p>Since project contains a lot of files, creating small snippet seems a difficult task to me. Can to suggest me how to debug the code for multi-gpu support ?</p>
<p>Can you think of any other reason why DataParallel not working based on your experience with pytorch and cuda ?</p>
<p><a class="mention" href="/u/ptrblck">@ptrblck</a> Thanks a lot for taking time to help me out <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=5" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

manish: <p>Hey I am using <a href="https://github.com/pentiumx/gte-vnmt-pytorch/" rel="nofollow noopener">https://github.com/pentiumx/gte-vnmt-pytorch/</a> as a starting point to my project. Running this in multi-gpu will be helpful.</p>
<p>Can you have a look ?</p>

ptrblck: <p>Could you walk me through the code a bit, so that it won’t take that much time to read all functions.<br>
First I suppose I have to run <code>gte_vae_pretrain.py</code> and then just <code>gte.py</code>?</p>

manish: <p>just run <code>gte_vae.py</code> only… there are many extra models tried… (so other names may be misleading) .</p>
<p>look into<code>gte_vae.py</code> only and different pytorch modules called from there.</p>

ptrblck: <p>Thanks for the info, unfortunately I need some additional files:</p>
<ul>
<li>glove.42B.300d</li>
<li>.vector_cache/input_vectors.pt</li>
<li>.data/snli/snli_1.0_entail/snli_1.0_train_entail.jsonl</li>
</ul>
<p>Could I create some of them with random values, since I don’t need the accuracy just an executable code?</p>

manish: <p>Make snli_1.0_train_entail.jsonl a line seperated file in the required directory , like :</p>
<pre><code class="lang-auto">{sentence1 : "Ram is Good" , sentence2 : "Shyam is good", label : "ab"}
{sentence1 : "Food was awesome" , sentence2 : "Any sentence", label : "bc"}
</code></pre>
<p><code>sentence1 , sentence2 , label</code> are keys. And same goes with test and  dev files.</p>
<p>and make random values for others</p>

Vivek_Malipatel: <p>HI, Were you able to solve this issue?</p>