How to parallelise a pytorch model on GPU?

username: ranger
cooked: <p>Hi!</p>
<p>My model fits on single gpu and works nicely but it is slow. In order to scale the computations, I want to parallelise the model so that I can use multiple gpus.</p>
<p>I have done<br>
model = torch.nn.DataParallel(model).cuda()</p>
<p>in place of model.cuda()</p>
<p>When I run this with 2 gpus, it is working with batch-size=1 but only using single gpu and when I increase the batch-size it says out of memory.</p>
<p>Looking for response.</p>
<p>Thanks <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=6" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: kaixin
cooked: <p>You need to specify the argument <code>device_ids</code> in <code>nn.DataParallel()</code>. The default is to use a single GPU.</p>
<p>Best</p>

username: ranger
cooked: <p>model = torch.nn.DataParallel(model,device_ids=[0,1]).cuda()</p>
<p>Changed to this, Still it is working on the first gpu only and then exploding which I am giving in CUDA_VISIBLE_DEVICES.</p>
<p>Giving the same error.</p>

username: kaixin
cooked: <p>Did you give the device_ids (e.g. [0,1]) in CUDA_VISIBLE_DEVICES?</p>

username: ranger
cooked: <p>I ran command for training as CUDA_VISIBLE_DEVICES=0,1 as well inside code I have this line</p>
<p>model = torch.nn.DataParallel(model,device_ids=[0,1]).cuda()</p>

username: kaixin
cooked: <p>Usually I do</p>
<pre><code class="lang-python">import os
os.environ['CUDA_VISIBLE_DEVICES'] = '0, 1'
</code></pre>
<p>and everything works fine. Maybe you can try this.</p>
<p>Hope it works.</p>

username: ptrblck
cooked: <p>Maybe you are running out of memory on the default device which will gather and scatter some parameters, thus usually using a bit more memory than the other devices.</p>
<p>This behavior is described in <a class="mention" href="/u/thomas_wolf">@Thomas_Wolf</a>’s <a href="https://medium.com/huggingface/training-larger-batches-practical-tips-on-1-gpu-multi-gpu-distributed-setups-ec88c3e51255">blog post</a>.</p>

username: ranger
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> With the help of nvidia-smi, I am monitoring the gpu memory usage. There I am getting only one of the gpus getting ~11 GB of memory and then it says out of memory and the other gpu has a basic memory usage of 2 MBs.</p>
<p>I hope that should not be the case as I understand from the blog.</p>

username: ptrblck
cooked: <p>How much memory is one GPU using for single GPU training? Is it also approx. 11GB or much lower?</p>

username: ranger
cooked: <p>8 GB usage for batch-size=1</p>

username: ranger
cooked: <p>My thought is that the network should get loaded on both the gpu(if using 2) and then may be the error comes.</p>

username: ptrblck
cooked: <p>How large is your batch size using <code>nn.DataParallel</code>?<br>
Would this batch with the model fit on a single GPU without running the forward and backward pass?</p>
<p>I would assume the same, i.e. that the gather on the “master” GPU might run OOM, but not during the initialization.</p>

username: ranger
cooked: <p>I am trying with batch-size=2 and giving 2 gpus for parallel.<br>
Any more math i should do regarding the extra computation on gpu 0?</p>

username: ptrblck
cooked: <p>Just for the sake of completeness:<br>
Based on a small chat, it seems <a href="https://github.com/mingyuliutw/UNIT">this code base</a> is used.<br>
Currently the <a href="https://github.com/mingyuliutw/UNIT/blob/master/trainer.py">Trainer class</a> provides convenient methods to train the model. However, skimming through the code it looks like some refactoring would be needed to make this code executable for <code>nn.DataParallel</code>, e.g. since the optimizer seems to be embedded in the trainer class.</p>
<p>I’m also not sure how <a href="https://github.com/mingyuliutw/UNIT/blob/25e99afe267df6eea2c97d23b05a42683d75e53c/trainer.py#L69">these lines of code</a> would be handled by <code>nn.DataParallel</code>, since no GPU id is passed to the <code>cuda</code> calls. It’s currently a guess, but I think this might also cause the OOM issue in this case.</p>

username: wen75741
cooked: <p>Does this mean the nn.DataParallel needs torch.optim.Optimizer to indicate where to scatter and gather but a single .backward() cannot be handled?</p>

username: ptrblck
cooked: <p>No, <code>DataParallel</code> in its basic form is just applied on the model, such that the input batch will be split in dim0 and each specified GPU will get a chunk. The forward and backward passes are executed in parallel and all necessary gradients etc. are finally gathered on the default device.<br>
Your training routine should not change if you are using <code>DataParallel</code>.</p>
<p>However, in the mentioned code base, the <code>Trainer</code> class is handling the models, optimizers etc.<br>
Using <code>DataParallel</code> by just wrapping your model in it and leaving all other code snippets as they were, is thus most likely not possible, and would need some code refactoring.</p>

username: zepmck
cooked: <p>Is there a way to slipt the Model into several sub models and make them parallel gradually? Handling a complex Model as a whole is an almost impossible task. Thanks.</p>