How to use multiple GPUs (DataParallel) for training a model that used to use one gpu

isalirezag: <p>If I want to use multiple gpus for a network should i specifically write my network in a way that it is designed to be train on multiple gpus or can i just add some comment to switch between one gpu and multiple gpus?</p>
<p>For example, lets say im doing this example (training a classifier: <a href="https://pytorch.org/tutorials/beginner/blitz/cifar10_tutorial.html" rel="nofollow noopener">https://pytorch.org/tutorials/beginner/blitz/cifar10_tutorial.html</a>) and I used to train it on one gpu, can i easily do something to train it on multiple gpus?</p>
<p>If the answer is no, please consider this for as your possible future features.</p>
<p>Thanks</p>

ptrblck: <p>You can just wrap your model in <code>DataParallel</code> and specify the <code>device_ids</code> you would like to use.<br>
The data will be split among the batch dimensions. See <a href="https://pytorch.org/tutorials/beginner/blitz/data_parallel_tutorial.html">the data parellel tutorial</a> for more information.</p>

isalirezag: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> So I easily should just add:</p>
<pre><code class="lang-auto">model = Model(input_size, output_size)
if torch.cuda.device_count() &gt; 1:
  print("Let's use", torch.cuda.device_count(), "GPUs!")
  # dim = 0 [30, xxx] -&gt; [10, ...], [10, ...], [10, ...] on 3 GPUs
  model = nn.DataParallel(model)

model.to(device)
</code></pre>
<p>to my code and that is it?</p>
<p>BTW the out put of that command in my system is:</p>
<pre><code class="lang-auto">("Let's use",3L,'GPUs!')

</code></pre>
<p>I am not sure what is the meaning of L after 3, but i guess that is no biggi for now, i understood what it means…</p>
<p>One more question.<br>
Lets say I have 3GPUs but I just want to use 2 of them. how should i specify that? <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=5" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

ptrblck: <p>The <code>3L</code> is just printing as a long type.<br>
If you would like to use your first two GPUs, just pass the <code>device_ids</code> to <code>DataParallel</code>:</p>
<pre><code>model = nn.DataParallel(model, device_ids=[0, 1])</code></pre>

OrNot: <p>I can not distribute the model to multiple specified gpus suppose I pass 1,2,3,4 from args.</p>
<p>Colud you pls help me on this ?</p>
<p>Thanks</p>
<pre><code class="lang-auto">use_cuda = torch.cuda.is_available()
if use_cuda:
     gpu_ids = list(map(int, args.gpu_ids.split(',')))
     cuda='cuda:'+ str(gpu_ids[0])
     model = DataParallel(model,device_ids=gpu_ids)

device= torch.device(cuda if use_cuda else 'cpu')
model.to(device)
</code></pre>

ptrblck: <p>The GPU mapping starts at index0, so try to pass <code>0,1,2,3</code> as the argument.</p>

OrNot: <p>suppose the GPU 0 is running other users’   script . what should I do?</p>

ptrblck: <p>If your system has &gt;=5 GPUs, your code should work.<br>
Could you print <code>gpu_ids</code> before passing it to <code>nn.DataParallel</code>?<br>
Also, what does <code>nvidia-smi</code> show?<br>
Do you see any utilization in the four GPUs?</p>

OrNot: <ol>
<li>8 GPUs in machine.</li>
<li>I have printed gpu_ids . it shows as I passed from argument, say, 0,1,2,3,4</li>
<li>nvidia-smi  shows only  GPU 0 is working</li>
<li>No other scripts running</li>
</ol>

ptrblck: <p>Are you passing a batch with more than 5 samples to the model?<br>
Note that each chunk of the batch will be send to each GPU, so you should at least pass one sample for each GPU.</p>

OrNot: <p>oops…   maybe you are right.  For some reason, I have to take a prediction  before batch training.</p>
<p>I will check  my code.</p>
<p>Thanks a lot</p>

drevicko: <p>In the tutorials, it mentions nothing about training (ie: no loss function, loss.backward(), optimizer.step() ).</p>
<p>When the model is converted to a DataParallel model, does the backprop get seamlessly handled behind the scenes? I guess the same gradients would be passed to all instances of the model across gpus, is that right? Is there another mechanism to synchronise the model instances across gpus?</p>

ptrblck: <p>The underlying workflow of <code>DataParallel</code> is described in <a href="https://medium.com/huggingface/training-larger-batches-practical-tips-on-1-gpu-multi-gpu-distributed-setups-ec88c3e51255" rel="nofollow noopener">this blog post</a> by <a class="mention" href="/u/thomas_wolf">@Thomas_Wolf</a> in a detailed way.</p>
<p>You could use the functional API and apply the scatter and gather methods manually, if you need more control over what’s being executed.</p>

che85: <p>When using DataParallel, is there is a way to set maximum allocated memory for single GPU?</p>
<p>When running my custom loss, additional memory gets allocated during the forward pass which is NOT only a few hundred megabytes. DataParallel is leaving some additional space on my GPU memory, but it’s not enough for running the forward loss.</p>
<p>Other than that, how does DataParallel decide how to split data?</p>
<p>Best,<br>
Christian</p>

ptrblck: <aside class="quote no-group" data-username="che85" data-post="14" data-topic="20150">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/che85/48/11493_2.png" class="avatar"> che85:</div>
<blockquote>
<p>When using DataParallel, is there is a way to set maximum allocated memory for single GPU?</p>
</blockquote>
</aside>
<p>Not that I’m aware of. Did the work around mentioned in the blog post not work for you?</p>
<aside class="quote no-group" data-username="che85" data-post="14" data-topic="20150">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/che85/48/11493_2.png" class="avatar"> che85:</div>
<blockquote>
<p>Other than that, how does DataParallel decide how to split data?</p>
</blockquote>
</aside>
<p>The batch will be chunked in dim0 based on the number of available GPUs.</p>

che85: <blockquote>
<p>Not that I’m aware of. Did the work around mentioned in the blog post not work for you?</p>
</blockquote>
<p>I implemented everything as mentioned in the blog post but GPU <span class="hashtag-raw">#1</span> still runs out of memory when increasing the batch_size. I notice a significant increase in memory inside the loss forward method.</p>
<p>So I think initialization is fine but at runtime it just uses more memory with every computation added to the graph inside the loss forward method.</p>
<p>DataParallel has to have a mechanism to predict how much memory will be used by a model as well as the loss(criterion), right? So I should expose that information to DataParallelCriterion when initializing or change my loss function.</p>
<p>What’s the correct way of doing this?</p>

ptrblck: <p>Is your training running fine on a single GPU?<br>
If the loss is consuming so much memory, I would assume a single GPU should consume the same amount.</p>
<aside class="quote no-group" data-username="che85" data-post="16" data-topic="20150">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/che85/48/11493_2.png" class="avatar"> che85:</div>
<blockquote>
<p>DataParallel has to have a mechanism to predict how much memory will be used by a model as well as the loss(criterion), right?</p>
</blockquote>
</aside>
<p>Not really, as the computation graph is defined dynamically based on your forward pass.</p>

che85: <aside class="quote no-group" data-username="ptrblck" data-post="17" data-topic="20150">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>Is your training running fine on a single GPU?<br>
If the loss is consuming so much memory, I would assume a single GPU should consume the same amount.</p>
</blockquote>
</aside>
<p>yes on a single GPU with smaller batch-size works fine.</p>

zepmck: <p>I am trying to parallelize an existing model (Transformer) on muliple GPUs. I have a NVIDIA DGX-1 with 8 Volta.</p>
<p>These are the issues/questions I ran into:</p>
<ul>
<li>not clear which value should I set for model.to(device). currently trying with simply .to(‘cuda’)</li>
<li>the model is using a lambda LR scheduler but with GPUs &gt; 1 it exits with an execution error “<strong>UnboundLocalError: local variable ‘values’ referenced before assignment</strong>”</li>
<li>how can I effectlvely check tensor boundaries limit? I am getting the impression that moving a model from 1 to multiple GPUs requires a profound refactoring of the code. It is extremely difficult to understand where indexs can go out of bounds.</li>
</ul>
<p>Thanks</p>

Melody_doudou: <p>I came across the same problem as you did. Could you please tell me how you fixed it? THANKS</p>