Run Pytorch on Multiple GPUs

andrew_su: <p>Hello</p>
<p>Just a noobie question on running pytorch on multiple GPU.<br>
If I simple specify this:</p>
<p><code>device = torch.device("cuda:0")</code>,</p>
<p>this only runs on the <strong>single</strong> GPU unit right?</p>
<p>If I have multiple GPUs, and I want to utilize <strong>ALL OF THEM</strong>. What should I do?<br>
Will below’s command automatically utilize all GPUs for me?</p>
<pre><code class="lang-auto">    use_cuda = not args.no_cuda and torch.cuda.is_available()
    device = torch.device("cuda" if use_cuda else "cpu")
</code></pre>

ptrblck: <p>Wrapping your model in <code>nn.DataParallel</code> is an easy way to use your GPUs.<br>
Have a look at the <a href="https://pytorch.org/tutorials/beginner/former_torchies/parallelism_tutorial.html">parallelism tutorial</a>.</p>

andrew_su: <p>Yes, I have browsed through the topic. But I didn’t find info answering the multiple GPUs question</p>

ptrblck: <p><a href="https://pytorch.org/tutorials/beginner/blitz/data_parallel_tutorial.html#create-model-and-dataparallel">This tutorial</a> might explain it better. Let me know, if this helps you.</p>

KanZa: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a></p>
<p>I am trying to run this project: <a href="https://github.com/chaoyuaw/pytorch-coviar/blob/master/train.py" class="inline-onebox" rel="noopener nofollow ugc">pytorch-coviar/train.py at master · chaoyuaw/pytorch-coviar · GitHub</a> with multiple GPUs. It stops after uploading the videos. Any suggestion?</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/2/2637d1ff8b344d8607c33dd83357d4731365d4d6.png" data-download-href="https://discuss.pytorch.org/uploads/default/2637d1ff8b344d8607c33dd83357d4731365d4d6" title="COVIAR%20on%20GPUs"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/2/2637d1ff8b344d8607c33dd83357d4731365d4d6_2_568x500.png" alt="COVIAR%20on%20GPUs" data-base62-sha1="5s5JM8JSuC9WQKuWdSKathVOC0u" width="568" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/2/2637d1ff8b344d8607c33dd83357d4731365d4d6_2_568x500.png, https://discuss.pytorch.org/uploads/default/optimized/2X/2/2637d1ff8b344d8607c33dd83357d4731365d4d6_2_852x750.png 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/2/2637d1ff8b344d8607c33dd83357d4731365d4d6.png 2x" data-dominant-color="3A182F"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">COVIAR%20on%20GPUs</span><span class="informations">1046×920 134 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>Does this code run with a single GPU?<br>
If so, could you try to set <code>num_workers=0</code> for the <code>DataLoaders</code> in the multi GPU setup and try it again?</p>

KanZa: <p>Yes it is working with single GPU but when the training are with resnet18 architecture and less batch size not with resnet152 and original batch size described by author.</p>

ptrblck: <p>So using a single GPU the code also get’s stuck for resnet152 and the original batch size?</p>

KanZa: <p>yes you are getting me correctly.</p>

KanZa: <p>Do you mean in train.py author is using dataloader for val_loader and train_loader which is like this:</p>
<p>batch_size=args.batch_size, shuffle=False,       num_workers=args.workers, pin_memory=True)</p>
<p>I shall change both val_loader and train_loader like this:</p>
<p>batch_size=args.batch_size, shuffle=False,<br>
num_workers=0, pin_memory=True)</p>
<p>Sorry I am new in deep learning and Pytorch</p>

ptrblck: <p>Yes, I meant exactly this line of code. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=6" title=":wink:" class="emoji" alt=":wink:"><br>
Could you try that, although your error seems to be a bit strange as resnet18 is running while resnet152 gets stuck.</p>

KanZa: <p>The author has used Tesla P100 GPU (FYI).</p>
<p>It has given error I have also tried with both of these too with changes you have described.</p>
<p>model = torch.nn.DataParallel(model, device_ids=None).cuda()</p>
<p>model = torch.nn.DataParallel(model, device_ids=args.gpus).cuda()</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/0/062a074380549e29d7dbc1c9c4a869ce83d7a447.png" data-download-href="https://discuss.pytorch.org/uploads/default/062a074380549e29d7dbc1c9c4a869ce83d7a447" title="With%20MGPUs"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/0/062a074380549e29d7dbc1c9c4a869ce83d7a447_2_690x421.png" alt="With%20MGPUs" data-base62-sha1="SwUqNHAvyToDylMNPsiKY3RyKj" width="690" height="421" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/0/062a074380549e29d7dbc1c9c4a869ce83d7a447_2_690x421.png, https://discuss.pytorch.org/uploads/default/optimized/2X/0/062a074380549e29d7dbc1c9c4a869ce83d7a447_2_1035x631.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/2X/0/062a074380549e29d7dbc1c9c4a869ce83d7a447_2_1380x842.png 2x" data-dominant-color="442037"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">With%20MGPUs</span><span class="informations">1405×859 160 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>Your GPU is out of memory. Probably the model is just too large for your GPU.<br>
You could try to use <a href="https://pytorch.org/docs/stable/checkpoint.html">torch.utils.checkpoint</a> to trade compute for memory.</p>

KanZa: <p>OK Thank you so much for your help.</p>

DoubtWang: <p>Hi,<br>
I have a look at <a href="https://pytorch.org/tutorials/beginner/blitz/data_parallel_tutorial.html#create-model-and-dataparallel" rel="nofollow noopener">data_parallel_tutorial</a>  and <a href="https://pytorch.org/tutorials/beginner/former_torchies/parallelism_tutorial.html" rel="nofollow noopener">parallelism_tutorial</a>.<br>
And I have write my code, but fail to use GPUs.</p>
<p>if opt.gpus = [0, 1]</p>
<pre><code class="lang-auto">class Model(nn.Module):
  def sample(self, input):
    input = input.cuda()
    output = self.other_function(input)
    return output

if len(opt.gpus) &gt; 1:
    print("Let's use", len(opt.gpus), "GPUs!")
    os.environ["CUDA_VISIBLE_DEVICES"] = ','.join(str(x) for x in opt.gpus)
else:
    torch.cuda.set_device(opt.gpus[0])
torch.cuda.manual_seed(opt.seed)

model = Model()
if len(opt.gpus) &gt; 1:
    model = nn.DataParallel(model, device_ids=opt.gpus, dim=0)
if use_cuda:
    model = model.cuda()
 
model.train()
out = model.module.sample(data)

</code></pre>
<p>data.shape : (Batch_size, time_step)</p>
<p>However, model only use the gpus 0 when I run my code .<br>
I don’t  know why ?<br>
Could you explain it ?<br>
Thanks.</p>

ptrblck: <p>How large is your batch size?<br>
Using <code>nn.DataParallel</code>your data will be split in dim0 and each chunk of data will be send to a device.<br>
If the batch size is too small, some GPU won’t be able to get a data chunk.</p>

DoubtWang: <p>Thank you for the quick reply (appreciated).<br>
My batch_size = 8,<br>
because the length of train_data sample is very long.<br>
I will get the error message (<code> out of memory</code> ) when I only use one gpu.<br>
So,  I plan to use the multi-gpus to train my model.<br>
However, the model only use the gpus 0 when gpus= [0, 1].</p>
<aside class="quote no-group" data-username="ptrblck" data-post="17" data-topic="20932">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>If the batch size is too small, some GPU won’t be able to get a data chunk.</p>
</blockquote>
</aside>
<p>In general, batch_size be equal to ?<br>
Could you give me some advice, thank you.</p>

ptrblck: <p>If your batch size is 8, each GPU should get 4 samples each.<br>
Do you see the GPU1 completely empty without any utilization in <code>nvidia-smi</code> or a similar tool?</p>

DoubtWang: <p>Sorry, I have been late because of the time zone.</p>
<p>I use the <code>watch -n 1 -d nvidia-smi</code> when I run my model.<br>
However, I see the the GPU1 completely empty.<br>
I am confused why this happens.<br>
Can you explain it in your experience?<br>
Thanks.</p>
<p>Besides,<br>
I output the <strong>data</strong>  tensor.shape(time_step, Batch_size_data, embedding_dim)<br>
before <code>out = model.module.sample(data)</code>.<br>
And I also output the <strong>output</strong> tensor.shape(time_step, Batch_size_output, hidden_dim) after the other_function.</p>
<pre><code class="lang-auto">class Model(nn.Module):
        def sample(self, input):
               input = input.cuda()
               output = self.other_function(input)
               return output

</code></pre>
<p>However, the Batch_size_data == Batch_size_output.<br>
In theory, I think  Batch_size_data should be equal to 2*Batch_size_output when using the GPU0 and GPU1.</p>
<p>Note that I modified parameter setting<br>
model = nn.DataParallel(model, device_ids=opt.gpus, <strong>dim=1</strong>)</p>

DoubtWang: <p>I found that just modifying the function name will solve my problem.</p>
<p>I changed this part:</p>
<pre><code class="lang-auto">class Model(nn.Module):
  def forward(self, input):
    input = input.cuda()
    output = self.other_function(input)
    return output
</code></pre>
<pre><code class="lang-auto">model.train()
out = model(data)
</code></pre>