How to use multiple GPUs in Pytorch?

username: Rafael_R
cooked: <p>Hi,</p>
<p>I have a Pytorch model for machine translation. I rented a 4 GPU machine.</p>
<p>The dataset is very large.</p>
<p>How to train my model using the 4 GPUs?</p>
<p>I see the model uses only 1 GPU.</p>
<p>So, my question is:</p>
<ol>
<li>
<p>What are the mechanics of training on 4 GPUs?</p>
</li>
<li>
<p>Any way I can make my model run on the 4 GPUs?</p>
</li>
</ol>
<p>Thanks.</p>

username: Rafael_R
cooked: <p>hi, any help will save me a lot of money <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: ptrblck
cooked: <p>Have a look at <a href="https://pytorch.org/docs/master/nn.html#torch.nn.parallel.DistributedDataParallel" rel="nofollow noopener">torch.nn.parallel.DistributedDataParallel</a>. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

username: pinocchio
cooked: <p>so pytorch or machines with multiple GPUs do not use the multiple GPUs by themselves?</p>

username: ptrblck
cooked: <p>That’s right. You can use a one-liner and wrap your model in <code>nn.DataParallel</code> or use the recommended DDP approach.<br>
Alternatively, you could also use model sharding and split the model among all GPUs in case you are working with a huge model.</p>

username: Rafael_R
cooked: <p>thanks, i am having some issues with mult-gpu training. i have multiple encoders, and the data parallel module is splitting along different dimensions for different encoders, so i get error in decoder.</p>
<p>does data parallel module assume the first dimension is the batch? LSTM module assumes Batch is the middle dimension.</p>
<p><img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="Rafael_R" data-post="6" data-topic="44776">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/rafael_r/48/9835_2.png" class="avatar"> Rafael_R:</div>
<blockquote>
<p>does data parallel module assume the first dimension is the batch?</p>
</blockquote>
</aside>
<p>Yes, the batch will be chunked in dim0.<br>
You could try to permute the data or use <code>batch_first=True</code> in your LSTM.</p>

username: aksg87
cooked: <p>Hello,</p>
<p>I have been trying to train additional models / do work on a second GPU of a machine but am running into issues. I have confirmed that torch.cuda recognizes 2 GPUs but I cannot switch to second GPU to train different models in parallel.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/1/153b065a87368c823fc258434f0c951dad16f267.png" data-download-href="https://discuss.pytorch.org/uploads/default/153b065a87368c823fc258434f0c951dad16f267" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/1/153b065a87368c823fc258434f0c951dad16f267_2_690x420.png" alt="image" data-base62-sha1="31OubUJ346LP1N4Mmjo6kLiqPhd" width="690" height="420" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/1/153b065a87368c823fc258434f0c951dad16f267_2_690x420.png, https://discuss.pytorch.org/uploads/default/original/2X/1/153b065a87368c823fc258434f0c951dad16f267.png 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/1/153b065a87368c823fc258434f0c951dad16f267.png 2x" data-dominant-color="F4F4F4"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">713×434 36.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: aksg87
cooked: <p>I think I solved this by adding:</p>
<pre><code class="lang-python">torch.backends.cudnn.enabled=True</code></pre>

username: bao18
cooked: <p>Many thanks for your help.<br>
I got the following error message, could you tell me how to fix it? thanks a lot.</p>
<p><code>RuntimeError: expected device cuda:3 and dtype Float but got device cuda:0 and dtype Float</code></p>

username: ptrblck
cooked: <p>If you are using <code>nn.DataParallel</code>, this error is often raised if new tensors are created in the <code>forward</code> method and pushed to the default device (<code>cuda:0</code>).<br>
Could you post the model definition so that we could have a look?</p>

username: bao18
cooked: <p>Many thanks for your prompt reply <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"><br>
Here is my forward definition. The network is quite big, but it is a UNet-based for semantic segmentation using a shared encoder.</p>
<pre><code class="lang-auto">    def forward(self, x):
        features_x1 = self.encoder(x[:, : self.in_channels // 2, :, :])
        features_x2 = self.encoder(x[:, self.in_channels // 2 :, :, :])
        features = [torch.cat([x1, x2], 1) for x1, x2 in zip(features_x1, features_x2)]
        features = [self.res[i].to(x.device)(x) for i, x in enumerate(features)]

        decoder_output = self.decoder(*features)
        masks = self.segmentation_head(decoder_output)

        return masks
</code></pre>

username: ptrblck
cooked: <p>Is it necessary to call <code>self.res[i].to(x.device)</code> in the list comprehension?<br>
Based on the code, it should be a no-op, since <code>self.res</code> should already be on the corresponding device, if it’s registered as a module.<br>
Could you post the complete error message including the line of code, which raises this error?</p>

username: bao18
cooked: <p>Hello (sorry for the late reply)</p>
<p>Regarding your first question, <code>self.res[i].to(x.device)</code>.<br>
I do this because I need to reduce the size (<code>c*2</code>) of the concatenated features in the previous step.<br>
Here, <code>self.res() = nn.Conv2d(c*2, c, stride=1, 0)</code> where <code>c</code> is the size of <code>x1</code> and <code>x2</code></p>
<p>As for the full error messages:</p>
<pre><code class="lang-auto">Epoch: 1
train:   0%|                                                                       | 0/146 [00:12&lt;?, ?it/s]
Traceback (most recent call last):
  File "train_network.py", line 112, in &lt;module&gt;
    train_logs, *_ = train_epoch.run(train_dataloader)
  File "/uge_mnt/home/bruno/parallel/codes/tools.py", line 125, in run
    loss, y_pred = self.batch_update(x, y)
  File "/uge_mnt/home/bruno/parallel/codes/tools.py", line 180, in batch_update
    loss.backward()
  File "/home/xxx/apps/intelpython3/lib/python3.6/site-packages/torch/tensor.py", line 118, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "/home/xxx/apps/intelpython3/lib/python3.6/site-packages/torch/autograd/__init__.py", line 93, in backward
    allow_unreachable=True)  # allow_unreachable flag
RuntimeError: expected device cuda:3 and dtype Float but got device cuda:2 and dtype Float
</code></pre>
<p>many thanks in advance for your help <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: ptrblck
cooked: <p>Thanks for the follow-up.<br>
Could you post a minimal, executable code snippet to reproduce this error please, as I cannot see any issues in the current code?</p>