Simple computational multi gpu tutorial

heavyfranz: <p>Hi there!<br>
I’m using pytorch as an autograd library.  Can someone provide a simple tutorial or a snippet for a simple example of multi-gpu processing?</p>
<p>If no gradient have to be generated the example in <a href="https://discuss.pytorch.org/t/pytorch-how-to-parallelize-over-multiple-gpu-using-multiprocessing-pool/90411/4" class="inline-onebox">PyTorch: How to parallelize over multiple GPU using multiprocessing.pool</a><br>
seems reasonable, but what to do if the gradients are needed?<br>
Is there a tutorial on simple “mpi-like” calls reported in the doc? <a href="https://pytorch.org/tutorials/beginner/former_torchies/parallelism_tutorial.html" rel="noopener nofollow ugc">https://pytorch.org/tutorials/beginner/former_torchies/parallelism_tutorial.html</a><br>
How to use the data_parallel function reported there?</p>
<p>I tried to apply the concepts in the DataParallel guide, but actually the code is slower if the model is encapsulated in the nn.DataPrallel. Clearly something is missing on my side.</p>
<pre><code class="lang-auto">import torch
from torch import Tensor
import torch.nn as nn
from torch.nn.parameter import Parameter
import time

class ModDef(nn.Module):
    def __init__(self, input_size=100, output_size=100) -&gt; None:
        super(ModDef, self).__init__()
        self.w1 = Parameter(torch.randn(1024,1024))
        self.w2 = Parameter(torch.randn(1024,1024))


    def forward(self, X: Tensor) -&gt; Tensor:
        output = torch.exp(100+(self.w1 * X - X.mean() / (self.w2 + 1))**2)
        return output


if __name__ == '__main__':
    device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
    model = ModDef()
    print('send')
    my = nn.DataParallel(model).to(device)
    
    optimizer = torch.optim.SGD(my.parameters(), lr=1e-4)
    
    datain = torch.randn(100,1024,1024).to(device)
    d2 = torch.randn(100,1024,1024).to(device)
    print('start')
    ta = time.time()
    for ii in range(100):
        optimizer.zero_grad()
        out = my(datain)
        loss = (out - d2).sum()
        loss.backward()
        optimizer.step()
    tb = time.time()
    print(tb-ta)
    print('end')
    

</code></pre>

mrshenli: <p>Hey <a class="mention" href="/u/heavyfranz">@heavyfranz</a></p>
<p>Please see this section in the overview: <a href="https://pytorch.org/tutorials/beginner/dist_overview.html#data-parallel-training" rel="noopener nofollow ugc">https://pytorch.org/tutorials/beginner/dist_overview.html#data-parallel-training</a></p>
<p><code>DataParallel</code> can be slower especially if the model is large, as it will do model replication, input scattering, and output gathering in each forward pass. <code>DistributedDataParallel</code> is expected to be faster in this case.</p>

heavyfranz: <p>Dear <a class="mention" href="/u/mrshenli">@mrshenli</a>,<br>
thank you for the kind and fast answer <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"> Yeah, you’re right.<br>
The following in your reply makes me drop my jaw:</p>
<blockquote>
<p>it will do model replication […] in each forward pass</p>
</blockquote>
<p>It was not clear for me from the beginning.</p>
<p>Somewhere I read that at least the scattering operation is done automatically segmenting the input tensor on the first axis, is it true?</p>
<p>Finally, just to know, in your opinion, is the code is at least correct for the DataParallel construct?</p>

mrshenli: <aside class="quote no-group" data-username="heavyfranz" data-post="3" data-topic="98341">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/h/3bc359/48.png" class="avatar"> heavyfranz:</div>
<blockquote>
<p>Somewhere I read that at least the scattering operation is done automatically segmenting the input tensor on the first axis, is it true?</p>
</blockquote>
</aside>
<p>Yes, this is truth. <code>DataParallel</code> will consider the first dimension as the batch dimension.</p>
<blockquote>
<p>Finally, just to know, in your opinion, is the code is at least correct for the DataParallel construct?</p>
</blockquote>
<p>It should work I think. But, usually the model is moved to the the target device before passing it to <code>DataParallel</code> ctor, like:</p>
<pre><code class="lang-auto">my = nn.DataParallel(model.to(device))
</code></pre>