How pytorch's parallel method and distributed method works?

username: fantasticfears
cooked: <p>\I’m no expert in distributed system and CUDA. But there is one really interesting feature that PyTorch support which is <code>nn.DataParallel</code> and <code>nn.DistributedDataParallel</code>. How they are actually implemented? How they separate common embeddings and synchronize data?</p>
<p>Here is a basic example of <code>DataParallel</code>.</p>
<pre><code>import torch.nn as nn
from torch.autograd.variable import Variable
import numpy as np

class Model(nn.Module):
    def __init__(self):
        super().__init__(
            embedding=nn.Embedding(1000, 10),
            rnn=nn.Linear(10, 10),
        )

    def forward(self, x):
        x = self.embedding(x)
        x = self.rnn(x)
        return x

model = nn.DataParallel(Model())
model.forward(Variable.from_numpy(np.array([1,2,3,4,5,6], dtype=np.int64)).cuda()).cpu()
</code></pre>
<p>PyTorch can split the input and send them to many GPUs and merge the results back.</p>
<p>How does it manage embeddings and synchronization for a parallel model or a distributed model? <br><br>
I wandered around PyTorch’s code but it’s very hard to know how the fundamentals work.</p>
<hr>
<p>I’ve posted this question on <a href="https://stackoverflow.com/questions/53375422/how-pytorchs-parallel-method-and-distributed-method-works" rel="nofollow noopener">StackOverflow</a>.</p>

username: rasbt
cooked: <p>I am not sure about <code>DistributedParallel</code> but in <code>DataParallel</code> each GPU gets a copy of the model, so, the parallelization is done via splitting the minibatches, not the layers/weights.</p>
<p>Here’s a sketch of how DataParallel works, assuming 4 GPUs where GPU:0 is the default GPU.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/e/e153c9fa07a2688c77fc1d7a572f26d12828528d.png" data-download-href="https://discuss.pytorch.org/uploads/default/e153c9fa07a2688c77fc1d7a572f26d12828528d" title="dataparallel"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/e/e153c9fa07a2688c77fc1d7a572f26d12828528d_2_275x500.png" alt="dataparallel" data-base62-sha1="w9kWLHLiy0iUDAkiTTrmktaGqLj" width="275" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/e/e153c9fa07a2688c77fc1d7a572f26d12828528d_2_275x500.png, https://discuss.pytorch.org/uploads/default/optimized/2X/e/e153c9fa07a2688c77fc1d7a572f26d12828528d_2_412x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/2X/e/e153c9fa07a2688c77fc1d7a572f26d12828528d_2_550x1000.png 2x" data-dominant-color="352842"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">dataparallel</span><span class="informations">1275×2312 660 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: gjpicker
cooked: <p>dear <a class="mention" href="/u/rasbt">@Rasbt</a><br>
I am so sorry to recognize the step5 draft words , could you pls re-type here for reply . thanks a lot</p>

username: rasbt
cooked: <p>Haha no problem</p>
<blockquote>
<p>Compute the loss with respect to the network outputs on the default GPU and scatter the losses back to the individual GPUs to compute the gradients with respect to the leaf nodes</p>
</blockquote>

username: ranger
cooked: <p>Can every network be ported for multi gpu if the model fits on single gpu?</p>

username: rasbt
cooked: <aside class="quote no-group" data-username="ranger" data-post="5" data-topic="30349" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ranger/48/12593_2.png" class="avatar"> ranger:</div>
<blockquote>
<p>Can every network be ported for multi gpu if the model fits on single gpu?</p>
</blockquote>
</aside>
<p>Yes, I believe so. If your model does not fit on a single GPU though, there’s even another method that you can use (instead of DataParallel). E.g.,</p>
<pre data-code-wrap="python"><code class="lang-python">class Network(nn.Module):
    def __init__(self, split_gpus):
        self.module1 = (some layers)
        self.module2 = (some layers)

        self.split_gpus = split_gpus
        if self.split_gpus:
            self.module1.device("cuda:0")
            self.module2.device("cuda:1")

    def forward(self, x):
        x = self.module1(x)
        if self.split_gpus:
            x = x.device("cuda:1") 
        return self.module2(x)
</code></pre>

username: ranger
cooked: <p><a class="mention" href="/u/rasbt">@rasbt</a></p>
<p>Any views on this thread<br>
<a href="https://discuss.pytorch.org/t/how-to-parallelise-a-pytorch-model-on-gpu/31334/14">How to parallelise a pytorch model on GPU?</a></p>

username: gjpicker
cooked: <aside class="quote no-group" data-username="ranger" data-post="7" data-topic="30349">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ranger/48/12593_2.png" class="avatar"> ranger:</div>
<blockquote>
<p>How to parallelise a pytorch model on GPU?</p>
</blockquote>
</aside>
<p>why to gather all outputs into GPU0  in step5 ,  it merely calculate distance of output_i with target_i ?  why not calculate it within each GPU separately   ?</p>
<p>one possible reason  :  all target (ground truth) are just stored in GPU0 , however ,  target can also be scattered as a addition of minibatch, like scatter data chunk  , in step1</p>

username: rasbt
cooked: <blockquote>
<p>why to gather all outputs into GPU0 in step5 , it merely calculate distance of output_i with target_i ? why not calculate it within each GPU separately ?</p>
</blockquote>
<p>It’s more like an implementation side-effect. In fact, you can do what you proposed, but you would have to rewrite your code then to compute the loss inside the model – usually, the loss is computed in the training loop based on the outputs of the model.</p>
<p>I mean, it’s not a problem at all to do what you propose, but it’s a bit less convenient, because when you decide to use DataParallel (occasionally), you would have to modify your model code as well.</p>
<p>Also, computing the loss is super cheap, so there is not much speed gain in distributing it across the devices, I suppose.</p>

username: gjpicker
cooked: <p>sorry ,I can’t agree with this explanation , thrifting or economy the distributing cluster’s  compute energy . Because transportation data is not cheaper than calculating within micro chip , in my opinion .</p>
<p>is there some welcomed link or chart  to strengthen  ?</p>

username: rasbt
cooked: <blockquote>
<p>sorry ,I can’t agree with this explanation , thrifting or economy the distributing cluster’s compute energy . Because transportation data is not cheaper than calculating within micro chip , in my opinion .</p>
<p>is there some welcomed link or chart to strengthen ?</p>
</blockquote>
<p>I don’t have a chart handy, but when I ran the code on my workstation (only 4x 1080Tis), I didn’t notice any difference between letting the GPU handle the loss computation versus distributing the results, calculating the results on the individual GPUs, and then gathering them. Either way, I get the same ~3x speedup when using 4 instead of 1 GPU. This is a small hardware setup, and the scenario is probably different for large datacenters, like you said… Also, the way I implemented it,</p>
<pre data-code-wrap="python"><code class="lang-python">model = MyModel(num_features=num_features, num_classes=num_classes)
if torch.cuda.device_count() &gt; 1:
    print("Using", torch.cuda.device_count(), "GPUs")
    model = nn.DataParallel(model)
</code></pre>
<p>and then</p>
<pre data-code-wrap="python"><code class="lang-python">for epoch in range(NUM_EPOCHS):
    
    model.train()
    for batch_idx, (features, targets) in enumerate(train_loader):
        
        features = features.to(DEVICE)
        targets = targets.to(DEVICE)
            
        ### FORWARD AND BACK PROP
        logits, probas = model(features)
        cost = cost_fn(logits, targets)
        optimizer.zero_grad()
        
        cost.backward()
        
        ### UPDATE MODEL PARAMETERS
        optimizer.step()
</code></pre>
<p>Is basically such that I don’t have to rewrite my model when I decide to use DataParallel, which is most convenient for my use cases since I do not always run DataParallel.</p>

username: ranger
cooked: <p>Thanks for the detailing here.<br>
How can we put the computation part of the losses on another gpu(GPU:4) rather than GPU:0 which happens by default.</p>
<p>It will be great to have the method as many networks are designed such that they cover ~11 gigs and this things holds you back while parallelizing such a network.</p>

username: fantasticfears
cooked: <p>It doesn’t makes sense. If the loss aggregation is the bottleneck, what’s the benefit to compute on a GPU? The computation of addition shouldn’t get faster because of devices.</p>

username: ranger
cooked: <p>Computation of addition is not the roadblock. There is no space left for the computation on the gpu. That is the issue which I am trying to solve.</p>
<p>I hope it’s clear now.</p>

username: rasbt
cooked: <p>You actually can use a different device as default device for data parallelism. However, this must be one of the GPUs that is also used by DataParallel. You can set it via the <code>output_device</code> parameter.</p>
<p>Regarding using a GPU that is not wrapped by DataParallel, that’s currently not possible (see discussion here: <a href="https://discuss.pytorch.org/t/uneven-gpu-utilization-during-training-backpropagation/36117/3" class="inline-onebox">Uneven GPU utilization during training backpropagation</a>)</p>

username: fantasticfears
cooked: <p>I finally did the code reading on <code>DataParallel</code> part.</p>
<aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://erickguan.me/favicon-32x32.png" class="site-icon" width="32" height="32">
      <a href="https://erickguan.me/2019/pytorch-parallel-model" target="_blank" title="12:00AM - 05 March 2019" rel="nofollow noopener">Blog – 5 Mar 19</a>
  </header>
  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:200/40;"><img src="https://erickguan.me/assets/images/2019/Pytorch_logo-200.png" class="thumbnail"></div>

<h3><a href="https://erickguan.me/2019/pytorch-parallel-model" target="_blank" rel="nofollow noopener">How PyTorch implements DataParallel?</a></h3>

<p>PyTorch can send batches and models to different GPUs automatically with DataParallel(model). How is it possible? I assume you know PyTorch uses dynamic computational graph. And PyTorch version is v1.0.1.</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>


username: dambo
cooked: <p>Hello,<br>
Is there any analogous version of DataParallel in C++/Libtorch?<br>
Whatever I do on an exported traced model, it always uses only a single GPU.<br>
Refer to: <a href="https://discuss.pytorch.org/t/automatic-parallelization-of-models-onto-multiple-gpus/48125" class="inline-onebox">"Automatic parallelization of models onto multiple GPUs"</a></p>

username: Zhihan_Yang
cooked: <p>As shown in the diagram you posted earlier on, “Step 5” says that “loss [is computed] with respect to network outputs on default GPU”. My question is, are losses computed for each network eventually averaged or summed over? I am not sure what exactly my loss function is printing.</p>