Issue with using DataParallel (includes minimal code)

blade: <p>I have the following (minimal) code that runs on GPU and I’m trying to run it in multiple GPUs using <code>nn.DataParallel</code>:</p>
<pre><code class="lang-auto">import math
import torch
import pickle
import time

import numpy as np
import torch.optim as optim

from torch import nn

print('device_count()', torch.cuda.device_count())
for i in range(torch.cuda.device_count()):
    print('get_device_name', torch.cuda.get_device_name(i))

def _data(dimension, num_examples):
    num_mislabeled_examples = 20

    ground_truth_weights = np.random.normal(size=dimension) / math.sqrt(dimension)
    ground_truth_threshold = 0

    features = np.random.normal(size=(num_examples, dimension)).astype(
        np.float32) / math.sqrt(dimension)
    labels = (np.matmul(features, ground_truth_weights) &gt;
              ground_truth_threshold).astype(np.float32)
    mislabeled_indices = np.random.choice(
        num_examples, num_mislabeled_examples, replace=False)
    labels[mislabeled_indices] = 1 - labels[mislabeled_indices]

    return torch.tensor(labels), torch.tensor(features)

class tools:
    def __init__(self):
        self.name = 'x_2'

    def SomeFunc(self, model, input_):
        print(model.first_term(input_)[0])        


class predictor(nn.Module):
    def __init__(self, dim):
        super(predictor, self).__init__()
        self.weights = torch.nn.Parameter(torch.zeros(dim, 1, requires_grad=True))
        self.threshold = torch.nn.Parameter(torch.zeros(1, 1, requires_grad=True))

    def first_term(self, features):
        return features @ self.weights

    def forward(self, features):
        return self.first_term(features) - self.threshold

class HingeLoss(nn.Module):

    def __init__(self):
        super(HingeLoss, self).__init__()
        self.relu = nn.ReLU()

    def forward(self, output, target):
        all_ones = torch.ones_like(target)
        labels = 2 * target - all_ones
        losses = all_ones - torch.mul(output.squeeze(1), labels)

        return torch.norm(self.relu(losses))

class function(object):
    def __init__(self, epochs):

        dim = 10
        N = 100
        self.target, self.features = _data(dim, N)

        self.epochs = epochs 
        self.model = predictor(dim).to('cuda')
        self.optimizer = optim.SGD(self.model.parameters(), lr=1e-3)
        self.target = self.target.to('cuda')
        self.features = self.features.to('cuda')
        self.loss_function = HingeLoss().to('cuda')

        self.tools = tools()

    def train(self):

        self.model.train()
        for epoch in range(self.epochs):
            self.optimizer.zero_grad()
            output = self.model(self.features)
            # self.tools.SomeFunc(self.model, self.features)
            print(output.is_cuda)
            loss = self.loss_function(output, self.target)
            loss.backward()
            print('For epoch {}, loss is: {}.'.format(epoch, loss.item()))
            self.optimizer.step()

def main():
    model = function(1000)
    print(torch.cuda.device_count())
    if False: # This is Flag
        if torch.cuda.device_count() &gt; 1:
            model.model = nn.DataParallel(model.model)
    t = time.time()
    model.train()
    print('elapsed: {}'.format(time.time() - t))

if __name__ == '__main__':
    main()
</code></pre>
<ol>
<li>
<p>I have 4 GPU cards (device_count = 4). When I set the flag (indicated with the comment <code>This is Flag</code>) to <code>True</code>, it takes 15.78 seconds to run the code. When I set it to False, it takes 0.71 seconds. Why? How could it be fixed?</p>
</li>
<li>
<p>When I uncomment the line <code>self.tools.SomeFunc(self.model, self.features)</code> and set the flag to <code>True</code>, I receive the following error:</p>
</li>
</ol>
<blockquote>
<p>AttributeError: ‘DataParallel’ object has no attribute ‘first_term’</p>
</blockquote>
<p>How should I fix this? Thanks!</p>

mrshenli: <aside class="quote no-group" data-username="blade" data-post="1" data-topic="87220">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/blade/48/23861_2.png" class="avatar"> blade:</div>
<blockquote>
<p>I have 4 GPU cards (device_count = 4). When I set the flag (indicated with the comment <code>This is Flag</code> ) to <code>True</code> , it takes 15.78 seconds to run the code. When I set it to False, it takes 0.71 seconds. Why? How could it be fixed?</p>
</blockquote>
</aside>
<p>One thing is that you will need a <code>torch.cuda.synchronize()</code> before calling <code>time.time()</code> to make sure all pending CUDA kernals in the stream are finished.</p>
<p>You can also use <a href="https://pytorch.org/docs/stable/cuda.html#torch.cuda.Event.elapsed_time" rel="noopener nofollow ugc"><code>elapsed_time</code></a> to measure. See discussion <a href="https://discuss.pytorch.org/t/does-it-ever-make-sense-to-try-model-parallelism-even-if-the-model-fits/86432/14">here</a>.</p>
<p>If you are looking for the most performant solution, <code>DistributedDataParallel</code> should be the way to go. [<a href="https://pytorch.org/docs/master/notes/ddp.html" rel="noopener nofollow ugc">example</a>]</p>
<blockquote>
<p>When I uncomment the line  <code>self.tools.SomeFunc(self.model, self.features)</code>  and set the flag to  <code>True</code> , I receive the following error:</p>
</blockquote>
<p>Looks like <code>self.model</code> is a <code>DataParallel</code> instance? If so, <code>DataParallel</code> does not have the <code>first_term</code> attribute. If this attribute is on the model instance you passed to <code>DataParallel</code>, you can access the original model instance through <code>self.model.module</code> (see DataParallel code <a href="https://github.com/pytorch/pytorch/blob/df8d6eeb19423848b20cd727bc4a728337b73829/torch/nn/parallel/data_parallel.py#L131" rel="noopener nofollow ugc">here</a>) which should have the <code>first_term</code> attribute.</p>

blade: <p>Regarding the first part, calling <code>torch.cuda.synchronize()</code> did not help. It still seems that using 4 GPUs instead of 1 makes the code 15 times slower.</p>
<p>After fixing the second part, my output for</p>
<pre><code class="lang-auto">print(model.module.first_term(input_)[0])
</code></pre>
<p>is always on the first worker:</p>
<pre><code class="lang-auto">tensor([0.0020], device='cuda:0', grad_fn=&lt;SelectBackward&gt;)
</code></pre>
<p>So it is not even sharing the work between the different worker.</p>

mrshenli: <aside class="quote no-group" data-username="blade" data-post="3" data-topic="87220">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/blade/48/23861_2.png" class="avatar"> blade:</div>
<blockquote>
<p>So it is not even sharing the work between the different worker.</p>
</blockquote>
</aside>
<p>What is “worker” here? Do you mean GPU? If so, isn’t that expected?  IIUC, in your code, <code>model</code> is the <code>DataParallel</code> instance. So only the forward function of <code>model</code> would utilize multiple GPUs. See the data parallel implementation below:</p>
<aside class="onebox githubblob" data-onebox-src="https://github.com/pytorch/pytorch/blob/4104ab8b187d6023b0cc80b77e6944126009c532/torch/nn/parallel/data_parallel.py#L141-L156">
  <header class="source">

      <a href="https://github.com/pytorch/pytorch/blob/4104ab8b187d6023b0cc80b77e6944126009c532/torch/nn/parallel/data_parallel.py#L141-L156" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/4104ab8b187d6023b0cc80b77e6944126009c532/torch/nn/parallel/data_parallel.py#L141-L156" target="_blank" rel="noopener nofollow ugc">pytorch/pytorch/blob/4104ab8b187d6023b0cc80b77e6944126009c532/torch/nn/parallel/data_parallel.py#L141-L156</a></h4>



    <pre class="onebox"><code class="lang-py">
      <ol class="start lines" start="141" style="counter-reset: li-counter 140 ;">
          <li>def forward(self, *inputs, **kwargs):</li>
          <li>    if not self.device_ids:</li>
          <li>        return self.module(*inputs, **kwargs)</li>
          <li></li>
          <li>    for t in chain(self.module.parameters(), self.module.buffers()):</li>
          <li>        if t.device != self.src_device_obj:</li>
          <li>            raise RuntimeError("module must have its parameters and buffers "</li>
          <li>                               "on device {} (device_ids[0]) but found one of "</li>
          <li>                               "them on device: {}".format(self.src_device_obj, t.device))</li>
          <li></li>
          <li>    inputs, kwargs = self.scatter(inputs, kwargs, self.device_ids)</li>
          <li>    if len(self.device_ids) == 1:</li>
          <li>        return self.module(*inputs[0], **kwargs[0])</li>
          <li>    replicas = self.replicate(self.module, self.device_ids[:len(inputs)])</li>
          <li>    outputs = self.parallel_apply(replicas, inputs, kwargs)</li>
          <li>    return self.gather(outputs, self.output_device)</li>
      </ol>
    </code></pre>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>


mrshenli: <p>Regarding the slowdown, I can reproduce it locally with two GPUs:</p>
<p>Using DataParallel on 2 GPUs</p>
<pre><code class="lang-auto">For epoch 999, loss is: 8.4603910446167.
elapsed: 3.1627652645111084
</code></pre>
<p>Not using DataParallel</p>
<pre><code class="lang-auto">For epoch 999, loss is: 8.323615074157715.
elapsed: 1.192000389099121
</code></pre>
<p>Then I go back to inspect the model:</p>
<pre><code class="lang-python">    def __init__(self, dim):
        super(predictor, self).__init__()
        self.weights = torch.nn.Parameter(torch.zeros(dim, 1, requires_grad=True))
        self.threshold = torch.nn.Parameter(torch.zeros(1, 1, requires_grad=True))
</code></pre>
<p>My suspicion is that the model parameters are so light that the overhead of GIL contention, input scattering, output gathering, and model replication in DataParallel forward pass overshadows the speedup brought by multi-gpu training. Are these parameters used in real applications or are you trying to profile <code>DataParallel</code> performance?</p>

blade: <aside class="quote no-group" data-username="mrshenli" data-post="4" data-topic="87220">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mrshenli/48/12220_2.png" class="avatar"> mrshenli:</div>
<blockquote>
<p>See the data parallel implementation below:</p>
</blockquote>
</aside>
<p>I didn’t quite get your point on this one.</p>
<aside class="quote no-group" data-username="mrshenli" data-post="5" data-topic="87220">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mrshenli/48/12220_2.png" class="avatar"> mrshenli:</div>
<blockquote>
<p>My suspicion is that the model parameters are so light that the overhead of GIL contention, input scattering, output gathering, and model replication in DataParallel forward pass overshadows the speedup brought by multi-gpu training.</p>
</blockquote>
</aside>
<p>That makes sense. Let me try it on a more sophisticated code and update the answer.</p>
<aside class="quote no-group" data-username="mrshenli" data-post="5" data-topic="87220">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mrshenli/48/12220_2.png" class="avatar"> mrshenli:</div>
<blockquote>
<p>Are these parameters used in real applications or are you trying to profile <code>DataParallel</code> performance?</p>
</blockquote>
</aside>
<p>I’m just making sure I understand how to use DataParallel correctly. Although way simpler, this sample code mimics the general structure of my actual code. So, there is nothing wrong with my implementation?</p>
<aside class="quote no-group" data-username="mrshenli" data-post="2" data-topic="87220">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mrshenli/48/12220_2.png" class="avatar"> mrshenli:</div>
<blockquote>
<p>you can access the original model instance through <code>self.model.module</code> (see DataParallel code <a href="https://github.com/pytorch/pytorch/blob/df8d6eeb19423848b20cd727bc4a728337b73829/torch/nn/parallel/data_parallel.py#L131" rel="noopener nofollow ugc">here </a>) which should have the <code>first_term</code> attribute.</p>
</blockquote>
</aside>
<p>Thanks, based on your response I used <a href="https://github.com/pytorch/pytorch/issues/16885#issuecomment-551779897" rel="noopener nofollow ugc">this wrapper</a> to get access to attributes without altering the code itself.</p>

mrshenli: <aside class="quote no-group" data-username="blade" data-post="6" data-topic="87220">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/blade/48/23861_2.png" class="avatar"> blade:</div>
<blockquote>
<p>I didn’t quite get your point on this one.</p>
</blockquote>
</aside>
<p>I might have misunderstood the original question.</p>
<blockquote>
<p>After fixing the second part, my output for</p>
<pre><code class="lang-auto">print(model.module.first_term(input_)[0])
</code></pre>
<p>is always on the first worker:</p>
<pre><code class="lang-auto">tensor([0.0020], device='cuda:0', grad_fn=&lt;SelectBackward&gt;)
</code></pre>
</blockquote>
<p>Is the question about why the output of <code>print(model.module.first_term(input_)[0])</code> always on <code>cuda:0</code>?</p>

blade: <p>Yes: I’m under impression that there are 2 ways of parallelizing a PyTorch code: <code>DistributedDataParallel</code> and <code>DataParallel</code>. In the former each layer of the network is assigned to a particular processor while in the latter, each processor takes a portion of the training data and all the processors go through all the code (<a href="https://pytorch.org/tutorials/beginner/blitz/data_parallel_tutorial.html#results" rel="nofollow noopener">like here</a>). Although <code>DistributedDataParallel</code> is preferred (though I’m not sure why, except for multi-node processing, perhaps?), it looks hairy and I decided to start with <code>DataParallel</code>. Hence, I expected all the processors call <code>first_term()</code> when they get to that part of the code. What am I missing?</p>

mrshenli: <aside class="quote no-group quote-modified" data-username="blade" data-post="8" data-topic="87220">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/blade/48/23861_2.png" class="avatar"> blade:</div>
<blockquote>
<p>Although <code>DistributedDataParallel</code> is preferred (though I’m not sure why)</p>
</blockquote>
</aside>
<p>This is mostly due to performance reasons. As of today, DataParallel (DP) replicates model in its forward pass, while DistributedDataParallel (DDP) replicates models in its ctor. That means DP would replicate model once in every iteration. Besides, DP also suffers from GIL contention as it is single-process-multi-thread. DDP does not hit this problem, as each model replica runs in its own process. More info about DDP can be found <a href="https://pytorch.org/docs/master/notes/ddp.html" rel="noopener nofollow ugc">here</a> and <a href="https://arxiv.org/abs/2006.15704" rel="noopener nofollow ugc">here</a>.</p>
<blockquote>
<p>Hence, I expected all the processors call  <code>first_term()</code>  when they get to that part of the code. What am I missing?</p>
</blockquote>
<p>What happens in DP’s forward function is: 1) replicate model to all devices 2) scatter inputs to all devices 3) launch multiple threads in parallel, where each threads processes an input split using one model replica on one device 4) gather outputs to the same device.</p>
<p>Given the above, if you change the <code>predictor</code> code to the following. You will see it prints multiple devices.</p>
<pre data-code-wrap="python"><code class="lang-python">class predictor(nn.Module):
     ....

    def forward(self, features):
        print(self.first_term(features).device)
        return self.first_term(features) - self.threshold
</code></pre>
<p>However, for the following code:</p>
<pre data-code-wrap="python"><code class="lang-python">    def SomeFunc(self, model, input_):
        print(model.first_term(input_)[0])
</code></pre>
<p>If it is called outside of a forward pass or if the <code>model</code> argument is not a model replica (the <code>self</code> argument in <code>predictor.forward</code> method), then it won’t show different devices.</p>

blade: <aside class="quote no-group quote-modified" data-username="blade" data-post="6" data-topic="87220">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/blade/48/23861_2.png" class="avatar"> blade:</div>
<blockquote>
<aside class="quote no-group" data-username="mrshenli" data-post="5" data-topic="87220">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mrshenli/48/12220_2.png" class="avatar"> mrshenli:</div>
<blockquote>
<p>My suspicion is that the model parameters are so light that the overhead of GIL contention, input scattering, output gathering, and model replication in DataParallel forward pass overshadows the speedup brought by multi-gpu training.</p>
</blockquote>
</aside>
<p>That makes sense. Let me try it on a more sophisticated code and update the answer.</p>
</blockquote>
</aside>
<p>So I tried on another code, with 1 GPU my code ran in 434 sec while with 2 GPUs it took 864 sec. So it shouldn’t be from the price we pay for parallelization. Also, using your line <code>print(self.first_term(features).device)</code> it uses all processors at each step so the code is not running in series by each GPU.</p>

mrshenli: <aside class="quote no-group" data-username="blade" data-post="10" data-topic="87220">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/blade/48/23861_2.png" class="avatar"> blade:</div>
<blockquote>
<p>So I tried on another code, with 1 GPU my code ran in 434 sec while with 2 GPUs it took 864 sec.</p>
</blockquote>
</aside>
<p>Can we profile how much of the 434s are spent in the forward pass when DP is <strong>not</strong> present? And how much of that is spent on GPU? This can be measured using <a href="https://pytorch.org/docs/stable/cuda.html#torch.cuda.Event.elapsed_time" rel="noopener nofollow ugc"><code>elapsed_time</code></a> . See <a href="https://discuss.pytorch.org/t/does-it-ever-make-sense-to-try-model-parallelism-even-if-the-model-fits/86432/14">this discussion</a>.</p>
<p>Note that multi-thread cannot parallelize normal Python ops due to Python GIL, and the parallelism only kicks in when the execution does not require GIL (e.g., CPU/GPU ops that explicitly drops GIL).</p>

blade: <p>Not sure if I’m doing right. Am I doing it right for the sample code below?</p>
<pre><code class="lang-auto">def main():
    model = function(1000)
    print(torch.cuda.device_count())
    if True:
        if torch.cuda.device_count() &gt; 1:
            model.model = MyDataParallel(model.model)

    start = time.monotonic()
    s = torch.cuda.current_stream()
    e_start = torch.cuda.Event(enable_timing=True)
    e_finish = torch.cuda.Event(enable_timing=True)
    s.record_event(e_start)

    model.train()

    torch.cuda.synchronize()
    s.record_event(e_finish)
    e_finish.synchronize()
    end = time.monotonic()

    print('Forward latency is: {} '.format(e_start.elapsed_time(e_finish)))
    print("end - start = ", end - start)

if __name__ == '__main__':
    main()
</code></pre>

mrshenli: <p>This looks correct to me. Does <code>MyDataParallel</code> use <code>DataParallel</code> internally?</p>

blade: <p>Yes, it’s just a wrapper so that I can access the attributes (so I don’t need to add <code>.module</code>). If so, below is the output for my actual code:</p>
<blockquote>
<p>Forward latency is: 437033.9375<br>
end - start =  437.031393879035</p>
</blockquote>
<p>So what do you think is wrong with the parallel implementation that takes double as much time to run with two GPUs?</p>

mrshenli: <aside class="quote no-group" data-username="blade" data-post="14" data-topic="87220">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/blade/48/23861_2.png" class="avatar"> blade:</div>
<blockquote>
<p>So what do you think is wrong with the parallel implementation that takes double as much time to run with two GPUs?</p>
</blockquote>
</aside>
<p>This is surprising to me. I would assume at least the CUDA ops can run in parallel. Could you please share the full code used in this test? We will investigate.</p>

blade: <p>Sure, I can share it privately. Could you please send me your Github id?</p>

mrshenli: <p>Thanks, it’s <code>mrshenli</code>.</p>

blade: <p>Done, let me know if you received it.</p>

mrshenli: <p>Got it and will investigate. Thanks for sharing.</p>

blade: <p>Hello <a class="mention" href="/u/mrshenli">@mrshenli</a>, I was wondering if you have any comments yet as to why this happens? Thanks.</p>