Distributed data parallel slower than data parallel?

kirk86: <p>I’ve come up across this strange thing where in a simple setting training vgg16 for 10 epochs is fater with data parallel than distributed data parallel.</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/b/a/bac238616f8c181e2505e2a5ba76be239b409230.png" alt="image" data-base62-sha1="qE8VjykBLyAv4GoyH1kXEqJTXZC" width="585" height="240"><br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/1/b/1b829949165ba5b366bb351cf935f3d65b393171.png" alt="image" data-base62-sha1="3VmHibzjfgFGTdX23GoMM6IfH57" width="287" height="25"></p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/3/9/3930cb6b3a1427b0be3c99d7d052f43c0b691bc3.png" alt="image" data-base62-sha1="89VKYSRYXux2NFIJoARMyLYYuwb" width="517" height="233"><br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/6/a/6a5e9ab98bb58d008f0314bc783210c69e9b9912.png" alt="image" data-base62-sha1="faZiEs6gJcibPDshAGWNUmw1LDI" width="299" height="26"></p>
<p>MWE:</p>
<pre><code class="lang-auto">device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
model = vgg16(...)
if dist and torch.distributed.is_available():
   torch.distributed.init_process_group(backend='nccl', init_method='env://')
   sampler = torch.utils.data.distributed.DistributedSampler
else:
   sampler = torch.utils.data.SubsetRandomSampler

db = Datasets(workers=4, pin_memory=True, sampler=sampler)
optimizer = Adam()
if torch.distributed.is_initialized():
   model.to(device)
   model = torch.nn.parallel.DistributedDataParallel(model)
elif torch.cuda.device_count() &gt; 1:
     model = torch.nn.parallel.DataParallel(model)
     model.to(device)
else:
     model.to(device)

training loop.....
</code></pre>
<p>Launching distributed training with</p>
<pre><code class="lang-auto">python -m torch.distributed.launch main.py
</code></pre>
<p>The <a href="https://pytorch.org/docs/stable/nn.html#torch.nn.parallel.DistributedDataParallel" rel="noopener nofollow ugc">docs</a> identify 2 cases.</p>
<ol>
<li>Single-Process Multi-GPU</li>
<li>Multi-Process Single-GPU</li>
</ol>
<p>I believe that my code fall under 1.</p>
<p>But to achieve 2. which says is faster it describes the following changes that should be made</p>
<pre><code class="lang-auto">torch.distributed.init_process_group(backend='nccl', world_size=4, init_method='...')
model = DistributedDataParallel(model, device_ids=[i], output_device=i)
</code></pre>
<p>My understanding is that I should change the following in my code<br>
<code>torch.distributed.init_process_group(backend='nccl', init_method='env://')</code><br>
into<br>
<code>torch.distributed.init_process_group(backend='nccl', world_size=4, init_method='...')</code></p>
<p>So  far so good.</p>
<p>But, I’m lost in this line<br>
<code>model = DistributedDataParallel(model, device_ids=[i], output_device=i)</code></p>
<p>In the docs it says that <code>i</code> corresponds to a particular gpu.<br>
Suppose we have 2 gpus for the sake of example and I want to run the above MWE (moved into file main.py).</p>
<p>Usually I would run <code>main.py</code> using something like <code>python -m torch.distributed.launch main.py --i=1</code>.<br>
Now my confusion arises from <code>i</code> which only specifies one of the 2 available gpus, how’s that distributed training? Or should I specify <code>--i=[0, 1]</code>?</p>
<p>Any clarifications or pointers to mistakes or misunderstandings that I’ve made are highly appreciated it.</p>
<p>Thanks.</p>

kirk86: <p>I made a mistake above when launching distributed training with <code>torch.distributed.launch</code> I should have specified <code>--nproc_pre_node</code>.</p>
<p>So I did that running <code>torch.distributed.launch --nproc_per_node=2</code> where 2 corresponds to num of process for each gpu in the system, and the <code>launch</code> help manual recommends that but I got even worse results not faster.</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/b/1/b13f3e6dd7fd84f42d01a40208edab368ea5f72f.png" alt="image" data-base62-sha1="pi00IFURXvLHNhjAhbQqeDfFCGz" width="572" height="260"><br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/f/9/f90d22105065bcdba9d311de4de7561f6282cfb8.png" alt="image" data-base62-sha1="zxd1I4WcVf1Cn6fhO3vRJQDf3yU" width="306" height="24"></p>
<p>Also this message<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/8/d82a72af22a688c1e8d963a776ac400e053b596f.png" data-download-href="https://discuss.pytorch.org/uploads/default/d82a72af22a688c1e8d963a776ac400e053b596f" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/d/8/d82a72af22a688c1e8d963a776ac400e053b596f_2_690x25.png" alt="image" data-base62-sha1="uQi4PQNKWBbEbSVfujhdqLaVJIr" width="690" height="25" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/d/8/d82a72af22a688c1e8d963a776ac400e053b596f_2_690x25.png, https://discuss.pytorch.org/uploads/default/optimized/3X/d/8/d82a72af22a688c1e8d963a776ac400e053b596f_2_1035x37.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/d/8/d82a72af22a688c1e8d963a776ac400e053b596f_2_1380x50.png 2x" data-dominant-color="161616"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1894×70 23.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Unless I’m missing something fundamental here why is it said that distributed parallel is faster than data parallel?</p>

ptrblck: <p>We recommend the usage of DDP with a single process per GPU.<br>
The <a href="https://github.com/pytorch/pytorch/blob/master/torch/distributed/launch.py">launch.py</a> script has some example usages for it.<br>
E.g. you could use <a href="https://github.com/pytorch/pytorch/blob/2b79bab029ec387618f6b9724a4027b81fbda33c/torch/distributed/launch.py#L71-L96"><code>--local_rank</code></a> in your script to specify the GPU.</p>

kirk86: <p>Thanks for the reply, please correct me if I’m wrong but the purpose of distributed training is to be used for:</p>
<ol>
<li>Multi-node &amp; multi-gpu training</li>
<li>Single-node &amp; multi-gpu training → my use case scenario</li>
</ol>
<aside class="quote no-group" data-username="ptrblck" data-post="3" data-topic="72052">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>The <a href="https://github.com/pytorch/pytorch/blob/master/torch/distributed/launch.py" rel="noopener nofollow ugc">launch.py</a> script has some example usages for it.</p>
</blockquote>
</aside>
<p>I’ve seen them and according to the recommendation shown below I launch my script for my scenario as explained above.<br>
<strong>How to use this module:</strong></p>
<ol>
<li>Single-Node multi-process distributed training<br>
::
<blockquote>
<blockquote>
<blockquote>
<p>python -m torch.distributed.launch --nproc_per_node=NUM_GPUS_YOU_HAVE<br>
YOUR_TRAINING_SCRIPT.py (–arg1 --arg2 --arg3 and all other<br>
arguments of your training script)</p>
</blockquote>
</blockquote>
</blockquote>
</li>
</ol>
<aside class="quote no-group" data-username="ptrblck" data-post="3" data-topic="72052">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>E.g. you could use <a href="https://github.com/pytorch/pytorch/blob/2b79bab029ec387618f6b9724a4027b81fbda33c/torch/distributed/launch.py#L71-L96" rel="noopener nofollow ugc"> <code>--local_rank</code> </a> in your script to specify the GPU.</p>
</blockquote>
</aside>
<p>This is what I don’t understand why do I need to specify gpu for distrubed training on single-node multi-gpu?</p>
<p>Obviously I want to use all gpus do I still need <code>--local_rank</code> I thought it was used to specify gpus in multi-node scenario where nodes might have different number of gpus.</p>
<p>When launching <code>launch.py</code> with <code>--nproc_per_node=2</code> where 2 is the num of gpus it returns <code>--local_rank=0</code> shouldn’t that be 2 instead for each gpu?</p>
<p>Even if I parse the <code>--local-rank</code> argument in my script and set the device to <code>--local_rank</code> that would still be using only one of the available gpus. I need to use all, that where there’s confusion  in my understanding. How do we do that?</p>