DistributedDataParallel with single-process slower than sing-gpu

Jack5358: <p>Hi. I have a machine with multi-GPU.<br>
And I a wrote training code  with <strong>Single-Process</strong> Multi-GPU according to this <a href="https://pytorch.org/docs/stable/nn.html#torch.nn.parallel.DistributedDataParallel" rel="noopener nofollow ugc">docs</a>.</p>
<blockquote>
<p>Single-Process Multi-GPU<br>
In this case, a single process will be spawned on each host/node and each process will operate on all the GPUs of the node where it’s running. To use  <code>DistributedDataParallel</code>  in this way, you can simply construct the model as the following:</p>
</blockquote>
<p>But I found that it slower than just using single gpu. There must be something wrong in my code.</p>
<h2><a name="code-1" class="anchor" href="#code-1"></a>code</h2>
<pre data-code-wrap="python"><code class="lang-python"># file: main.py
torch.distributed.init_process_group(backend="nccl")

# dataset
class RandomDataset(Dataset):
    def __getitem__(self, index):
        return torch.randn(3,255,255),0

    def __len__(self):
        return 100
datasets = RandomDataset()
sampler = DistributedSampler(datasets)
dataloader = DataLoader(datasets,16,sampler=sampler)

# model
model = torch.nn.Sequential(
  torchvision.models.resnet101(False),
  torch.nn.Linear(1000,2)
).cuda()
model = DistributedDataParallel(model)

begin_time = time.time()
# training loop
for i in range(10):
    for x, y in dataloader:
        x = x.cuda()
        y = y.reshape(-1).cuda()
        optimizer.zero_grad()

        output = model(x)
        loss = critertion(output,y)
        loss.backward()
        optimizer.step()
print('Cost:',time.time()-begin_time)
</code></pre>
<h2><a name="launch-with-2" class="anchor" href="#launch-with-2"></a>launch with</h2>
<pre data-code-wrap="bash"><code class="lang-bash">CUDA_VISIBLE_DEVICES=0,1 python -m torch.distributed.launch --nproc_per_node=1 main.py
</code></pre>
<h2><a name="time-cost-3" class="anchor" href="#time-cost-3"></a>Time cost:</h2>
<div class="md-table">
<table>
<thead>
<tr>
<th>DistributedDataParallel with single-process and 2-gpu</th>
<th>Single-gpu</th>
</tr>
</thead>
<tbody>
<tr>
<td>22s</td>
<td>19s</td>
</tr>
</tbody>
</table>
</div><h2><a name="gpu-memory-cost-4" class="anchor" href="#gpu-memory-cost-4"></a>GPU memory cost:</h2>
<div class="md-table">
<table>
<thead>
<tr>
<th>DistributedDataParallel with single-process and 2-gpu</th>
<th>Single-gpu</th>
</tr>
</thead>
<tbody>
<tr>
<td>3101MiB(GPU 0) /  2895MiB(GPU 1)</td>
<td>4207MiB</td>
</tr>
</tbody>
</table>
</div><p>I’ve been debuging and looking docs for hours. I’ll be appreciated that somebody have a look.<br>
thanks.</p>

mrshenli: <p>Hi <a class="mention" href="/u/jack5358">@Jack5358</a></p>
<p>DistributedDataParallel’s single-process-multi-gpu mode is not recommended, because it does parameter replication, input split, output gather, etc. in every iteration, and Python GIL might get in the way. If you just have one machine, with one process per machine, then it will be very similar to DataParallel.</p>
<p>The recommended solution is to use single-process-single-gpu, which means, in your use case with two GPUs, you can spawn two processes, and each process exclusively works on one GPU. This should be faster than the current setup.</p>

gmondaut: <p>Hi <a class="mention" href="/u/mrshenli">@mrshenli</a><br>
Do you have code reference for this recommended solution you are proposing? single-process-single-gpu?<br>
Thanks<br>
gmondaut</p>

pritamdamania87: <p>You can find some examples under the “Multi-Process Single-GPU” section in <a href="https://pytorch.org/docs/stable/nn.html#distributeddataparallel" rel="nofollow noopener">https://pytorch.org/docs/stable/nn.html#distributeddataparallel</a>.</p>

xiefeiwhu: <p>Hello, I meet the same slower question. I use multi nodes and multi gpus, also with spawn. After checking the code, I find the time costs heavily in optimizer.step(). Any solutions?</p>

mrshenli: <p>Hey <a class="mention" href="/u/xiefeiwhu">@xiefeiwhu</a></p>
<p><code>optimizer.step()</code> is not part of DDP forward-backward. Which optimizer are you using? and do you observe the same slowness in local training.</p>
<p>BTW, how did you measure the delay? You might need to use <a href="https://pytorch.org/docs/stable/cuda.html#torch.cuda.Event.elapsed_time" rel="nofollow noopener">CUDA events</a> to get accurate timing measures, as there could be pending ops in the CUDA stream so that <code>time.time()</code> cannot faithfully represent the time cost.</p>

mrshenli: <p>Sorry, I missed this. Yes please checkout <a href="https://pytorch.org/docs/master/notes/ddp.html#example" rel="nofollow noopener">this example</a> that uses <code>device_ids=[rank]</code> to specify which device DDP should use.</p>

xiefeiwhu: <p>I use Adam as the optimizer. After a carefully check with the code, the slowness seems to be the communication time between the nodes as my model is about 180M params, i.e. 760MB. The computation time is faster than the communication time. Then, I expand the nodes from 2 to 4 and the communication time is bigger but not 2 times which accelerates the training procedure to some extent.</p>

pritamdamania87: <p>You can try ProcessGroupRoundRobin and see if it helps in reducing the communication time. Example usage: <a href="https://github.com/pytorch/pytorch/blob/master/test/distributed/test_c10d.py#L1511" rel="nofollow noopener">https://github.com/pytorch/pytorch/blob/master/test/distributed/test_c10d.py#L1511</a>. Note that this API is not officially supported yet.</p>