DistributedDataParallel backward much slower than single GPU

username: Olafyii
cooked: <p>Hi</p>
<p>I’m training a model on multiple GPUs on a single machine, and I found that the backward call is taking much longer time when distributed training, compared to single GPU training.</p>
<p>A script to reproduce:</p>
<pre><code class="lang-auto">import torch.distributed as dist
from argparse import ArgumentParser
import timm
import torch
import os
import time

def main(args):
    if "RANK" in os.environ and "WORLD_SIZE" in os.environ:
        args.rank = int(os.environ["RANK"])
        args.world_size = int(os.environ["WORLD_SIZE"])
        args.gpu = int(os.environ["LOCAL_RANK"])
        args.distributed = True
        rank = args.gpu
        torch.cuda.set_device(args.gpu)
        args.dist_backend = "nccl"
        dist.init_process_group(
            backend=args.dist_backend, init_method=args.dist_url, world_size=args.world_size, rank=args.rank
        )
        dist.barrier()
    else:
        args.distributed = False
        rank = 0

    model = timm.create_model('resnet101').cuda()
    if args.distributed:
        print('distributed')
        model = torch.nn.parallel.DistributedDataParallel(model, device_ids=[int(os.environ["LOCAL_RANK"])])
    else:
        print('not distributed')

    bs = 16
    x = torch.randn(bs, 3, 224, 224).cuda()
    label = torch.randint(0, 1000, (bs,)).cuda()
    durations = []

    for i in range(20):
        out = model(x)
        loss = torch.nn.functional.cross_entropy(out, label)
        t = time.time()
        loss.backward()
        duration = time.time() - t
        print(rank, duration)
        durations.append(duration)
    print(rank, "avg time:", sum(durations)/len(durations))

if __name__ == "__main__":
    parser = ArgumentParser()
    parser.add_argument("--world-size", default=1, type=int)
    parser.add_argument("--dist-url", default="env://")
    args = parser.parse_args()
    main(args)
</code></pre>
<p>If I run</p>
<pre><code class="lang-auto">CUDA_VISIBLE_DEVICES=6,7 torchrun --nproc_per_node=2 main.py
</code></pre>
<p>the output looks like:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/9/d9663271013087d60c995c87aac87d49b313e891.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/d9663271013087d60c995c87aac87d49b313e891" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/d/9/d9663271013087d60c995c87aac87d49b313e891_2_291x500.jpeg" alt="image" data-base62-sha1="v1czaqylzABOg0N670IpOUHOMvL" width="291" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/d/9/d9663271013087d60c995c87aac87d49b313e891_2_291x500.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/d/9/d9663271013087d60c995c87aac87d49b313e891_2_436x750.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/d/9/d9663271013087d60c995c87aac87d49b313e891_2_582x1000.jpeg 2x" data-dominant-color="313131"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">838×1438 111 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
Avg time for backward call is about 0.5s</p>
<p>If I run</p>
<pre><code class="lang-auto">python main.py
</code></pre>
<p>The output is<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/b/8b9e5f696d8e496bb454cb5936e0537739217ae6.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/8b9e5f696d8e496bb454cb5936e0537739217ae6" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/b/8b9e5f696d8e496bb454cb5936e0537739217ae6_2_368x500.jpeg" alt="image" data-base62-sha1="jV7GbeFSa5gVuzgaOJCqjihL4A6" width="368" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/b/8b9e5f696d8e496bb454cb5936e0537739217ae6_2_368x500.jpeg, https://discuss.pytorch.org/uploads/default/original/3X/8/b/8b9e5f696d8e496bb454cb5936e0537739217ae6.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/8/b/8b9e5f696d8e496bb454cb5936e0537739217ae6.jpeg 2x" data-dominant-color="3D3D3D"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">458×622 96.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
Avg time for backward call is about 0.03s</p>
<p>This behavior is similar to what happened to my main project, where if I use a single GPU it takes about 2 days to train one epoch, but if I use 4 GPUs, it would take about 7 days to train one epoch. And the GPU utilization is always 100% in both cases.</p>
<p>What might be the problem for this slow backward call? Is this behavior in this small example actually expected?</p>

username: Yanli_Zhao
cooked: <p>DDP backward will launch a collective call to synchronize the gradients, that will increase communication cost for your training.</p>
<p>Usually you can increase batch size for single GPU to achieve best performance. But if you want to train larger batch size and single GPU memory is not sufficient, you can use multiple GPUs to scale the global batch size. Increase batch size can improve DDP performance as well, as it will increase communication and computation overlapping.</p>