DDP losses (train and val) are much worse than in single GPU training

xedwin23x: <aside class="quote quote-modified" data-post="9" data-topic="121624">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mrshenli/48/12220_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/distributeddataparallel-training-not-efficient/121624/9">DistributedDataParallel training not efficient</a> <a class="badge-category__wrapper " href="/c/distributed/12"><span data-category-id="12" style="--category-badge-color: #0088CC; --category-badge-text-color: #FFFFFF;" data-drop-close="true" class="badge-category "><span class="badge-category__name">distributed</span></span></a>
  </div>
  <blockquote>
    The epochs in a training with 4 GPUs are 4x shorter (because each GPU works with only 1/4 of the dataset) but we need 4x the amount of epochs to get the same result. 


With DataParallel this issue doesn’t occur. 

Putting these two together, looks like the loss function might play a role here. With DDP, the gradient synchronization only occurs during the backward pass after loss computation, which means that each process/GPU independently computes the loss using its local input split. In contr…
  </blockquote>
</aside>
<p>
Basically the same issue as the one described in the above thread, where the results for training and evaluation are much better when using a single GPU than when using multiple GPUs.</p>
<p>The only changes i make when using DDP are initializing the distributed processes, wrapping the model in DDP, and using the DistributedSampler for training, and then using the sampler to set the epoch during training :</p>
<pre><code class="lang-auto">if 'WORLD_SIZE' in os.environ:
        args.local_rank = int(os.environ['LOCAL_RANK'])
        args.distributed = int(os.environ['WORLD_SIZE']) &gt; 1

    if args.distributed:
        args.device = 'cuda:%d' % args.local_rank
        torch.cuda.set_device(args.local_rank)
        torch.distributed.init_process_group(backend='nccl', init_method='env://')
        args.world_size = torch.distributed.get_world_size()
        args.rank = torch.distributed.get_rank()

sampler = data.DistributedSampler(ds, num_replicas=args.world_size, rank=args.rank, shuffle=True, 
    drop_last=True)
data_loader = data.DataLoader(ds, batch_size=args.batch_size, num_workers=args.cpu_workers,
    pin_memory=args.pin_memory, sampler=sampler)

model = DDP(model, device_ids=[args.local_rank], output_device=args.local_rank)

if self.args.distributed:
    self.train_loader.sampler.set_epoch(self.epoch)

if self.args.distributed:
     # this is taken from https://github.com/rwightman/pytorch-image-models/blob/master/timm/utils/distributed.py
     distribute_bn(self.model, self.args.world_size, True)
</code></pre>
<p>Asides from that mostly everything else is the same (I wrapped print statements so that they only print in local_rank == 0, but asides from that it’s the same). Validation is done using SequentialSampler and only done in local_rank == 0.<br>
However in that thread the OP never commented on a solution, and I find myself facing the same issue.</p>
<p>I have tried changing LR/batch size, changing the shuffle and drop_last parameters in the sampler, with or without using the sampler.set_epoch(epoch) method, with or without using the distribute_bn method from timm, different datasets/models, and different number of GPUs (1, 2, and 4, all in a single node using <code>torchrun --nproc_per_node=N train.py</code>). They all have the same problem where the training lags severely behind the single GPU training. Any help or advice would be sincerely appreciated. Thanks in advance.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/3/0309b0c8e04ecdd8780646519e6a881ab26b58fc.png" data-download-href="https://discuss.pytorch.org/uploads/default/0309b0c8e04ecdd8780646519e6a881ab26b58fc" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/0/3/0309b0c8e04ecdd8780646519e6a881ab26b58fc_2_690x413.png" alt="image" data-base62-sha1="qSc35niahTD68GsCj8vePXrskY" width="690" height="413" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/0/3/0309b0c8e04ecdd8780646519e6a881ab26b58fc_2_690x413.png, https://discuss.pytorch.org/uploads/default/optimized/3X/0/3/0309b0c8e04ecdd8780646519e6a881ab26b58fc_2_1035x619.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/0/3/0309b0c8e04ecdd8780646519e6a881ab26b58fc_2_1380x826.png 2x" data-dominant-color="FEFEFE"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1587×950 71.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

agu: <p>May I ask what loss function you are using?</p>

xedwin23x: <p>I’m using CrossEntropyLoss (<a href="https://pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html" class="inline-onebox" rel="noopener nofollow ugc">CrossEntropyLoss — PyTorch 1.11.0 documentation</a>) or the SmoothedCrossEntropy from timm (<a href="https://github.com/rwightman/pytorch-image-models/blob/master/timm/loss/cross_entropy.py" class="inline-onebox" rel="noopener nofollow ugc">pytorch-image-models/cross_entropy.py at master · rwightman/pytorch-image-models · GitHub</a>)</p>

cabezon: <p>I am facing the same issue and also using CrossEntropyLoss. Is there a bug in PyTorch DDP source somewhere?</p>

mrshenli: <p>DDP’s loss is local to each process, and gradients are averaged across all processes, which is different from local training or DataParallel training where the loss is global. In your use case, do you need a global loss to achieve good results?</p>

HrealcodeH: <p>Did you find the reason? How did you solve this problem? <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=12" title=":smiley:" class="emoji" alt=":smiley:" loading="lazy" width="20" height="20"></p>

xedwin23x: <p>In my case the issue was caused by a mistake I made.<br>
I forgot the details but IIRC I was using a learning rate scheduler which updated the LR per step/epoch in the host GPU but not on the others causing issues with the gradients and the whole training.</p>

Woosung: <p>Do you have any updates on this?<br>
I faced same issue too</p>