DDP and Dropout behave the same across GPUs

username: LQT
cooked: <p>Hi, correct me if I’m wrong but I found that Dropout behaves similarly (correlated) across different GPUs when using DDP. In other word, cells at same position in tensor in different GPUs get all dropped out or not dropped out.</p>
<p>I believe this (might) makes the training loss reduce slower than when using single GPU training.</p>
<p>Is this a bug? and is there a quick fix to make dropout layers operate independently across GPUs?</p>
<p>Example code</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
import torch.nn.functional as F
import numpy as np
from torch.utils.data import Dataset, DataLoader
from torch.utils.data.distributed import DistributedSampler
from torch.nn.parallel import DistributedDataParallel as DDP
import random
import torch.distributed as dist
import os

def set_seed(seed):
    random.seed(seed)
    np.random.seed(seed)
    torch.manual_seed(seed)
    torch.cuda.manual_seed_all(seed)

class CustomDataset(Dataset):
    def __init__(self, size=10) -&gt; None:
        super().__init__()
        self.size=10
    
    def __getitem__(self, i):
        return torch.Tensor([i]), torch.Tensor([1, i , 2])
    
    def __len__(self):
        return self.size

class CustomModel(nn.Module):
    def __init__(self):
        super().__init__()
        self.linear1 = nn.Linear(1, 3)
        self.dropout = nn.Dropout(p=0.5)

    def forward(self, x):
        return self.dropout(self.linear1(x))


def main():
    local_process_index = int(os.environ.get("LOCAL_RANK", -1))
    dist.init_process_group(backend="nccl",
                            world_size=2, # Use 2 GPUs
                            rank=local_process_index)
    set_seed(66)
    device = torch.device("cuda", local_process_index)
    dataset = CustomDataset(size=10)
    dataloader = DataLoader(dataset, batch_size=5, shuffle=False,
                            sampler=DistributedSampler(dataset))
    model = DDP(CustomModel().to(device), 
                device_ids=[local_process_index],
                output_device=local_process_index)
    model.train()
    for input_, output_ in dataloader:
        input_, output_ = input_.to(device), output_.to(device)
        res = model(input_)
        print(f'{res} : {input_}')
if __name__ == '__main__':
    main()
-------
Result:
tensor([[ 0.0000, -0.5181, -0.2358],
        [ 0.0000,  0.0000,  0.0000],
        [ 0.0000,  0.0000, -8.9689],
        [ 0.2071,  0.0000,  0.0000],
        [ 0.0000, -2.4261,  0.0000]], device='cuda:1',
       grad_fn=&lt;FusedDropoutBackward&gt;)
tensor([[  0.0000,  -6.2421,  -3.5107],
        [  0.0000,   0.0000,   0.0000],
        [  0.0000,   0.0000,  -2.4191],
        [  0.2832,   0.0000,   0.0000],
        [  0.0000, -10.0581,   0.0000]], device='cuda:0',
       grad_fn=&lt;FusedDropoutBackward&gt;)
</code></pre>

username: ptrblck
cooked: <p>I think this would be expected, since you are manually seeding the script, wouldn’t it?</p>

username: LQT
cooked: <p>I see, thanks. Then I guess setting seed with local process <code>seed(local_process_index)</code> can make it still reproducible and the dropout operates independently across process. Would there be any hidden issue with this?</p>

username: ptrblck
cooked: <p>I don’t think there would be any hidden issue with it, as each process would then get its own seed.<br>
Nevertheless, I would recommend to check the results using a small code snippet and make sure the results are expected.</p>

username: LQT
cooked: <p>Is there any pytorch convention about using which random number generator?<br>
e.g Are all pytorch operations by default use torch RNG?, or are there operations use python built-in random RNG ? … And how can I look for which RNG is used? …</p>
<p>Thanks</p>

username: ptrblck
cooked: <p>You can <code>grep -r "import random"</code> in the pytorch source to check the usage in the code base.<br>
Currently a lot of tests are using the Python <code>random</code> package (which should be uninteresting for you), the <code>worker_init_fn</code> seeds the <code>random</code> package for each worker in the <code>DataLoader</code>, and the <code>distributed/elastic/rendezvous</code> methods seem to use it for a random delay.<br>
Besides that the internal methods should use the PyTorch pseudorandom number generator.</p>

username: vanshil_shah
cooked: <p>If I seed each process differently, then dropout would behave differently for each process. So conceptually the tensor that is dropped in one process is picked in another then when we are doing all reduce on gradients there would be no zeroed-out gradient which is different from single GPU training if we use dropout. So how can we set different manual seed if we use a dropout layer across all the processes to make it conceptually work?</p>

username: eqy
cooked: <p>I’m not sure I understand the issue here: why is having no zero’d out gradient after allreduce a problem? Wouldn’t this be possible in a single GPU data-parallel case anyway, as different samples in the same batch can have different parts of their activations drop’d out?</p>
<p>I don’t know what you mean by “conceptually” work, but you could simply set a different seed depending on the rank in a distributed training setup to ensure that dropout is behaving differently across the different ranks.</p>

username: vanshil_shah
cooked: <p>sorry to explain more clearly<br>
assume 2 gpus with different seeds<br>
like what i meant is lets say the input to dropout is of dim=4 with activation being [a0,a1,a2,a3]<br>
in forward pass gpu 1 first two are dropped out<br>
[0,a1,0,a3]<br>
in forward pass gpu 2 the other two are dropped out<br>
[a0,0,a2,0]</p>
<p>now when we do backward pass with all reduce there would not be any gradient which is dropped out<br>
but this would be different from a single gpu(no data parellism) case right because there would always be some activation dropped out so wouldnt the behaviour diverge from here? I am trying to compare the behaviour of dropout with DDP and<br>
with no DDP.</p>

username: eqy
cooked: <p>If you mean batch-size one with no data parallelism, then sure, it is different (but this is typical and expected behavior for training).</p>
<p>If you are trying to force e.g., all GPUs to dropout identically, did you check that</p>
<pre><code class="lang-auto">    torch.manual_seed(seed)
    torch.cuda.manual_seed_all(seed)
</code></pre>
<p>didn’t produce the behavior expected?</p>

username: vanshil_shah
cooked: <p>yes so if set the manual seed then it would have the same behaviour. But with respect to the answer to this comment above.</p><aside class="quote" data-post="3" data-topic="125052">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/lqt/48/30435_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/ddp-and-dropout-behave-the-same-across-gpus/125052/3">DDP and Dropout behave the same across GPUs</a> <a class="badge-category__wrapper " href="/c/distributed/12"><span data-category-id="12" data-drop-close="true" class="badge-category "><span class="badge-category__name">distributed</span></span></a>
  </div>
  <blockquote>
    I see, thanks. Then I guess setting seed with local process seed(local_process_index) can make it still reproducible and the dropout operates independently across process. Would there be any hidden issue with this?
  </blockquote>
</aside>
<p>
if i consider the case which i highlighted then there is a problem of unstable training right, when scaling a model with dropout with multiple gpu’s ?</p>

username: vanshil_shah
cooked: <p>Wouldnt this cause unstable training as during the all reduce operation gradients from a dropout with different masks would be averaged?</p>

username: ptrblck
cooked: <p>Wouldn’t this be the same case for a single model using a batch size &gt; 1?</p>

username: vanshil_shah
cooked: <p>I understand that for a mini batch each datum has a different mask.</p><aside class="onebox stackexchange" data-onebox-src="https://stats.stackexchange.com/questions/335690/understanding-dropout-method-one-mask-per-batch-or-more/541524#541524">
  <header class="source">

      <a href="https://stats.stackexchange.com/questions/335690/understanding-dropout-method-one-mask-per-batch-or-more/541524#541524" target="_blank" rel="noopener nofollow ugc">stats.stackexchange.com</a>
  </header>

  <article class="onebox-body">
      <a href="https://stats.stackexchange.com/users/103153/lerner-zhang" target="_blank" rel="noopener nofollow ugc">
    <img alt="Lerner Zhang" src="https://i.stack.imgur.com/Lr5jY.jpg?s=256&amp;g=1" class="thumbnail onebox-avatar" width="256" height="256">
  </a>

<h4>
  <a href="https://stats.stackexchange.com/questions/335690/understanding-dropout-method-one-mask-per-batch-or-more/541524#541524" target="_blank" rel="noopener nofollow ugc">Understanding dropout method: one mask per batch, or more?</a>
</h4>

<div class="tags">
  <strong>neural-networks, dropout</strong>
</div>

<div class="date">
  
  answered by
  <a href="https://stats.stackexchange.com/users/103153/lerner-zhang" target="_blank" rel="noopener nofollow ugc">
    Lerner Zhang
  </a>
  on <a href="https://stats.stackexchange.com/questions/335690/understanding-dropout-method-one-mask-per-batch-or-more/541524#541524" target="_blank" rel="noopener nofollow ugc">01:20PM - 26 Aug 21 UTC</a>
</div>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
<p>
But the backward passes that happen on each mini-batch are independent in each forward-backward pass and no gradient reduction thats happening across different mini batch</p>
<p>But for multi gpu setting what is of concern to me is that different masks are there across different gpus  and we are averaging the gradient. So it would be like combining gradients from different models at the same time.  ?</p>