Use Distributed Data Parallel correctly

username: bing
cooked: <p>I am trying to run distributed data-parallel on a single node with 3 GPUs to maximise GPU utility which is currently very low.<br>
After following multiple tutorials, the following is my code(I have tried to add a minimal example, let me know if anything is not clear and I’ll add more) but it is exiting without doing anything on running -<br>
<span class="hashtag">#:</span> before any statement represents minimal code I have provided</p>
<pre><code class="lang-auto">#All the required imports
#setting of environment variables

def train(world_size, args):
        #setting of all the seeds for deterministic behaviour

         rank = args.nr * args.gpus + args.gpu
         dist.init_process_group(backend='nccl', init_method='env://', world_size=args.world_size, rank=rank)
         gpu = args.world_size


         #data_transform = 
        # root_dir = 
         #image_dataset = dataset.ImageFolder(train_dir)
         
         train_loader = torch.utils.data.DataLoader(image_dataset['train'], batch_size=batch_size,shuffle = True,num_workers = 0,sampler = train_sampler)
         valid_loader = torch.utils.data.DataLoader(image_dataset['valid'], batch_size=batch_size,shuffle = True,num_workers = 0,sampler= train_sampler)
         model_transfer = models.resnet18(pretrained=True)
         criterion = nn.CrossEntropyLoss().cuda(gpu)
         optimizer = optim.SGD(model_transfer.fc.parameters(),lr = 0.001)
         ddp_model_transfer = DDP(model_transfer, device_ids = [rank])


          for epoch in range(1, args.epochs+1):
               for batch_idx, (data, target) in enumerate(data_transfer['train']):
            if use_cuda:
                data, target = data.cuda(), target.cuda()
            optimizer.zero_grad()
            output = ddp_model_transfer(data)
            loss = criterion(output,target)
            loss.backward()
            optimizer.step()
            
            train_loss = train_loss + ((1 / (batch_idx + 1)) * (loss.data - train_loss))
##Vadlidation
        model.eval()
        for batch_idx, (data, target) in enumerate(data_transfer['valid']):
            # move to GPU
            if use_cuda:
                data, target = data.cuda(), target.cuda()
            ## update the average validation loss
            output = ddp_model_transfer(data).to(rank)
            loss = criterion(output,target)
            valid_loss +=  valid_loss + ((1 / (batch_idx + 1)) * (loss.data - valid_loss))
            pred = output.data.max(1, keepdim=True)[1]
            correct += np.sum(np.squeeze(pred.eq(target.data.view_as(pred))).cpu().numpy())
            total += data.size(0)

            if valid_loss &lt;= valid_loss_min:
                  print('Validation loss decreased ({:.6f} --&gt; {:.6f}).  Saving model ...'.format(
                  valid_loss_min,valid_loss))
                  torch.save(ddp_model_transfer.state_dict(), 'case_1_model.pt')
                  valid_loss_min = valid_loss

def main():
    parser = argparse.ArgumentParser()
    parser.add_argument('--epochs', default=2, type=int,metavar = 'N', help='number of total epochs to run')
    parser.add_argument('-g', '--gpus', default=1, type=int,help='number of gpus per node')
    parser.add_argument('-nr', '--nr', default=0, type=int,help='ranking within the nodes')
    args = parser.parse_args()
    world_size = 3
    mp.spawn(train,args = (world_size,args),nprocs = world_size,join=True)

if __name__=="main":
    main()





</code></pre>

username: mrshenli
cooked: <p>Hey <a class="mention" href="/u/bing">@bing</a></p>
<p>I see a few errors in the code snippet:</p>
<ol>
<li>It is <code>if __name__=="__main__"</code> instead of <code>if __name__=="main"</code>. I think this is also the reason why you see no output, as nothing is executed.</li>
<li>
<code>mp.spawn</code> will automatically provide the rank as the first argument to the target function, so the <code>train</code> function needs to take one additional argument before <code>world_size</code>. Please check the API doc here: <a href="https://pytorch.org/docs/stable/multiprocessing.html#torch.multiprocessing.spawn" rel="nofollow noopener">https://pytorch.org/docs/stable/multiprocessing.html#torch.multiprocessing.spawn</a>
</li>
<li>Don’t need to move the loss function to gpu as it does not contain any parameters.<pre><code class="lang-auto">gpu = args.world_size
...
criterion = nn.CrossEntropyLoss().cuda(gpu)
</code></pre>
</li>
<li>You need to use <code>rank</code> instead of the current device, otherwise all processes will use the same device if you didn’t set <code>CUDA_VISIBLE_DEVICE</code> or run <code>torch.cuda.set_device(rank)</code><pre><code class="lang-auto">if use_cuda:
             data, target = data.cuda(), target.cuda()
</code></pre>
</li>
</ol>
<p>I tried to fix all above and run your code, but then realized I don’t have the dataset. It would be helpful to have a minimum repro. Or is it hard to create a min repro, I would suggest to expand this <a href="https://pytorch.org/docs/master/notes/ddp.html#example" rel="nofollow noopener">example</a> to create your own application.</p>

username: bing
cooked: <p>Hello Mr. Shen,<br>
Thanks a lot for replying.<br>
I added all the suggested modifications and now, I am getting -<br>
<code>ValueError: Error initializing torch.distributed using env:// rendezvous: environment variable MASTER_ADDR expected, but not set </code><br>
Full traceback -</p>
<p>Traceback (most recent call last):</p>
<pre><code class="lang-auto"> File "Distributed_code1.py", line 212, in &lt;module&gt;
    main()
  File "Distributed_code1.py", line 209, in main
    mp.spawn(train,args = (world_size,),nprocs = world_size,join=True)
  File "/opt/conda/lib/python3.6/site-packages/torch/multiprocessing/spawn.py", line 200, in spawn
    return start_processes(fn, args, nprocs, join, daemon, start_method='spawn')
  File "/opt/conda/lib/python3.6/site-packages/torch/multiprocessing/spawn.py", line 158, in start_processes
    while not context.join():
  File "/opt/conda/lib/python3.6/site-packages/torch/multiprocessing/spawn.py", line 119, in join
    raise Exception(msg)
Exception: 

-- Process 0 terminated with the following error:
Traceback (most recent call last):
  File "/opt/conda/lib/python3.6/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap
    fn(i, *args)
  File "/workdir/script/Distributed_code1.py", line 31, in train
    dist.init_process_group(backend='nccl',init_method = 'env://', world_size=world_size, rank=rank)
  File "/opt/conda/lib/python3.6/site-packages/torch/distributed/distributed_c10d.py", line 393, in init_process_group
    store, rank, world_size = next(rendezvous_iterator)
  File "/opt/conda/lib/python3.6/site-packages/torch/distributed/rendezvous.py", line 159, in _env_rendezvous_handler
    raise _env_error("MASTER_ADDR")
ValueError: Error initializing torch.distributed using env:// rendezvous: environment variable MASTER_ADDR expected, but not set
</code></pre>
<p>I also tried running the code <code>python -m torch.distributed.launch mycode.py</code> as suggested <a href="https://stackoverflow.com/questions/49947814/python-threading-error-must-be-an-iterable-not-int" rel="nofollow noopener">here</a> but it is not working.</p>

username: mrshenli
cooked: <p>Hey Mr. Bing,</p>
<p>If you use <code>env://</code> as the <code>init_method</code>, it would look for <code>MASTER_ADDR</code> and <code>MASTER_PORT</code> env vars. You can use sth like <code>init_method='tcp://ip:port'</code> or directly set the env var. E.g.:</p>
<pre><code class="lang-python">    os.environ['MASTER_ADDR'] = 'localhost'
    os.environ['MASTER_PORT'] = '29500'
</code></pre>

username: bing
cooked: <p>Thank You.</p>
<p>I just added the suggested modification. Now, I am getting -<br>
<code>RuntimeError: NCCL error in: ../torch/lib/c10d/ProcessGroupNCCL.cpp:514, invalid usage, NCCL version 2.6.3 </code><br>
Full traceback -</p>
<pre><code class="lang-auto">
Traceback (most recent call last):
  File "Distributed_code1.py", line 217, in &lt;module&gt;
    main()
  File "Distributed_code1.py", line 214, in main
    mp.spawn(train,args = (world_size,),nprocs = world_size,join=True)
  File "/opt/conda/lib/python3.6/site-packages/torch/multiprocessing/spawn.py", line 200, in spawn
    return start_processes(fn, args, nprocs, join, daemon, start_method='spawn')
  File "/opt/conda/lib/python3.6/site-packages/torch/multiprocessing/spawn.py", line 158, in start_processes
    while not context.join():
  File "/opt/conda/lib/python3.6/site-packages/torch/multiprocessing/spawn.py", line 119, in join
    raise Exception(msg)
Exception: 

-- Process 1 terminated with the following error:
Traceback (most recent call last):
  File "/opt/conda/lib/python3.6/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap
    fn(i, *args)
  File "/workdir/script/Distributed_code1.py", line 110, in train
    ddp_model_transfer = DDP(model_transfer, device_ids = [device])
  File "/opt/conda/lib/python3.6/site-packages/torch/nn/parallel/distributed.py", line 303, in __init__
    self.broadcast_bucket_size)
  File "/opt/conda/lib/python3.6/site-packages/torch/nn/parallel/distributed.py", line 485, in _distributed_broadcast_coalesced
    dist._broadcast_coalesced(self.process_group, tensors, buffer_size)
RuntimeError: NCCL error in: ../torch/lib/c10d/ProcessGroupNCCL.cpp:514, invalid usage, NCCL version 2.6.3
</code></pre>
<p>I also did a change in how the model is loading in the following way -</p>
<pre><code>device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
model_transfer = model_transfer.to(device)
ddp_model_transfer = DDP(model_transfer, device_ids = [device])</code></pre>

username: mrshenli
cooked: <aside class="quote no-group quote-modified" data-username="bing" data-post="5" data-topic="82500">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/b/839c29/48.png" class="avatar"> bing:</div>
<blockquote>
<p>RuntimeError: NCCL error in: …/torch/lib/c10d/ProcessGroupNCCL.cpp:514, invalid usage, NCCL version 2.6.3</p>
</blockquote>
</aside>
<p>Are you using system NCCL? As of today, the master is still on NCCL 2.4.8 IIUC. Let’s first get DDP run correctly and then deal with system NCCL. Can you try changing <code>backend='nccl'</code> to <code>backend='gloo'</code>?</p>

username: bing
cooked: <blockquote>
<p>Are you using system NCCL?</p>
</blockquote>
<p>How to figure that out?</p>
<p>I changed to “gloo” and now I am getting -</p>
<pre><code class="lang-auto">AssertionError: DistributedDataParallel device_ids and output_device arguments only work with single-device CUDA modules, but got device_ids [device(type='cuda', index=0)], output_device None, and module parameters {device(type='cuda', index=0), device(type='cpu')}.
</code></pre>

username: mrshenli
cooked: <aside class="quote no-group" data-username="bing" data-post="7" data-topic="82500">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/b/839c29/48.png" class="avatar"> bing:</div>
<blockquote>
<pre><code class="lang-auto">AssertionError: DistributedDataParallel device_ids and output_device arguments only work with single-device CUDA modules, but got device_ids [device(type='cuda', index=0)], output_device None, and module parameters {device(type='cuda', index=0), device(type='cpu')}.
</code></pre>
</blockquote>
</aside>
<p>The error message means the model you passed to DDP ctor lives on both CPU and GPU. Is this intentional?</p>
<blockquote>
<p>module parameters {device(type=‘cuda’, index=0), device(type=‘cpu’)}</p>
</blockquote>

username: bing
cooked: <p>No,<br>
Below is my model loading process -</p>
<pre><code class="lang-auto">    model_transfer = models.resnet18(pretrained=True)
    device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
    for name,param in model_transfer.named_parameters():
        if ("bn" not in name):
            param.requires_grad = False
    print(model_transfer.fc.in_features)
    n_inputs = model_transfer.fc.in_features
    model_transfer.fc = nn.Linear(n_inputs,3)
    ddp_model_transfer = DDP(model_transfer, device_ids = [device])
</code></pre>

username: mrshenli
cooked: <p>If you replace</p>
<pre><code class="lang-python">ddp_model_transfer = DDP(model_transfer, device_ids = [device])
</code></pre>
<p>with</p>
<pre><code class="lang-python">ddp_model_transfer = DDP(model_transfer.to(device), device_ids = [device])
</code></pre>
<p>then, this error would disappear I think.</p>
<p>Honestly, I would recommend starting from a simpler version and gradually adding complexities to it. We could try:</p>
<ol>
<li>See if <a href="https://pytorch.org/docs/master/notes/ddp.html#example" rel="nofollow noopener"><code>this example</code></a> works in your env (after setting MASTER_ADDR and MASTER_PORT).</li>
<li>Replace the model in that example with a resnet 18 pretrained model</li>
<li>Adding data loaders</li>
</ol>

username: bing
cooked: <blockquote>
<p>Honestly, I would recommend starting from a simpler version and gradually adding complexities to it.</p>
</blockquote>
<p>Okay, I just did the above using the example you provided. It is working now, even when I use <code>backend='nccl'</code> too.<br>
But here is an issue,  how to merge the metrics provided by different GPUs together. Now, I am getting 3 outputs like below -</p>
<pre><code class="lang-auto">

Epoch: 1 	Training Loss: 1.044399 	Validation Loss: 21.517847

Test Accuracy: 58% (106/180)
Validation loss decreased (inf --&gt; 21.517847).  Saving model ...
Epoch: 1 	Training Loss: 1.036141 	Validation Loss: 23.852943

Test Accuracy: 47% (86/180)
Epoch: 1 	Training Loss: 1.040907 	Validation Loss: 21.792797

Test Accuracy: 56% (101/180)
Validation loss decreased (inf --&gt; 23.852943).  Saving model ...
Validation loss decreased (inf --&gt; 21.792797).  Saving model ...
Epoch: 2 	Training Loss: 0.971241 	Validation Loss: 19.095814

Test Accuracy: 62% (113/180)
Validation loss decreased (inf --&gt; 19.095814).  Saving model ...

wandb: Waiting for W&amp;B process to finish, PID 30011
Epoch: 2 	Training Loss: 0.965157 	Validation Loss: 20.114439

Test Accuracy: 55% (99/180)
Epoch: 2 	Training Loss: 0.966274 	Validation Loss: 21.474331

Test Accuracy: 56% (101/180)
Validation loss decreased (inf --&gt; 20.114439).  Saving model ...
Validation loss decreased (inf --&gt; 21.474331).  Saving model ...
</code></pre>

username: mrshenli
cooked: <p>Great!</p>
<p>For merging logs, I see a few options. There might be more.</p>
<ol>
<li>Let one rank do the printing (e.g., <code>if rank == 0: ...</code>)</li>
<li>If you need logs from all ranks, you can redirect then to write to different files.</li>
<li>Convert strings to tensors, and then use <code>all_gather</code>/<code>gather</code> to collect them (<a href="https://pytorch.org/docs/stable/distributed.html#torch.distributed.gather" rel="nofollow noopener">https://pytorch.org/docs/stable/distributed.html#torch.distributed.gather</a>). This might be a little more complex than it sounds. <a href="https://github.com/pytorch/pytorch/blob/1294e55c15b4d6a0e643c8414497223191b3df66/torch/distributed/rpc.py#L27-L51" rel="nofollow noopener">This</a> can serve as an example.</li>
</ol>

username: rwightman
cooked: <p><a class="mention" href="/u/bing">@bing</a> it might be a good idea to try a framework. Lightning, Ignite, or Catalyst should take care of most of the boilerplate and get the DDP details right (not always easy).</p>
<p>Not following my own advice, I have iterated my own train script from the original PyTorch examples over various projects and experiments. It’s a bit sprawling, but fast and easy to hack once you are familiar. It supports both torch native and Nvidia Apex DDP. I haven’t done any multi-node work with it, but lots of multi-gpu single node. I know quite a few researchers and engineers have used it for training large nets on 4-8 GPU V100 systems. A version of it tailored for ImageNet w/ RandAugment, Mixup, Random Erasing here: <a href="https://github.com/rwightman/pytorch-image-models/blob/master/train.py" rel="nofollow noopener">https://github.com/rwightman/pytorch-image-models/blob/master/train.py</a></p>
<p>An example of hacking it for object detection training with MSCOCO: <a href="https://github.com/rwightman/efficientdet-pytorch/blob/master/train.py" rel="nofollow noopener">https://github.com/rwightman/efficientdet-pytorch/blob/master/train.py</a></p>
<p>Those scripts have no problem maxing out GPU utilization, so long as the drives for the dataset are fast enough, the data/network is large enough, and the CPU has enough cores for your augmentation/data decoding. I use a shell script for launching the distributed training mode of the train script to keep it simple.</p>

username: bing
cooked: <p>Thank You. I am working on it and will get back to you, if it doesn’t work out.</p>

username: bing
cooked: <p>Thank You, I’ll try to implement and will get back , if I face any problem.</p>

username: karan_purohit
cooked: <p><strong>Not getting whats wrong here</strong></p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "train.py", line 14, in &lt;module&gt;
    cli_main()
  File "/data/fairseq/fairseq_cli/train.py", line 343, in cli_main
    distributed_utils.call_main(args, main)
  File "/data/fairseq/fairseq/distributed_utils.py", line 172, in call_main
    args.distributed_world_size,
  File "/data/wav/lib/python3.6/site-packages/torch/multiprocessing/spawn.py", line 200, in spawn
    return start_processes(fn, args, nprocs, join, daemon, start_method='spawn')
  File "/data/wav/lib/python3.6/site-packages/torch/multiprocessing/spawn.py", line 158, in start_processes
    while not context.join():
  File "/data/wav/lib/python3.6/site-packages/torch/multiprocessing/spawn.py", line 119, in join
    raise Exception(msg)
Exception:

-- Process 2 terminated with the following error:
Traceback (most recent call last):
  File "/data/wav/lib/python3.6/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap
    fn(i, *args)
  File "/data/fairseq/fairseq/distributed_utils.py", line 154, in distributed_main
    main(args, **kwargs)
  File "/data/fairseq/fairseq_cli/train.py", line 65, in main
    task.load_dataset(valid_sub_split, combine=False, epoch=1)
  File "/data/fairseq/fairseq/tasks/audio_pretraining.py", line 110, in load_dataset
    for line in f:
  File "/data/wav/lib/python3.6/encodings/ascii.py", line 26, in decode
    return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 4924: ordinal not in range(128)
</code></pre>
<p><strong>train.tsv</strong> :</p>
<pre><code class="lang-auto">
/data/fairseq/IE/
1143-GR/M1IE041143GENC0005.wav	21440
1143-GR/M1IE041143DGSQ1143.wav	69120
1143-GR/M1IE041143GENC0004.wav	17920
</code></pre>
<p>Any thoughts?</p>