DDP with multiple models

hnt4499: <p>Hi,</p>
<p>I’m trying to train two models A and B on 4 GPUs, each being trained on 2 GPUs (and thus DDP is needed). The two models are independent, but they need to exchange some information during training (not gradients), hence I would like to execute a single command <code>torch.distributed.launch</code> so that the workers can communicate to each other. What is the correct way to do it?</p>
<p>Thanks in advance!</p>

pritamdamania87: <p>You can launch 4 processes (1 per GPU) and initialize a process group of world_size 4. You can use this process group to exchange data between A and B.</p>
<p>Then for each model, create a new subprocess group using <a href="https://pytorch.org/docs/stable/distributed.html#torch.distributed.new_group" class="inline-onebox" rel="noopener nofollow ugc">Distributed communication package - torch.distributed — PyTorch 1.9.0 documentation</a> on GPUs 0,1 and GPUs 2,3 respectively and pass that process group to DDP.</p>

mlearn: <p>Hi! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"> I am currently trying the same thing for a GAN.</p>
<p><a class="mention" href="/u/pritamdamania87">@pritamdamania87</a>  Can you maybe elaborate how I would pass that new process group to DDP after creating it?</p>
<p>I am currently trying it this way:</p>
<pre><code>if "WORLD_SIZE" in os.environ:
    world_size = int(os.environ["WORLD_SIZE"])
ngpus_per_node = torch.cuda.device_count()

if world_size &gt; 1:
    if 'SLURM_PROCID' in os.environ:  # for slurm scheduler
        rank = int(os.environ['SLURM_PROCID'])
        gpu = rank % torch.cuda.device_count()

          dist.init_process_group(backend='nccl', init_method='env://',
                                          world_size=world_size, rank=rank)
          
          model.cuda(gpu)#(gpu)
          group_model = torch.distributed.new_group(ranks=[0,1])
          model = torch.nn.parallel.DistributedDataParallel(model, device_ids=[0,1])
          model2.cuda(gpu)
          group_model2 = torch.distributed.new_group(ranks=[2,3])
          D = torch.nn.parallel.DistributedDataParallel(model2, device_ids=[2,3])
</code></pre>
<p>But I am running in various kinds of errors.</p>

pritamdamania87: <p>You need to make these changes:</p>
<pre><code class="lang-auto">model = torch.nn.parallel.DistributedDataParallel(model, device_ids=[0,1], process_group=group_model)
D = torch.nn.parallel.DistributedDataParallel(model2, device_ids=[2,3], process_group=group_model2)
</code></pre>

mlearn: <p>Thank you so much! That at least helped me at some point but I am having now more troubles I am running into. So I try to run my code on an HPC Cluster with slurm. My whole code looks:</p>
<pre><code>import os
import torch
import torch.nn as nn
import torch.distributed as dist


class Generator(nn.Module):

def __init__(self):
    super(Generator, self).__init__()
    self.main = nn.Sequential(
        nn.ConvTranspose2d(512, 256, (4, 4), stride=(2, 2), padding=(0, 0)),
        nn.BatchNorm2d(256),
        nn.LeakyReLU(),
        nn.ConvTranspose2d(256, 128, (5, 5), (1, 1)),
        nn.BatchNorm2d(128),
        nn.LeakyReLU(),
        nn.ConvTranspose2d(128, 64, (2, 2), (2, 2)),
        nn.BatchNorm2d(64),
        nn.LeakyReLU(),
        nn.ConvTranspose2d(64, 32, (2, 2), stride=(2, 2)),
        nn.BatchNorm2d(32),
        nn.LeakyReLU(),
        nn.ConvTranspose2d(32, 2, (1, 1), stride=(1, 1)),
        nn.BatchNorm2d(2),
        nn.LeakyReLU(),
        nn.Sigmoid()
    )

def forward(self, x):
    return self.main(x)

class Discriminator(nn.Module):
def __init__(self):
    super(Discriminator, self).__init__()
    self.main = nn.Sequential(
        nn.Conv2d(2, 32, (1,1), stride=(1,1), padding=0, padding_mode='circular'),
        nn.LayerNorm(32),
        nn.LeakyReLU(0.2),
        nn.Dropout(0.5),
        nn.Conv2d(32, 64, (5,5), stride=(2,2), padding=2, padding_mode='circular'),
        nn.LayerNorm(16),
        nn.LeakyReLU(0.2),
        nn.Conv2d(64, 128, (5,5), stride=(2,2), padding=2, padding_mode='circular'),
        nn.LayerNorm(8),
        nn.LeakyReLU(0.2),
        nn.Conv2d(128, 256, (5,5), stride=(2,2), padding=2, padding_mode='circular'),
        nn.LayerNorm(4),
        nn.LeakyReLU(0.2),
        nn.Conv2d(256, 1, (4,4), stride=(1,1), padding=0, padding_mode='circular')
    )

def forward(self, x):
    return self.main(x)



if __name__ == '__main__':
# DDP setting
if "WORLD_SIZE" in os.environ:
    world_size = int(os.environ["WORLD_SIZE"])
ngpus_per_node = torch.cuda.device_count()

if world_size &gt; 1:
    if 'SLURM_PROCID' in os.environ:  # for slurm scheduler
        rank = int(os.environ['SLURM_PROCID'])
        gpu = rank % torch.cuda.device_count()
        print("gpu", gpu, "rank", rank, "ngpus_per_node", ngpus_per_node)
    dist.init_process_group(backend='nccl', init_method='env://',
                            world_size=world_size, rank=rank)
    group_g = torch.distributed.new_group(ranks=[0, 1])
    group_d = torch.distributed.new_group(ranks=[2, 3])

    if gpu is not None:
        torch.cuda.set_device(gpu)
        with torch.cuda.device(0):
            G = Generator().cuda()
            D = Discriminator().cuda()
        G = torch.nn.parallel.DistributedDataParallel(G, device_ids=[0,1], process_group=group_g,broadcast_buffers=True)
        D = torch.nn.parallel.DistributedDataParallel(D, device_ids=[2,3],process_group=group_d,broadcast_buffers=True)</code></pre>

mlearn: <p>and the error I get is this:</p>
<p>Traceback (most recent call last):<br>
File “mini.py”, line 85, in <br>
G = torch.nn.parallel.DistributedDataParallel(G, device_ids=[0,1], process_group=group_g,broadcast_buffers=True)<br>
File “/mpcdf/soft/SLE_15/packages/skylake/pytorch/gpu-cuda-11.2/anaconda_3_2020.02-2020.02/1.8.1/lib/python3.7/site-packages/torch/nn/parallel/distributed.py”, line 446, in <strong>init</strong><br>
self._sync_params_and_buffers(authoritative_rank=0)<br>
File “/mpcdf/soft/SLE_15/packages/skylake/pytorch/gpu-cuda-11.2/anaconda_3_2020.02-2020.02/1.8.1/lib/python3.7/site-packages/torch/nn/parallel/distributed.py”, line 460, in _sync_params_and_buffers<br>
authoritative_rank)<br>
File “/mpcdf/soft/SLE_15/packages/skylake/pytorch/gpu-cuda-11.2/anaconda_3_2020.02-2020.02/1.8.1/lib/python3.7/site-packages/torch/nn/parallel/distributed.py”, line 1156, in _distributed_broadcast_coalesced<br>
self.process_group, tensors, buffer_size, authoritative_rank<br>
TypeError: _broadcast_coalesced(): incompatible function arguments. The following argument types are supported:<br>
1. (process_group: torch._C._distributed_c10d.ProcessGroup, tensors: List[at::Tensor], buffer_size: int, src: int = 0) → None</p>
<p>Invoked with: &lt;object object at 0x14c74759bb00&gt;, [tensor([[[[-4.2067e-03, -7.6654e-03,  1.2881e-02, -1.5023e-02],<br>
[ 6.4325e-03,  1.0096e-02,  1.0314e-02,  4.2596e-03],<br>
…<br>
[[ 0.1669]]]], device=‘cuda:0’), tensor([ 0.2087, -0.3322], device=‘cuda:0’), tensor([1., 1.], device=‘cuda:0’), tensor([0., 0.], device=‘cuda:0’), tensor([0., 0.], device=‘cuda:0’), tensor([1., 1.], device=‘cuda:0’), tensor(0, device=‘cuda:0’)], 262144000, 0<br>
srun: error: ravg1017: tasks 0-1,2,3: Exited with exit code 1<br>
slurmstepd: error: *** STEP 1387251.0 ON ravg1017 CANCELLED AT 2022-02-24T15:14:14 ***</p>

mlearn: <p>So I solved it by trial and error. Changing the main to the following works:</p>
<pre><code>if __name__ == '__main__':
# DDP setting
   if "WORLD_SIZE" in os.environ:
        world_size = int(os.environ["WORLD_SIZE"])
    ngpus_per_node = torch.cuda.device_count()

if world_size &gt; 1:
    if 'SLURM_PROCID' in os.environ:  # for slurm scheduler
        rank = int(os.environ['SLURM_PROCID'])
        gpu = rank % torch.cuda.device_count()
        print("gpu", gpu, "rank", rank, "ngpus_per_node", ngpus_per_node)
    torch.cuda.set_device(gpu)
    dist.init_process_group(backend='nccl', init_method='env://',
                            world_size=world_size, rank=rank)
    group_g = torch.distributed.new_group(ranks=[0,1,2,3])
    group_d = torch.distributed.new_group(ranks=[0,1,2,3])

    if gpu is not None:
        #torch.cuda.set_device(gpu)
        #with torch.cuda.device(0):
        G = Generator().cuda()
        D = Discriminator().cuda()
        G = torch.nn.parallel.DistributedDataParallel(G, device_ids=[gpu], process_group=group_g,broadcast_buffers=True)
        D = torch.nn.parallel.DistributedDataParallel(D, device_ids=[gpu], process_group=group_d,broadcast_buffers=True)</code></pre>

alexfoo: <p>Hi <a class="mention" href="/u/pritamdamania87">@pritamdamania87</a>, thanks for your input on this. I have tried launching several independent experiments (models A,B,C,D…) on different GPUs and they work fine.</p>
<p>Now in order to utilize my GPU ram better, I have tried overloading several experiments on the same GPUs (e.g. A, B with DDP on GPUs 0,1) but notice a significant slow-down during backprop and slight slow-down for forward pass for both A and B despite getting expected training results. The total spawned processes are also well within the number of CPU cores. Can I check if there is any way to optimize this for speed or is this a bottleneck on how NCCL passes messages between GPUs?</p>
<p>Thanks in advance!</p>

pritamdamania87: <blockquote>
<p>Now in order to utilize my GPU ram better, I have tried overloading several experiments on the same GPUs (e.g. A, B with DDP on GPUs 0,1)</p>
</blockquote>
<p>I wouldn’t recommend doing this unless you somehow ensure only 1 DDP instance is utilizing the GPUs at a time. Running multiple processes using the same GPUs can result in deadlocks.</p>

alexfoo: <p>Gotcha, thanks so much for your input!</p>