torch.nn.parallel.DistributedDataParallel() problem about "NoneType Error"\ CalledProcessError\backward

username: lzkzls
cooked: <p>In a GAN-based model, contains one generator model and three discriminator model, all the models are wrapped in torch.nn.parallel.DistributedDataParallel() with different argument process_group, the loss function contains two parts, like this:<br>
d_total_loss = d_real_loss + d_fake_loss<br>
and the backpropgation is : d_total_loss.backward()<br>
when I run the program, the error is:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/6/66ba0ca7ff774a089fb5d02bdf07d670dd959392.png" data-download-href="https://discuss.pytorch.org/uploads/default/66ba0ca7ff774a089fb5d02bdf07d670dd959392" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/6/6/66ba0ca7ff774a089fb5d02bdf07d670dd959392.png" alt="image" data-base62-sha1="eELjhKwA88pJgETYkhyNm2lz7iy" width="690" height="115" data-dominant-color="191919"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">2008×335 25.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
But, when I run d_real_loss.backward() or d_fake_loss.backward(), the program could run normally.<br>
What’s more, I have another problem that is when I use generator_model.train() in my program and run it, there will be an error:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/0/c08105476b3ec8fa3c256905533510b11166c865.png" data-download-href="https://discuss.pytorch.org/uploads/default/c08105476b3ec8fa3c256905533510b11166c865" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/c/0/c08105476b3ec8fa3c256905533510b11166c865.png" alt="image" data-base62-sha1="rsY6bmTQzZuiB2umsNSfUKv1Ch7" width="690" height="173" data-dominant-color="0F0F0F"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">2018×507 29.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
Could you give me some advice to solve these problems?</p>

username: mrshenli
cooked: <p>Hey <a class="mention" href="/u/lzkzls">@lzkzls</a> could you please share a minimal reproduce-able example code?</p>
<p>The first error picture does not seem to be a DDP error. Does the code run correctly without DDP? Looks like the autograd graph generating <code>d_real_loss</code> and <code>d_fake_loss</code> share some operators/parameters.</p>
<p>The second error picture seems to suggest the <code>generator_model</code> is a None object? It will be helpful to see a self-contained repro of this error.</p>

username: lzkzls
cooked: <p>When I use torch.nn.DataParallel(), the code run correctly.<br>
Thank you so much!<br>
Here is a minimal reproduce-able example:</p><aside class="onebox githubblob" data-onebox-src="https://github.com/lzkzls/DDPtest/blob/master/DDP_test.py">
  <header class="source">

      <a href="https://github.com/lzkzls/DDPtest/blob/master/DDP_test.py" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/lzkzls/DDPtest/blob/master/DDP_test.py" target="_blank" rel="noopener nofollow ugc">lzkzls/DDPtest/blob/master/DDP_test.py</a></h4>


      <pre><code class="lang-py">import torch, torchvision
import torch.nn as nn
import torch.distributed as dist
import torchvision.transforms as transforms
import torch.optim as optim


#input (1,28,28)
class Discriminator(nn.Module):
    def __init__(self):
        super(Discriminator, self).__init__()
        self.conv2 = nn.ModuleList()
        self.conv2.append(nn.Sequential(nn.Conv2d(1, 16, 3, stride=2, padding=1),
                                        nn.BatchNorm2d(16),
                                        nn.LeakyReLU(negative_slope=0.2)
        ))
        
        self.conv2.append(nn.Sequential(nn.Conv2d(16, 32, 3, stride=2, padding=1),
                                        nn.BatchNorm2d(32),
                                        nn.LeakyReLU(negative_slope=0.2)
</code></pre>



  This file has been truncated. <a href="https://github.com/lzkzls/DDPtest/blob/master/DDP_test.py" target="_blank" rel="noopener nofollow ugc">show original</a>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
<p>
and I run this code with command: sudo CUDA_VISIBLE_DEVICES=0,1,2,3 python3 -m torch.distributed.launch --nproc_per_node=4 DDP_test.py.<br>
note: When I remove the comment of line 77 and 78, there will be an error:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/e/ee53c8d3d89a5dd3e53786ba51488cdcfe201e82.png" data-download-href="https://discuss.pytorch.org/uploads/default/ee53c8d3d89a5dd3e53786ba51488cdcfe201e82" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/e/e/ee53c8d3d89a5dd3e53786ba51488cdcfe201e82.png" alt="image" data-base62-sha1="y0l90TaCFpV9ZOyO1u6kiVNHIEa" width="690" height="81" data-dominant-color="181818"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1348×160 8.39 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: mrshenli
cooked: <p>Hey <a class="mention" href="/u/lzkzls">@lzkzls</a></p>
<p>The following code works for me. I found two errors:</p>
<ol>
<li>The original code didn’t set local_rank correctly. It needs be read the local_rank argument instead of hardcoding to 0.</li>
<li>For DDP, you need to call forward and backward interleavingly, instead of two forward followed by one backward. This is fixed by letting the forward function of <code>Discriminator</code> taking both fake and real images.</li>
</ol>
<pre><code class="lang-python">import argparse
import torch, torchvision
import torch.nn as nn
import torch.distributed as dist
import torchvision.transforms as transforms
import torch.optim as optim


#input (1,28,28)
class Discriminator(nn.Module):
    def __init__(self):
        super(Discriminator, self).__init__()
        self.conv2 = nn.ModuleList()
        self.conv2.append(nn.Sequential(nn.Conv2d(1, 16, 3, stride=2, padding=1),
                                        nn.BatchNorm2d(16),
                                        nn.LeakyReLU(negative_slope=0.2)
        ))

        self.conv2.append(nn.Sequential(nn.Conv2d(16, 32, 3, stride=2, padding=1),
                                        nn.BatchNorm2d(32),
                                        nn.LeakyReLU(negative_slope=0.2)
                        ))
        self.conv2.append(nn.Sequential(nn.Conv2d(32, 64, 3, stride=2, padding=1),
                                        nn.BatchNorm2d(64),
                                        nn.LeakyReLU(negative_slope=0.2)
        ))
        self.conv2.append(nn.Sequential(nn.Conv2d(64, 1, 3, stride=2),
                                        nn.BatchNorm2d(1),
                                        nn.LeakyReLU(negative_slope=0.2)
        ))
    def forward(self, fake, real):
        for conv_layer in self.conv2:
            fake = conv_layer(fake)
            real = conv_layer(real)

        return fake.view(-1,1), real.view(-1, 1)

class Generator(nn.Module):
    def __init__(self):
        super(Generator, self).__init__()
        self.deconv2 = nn.ModuleList()
        self.deconv2.append(nn.Sequential(nn.ConvTranspose2d(64, 32, kernel_size=3, stride=2,padding=1),
                            nn.BatchNorm2d(32),
                            nn.LeakyReLU()
        ))
        self.deconv2.append(nn.Sequential(nn.ConvTranspose2d(32, 16, kernel_size=3, stride=2,padding=1),
                            nn.BatchNorm2d(16),
                            nn.LeakyReLU()
        ))
        self.deconv2.append(nn.Sequential(nn.ConvTranspose2d(16, 1, kernel_size=3, stride=2,padding=1),
                            nn.BatchNorm2d(1),
                            nn.LeakyReLU()
        ))
    def forward(self, x):
        for layer in self.deconv2:
            x = layer(x)

        return x

parser = argparse.ArgumentParser()
parser.add_argument("--local_rank", type=int, default=0)
parser.add_argument("--local_world_size", type=int, default=1)
args = parser.parse_args()

local_rank = args.local_rank
dist.init_process_group(backend='nccl', init_method='env://')

disciminator_model = Discriminator()
generator_model = Generator()

torch.cuda.set_device(local_rank)
disciminator_model.cuda(local_rank)
generator_model.cuda(local_rank)

pg1 = dist.new_group(range(dist.get_world_size()))
pg2 = dist.new_group(range(dist.get_world_size()))
disciminator_model = torch.nn.parallel.DistributedDataParallel(disciminator_model, device_ids=[local_rank],
                                                                output_device=local_rank, process_group=pg1)
generator_model = torch.nn.parallel.DistributedDataParallel(generator_model, device_ids=[local_rank],
                                                                output_device=local_rank, process_group=pg2)

# disciminator_model = disciminator_model.train()
# generator_model = generator_model.train()

g_optimizer = optim.Adam(params=generator_model.parameters(), lr=1e-4)
d_optimizer = optim.Adam(params=disciminator_model.parameters(), lr =1e-4)
bcelog_loss = nn.BCEWithLogitsLoss().cuda(local_rank)

train_dataset = torchvision.datasets.MNIST(root='../../data',
                                           train=True,
                                           transform=transforms.ToTensor(),
                                           download=True)

train_sampler = torch.utils.data.distributed.DistributedSampler(train_dataset)
batch_size = 8
train_loader = torch.utils.data.DataLoader(train_dataset,
                                           batch_size=batch_size,
                                           shuffle=False,
                                           num_workers=4,
                                           pin_memory=True,
                                           sampler=train_sampler)

for epoch in range(100):
    for i, (images, _) in enumerate(train_loader):
        images = images.cuda(local_rank, non_blocking=True)
        real_tensor = torch.full((batch_size,1), 1, dtype=torch.float32).cuda(local_rank)
        fake_tensor = torch.zeros((batch_size,1), dtype=torch.float32).cuda(local_rank)
        noise_tensor = torch.rand((batch_size, 64, 4, 4))
        gen_image = generator_model(noise_tensor)

        d_fake, d_real = disciminator_model(gen_image, images)
        #d_real = disciminator_model(images)

        d_fake_loss = bcelog_loss(d_fake, fake_tensor)
        d_real_loss = bcelog_loss(d_real, real_tensor)

        d_total_loss = d_fake_loss + d_real_loss

        g_optimizer.zero_grad()
        d_optimizer.zero_grad()

        d_total_loss.backward()
        g_optimizer.step()
        d_optimizer.step()
        if i % 10 == 0:
            print(f"processed {i} images")
    print("current epoch: ", epoch)
</code></pre>

username: lzkzls
cooked: <p>Thank you! You are so great! What should I do if I have two different Discriminator class?</p>

username: mrshenli
cooked: <blockquote>
<p>What should I do if I have two different Discriminator class?</p>
</blockquote>
<p>As long as forward and backward on one DDP instance is called alternatively, it should work. So, there are at least two options:</p>
<ol>
<li>Wrap the two Discriminators into one <code>nn.Module</code>, say <code>CombinedDiscriminator</code>, and then pass the <code>CombinedDiscriminator</code> to DDP ctor.</li>
<li>Create a dedicated DDP instance (with dedicated ProcessGroup instance) for each Discriminator.</li>
</ol>