[BUG?] DistributedDataParallel cannot be destroyed

username: CuriousCat-7
cooked: <p>My code involves two stage, like<br>
model_a -&gt; do something and model_b -&gt; do something<br>
And I use DistributedDataParallel(DDP) to accelerate them rather than DP. So I do something like that:</p>
<pre><code class="lang-auto">model_a -&gt; model_a = setup DDP and DDP(model_a) -&gt; model_b -&gt; setup DDP and model_b = DDP(model_b)
</code></pre>
<p><strong>This will cause problem because you cannot open start two DDP in one process.</strong></p>
<p>So I use dist.destroy_process_group(). But when I use this function, <strong>the program will wait for endless time.</strong></p>
<p>Here are my start DDP and destroy DDP codes:</p>
<pre><code class="lang-python"> def setup(rank, world_size):
     os.environ['MASTER_ADDR'] = 'localhost'
     os.environ['MASTER_PORT'] = '12355'

     # initialize the process group
     dist.init_process_group("gloo", rank=rank, world_size=world_size)
     #dist.init_process_group("nccl", rank=rank, world_size=world_size)

     # Explicitly setting seed to make sure that models created in two processes
     # start from same random weights and biases.
     torch.manual_seed(42)


 def cleanup():
     dist.destroy_process_group()
</code></pre>
<p>I’ve tried either gloo backend or nccl backend</p>
<p>For Pytorch 1.1, python 3.6, <strong>CUDA 9.0</strong></p>
<p>Thank you</p>

username: pritamdamania87
cooked: <pre><code class="lang-auto">model_a -&gt; model_a = setup DDP and DDP(model_a) -&gt; model_b -&gt; setup DDP and model_b = DDP(model_b)
</code></pre>
<p>I’m not sure I follow this completely, does model_b use the output of model_a? Could you share some code about how model_a and model_b are initialized and trained using DDP?</p>
<p>Is it possible to create a single model with model_a and model_b as submodules and then use that as part of DDP?</p>

username: CuriousCat-7
cooked: <p>Sorry for late, I mean I setup DDP for model_a and setup DDP for model_b in the same time:<br>
Looks like:</p>
<pre><code class="lang-python">setup()
model_a = DDP(model_a)
setup()
model_b = DDP(model_b)
</code></pre>
<p>This will cause an error.</p>
<p>So I need to change like that:</p>
<pre><code class="lang-python">setup()
model_a = DDP(model_a)
cleanup()
setup()
model_b = DDP(model_b)
</code></pre>
<p>but, after the cleanup, the program will be blocked.<img src="https://discuss.pytorch.org/images/emoji/apple/joy.png?v=9" title=":joy:" class="emoji" alt=":joy:"></p>

username: pritamdamania87
cooked: <p>I tried the following program (which is a bit similar to what you were doing), but couldn’t reproduce the issue:</p>
<pre><code class="lang-auto">import os
import tempfile
import torch
import torch.distributed as dist
import torch.nn as nn
import torch.optim as optim
import torch.multiprocessing as mp

from torch.nn.parallel import DistributedDataParallel as DDP


def setup(rank, world_size):
    os.environ['MASTER_ADDR'] = 'localhost'
    os.environ['MASTER_PORT'] = '12355'

    # initialize the process group
    dist.init_process_group("gloo", rank=rank, world_size=world_size)

    # Explicitly setting seed to make sure that models created in two processes
    # start from same random weights and biases.
    torch.manual_seed(42)


def cleanup():
    dist.destroy_process_group()

class ToyModel(nn.Module):
    def __init__(self):
        super(ToyModel, self).__init__()
        self.net1 = nn.Linear(10, 10)
        self.relu = nn.ReLU()
        self.net2 = nn.Linear(10, 5)

    def forward(self, x):
        return self.net2(self.relu(self.net1(x)))

setup(0, 1)
print ("F1")
model_a = DDP(ToyModel())
print ("F2")
cleanup()
print ("F3")
setup(0, 1)
print ("F4")
model_b = DDP(ToyModel())
print ("F5")
</code></pre>
<p>The program prints:</p>
<pre><code class="lang-auto">F1
F2
F3
F4
F5
</code></pre>
<p>Could you share more details about your environment (OS, python version etc)? Also, do you know at which line after the <code>cleanup</code> the program blocks on?</p>