DDP: ProcessExitedException: process 0 terminated with exit code 1

turian: <p>On LambdaLabs, I spin up a two-GPU machine. I run the simple example code from the <a href="https://pytorch.org/docs/stable/notes/ddp.html" rel="noopener nofollow ugc">pytorch docs</a></p>
<p>However, I can’t even get the simple example to run:</p>
<pre><code class="lang-auto">---------------------------------------------------------------------------
ProcessExitedException                    Traceback (most recent call last)
&lt;ipython-input-1-e1523c2c83af&gt; in &lt;module&gt;
     34 
     35 if __name__=="__main__":
---&gt; 36     main()

&lt;ipython-input-1-e1523c2c83af&gt; in main()
     28 def main():
     29     world_size = 2
---&gt; 30     mp.spawn(example,
     31         args=(world_size,),
     32         nprocs=world_size,

~/.local/lib/python3.8/site-packages/torch/multiprocessing/spawn.py in spawn(fn, args, nprocs, join, daemon, start_method)
    228                ' torch.multiprocessing.start_process(...)' % start_method)
    229         warnings.warn(msg)
--&gt; 230     return start_processes(fn, args, nprocs, join, daemon, start_method='spawn')

~/.local/lib/python3.8/site-packages/torch/multiprocessing/spawn.py in start_processes(fn, args, nprocs, join, daemon, start_method)
    186 
    187     # Loop on join until it returns True or raises an exception.
--&gt; 188     while not context.join():
    189         pass
    190 

~/.local/lib/python3.8/site-packages/torch/multiprocessing/spawn.py in join(self, timeout)
    137                 )
    138             else:
--&gt; 139                 raise ProcessExitedException(
    140                     "process %d terminated with exit code %d" %
    141                     (error_index, exitcode),

ProcessExitedException: process 0 terminated with exit code 1
</code></pre>
<p>How can I get a simple DDP example to run?</p>

turian: <p>This is the sample code:</p>
<pre><code class="lang-auto">import torch
import torch.distributed as dist
import torch.multiprocessing as mp
import torch.nn as nn
import torch.optim as optim
from torch.nn.parallel import DistributedDataParallel as DDP


def example(rank, world_size):
    # create default process group
    dist.init_process_group("gloo", rank=rank, world_size=world_size)
    # create local model
    model = nn.Linear(10, 10).to(rank)
    # construct DDP model
    ddp_model = DDP(model, device_ids=[rank])
    # define loss function and optimizer
    loss_fn = nn.MSELoss()
    optimizer = optim.SGD(ddp_model.parameters(), lr=0.001)

    # forward pass
    outputs = ddp_model(torch.randn(20, 10).to(rank))
    labels = torch.randn(20, 10).to(rank)
    # backward pass
    loss_fn(outputs, labels).backward()
    # update parameters
    optimizer.step()

def main():
    world_size = 2
    mp.spawn(example,
        args=(world_size,),
        nprocs=world_size,
        join=True)

if __name__=="__main__":
    main()
</code></pre>
<p>I will note that it works if join=False. But why does this simple pytorch doc example not work as written?</p>

agolynski: <aside class="quote no-group" data-username="turian" data-post="2" data-topic="117800">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/turian/48/30217_2.png" class="avatar"> turian:</div>
<blockquote>
<pre><code class="lang-auto">    mp.spawn(example,
        args=(world_size,),
        nprocs=world_size,
        join=True)
</code></pre>
</blockquote>
</aside>
<p>Hi,</p>
<p>This works ok for me with join=True. Seems like your process 0 is dying for some reason, can you add logging to example function and see where is the problem?</p>
<p>(also seems like you aren’t using GPUs here)</p>

turian: <p><a class="mention" href="/u/agolynski">@agolynski</a> could you suggest how/where I should add logging?</p>
<p>Yes, I’m on a 2 GPU machine from <a href="https://lambdalabs.com/service/gpu-cloud" rel="noopener nofollow ugc">LambdaLabs</a> if you want to try and replicate. (I upgrade torch to the latest release when I create the instance.)</p>
<p>Do I need to add anything extra to use the GPUs?</p>

agolynski: <p>The code you have doesn’t use GPUs, it’s CPU only tensors.</p>
<p>I suggest add some print statements in your model before and after critical sections of the code, i.e.<br>
dist.init_process_group(“gloo”, rank=rank, world_size=world_size)<br>
forward pass<br>
backward pass<br>
optimizer.step()<br>
and see which line causes your error.</p>