Distributed Data Parallel example - "process 0 terminated with exit code 1"

dyukha: <p>I’m running <a href="https://pytorch.org/docs/stable/notes/ddp.html#example" rel="noopener nofollow ugc">Distributed Data Parallel example</a> in jupyter labs, and getting an error:</p>
<pre><code class="lang-auto">process 1 terminated with exit code 1
</code></pre>
<p>How can I fix it? Where should I look at? I tried using “nccl” or “mpi” in <code>dist.init_process_group</code>, no effect. Replacing the entire body of <code>example()</code> with <code>pass</code>: no effect.</p>
<p>Full stack trace:</p>
<pre><code class="lang-auto">---------------------------------------------------------------------------
Exception                                 Traceback (most recent call last)
&lt;ipython-input-29-703537cd0120&gt; in &lt;module&gt;
     33         join=True)
     34 
---&gt; 35 main()

&lt;ipython-input-29-703537cd0120&gt; in main()
     28 def main():
     29     world_size = 2
---&gt; 30     mp.spawn(example,
     31         args=(world_size,),
     32         nprocs=world_size,

/usr/local/lib/python3.8/site-packages/torch/multiprocessing/spawn.py in spawn(fn, args, nprocs, join, daemon, start_method)
    197                ' torch.multiprocessing.start_process(...)' % start_method)
    198         warnings.warn(msg)
--&gt; 199     return start_processes(fn, args, nprocs, join, daemon, start_method='spawn')

/usr/local/lib/python3.8/site-packages/torch/multiprocessing/spawn.py in start_processes(fn, args, nprocs, join, daemon, start_method)
    155 
    156     # Loop on join until it returns True or raises an exception.
--&gt; 157     while not context.join():
    158         pass
    159 

/usr/local/lib/python3.8/site-packages/torch/multiprocessing/spawn.py in join(self, timeout)
    108                 )
    109             else:
--&gt; 110                 raise Exception(
    111                     "process %d terminated with exit code %d" %
    112                     (error_index, exitcode)

Exception: process 1 terminated with exit code 1
</code></pre>

wanchaol: <p>Thanks for posting <a class="mention" href="/u/dyukha">@dyukha</a>, I am seeing that you are running the code inside a jupyter notebook, I think the error you hit here is not a problem with PyTorch distributed, this is related to the incompatibility between python multiprocessing module and Jupyter Notebook because multiprocessing module pickles data to send to processes. If you can try the script in a normal python file it should work fine.</p>

dyukha: <p>Thanks for the reply! Is there a way to make it work in Jupyter?</p>

Ali_Boushehri: <p>is this still an issue?</p>

wanchaol: <aside class="quote no-group" data-username="wanchaol" data-post="2" data-topic="127531">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/wanchaol/48/50343_2.png" class="avatar"> wanchaol:</div>
<blockquote>
<p>python multiprocessing module and Jupyter Notebook</p>
</blockquote>
</aside>
<p>You could try multiprocessing.Pool and see if that works <a href="https://stackoverflow.com/questions/50937362/multiprocessing-on-python-3-jupyter" class="inline-onebox" rel="noopener nofollow ugc">Multiprocessing on Python 3 Jupyter - Stack Overflow</a></p>