DistributedDataParallel on terminal vs jupyter notebook

username: praveen_91
cooked: <p>The code below works on Terminal but not on Jupyter Notebook</p>
<pre><code class="lang-auto">import os
from datetime import datetime
import argparse
import torch.multiprocessing as mp
import torchvision
import torchvision.transforms as transforms
import torch
import torch.nn as nn
import torch.distributed as dist
import torch.optim as optim
from torch.nn.parallel import DistributedDataParallel as DDP

os.environ["CUDA_VISIBLE_DEVICES"] = "6,7"

def setup(rank, world_size):
    os.environ['MASTER_ADDR'] = 'localhost'
    os.environ['MASTER_PORT'] = '12355'

    # initialize the process group
    dist.init_process_group("nccl", rank=rank, world_size=world_size)


def cleanup():
    dist.destroy_process_group()


class ToyModel(nn.Module):
    def __init__(self):
        super(ToyModel, self).__init__()
        self.net1 = nn.Linear(10, 10)
        self.relu = nn.ReLU()
        self.net2 = nn.Linear(10, 5)

    def forward(self, x):
        return self.net2(self.relu(self.net1(x)))


def demo_basic(rank, world_size):
    print(f"Running basic DDP example on rank {rank}.")
    setup(rank, world_size)

    # create model and move it to GPU with id rank
    model = ToyModel().to(rank)
    ddp_model = DDP(model, device_ids=[rank])

    loss_fn = nn.MSELoss()
    optimizer = optim.SGD(ddp_model.parameters(), lr=0.001)

    optimizer.zero_grad()

    outputs = ddp_model(torch.ones(200, 10))
    labels = torch.randn(200, 5).to(rank)

    loss = loss_fn(outputs, labels)
    print("Loss is ",loss.item())

    loss.backward()
    optimizer.step()

    cleanup()


if __name__ == '__main__':
    
    world_size = 2

    print("We have available ", torch.cuda.device_count(), "GPUs! but using ",world_size," GPUs")

    #########################################################
    mp.spawn(demo_basic, args=(world_size), nprocs=world_size, join=True)    
    #########################################################
</code></pre>
<p>Terminal output:</p>
<pre><code class="lang-auto">We have available  2 GPUs! but using  2  GPUs
Running basic DDP example on rank 1.
Running basic DDP example on rank 0.
Loss is  1.0888941287994385
Loss is  1.0354920625686646

</code></pre>
<p>Jupyter Notebook Output:</p>
<pre><code class="lang-auto">We have available  2 GPUs! but using  2  GPUs
---------------------------------------------------------------------------
Exception                                 Traceback (most recent call last)
&lt;ipython-input-2-52a9f6d32955&gt; in &lt;module&gt;
     68 
     69     #########################################################
---&gt; 70     mp.spawn(demo_basic, args=(world_size), nprocs=world_size, join=True)
     71     #########################################################

~/.conda/envs/praveen_tf/lib/python3.6/site-packages/torch/multiprocessing/spawn.py in spawn(fn, args, nprocs, join, daemon, start_method)
    198                ' torch.multiprocessing.start_process(...)' % start_method)
    199         warnings.warn(msg)
--&gt; 200     return start_processes(fn, args, nprocs, join, daemon, start_method='spawn')

~/.conda/envs/praveen_tf/lib/python3.6/site-packages/torch/multiprocessing/spawn.py in start_processes(fn, args, nprocs, join, daemon, start_method)
    156 
    157     # Loop on join until it returns True or raises an exception.
--&gt; 158     while not context.join():
    159         pass
    160 

~/.conda/envs/praveen_tf/lib/python3.6/site-packages/torch/multiprocessing/spawn.py in join(self, timeout)
    111                 raise Exception(
    112                     "process %d terminated with exit code %d" %
--&gt; 113                     (error_index, exitcode)
    114                 )
    115 

Exception: process 1 terminated with exit code 1

</code></pre>
<p>Can anyone explain why is this?</p>

username: praveen_91
cooked: <p><a class="mention" href="/u/ph0123">@ph0123</a> You had the similar post. Did you figure out why this happens?</p>

username: ph0123
cooked: <p>Hi,<br>
At that time, the distributed pytorch did not support for Window.<br>
Thanks,</p>

username: pritamdamania87
cooked: <p><a class="mention" href="/u/praveen_91">@praveen_91</a> Getting multiprocessing code to work with Jupyter Notebooks is tricky. You can probably try out solutions such as <a href="https://medium.com/@grvsinghal/speed-up-your-python-code-using-multiprocessing-on-windows-and-jupyter-or-ipython-2714b49d6fac" rel="noopener nofollow ugc">https://medium.com/@grvsinghal/speed-up-your-python-code-using-multiprocessing-on-windows-and-jupyter-or-ipython-2714b49d6fac</a> to see if it works for you.</p>
<p>There is an issue open for this on ipython as well: <a href="https://github.com/ipython/ipython/issues/12396" rel="noopener nofollow ugc">https://github.com/ipython/ipython/issues/12396</a>.</p>

username: praveen_91
cooked: <aside class="quote no-group" data-username="pritamdamania87" data-post="4" data-topic="101404">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/pritamdamania87/48/28891_2.png" class="avatar"> pritamdamania87:</div>
<blockquote>
<p>ing multiprocessing code to work with Jupyter Notebooks is trick</p>
</blockquote>
</aside>
<p>I am not on a windows machine, its a linux server and I have a 1 hop connection with jupyter notebook</p>

username: praveen_91
cooked: <p>I am on a windows machine with 1 hop connection from a linux server</p>

username: Felipe_bivort_haiek
cooked: <p>Hi could you get it working?</p>

username: Vio_Ubi
cooked: <p>No way to get DDP running in Jupyter?</p>