Pytorch isn't working with DistributedDataParallel for multi gpu training

i_m_a_q: <p>I am trying to train my model on multiple GPUS. I used the libraries and a added a code for it</p>
<pre><code class="lang-auto">from torch.nn.parallel import DistributedDataParallel as DDP
from torch.distributed import init_process_group, destroy_process_group
</code></pre>
<p>Initialization</p>
<pre><code class="lang-auto">def ddp_setup(rank: int, world_size: int):
   os.environ["MASTER_ADDR"] = "localhost"
   os.environ["MASTER_PORT"] = "12355"
   os.environ["TORCH_DISTRIBUTED_DEBUG"]="DETAIL"
   init_process_group(backend="gloo", rank=0, world_size=1)
</code></pre>
<p>my model</p>
<pre><code class="lang-auto"> model = CMGCNnet(config,
                     que_vocabulary=glovevocabulary,
                     glove=glove,
                     device=device)

    model = model.to(0)

    if -1 not in args.gpu_ids and len(args.gpu_ids) &gt; 1:
       model = DDP(model, device_ids=[0,1])
</code></pre>
<p><strong>it throws following error:</strong></p>
<blockquote>
<p>config_yml          : model/config_fvqa_gruc.yml<br>
cpu_workers         : 0<br>
save_dirpath        : exp_test_gruc<br>
overfit             : False<br>
validate            : True<br>
gpu_ids             : [0, 1]<br>
dataset             : fvqa<br>
Loading FVQATrainDataset…<br>
True<br>
done splitting<br>
Loading FVQATestDataset…<br>
Loading glove…<br>
Building Model…<br>
Traceback (most recent call last):<br>
File “trainfvqa_gruc.py”, line 512, in <br>
train()<br>
File “trainfvqa_gruc.py”, line 145, in train<br>
ddp_setup(0,1)<br>
File “trainfvqa_gruc.py”, line 42, in ddp_setup<br>
init_process_group(backend=“gloo”, rank=0, world_size=1)<br>
File “/home/seecs/miniconda/envs/mucko-edit/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py”, line 360, in init_process_group<br>
timeout=timeout)<br>
RuntimeError: [enforce fail at /opt/conda/conda-bld/pytorch_1544202130060/work/third_party/gloo/gloo/transport/tcp/device.cc:128] rp != nullptr. Unable to find address for: 127.0.0.1localhost.<br>
localdomainlocalhost</p>
</blockquote>
<p>I tried printing the issue with    <code>os.environ["TORCH_DISTRIBUTED_DEBUG"]="DETAIL"</code><br>
it outputs:</p>
<pre><code class="lang-auto">Loading FVQATrainDataset...
True
done splitting
Loading FVQATestDataset...
Loading glove...
Building Model...
**Segmentation fault**
</code></pre>
<p>with NCCL background it starts the training but get stuck and doesn’t go further than this <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
<blockquote>
<p>Training for epoch 0:<br>
0%|                                                                                                                                                                  | 0/2039 [00:00&lt;?, ?it/s]</p>
</blockquote>
<p>I found this solution but where to add these lines?<br>
<code>*GLOO_SOCKET_IFNAME** , for example </code>export GLOO_SOCKET_IFNAME=eth0`<br>
mentioned in<br>
(<a href="https://discuss.pytorch.org/t/runtime-error-using-distributed-with-gloo/16579/2" class="inline-onebox">Runtime error using Distributed with gloo - #2 by ifgovh</a>)</p>
<p>Can someone help me with this issue?</p>