RuntimeError with Distributed package using NCCL or GLOO backend

username: enisberk
cooked: <p>Hi,</p>
<p>I am trying to use <a href="https://pytorch.org/docs/master/distributed.html" rel="nofollow noopener">distributed package</a> with two nodes but I am getting runtime errors.<br>
I am using Pytorch nightly version with Python3. I have two scripts one for master and one for slave (code: <a href="https://gist.github.com/EnisBerk/1c1c10df476acd60cb28bffeab5bc2f7" rel="nofollow noopener">master</a>, <a href="https://gist.github.com/EnisBerk/f5a71482f14b4a127a53e3aab1fceff3" rel="nofollow noopener">slave</a>).  I tried both gloo and nccl backends and got the same errors.</p>
<p>I am getting following error on master:</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "s_testm.py", line 86, in &lt;module&gt;
    main()
  File "s_testm.py", line 83, in main
    init_processes(rank,size,run,master_ip)
  File "s_testm.py", line 57, in init_processes
    dist.init_process_group(backend)
  File "/opt/conda/envs/fastai/lib/python3.6/site-packages/torch/distributed/distributed_c10d.py", line 280, in init_process_group
    store, rank, world_size = next(rendezvous(init_method))
  File "/opt/conda/envs/fastai/lib/python3.6/site-packages/torch/distributed/rendezvous.py", line 131, in _env_rendezvous_handler
    store = TCPStore(master_addr, master_port, start_daemon)
RuntimeError: Connection timed out
</code></pre>
<p>And following error on the slave:</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "s_tests.py", line 76, in &lt;module&gt;
    main()
  File "s_tests.py", line 73, in main
    init_processes(rank,size,run,master_ip)
  File "s_tests.py", line 52, in init_processes
    dist.init_process_group(backend)
  File "/opt/conda/envs/fastai/lib/python3.6/site-packages/torch/distributed/distributed_c10d.py", line 283, in init_process_group
    _default_pg = ProcessGroupGloo(store, rank, world_size)
RuntimeError: Resource temporarily unavailable
</code></pre>
<p>I am using machines from <a href="https://www.paperspace.com/" rel="nofollow noopener">Paperspace</a>, I am using their shared file system to exchange IP addresses between master and slave. I am not using file system initialization because their file system does not support <a href="http://man7.org/linux/man-pages/man2/fcntl.2.html" rel="nofollow noopener">fcntl</a> locking.</p>
<p>To run the codes, I am using their <a href="https://github.com/Paperspace/paperspace-python" rel="nofollow noopener">command line python client</a>  as following:</p>
<pre><code class="lang-auto">#master
paperspace jobs create --project myproject --machineType P5000 --container paperspace/fastai:1.0-CUDA9.2-base --ports 29500:29500 --command 'apt-get update;apt-get install libibverbs1;echo "start";python3 s_testm.py' --workspace "./"

#slave
paperspace jobs create --project myproject --machineType P5000 --container paperspace/fastai:1.0-CUDA9.2-base --ports 29500:29500 --command 'apt-get update;apt-get install libibverbs1; python3 s_tests.py' --workspace "./"
</code></pre>
<p>As you can see, machine have P5000 gpu, I am using their fast-ai container with pytorch 1.0, and I am opening port 29500 to use for initialization.<br>
Container lacks <a href="https://packages.debian.org/sid/libibverbs1" rel="nofollow noopener">libibverbs1</a> library, so I am installing it before running my scripts.</p>
<p>I am open to all kinds of help, thanks in advance.</p>

username: zkzhu0110
cooked: <p>I met the same problem as this. I think that it may be related to the communication way of pytorch between different containers.</p>

username: Shirley_Han
cooked: <p>Hi,</p>
<p>Did you solve this problem?</p>
<p>I also ran into some error related to TCPStore.</p>

username: gradwolf
cooked: <p>Same issue here.</p>
<p>I am trying to run a job with distributed training in a single node with 4 GPUs. I have set the IP address of my instance and port number. I am getting this error. Any solutions?</p>
<pre><code class="lang-auto">File "/opt/conda/lib/python3.6/site-packages/torch/distributed/distributed_c10d.py", line 400, in init_process_group
    store, rank, world_size = next(rendezvous(url))
  File "/opt/conda/lib/python3.6/site-packages/torch/distributed/rendezvous.py", line 143, in _env_rendezvous_handler
    store = TCPStore(master_addr, master_port, world_size, start_daemon)
RuntimeError: Connection timed out
</code></pre>

username: enisberk
cooked: <p>It was an network setup issue related to docker so we could not solve it. You need to make sure nodes/threads can communicate through network.<br>
Paperspace solved this problem in their latest version by changing their networking setup.</p>