Using rpc on two computers

pajenk: <p>Hi.<br>
I’m trying to use two computers following the example:<br>
<a href="https://pytorch.org/tutorials/intermediate/rpc_async_execution.html#batch-updating-parameter-server" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/tutorials/intermediate/rpc_async_execution.html#batch-updating-parameter-server</a></p>
<p>First I tried very easy example, but I got an error.<br>
On computer1 (let’s say, ip address: 1.1.1.1)</p>
<pre><code class="lang-auto">import os
import torch.distributed.rpc as rpc

os.environ['MASTER_ADDR'] = '2.2.2.2'
os.environ['MASTER_PORT'] = '7271'
rpc.init_rpc("worker1", rank=1, world_size=2)
rpc.shutdown()
</code></pre>
<p>On computer2 (let’s say, ip address : 2.2.2.2)</p>
<pre><code class="lang-auto">import os
import torch.distributed.rpc as rpc

os.environ['MASTER_ADDR'] = 'localhost'
os.environ['MASTER_PORT'] = '7271'
rpc.init_rpc("worker0", rank=0, world_size=2)
rpc.shutdown()
</code></pre>
<p>Error:</p>
<pre><code class="lang-auto">RuntimeError: […/third_party/gloo/gloo/transport/tcp/pair.cc:799] connect [127.0.1.1]:14237: Connection refused
</code></pre>
<p>This error looks like the same as the error in the following link,</p><aside class="quote quote-modified" data-post="1" data-topic="100802">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/bbce88/48.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/connect-127-0-1-1-a-port-connection-refused/100802">Connect [127.0.1.1]:[a port]: Connection refused</a> <a class="badge-category__wrapper " href="/c/distributed/distributed-rpc/20"><span data-category-id="20" data-parent-category-id="12" data-drop-close="true" class="badge-category --has-parent"><span class="badge-category__name">distributed-rpc</span></span></a>
  </div>
  <blockquote>
    I am using the batch processing example for <a href="https://pytorch.org/tutorials/intermediate/rpc_async_execution.html#batch-updating-parameter-server" rel="noopener nofollow ugc">PS</a>. 
I put the PS and a worker on two machines. I changed the os.environ[‘MASTER_ADDR’] = ‘localhost’ 
os.environ[‘MASTER_PORT’] = ‘29500’ to the machine with rank 0. 
PS runs and waiting for worker. But the worker send an error: 
Traceback (most recent call last): 
File “/home/skh018/PycharmProjects/test_DDL_pytorch/worker1.py”, line 148, in  
run(1, world_size) 
File “/home/skh018/PycharmProjects/test_DDL_pytorch/worker1.py”, line 124, in run 
rpc.in…
  </blockquote>
</aside>
<p>
but It seems that the problem has not been solved.</p>
<p>Is it necessary to set up additional network settings or include options in the code?</p>
<p>=======================================<br>
In Computer 1 and Computer 2:</p>
<pre data-code-wrap="bash"><code class="lang-bash">$ ifconfig
eno1: ~~~
lo: ~~~

# I did this.
$ sudo ufw allow 7271
</code></pre>

H-Huang: <p>Hi <a class="mention" href="/u/pajenk">@pajenk</a>, a couple of questions:</p>
<ol>
<li>On computer2(ip address: 2.2.2.2) have you tried also settings the MASTER_ADDR as 2.2.2.2? e.g.</li>
</ol>
<pre><code class="lang-auto">import os
import torch.distributed.rpc as rpc

os.environ['MASTER_ADDR'] = '2.2.2.2' # instead of localhost
os.environ['MASTER_PORT'] = '7271'
rpc.init_rpc("worker0", rank=0, world_size=2)
rpc.shutdown()
</code></pre>
<ol start="2">
<li>
<p>Gloo error means the error is likely happening during init_rpc, curious which version of pytorch are you using?</p>
</li>
<li>
<p>Not sure of additional networking settings, but the main thing is to confirm that the network address and port are accessible for each computer.</p>
</li>
</ol>

pajenk: <p>Hi <a class="mention" href="/u/h-huang">@H-Huang</a>, Thanks for the questions !</p>
<ol>
<li>Yes, I tried but the same error occurred.</li>
<li>I tried ‘1.11.0+cu113’  and ‘1.11.0+cu102’ on both computers using virtual environment (using <code>python3 -m venv ~</code>).<br>
Should the python version be the same? The computers have different versions (3.8 and 3.9).</li>
<li>When I run the code in only one of the two computers, there is no error (just waiting), and if I run the another code in another computer, then the error occurred. I think they are connected somehow.</li>
</ol>