Strange behaviour of GLOO tcp transport

Oleg_Ivanov: <p>HI all.<br>
I have strange problem: I’m trying to run 2 tasks on 2 machines via following<br>
trivial script:</p>
<pre><code>dist.init_process_group(backend = "gloo",init_method = 'tcp://192.168.0.1:29500',rank = irank,world_size = iwsize)
arg = None
if(dist.get_rank()==0):
    arg = Dist_Trainer()
run(dist.get_rank(),dist.get_world_size(),arg)
</code></pre>
<p>When I run them on one machine, all works fine.<br>
But when I start process with rank = 0 on one machine,<br>
and process with rank = 1 on another machine,<br>
process with rank = 0 fails with the following  output:</p>
<p>python train_dist.py 0 2<br>
RANK: 0 wsize: 2<br>
terminate called after throwing an instance of ‘gloo::IoException’                                                                                                                                                   what():  [/opt/conda/conda-bld/pytorch_1544176307774/work/third_party/gloo/gloo/transport/tcp/pair.cc:724] connect [127.0.0.1]:45965: Connection refused</p>
<p>This happens <em>only</em> when I start process with rank=1. If I don’t started it,<br>
process with rank =0 is waiting for connection.</p>
<p>i.e.,I assume that tcp connection happens, but then process with rank = 0<br>
tries to work with 127.0.0.1?</p>
<p>Upd: I tried  setting export GLOO_SOCKET_IFNAME=enp2s0,<br>
the problem still remains.</p>

mrshenli: <p>Looks like rank 0 is working with [127.0.0.1]:45965. Have you unset <code>MASTER_ADDR</code> and <code>MASTER_PORT</code> environment vars before launching the script?</p>

Oleg_Ivanov: <p>Yes, they were unset.</p>
<p>By the way, if I swap scripts with rank=0 and rank=1 on these machines,<br>
then script with rank=1crashes:</p>
<p>python train_dist.py 1 2<br>
RANK: 1 wsize: 2<br>
terminate called after throwing an instance of ‘gloo::IoException’<br>
what():  [/opt/conda/conda-bld/pytorch_1544176307774/work/third_party/gloo/gloo/transport/tcp/pair.cc:724] connect [127.0.1.1]:3978: Connection refused</p>
<p>Script with rank=0 still waiting for connection</p>

ZiyiZhu: <p>Hi <a class="mention" href="/u/oleg_ivanov">@Oleg_Ivanov</a>,</p>
<p>Have you solved this problem? Same as here.</p>
<p>Thanks,<br>
Ziyi</p>

mrshenli: <p>If you just run the following without any other code, does it fail?</p>
<pre><code class="lang-python">import torch.distributed as dist

# on rank 0
dist.init_process_group(
    backend = "gloo",
    init_method = 'tcp://192.168.0.1:29500',
    rank = 0,
    world_size = 2
)

# on rank 1
dist.init_process_group(
    backend = "gloo",
    init_method = 'tcp://192.168.0.1:29500',
    rank = 1,
    world_size = 2
)
</code></pre>

mrshenli: <p>Hey <a class="mention" href="/u/ziyizhu">@ZiyiZhu</a>  Are you trying to run this with RPC? Currently <code>init_rpc</code> does not work together with <code>init_process_group</code>. There are work around to create non-default process groups. Or we can also add a fix to <code>init_rpc</code> if necessary. This is the tracking issue: <a href="https://github.com/pytorch/pytorch/issues/33583" rel="nofollow noopener">https://github.com/pytorch/pytorch/issues/33583</a></p>

ZiyiZhu: <p>Hi <a class="mention" href="/u/mrshenli">@mrshenli</a>,</p>
<p>This is different from the RPC problem. Back then I was using Google Cloud VMs. The torch.distributed and RPC worked fine there.</p>
<p>However, just recently we built up new servers with GPU in our lab and connect them using an electrical packet switch. They can ping each other using the internal IP. For me now it is 10.1.1.101 for rank 0 and 10.1.1.102 for rank 1. So I run the following:</p>
<pre><code class="lang-auto">import torch.distributed as dist

# on rank 0
dist.init_process_group(
    backend = "gloo",
    init_method = 'tcp://10.1.1.101:29500',
    rank = 0,
    world_size = 2
)
</code></pre>
<pre><code class="lang-auto">import torch.distributed as dist

# on rank 1
dist.init_process_group(
    backend = "gloo",
    init_method = 'tcp://10.1.1.101:29500',
    rank = 1,
    world_size = 2
)
</code></pre>
<p>However, it failed with</p>
<pre><code class="lang-auto">---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
&lt;ipython-input-1-532df564c254&gt; in &lt;module&gt;
      6     init_method = 'tcp://10.1.1.101:29500',
      7     rank = 1,
----&gt; 8     world_size = 2
      9 )

~/anaconda3/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py in init_process_group(backend, init_method, timeout, world_size, rank, store, group_name)
    401             store,
    402             group_name=group_name,
--&gt; 403             timeout=timeout)
    404 
    405     _pg_group_ranks[_default_pg] = {i: i for i in range(_default_pg.size())}

~/anaconda3/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py in _new_process_group_helper(world_size, rank, group_ranks, backend, store, group_name, timeout)
    469                 rank,
    470                 world_size,
--&gt; 471                 timeout=timeout)
    472             _pg_map[pg] = (Backend.GLOO, store)
    473             _pg_names[pg] = group_name

RuntimeError: [/opt/conda/conda-bld/pytorch_1587428398394/work/third_party/gloo/gloo/transport/tcp/pair.cc:769] connect [127.0.0.1]:31662: Connection refused
</code></pre>
<p>Which I guess is the same problem for <a class="mention" href="/u/oleg_ivanov">@Oleg_Ivanov</a> too. In terms of</p>
<pre><code class="lang-auto">export GLOO_SOCKET_IFNAME=eno2
</code></pre>
<p>Should I simply do it in any terminal? eno2 is my NIC.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/c/f/cf593cd47885bd0ea96d176c8225a5b754f87768.png" alt="image" data-base62-sha1="tAi1G9Fo6wfuElbqHsbFZy9tNLq" width="597" height="154"></p>
<p>Please let me know if you have any thoughts. Thank you very much for your help!</p>

mrshenli: <aside class="quote no-group" data-username="ZiyiZhu" data-post="7" data-topic="66651">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/z/ce7236/48.png" class="avatar"> ZiyiZhu:</div>
<blockquote>
<p>Should I simply do it in any terminal?</p>
</blockquote>
</aside>
<p>Yes, either set it in terminal or pass <code>GLOO_SOCKET_IFNAME=eno2</code> as a prefix to the command that launches the process.</p>
<p>Another cause might be hostname to ip mapping. IIUC, Gloo would try to resolve the the ip using the hostname. What does the following command return for you?</p>
<pre><code class="lang-auto">getent hosts `hostname`
</code></pre>

ZiyiZhu: <p>Hi <a class="mention" href="/u/mrshenli">@mrshenli</a>,</p>
<p>oh. This may be the problem. In the new servers (10.1.1.101 &amp; 10.1.1.102), getent hosts <code>hostname</code> returns nothing.</p>
<p>I am also testing the torch.distributed with some old servers in my lab now and they can work. In one of the old servers, it does return IPs currently in use.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/a/f/af81d428b2fe9093551180df9a72e018913bed57.png" alt="image" data-base62-sha1="p2BIh89e9YHcTuaUPCCe1QYHWGr" width="368" height="37"><br>
Where I am using <strong>10.0.1.101</strong> and <strong>10.0.1.102</strong> for testing the torch.distributed (old servers).</p>
<p>I will figure this out in the new servers and let you know! Thank you!</p>
<p>Best,<br>
Ziyi</p>

ZiyiZhu: <p>Hi <a class="mention" href="/u/mrshenli">@mrshenli</a>,</p>
<p>Problem solved. Like in the old server, I made <code>10.1.1.101</code> as a host in <code>/etc/hosts</code> and updated it in <code>/etc/hostname</code>. Now if I run <code>getent hosts hostname</code>, <code>10.1.1.101 host1</code> will pop up like in the screenshot below</p>
<aside class="quote no-group quote-modified" data-username="ZiyiZhu" data-post="9" data-topic="66651">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/z/ce7236/48.png" class="avatar"> ZiyiZhu:</div>
<blockquote>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/a/f/af81d428b2fe9093551180df9a72e018913bed57.png" alt="image" data-base62-sha1="p2BIh89e9YHcTuaUPCCe1QYHWGr" width="368" height="37"></p>
</blockquote>
</aside>
<p>However, it happens to be one of the NIC port (<code>eno2</code>)'s IP. What if I want to use another ethernet port which is 10.1.2.101 for another NIC port (<code>eno3</code>), do I need to change the <code>/etc/hostname</code> every time?</p>
<p>Thank you,</p>

mrshenli: <p>Looking at the code, this is not the expected behavior. It would always first try <code>GLOO_SOCKET_IFNAME</code> if that’s available. Somehow, it didn’t pick up the env var.</p>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/blob/945d7a7408891e25bc54a65015724f6d2de644e6/torch/csrc/distributed/c10d/init.cpp#L603-L615" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/945d7a7408891e25bc54a65015724f6d2de644e6/torch/csrc/distributed/c10d/init.cpp#L603-L615" target="_blank" rel="nofollow noopener">pytorch/pytorch/blob/945d7a7408891e25bc54a65015724f6d2de644e6/torch/csrc/distributed/c10d/init.cpp#L603-L615</a></h4>
<pre class="onebox"><code class="lang-cpp"><ol class="start lines" start="603" style="counter-reset: li-counter 602 ;">
<li>char* ifnameEnv = getenv(GLOO_SOCKET_IFNAME_ENV);</li>
<li>if (ifnameEnv) {</li>
<li>  for (const auto&amp; iface : split(',', ifnameEnv)) {</li>
<li>    options.devices.push_back(</li>
<li>        ::c10d::ProcessGroupGloo::createDeviceForInterface(iface));</li>
<li>  }</li>
<li>} else {</li>
<li>  // If no hostname is specified, this function looks up</li>
<li>  // the machine's hostname and returns a device instance</li>
<li>  // associated with the address that the hostname resolves to.</li>
<li>  options.devices.push_back(</li>
<li>      ::c10d::ProcessGroupGloo::createDefaultDevice());</li>
<li>}</li>
</ol></code></pre>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<pre><code class="lang-auto">            char* ifnameEnv = getenv(GLOO_SOCKET_IFNAME_ENV);
            if (ifnameEnv) {
              for (const auto&amp; iface : split(',', ifnameEnv)) {
                options.devices.push_back(
                    ::c10d::ProcessGroupGloo::createDeviceForInterface(iface));
              }
            } else {
              // If no hostname is specified, this function looks up
              // the machine's hostname and returns a device instance
              // associated with the address that the hostname resolves to.
              options.devices.push_back(
                  ::c10d::ProcessGroupGloo::createDefaultDevice());
            }
</code></pre>
<p>Can you try reading the <code>GLOO_SOCKET_IFNAME</code> env var immediately before <code>init_process_group</code> from Python and see if that gives the correct result?</p>
<p>Let me check Gloo code.</p>

mrshenli: <p>Gloo part looks correct to me:</p>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/facebookincubator/gloo/blob/7b58938c5d87380f88a5266035dda1041d45626e/gloo/transport/tcp/device.cc#L142-L162" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/facebookincubator/gloo/blob/7b58938c5d87380f88a5266035dda1041d45626e/gloo/transport/tcp/device.cc#L142-L162" target="_blank" rel="nofollow noopener">facebookincubator/gloo/blob/7b58938c5d87380f88a5266035dda1041d45626e/gloo/transport/tcp/device.cc#L142-L162</a></h4>
<pre class="onebox"><code class="lang-cc"><ol class="start lines" start="142" style="counter-reset: li-counter 141 ;">
<li>std::shared_ptr&lt;transport::Device&gt; CreateDevice(const struct attr&amp; src) {</li>
<li>  struct attr attr = src;</li>
<li>
</li>
<li>  if (attr.iface.size() &gt; 0) {</li>
<li>    // Initialize attributes using network interface name</li>
<li>    lookupAddrForIface(attr);</li>
<li>  } else {</li>
<li>    // Initialize attributes using hostname/IP address</li>
<li>    // If not already specified, use this machine's hostname</li>
<li>    if (attr.hostname.size() == 0) {</li>
<li>      std::array&lt;char, HOST_NAME_MAX&gt; hostname;</li>
<li>      auto rv = gethostname(hostname.data(), hostname.size());</li>
<li>      GLOO_ENFORCE_EQ(rv, 0);</li>
<li>      attr.hostname = hostname.data();</li>
<li>    }</li>
<li>    lookupAddrForHostname(attr);</li>
<li>  }</li>
<li>
</li>
<li>  auto device = std::make_shared&lt;Device&gt;(attr);</li>
<li>  return std::shared_ptr&lt;transport::Device&gt;(device);</li>
<li>}</li>
</ol></code></pre>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>


mrshenli: <p>Another way to test, is to use some non-exist interface, e.g.</p>
<pre><code class="lang-auto">export GLOO_SOCKET_IFNAME=nonexist
</code></pre>
<p>And then check if <code>init_process_group</code> throws the follow error for you:</p>
<pre><code class="lang-auto">    dist.init_process_group("gloo", rank=rank, world_size=world_size)
  File "/scratch/shenli/pytorch/torch/distributed/distributed_c10d.py", line 425, in init_process_group
    _default_pg = _new_process_group_helper(
  File "/scratch/shenli/pytorch/torch/distributed/distributed_c10d.py", line 499, in _new_process_group_helper
    pg = ProcessGroupGloo(
RuntimeError: [enforce fail at ../third_party/gloo/gloo/transport/tcp/device.cc:83] ifa != nullptr. Unable to find address for: nonexist
</code></pre>

ZiyiZhu: <p>Hi <a class="mention" href="/u/mrshenli">@mrshenli</a>,</p>
<p>After I</p>
<pre><code class="lang-auto">export GLOO_SOCKET_IFNAME=nonexist
</code></pre>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/c/9c09774b18f4a4382973db0c9f3e29d51edfbc9f.png" data-download-href="https://discuss.pytorch.org/uploads/default/9c09774b18f4a4382973db0c9f3e29d51edfbc9f" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/9/c/9c09774b18f4a4382973db0c9f3e29d51edfbc9f.png" alt="image" data-base62-sha1="mgmLvbZj5Kyb2NeKWY9S5iCXZ8P" width="690" height="441" data-dominant-color="F7F8F8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1107×708 36.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>This is the error I got. Does it seem that it bypasses the <code>nonexist</code> and look at some others? If I add the master_address then it will just hang there for the second rank to come in.</p>
<p>Thanks,</p>

mrshenli: <p>can you keep/uncomment the <code>init_method</code> line or set <code>MASTER_ADDR</code> and <code>MASTER_PORT</code>? It seems failed during Python land arg checking due to missing master addr/port, before entering C++ pybind methods.</p>

ZiyiZhu: <p>Hi <a class="mention" href="/u/mrshenli">@mrshenli</a>,</p>
<p>I guess I found the problem. If I do <code>export GLOO_SOCKET_IFNAME=nonexist</code> in the terminal then it does not become an environment variable in the Jupter Notebook. But see it in Python launched from that terminal directly.</p>
<p>So I guess I have to do the other way around and set that in the JupyterNotebook explicitly? Here is the result if I do what you suggested.</p>
<aside class="quote no-group" data-username="mrshenli" data-post="15" data-topic="66651" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mrshenli/48/12220_2.png" class="avatar"> mrshenli:</div>
<blockquote>
<p>can you keep/uncomment the <code>init_method</code> line or set <code>MASTER_ADDR</code> and <code>MASTER_PORT</code> ? It seems failed during Python land arg checking due to missing master addr/port, before entering C++ pybind methods.</p>
</blockquote>
</aside>
<pre><code class="lang-auto">import torch.distributed as dist
import os
​
print(os.environ.get('GLOO_SOCKET_IFNAME'))
​
os.environ['MASTER_ADDR'] = 'localhost'
os.environ['MASTER_PORT'] = '23456'
​
os.environ['GLOO_SOCKET_IFNAME']='nonexist'
print(os.environ.get('GLOO_SOCKET_IFNAME'))
# on rank 0
dist.init_process_group(
    backend = "gloo",
    init_method = 'tcp://10.1.1.101:29500',
    rank = 0,
    world_size = 1
)
​
​
​
None
nonexist
----------------------------------------------------------------------
RuntimeError                         Traceback (most recent call last)
&lt;ipython-input-1-ad5d77a63395&gt; in &lt;module&gt;
     14     init_method = 'tcp://10.1.1.101:29500',
     15     rank = 0,
---&gt; 16     world_size = 1
     17 )
     18 

~/anaconda3/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py in init_process_group(backend, init_method, timeout, world_size, rank, store, group_name)
    401             store,
    402             group_name=group_name,
--&gt; 403             timeout=timeout)
    404 
    405     _pg_group_ranks[_default_pg] = {i: i for i in range(_default_pg.size())}

~/anaconda3/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py in _new_process_group_helper(world_size, rank, group_ranks, backend, store, group_name, timeout)
    469                 rank,
    470                 world_size,
--&gt; 471                 timeout=timeout)
    472             _pg_map[pg] = (Backend.GLOO, store)
    473             _pg_names[pg] = group_name

RuntimeError: [enforce fail at /opt/conda/conda-bld/pytorch_1587428398394/work/third_party/gloo/gloo/transport/tcp/device.cc:83] ifa != nullptr. Unable to find address for: nonexist
</code></pre>
<p>Thanks,<br>
Ziyi</p>

mrshenli: <p>Ah, I see. Yep, setting it directly in notebook should work I think.</p>

ZiyiZhu: <p>Hi <a class="mention" href="/u/mrshenli">@mrshenli</a>,</p>
<p>To follow up with the issue below, I wonder if you could let me know more about why currently we cannot use the “nccl” as a backend for the communication for RCP? We have to explicitly copy the data to CPU and then do the transmission.</p>
<p>What are the concerns that the RPC does not do something similar to the DDP that has the GPU directly access to the NIC for the model parallel algorithm?</p>
<p>Thank you</p>
<aside class="quote no-group" data-username="mrshenli" data-post="6" data-topic="66651">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mrshenli/48/12220_2.png" class="avatar"> mrshenli:</div>
<blockquote>
<p>Hey <a class="mention" href="/u/ziyizhu">@ZiyiZhu</a> Are you trying to run this with RPC? Currently <code>init_rpc</code> does not work together with <code>init_process_group</code> . There are work around to create non-default process groups. Or we can also add a fix to <code>init_rpc</code> if necessary. This is the tracking issue:</p>
</blockquote>
</aside>

mrshenli: <blockquote>
<p>To follow up with the issue below, I wonder if you could let me know more about why currently we cannot use the “nccl” as a backend for the communication for RCP?</p>
</blockquote>
<p>This is because NCCL does not support p2p (send/recv) communication yet when we develop RPC. It is possible to use NCCL broadcast to mimic that send/recv, but that’s too hackish.</p>
<p>The p2p comm is coming to NCCL in v2.7. When that is ready, we probably can add it to <code>ProcessGroupAgent</code> or the new <code>TensorPipeAgent</code> (the latter is a more performant RPC agent implementation and should be able to use the best channels, e.g., IB/ETH/NvLink/etc.). See this PR: <a href="https://github.com/pytorch/pytorch/pull/35483" rel="noopener nofollow ugc">https://github.com/pytorch/pytorch/pull/35483</a></p>
<blockquote>
<p>We have to explicitly copy the data to CPU and then do the transmission.</p>
</blockquote>
<p>For Gloo backend even if application don’t copy the tensor from CUDA to CPU, Gloo would need to do that internally anyway. Hence, this explicit copy in application not a perf limitation when using Gloo backend.</p>
<p>We used to do that GPU-to-CPU copy implicitly in v1.4, but later realized that applications could run into unexpected errors if the destination device is not available on the callee. E.g., when I do <code>rpc.rpc_sync(...., args=(torch.zeros(2).to(3),))</code> and if <code>cuda:3</code> is not available on the callee, it would throw an error. So, we decided to make it explicit for applications.</p>
<blockquote>
<p>What are the concerns that the RPC does not do something similar to the DDP that has the GPU directly access to the NIC for the model parallel algorithm?</p>
</blockquote>
<p>From the API level, the difference is that DDP is supposed to run on a set of homogeneous servers, and RPC should be able to support heterogeneous clusters. So the device mismatch in RPC can be common. We are adding explicit device placement support (sth. similar to <code>map_location</code> on <code>torch.save</code> and <code>torch.load</code>) to the RPC API. <a href="https://github.com/pytorch/pytorch/issues/33991" rel="noopener nofollow ugc">This</a> is an early issue to track. <a class="mention" href="/u/osalpekar">@osalpekar</a> is working on a design RFC for that. Look forward to hear your comments when that RFC is posted. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

ZiyiZhu: <p>Thank you very much for your detailed explanations! I agree that explicit can avoid lots of unexpected errors and really look forward to seeing RFC design.</p>
<p>Best,<br>
Ziyi</p>