Getting Gloo error when connecting server and client over VPN from different systems

username: selineni
cooked: <p>i running rpc_sync between 1 server and 2 nodes connected over VPN.  when i run them as containers from single platform (like server or laptop), there is no error.  but if i divide the server container and node containers on a laptop and azure VM both connected to VPN, there is an error " RuntimeError: Gloo connectFullMesh failed with […/third_party/gloo/gloo/transport/tcp/pair.cc:144] no error"</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/c/8cf1086e95a341c4c844134f63126a441d32d235.png" data-download-href="https://discuss.pytorch.org/uploads/default/8cf1086e95a341c4c844134f63126a441d32d235" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/8/c/8cf1086e95a341c4c844134f63126a441d32d235.png" alt="image" data-base62-sha1="k6PfUaWrr55uytpeoCfhjy5dzbn" width="690" height="284" data-dominant-color="1B1C1D"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1722×710 46.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: selineni
cooked: <p>Solved this by adding os.environ[“TP_SOCKET_IFNAME”]=“tun0” os.environ[“GLOO_SOCKET_IFNAME”]=“tun0” to where i called init_rpc.  We were also tunnelling the communication through VPN.</p>

username: daviddwlee84
cooked: <aside class="onebox githubissue" data-onebox-src="https://github.com/pytorch/pytorch/issues/68726#issuecomment-1813807190">
  <header class="source">

      <a href="https://github.com/pytorch/pytorch/issues/68726#issuecomment-1813807190" target="_blank" rel="noopener nofollow ugc">github.com/pytorch/pytorch</a>
  </header>

  <article class="onebox-body">
    <div class="github-row">
  <div class="github-icon-container" title="Issue">
	  <svg width="60" height="60" class="github-icon" viewBox="0 0 14 16" aria-hidden="true"><path fill-rule="evenodd" d="M7 2.3c3.14 0 5.7 2.56 5.7 5.7s-2.56 5.7-5.7 5.7A5.71 5.71 0 0 1 1.3 8c0-3.14 2.56-5.7 5.7-5.7zM7 1C3.14 1 0 4.14 0 8s3.14 7 7 7 7-3.14 7-7-3.14-7-7-7zm1 3H6v5h2V4zm0 6H6v2h2v-2z"></path></svg>
  </div>

  <div class="github-info-container">
    <h4>
      <a href="https://github.com/pytorch/pytorch/issues/68726#issuecomment-1813807190" target="_blank" rel="noopener nofollow ugc">Failed to create Gloo new group after initialized with NCCL</a>
    </h4>

    <div class="github-info">
      <div class="date">
        opened <span class="discourse-local-date" data-format="ll" data-date="2021-11-22" data-time="10:20:41" data-timezone="UTC">10:20AM - 22 Nov 21 UTC</span>
      </div>


      <div class="user">
        <a href="https://github.com/zhuzilin" target="_blank" rel="noopener nofollow ugc">
          <img alt="zhuzilin" src="https://avatars.githubusercontent.com/u/10428324?v=4" class="onebox-avatar-inline" width="20" height="20">
          zhuzilin
        </a>
      </div>
    </div>

    <div class="labels">
        <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">
          oncall: distributed
        </span>
        <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">
          module: nccl
        </span>
    </div>
  </div>
</div>

  <div class="github-row">
    <p class="github-body-container">## 🐛 Bug

In our project [Tencent/PatrickStar](https://github.com/Tencent/Patr<span class="show-more-container"><a href="" rel="noopener" class="show-more">…</a></span><span class="excerpt hidden">ickStar), we need to create a NCCL comm group and a Gloo comm group in order to utilize both GPU reduce scatter and CPU comm operations. However, it seems that after initialized with NCCL, the Gloo group could not detect the master address and master port, but instead using localhost (127.0.0.1).

I'm afraid the reason is that the NCCL store and Gloo store are not compatible with each other so that the new Gloo group could not read the master addr saved by NCCL group.

The relevant error message is:

```bash
Traceback (most recent call last):
Traceback (most recent call last):
  File "test_new_group.py", line 4, in &lt;module&gt;
  File "test_new_group.py", line 4, in &lt;module&gt;
        cpu_comm = torch.distributed.new_group(backend="gloo")cpu_comm = torch.distributed.new_group(backend="gloo")

  File "/opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py", line 2843, in new_group
  File "/opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py", line 2843, in new_group
        pg = _new_process_group_helper(pg = _new_process_group_helper(

  File "/opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py", line 668, in _new_process_group_helper
  File "/opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py", line 668, in _new_process_group_helper
    pg = ProcessGroupGloo(prefix_store, rank, world_size, timeout=timeout)
    RuntimeErrorpg = ProcessGroupGloo(prefix_store, rank, world_size, timeout=timeout): 
[/opt/pytorch/pytorch/third_party/gloo/gloo/transport/tcp/pair.cc:799] connect [127.0.0.1]:1453: Connection refused
RuntimeError: [/opt/pytorch/pytorch/third_party/gloo/gloo/transport/tcp/pair.cc:799] connect [127.0.0.1]:4005: Connection refused
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 0 (pid: 6027) of binary: /opt/conda/bin/python3
```

## To Reproduce

```python
# test_new_group.py
import torch

torch.distributed.init_process_group(backend="nccl")
cpu_comm = torch.distributed.new_group(backend="gloo")
```

Run :

```bash
python3 -m torch.distributed.launch --nproc_per_node=1 \
               --nnodes=2 --master_addr=$MASTER_ADDR --master_port=$MASTER_PORT \
              --node_rank=$NODE_RANK \
              test_new_group.py
```

Note that the code should work with only one machine or the second new group is created with backend "nccl".

## Expected behavior

User should be allowed to create 2 comm group of different types.

## Environment

I'm using the NGC container: `nvcr.io/nvidia/pytorch:21.09-py3`

 - PyTorch Version (e.g., 1.0): 1.10.0
 - OS (e.g., Linux): linux
 - Python version: 3.8
 - CUDA/cuDNN version: 11.4
 - GPU models and configuration: A100


cc @pietern @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @SciPioneer @H-Huang</span></p>
  </div>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>