RPC + Torchrun hangs in ProcessGroupGloo

patataman: <p>Hello, I’m trying to run RPC with multiples nodes to implement model parallelism. When using only 1 node (local execution) everything works OK. However when trying to use other nodes with torchrun, the script hangs in rpc.init_rpc.</p>
<p>I have checked other threads and solutions, but usually they are in a 1 node with multiples GPUs (like this <a href="https://github.com/pytorch/pytorch/issues/85607" class="inline-onebox" rel="noopener nofollow ugc">[Distributed: RPC] Failed to initialize RPC with &gt;18 workers · Issue #85607 · pytorch/pytorch · GitHub</a> and this is not my case, only 1 GPU per node) or they use multiple nodes + multiple GPUs but the question and solution doesn’t include any minimal code (<a href="https://discuss.pytorch.org/t/torchrun-launched-jobs-hang-on-multiple-machines/145948" class="inline-onebox">Torchrun launched jobs hang on multiple machines</a>) So I cannot really compare.</p>
<p>Applying this “hack” (<a href="https://discuss.pytorch.org/t/remove-the-dependency-of-gloo-from-rpc/156573" class="inline-onebox">Remove the dependency of Gloo from RPC</a>) I manage to get it execute, but with some errors at the end (Highly probable because a race condition when finalizing the workers) Therefore, I’m pretty sure the problem is related with ProcessGroupGloo.</p>
<p>I’m using my university cluster, so <strong>I don’t have admin privileges</strong> and <strong>most of ports are closed</strong> for security reasons.</p>
<p>My minimal code is the same as the Github issue:</p>
<pre><code class="lang-auto"># rpc_test.py
# https://github.com/pytorch/pytorch/issues/85607

import os
import random

import numpy as np
import torch

import torch.distributed.rpc as rpc


def worker_init():
    rank = int(os.environ['RANK'])

    random.seed(rank)
    np.random.seed(rank)
    torch.manual_seed(rank)
    print(f'Rank {rank}')


def main():
    rpc_backend_options = rpc.TensorPipeRpcBackendOptions(
        init_method='tcp://compute-2-2:53555'
    )

    rank = int(os.environ['RANK'])
    world_size = int(os.environ['WORLD_SIZE'])
    
    rpc.init_rpc(name=f'worker{rank}', rank=rank, world_size=world_size,
        rpc_backend_options=rpc_backend_options
    )
    print("pos init")
    worker_init()
    
    # no-op
    rpc.shutdown()


if __name__ == '__main__':
    main()
</code></pre>
<p><strong>If not using rpc_backend_options</strong>: Gloo tries to use a forbidden port.</p>
<p>I searched and apparently, I might be able to tell Gloo which port use with the <code>rpc_backend_options</code> parameter. However, when using it (as in the code) with a valid port, the process is there indefinitely.</p>
<p>I run the scripts in both nodes with:<br>
Node 0: <code>torchrun --nnodes=2 --nproc_per_node=1 --node_rank=0 --rdzv_id=0 --rdzv_endpoint=compute-2-2:53554 minimal-rpc.py</code><br>
Node 1:  <code>torchrun --nnodes=2 --nproc_per_node=1 --node_rank=1 --rdzv_id=0 --rdzv_endpoint=compute-2-2:53554 minimal-rpc.py</code></p>
<p>I have tried also without the rdzv params and just using master_address and master_port but nothing seems to work.</p>
<p>I can confirm that the torchrun works, because I have successfully executed other scripts with torchrun + DDP between these machines, but DDP doesn’t solve my memory, so I want to try with model parallelism.</p>

patataman: <p>Seems like using the solution proposed here <a href="https://discuss.pytorch.org/t/getting-gloo-error-when-connecting-server-and-client-over-vpn-from-different-systems/191076/2" class="inline-onebox">Getting Gloo error when connecting server and client over VPN from different systems - #2 by selineni</a> fix the hanging…</p>
<p>Nothing like asking after fighting with the problem for a few hours to find a solution short after.</p>
<p>EDIT —</p>
<p>For completeness:</p>
<pre><code class="lang-auto"># rpc_test.py
# https://github.com/pytorch/pytorch/issues/85607

import os
import random

import numpy as np
import torch

import torch.distributed.rpc as rpc


def worker_init():
    rank = int(os.environ['RANK'])

    random.seed(rank)
    np.random.seed(rank)
    torch.manual_seed(rank)
    print(f'Rank {rank}')


def main():
    rank = int(os.environ['RANK'])
    world_size = int(os.environ['WORLD_SIZE'])
    
    print("initing rpc")
    rpc.init_rpc(name=f'worker{rank}', rank=rank, world_size=world_size)
    print("rpc inited - worker init")
    worker_init()
    print("worker inited")
    
    # no-op
    rpc.shutdown()


if __name__ == '__main__':
    main()
</code></pre>
<p>And the execution calls</p>
<pre><code class="lang-auto">TP_SOCKET_IFNAME=&lt;interface&gt; GLOO_SOCKET_IFNAME=&lt;interface&gt; torchrun --nnodes=2 --nproc_per_node=1 --node_rank=0 --rdzv_id=0 --rdzv_endpoint=compute-2-2:53554 minimal-rpc.py

TP_SOCKET_IFNAME=&lt;interface&gt; GLOO_SOCKET_IFNAME=&lt;interface&gt; torchrun --nnodes=2 --nproc_per_node=1 --node_rank=1 --rdzv_id=0 --rdzv_endpoint=compute-2-2:53554 minimal-rpc.py
</code></pre>