Can't init rpc on more than one machine

username: yueyilia
cooked: <p>When I use RPC on more than one machine, the code will get stuck in init_rpc. I try opt.num_servers == 1, it works when opt.num_envs_each_server&lt;30. I use python 3.7.10 and pytorch 1.8.1. Is there something wrong with this code？</p>
<pre><code class="lang-python"># server
def main():
    opt = Options().parse()
    if opt.num_servers == 1:
        num = NUM_TRAINER_PROCESSES + opt.num_envs_each_server
    else:
        num = NUM_TRAINER_PROCESSES
    mp.spawn(run_worker, args=(opt,), nprocs=num, join=True)


def run_worker(idx, opt):
    os.environ["MASTER_ADDR"] = opt.address
    os.environ["MASTER_PORT"] = opt.port
    backend = rpc.BackendType.TENSORPIPE
    rpc_backend_options = rpc.TensorPipeRpcBackendOptions(
        num_worker_threads=1, rpc_timeout=60
    )

    if idx == 0:
        s = socket.socket()
        s.bind((opt.address, opt.env_port))
        s.listen(opt.num_envs)
        for i in tqdm.trange(opt.num_envs, ascii=True):
            c, addr = s.accept()
            c.send(str(i).encode("utf-8"))
        time.sleep(1)
        s.close()

        logger = utils.get_logger("agent")
        logger.info("init rpc for ppo agent")
        rpc.init_rpc(
            "ppo_agent",
            rank=0,
            world_size=opt.world_size,
            backend=backend,
            rpc_backend_options=rpc_backend_options,
        )
    else:
        while 1:
            try:
                s = socket.socket()
                s.connect((opt.address, opt.env_port))
                rank = s.recv(1024)
                rank = int(rank.decode())
                s.close()
                break
            except:
                time.sleep(1)
                pass
        logger = utils.get_logger("env_{}".format(rank))
        logger.info("init rpc for env {}".format(rank))
        rpc.init_rpc(
            "env_{}".format(rank),
            rank=NUM_TRAINER_PROCESSES + rank,
            world_size=opt.world_size,
            backend=backend,
            rpc_backend_options=rpc_backend_options,
        )
        logger.info("env {} is waiting".format(rank))

    rpc.shutdown()

# client
def main():
    opt = Options().parse()
    mp.spawn(run_worker, args=(opt,), nprocs=opt.num_envs_each_server, join=True)


def run_worker(idx, opt):
    os.environ["MASTER_ADDR"] = opt.address
    os.environ["MASTER_PORT"] = opt.port
    backend = rpc.BackendType.TENSORPIPE
    rpc_backend_options = rpc.TensorPipeRpcBackendOptions(
        num_worker_threads=1, rpc_timeout=60
    )
    logger = utils.get_logger("{}".format(idx))

    while 1:
        try:
            s = socket.socket()
            s.connect((opt.address, opt.env_port))
            rank = s.recv(1024)
            rank = int(rank.decode())
            s.close()
            break
        except Exception as e:
            logger.info(e)
            time.sleep(1)
            pass

    logger.info("init rpc for env {}".format(rank))
    rpc.init_rpc(
        "env_{}".format(rank),
        rank=NUM_TRAINER_PROCESSES + rank,
        world_size=opt.world_size,
        backend=backend,
        rpc_backend_options=rpc_backend_options,
    )
    logger.info("env {} is waiting".format(rank))

    rpc.shutdown()
</code></pre>

username: mrshenli
cooked: <p>Hey <a class="mention" href="/u/yueyilia">@yueyilia</a>, would I be correct if I assume the <code>world_size</code> is <code>NUM_TRAINER_PROCESSES + 2 * num_envs_each_server</code>? Have you trired increasing the value of <code>num_worker_threads</code> to, say, 128?</p>
<p>BTW, when you replace <code>init_rpc</code> with <code>init_process_group</code> and use the same rank and world_size, port, and, address, does it work?</p>

username: yueyilia
cooked: <p>Hey, thanks for your reply.<br>
Specifically, in my simplified test example, opt.world_size=33, opt.num_servers = 2, opt.num_envs_each_server=16, NUM_TRAINER_PROCESSES=1.<br>
It does not work when modifying the value of num_worker_threads.<br>
When using init_process_group, clients are successfully initialized, but the server still gets stuck in init_process_group.</p>

username: mrshenli
cooked: <aside class="quote no-group" data-username="yueyilia" data-post="3" data-topic="130007">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/y/df788c/48.png" class="avatar"> yueyilia:</div>
<blockquote>
<p>When using init_process_group, clients are successfully initialized, but the server still gets stuck in init_process_group.</p>
</blockquote>
</aside>
<p>Hmm, this means some configuration might be wrong (<code>init_process_group</code> has been widely used in multi-machine training). The behavior suggests the client thinks all peers are joined, but the server is still waiting for some peers.</p>
<p>Curious, why do you need to use your own sockets to communicate ranks? Could you please print all ranks and see if those are expected?</p>
<p>If that still doesn’t give a clue, could you please share a min repro?</p>

username: yueyilia
cooked: <aside class="quote no-group" data-username="mrshenli" data-post="4" data-topic="130007">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mrshenli/48/12220_2.png" class="avatar"> mrshenli:</div>
<blockquote>
<p>Curious, why do you need to use your own sockets to communicate ranks? Could you please print all ranks and see if those are expected?</p>
</blockquote>
</aside>
<p>I plan to use 100 servers to build thousands of environments. I don’t know how to assign a unique rank without sockets. Is there any usage that I don’t know?<br>
In fact, I once ran the same code on pytorch1.6. There was no problem with the startup. But I could only run up to 1000 environments because of the port limit (can see <a href="https://discuss.pytorch.org/t/when-i-use-1024-nodes-in-rpc-i-meet-runtimeerror-listen-address-already-in-use/95642/6" class="inline-onebox">When I use 1024 nodes in rpc, I meet RuntimeError "listen: Address already in use" - #6 by yueyilia</a>). I don’t know why the error occurred now.<br>
I still haven’t found a solution, the following is a min repro.</p>
<pre data-code-wrap="python"><code class="lang-python">import os
import time
import tqdm
import socket
import argparse
import logging
import torch
import torch.multiprocessing as mp
import torch.distributed
import torch.distributed.rpc as rpc

parser = argparse.ArgumentParser()
parser.add_argument("--mode", type=str, default="trainer")
parser.add_argument("--address", type=str, default="10.90.224.127", help="")
parser.add_argument("--port", type=str, default="10088", help="")
parser.add_argument("--rank_port", type=int, default="10099", help="")
parser.add_argument("--num_envs_each_server", type=int, default=16, help="")
parser.add_argument("--num_servers", type=int, default=2, help="")
parser.add_argument("--num_envs", type=int, default=32, help="")
parser.add_argument("--world_size", type=int, default=33, help="")
opt = parser.parse_args()


def main():
    if opt.mode == "trainer":
        mp.spawn(run_worker, args=(opt,), nprocs=1, join=True)
    else:
        mp.spawn(run_worker, args=(opt,), nprocs=opt.num_envs_each_server, join=True)


def run_worker(idx, opt):
    os.environ["MASTER_ADDR"] = opt.address
    os.environ["MASTER_PORT"] = opt.port
    backend = rpc.BackendType.TENSORPIPE
    rpc_backend_options = rpc.TensorPipeRpcBackendOptions(
        num_worker_threads=1, rpc_timeout=60
    )

    if opt.mode == "trainer":
        s = socket.socket()
        s.bind((opt.address, opt.rank_port))
        s.listen(opt.num_envs)
        for i in tqdm.trange(opt.num_envs, ascii=True):
            c, addr = s.accept()
            c.send(str(i).encode("utf-8"))
        time.sleep(1)
        s.close()

        logger = get_logger("agent")
        logger.info("init rpc for ppo agent")
        rpc.init_rpc(
            "ppo_agent",
            rank=0,
            world_size=opt.world_size,
            backend=backend,
            rpc_backend_options=rpc_backend_options,
        )
        # torch.distributed.init_process_group(
        #     backend="gloo", rank=0, world_size=opt.world_size,
        # )
        logger.info("end")
    else:
        while 1:
            try:
                s = socket.socket()
                s.connect((opt.address, opt.rank_port))
                rank = s.recv(1024)
                rank = int(rank.decode())
                s.close()
                break
            except:
                time.sleep(1)
                pass
        logger = get_logger("env_{}".format(rank))
        logger.info("init rpc for env {}".format(rank))
        rpc.init_rpc(
            "env_{}".format(rank),
            rank=1 + rank,
            world_size=opt.world_size,
            backend=backend,
            rpc_backend_options=rpc_backend_options,
        )
        # torch.distributed.init_process_group(
        #     backend="gloo", rank=1 + rank, world_size=opt.world_size,
        # )
        logger.info("env {} is waiting".format(rank))

    rpc.shutdown()


def get_logger(name="", level=logging.INFO, stream=True, file=None):
    try:
        import absl.logging

        logging.root.removeHandler(absl.logging._absl_handler)
        absl.logging._warn_preinit_stderr = False
    except Exception as e:
        print("failed to fix absl logging bug", e)
        pass

    logger = logging.getLogger(name)
    logger.setLevel(level)

    if stream:
        stream_handler = logging.StreamHandler()
        stream_formatter = logging.Formatter("%(asctime)s - %(message)s")
        stream_handler.setFormatter(stream_formatter)
        logger.addHandler(stream_handler)

    if file:
        path = os.path.join(file, name + ".log")
        file_handler = logging.handlers.RotatingFileHandler(
            path, "a", 100 * 1024 * 1024, 1, encoding="utf-8"
        )
        file_formatter = logging.Formatter(
            "%(asctime)s %(levelname)s [%(filename)s: %(lineno)d] [%(processName)s: %(process)d] - %(message)s"
        )
        file_handler.setFormatter(file_formatter)
        logger.addHandler(file_handler)

    return logger


if __name__ == "__main__":
    main()

</code></pre>

username: mrshenli
cooked: <p>pulling in store rendezvous expert <a class="mention" href="/u/h-huang">@H-Huang</a> <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=10" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: H-Huang
cooked: <p>Hi <a class="mention" href="/u/yueyilia">@yueyilia</a>, I tried your code locally (python 3.8.5 and nightly build of pytorch which is 1.10 now), it worked for me and is able to get past init_rpc. I think the issue may be with how your cluster of machines is configured where they cannot establish connections with each other, but I’m not exactly sure.</p>
<aside class="quote no-group" data-username="yueyilia" data-post="5" data-topic="130007">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/y/df788c/48.png" class="avatar"> yueyilia:</div>
<blockquote>
<p>I plan to use 100 servers to build thousands of environments. I don’t know how to assign a unique rank without sockets. Is there any usage that I don’t know?</p>
</blockquote>
</aside>
<p>You can modify your script to use torchleastic via <a href="https://pytorch.org/docs/master/elastic/run.html#module-torch.distributed.run" class="inline-onebox" rel="noopener nofollow ugc">torchrun (Elastic Launch) — PyTorch master documentation</a>. This essentially performs the work you are doing by having all nodes rendezvous then assign ranks to each process, with added fault tolerance and elasticity if you choose to enable it. You need to modify your script so that it acts as a single process, then run the command:</p>
<pre><code class="lang-auto">python -m torch.distributed.run
    --nnodes=$NUM_NODES
    --nproc_per_node=$NUM_TRAINERS
    --rdzv_id=$JOB_ID
    --rdzv_backend=c10d
    --rdzv_endpoint=$HOST_NODE_ADDR
    YOUR_TRAINING_SCRIPT.py (--arg1 ... train script args...)
</code></pre>
<p>The rank is automatically assigned with torchelastic. Each process will be able to read it’s own rank as an environment variable (e.g. <code>rank = os.environ["RANK"]</code>). Here are the list of the environment variables that get be passed into your script: <a href="https://pytorch.org/docs/master/elastic/run.html#:~:text=Environment%20Variables%3A" class="inline-onebox" rel="noopener nofollow ugc">torchrun (Elastic Launch) — PyTorch master documentation</a></p>