RPC init fails and crashes when world_size is greater than 18
Labels: oncall: distributed, triaged, module: rpc

### 🐛 Describe the bug

Hi! When I use RPC, I find that when the `world_size` is greater than 18, the program will crash. I've tested it on two servers and got the same result.

Program to reproduce:

```python
import torch
import torch.multiprocessing as mp
import torch.distributed.rpc as rpc

def init_process(rank, tot_processes):
    print(f'here is rank {rank}', flush=True)

    rpc_backend_options = rpc.TensorPipeRpcBackendOptions(
        init_method=f'tcp://localhost:52521'
    )
    rpc.init_rpc(
        name=f'test_{rank}', rank=rank, world_size=tot_processes,
        rpc_backend_options=rpc_backend_options,
    )
    print(f'rank {rank} init successfully', flush=True)
    rpc.shutdown()


def main() -> None:
    mp.set_start_method('spawn')
    tot_processes = 19
    print(f'spawning {tot_processes} processes...')
    mp.spawn(
        fn=init_process,
        args=(tot_processes, ),
        nprocs=tot_processes,
        join=True,
    )

if __name__ == '__main__':
    main()
```

When `tot_processes = 18`, the program can exit without any error. But if `tot_processes = 19`, it will crash as below.

```shell
❯ python rpc.py
spawning 19 processes...
here is rank 7
here is rank 12
here is rank 4
here is rank 11
here is rank 2
here is rank 6
here is rank 15
here is rank 10
here is rank 9
here is rank 14
here is rank 8
here is rank 5
here is rank 13
here is rank 17
here is rank 3
here is rank 1
here is rank 16
here is rank 0
here is rank 18
[W tensorpipe_agent.cpp:863] RPC agent for test_12 encountered error when sending outgoing request #0 to test_0: connect: Resource temporarily unavailable (this error originated at tensorpipe/common/socket.cc:114)
[W tensorpipe_agent.cpp:863] RPC agent for test_5 encountered error when sending outgoing request #0 to test_0: connect: Resource temporarily unavailable (this error originated at tensorpipe/common/socket.cc:114)
[W tensorpipe_agent.cpp:863] RPC agent for test_16 encountered error when sending outgoing request #0 to test_0: connect: Resource temporarily unavailable (this error originated at tensorpipe/common/socket.cc:114)
[W tensorpipe_agent.cpp:863] RPC agent for test_15 encountered error when sending outgoing request #0 to test_0: connect: Resource temporarily unavailable (this error originated at tensorpipe/common/socket.cc:114)
[W tensorpipe_agent.cpp:863] RPC agent for test_3 encountered error when sending outgoing request #0 to test_0: connect: Resource temporarily unavailable (this error originated at tensorpipe/common/socket.cc:114)
[W tensorpipe_agent.cpp:863] RPC agent for test_13 encountered error when sending outgoing request #0 to test_0: connect: Resource temporarily unavailable (this error originated at tensorpipe/common/socket.cc:114)
[W tensorpipe_agent.cpp:492] RPC agent for test_0 encountered error when accepting incoming pipe: sendmsg: Broken pipe (this error originated at tensorpipe/common/socket.h:105)
[W tensorpipe_agent.cpp:682] RPC agent for test_0 encountered error when reading incoming request from test_16: sendmsg: Broken pipe (this error originated at tensorpipe/common/socket.h:105)
[W tensorpipe_agent.cpp:682] RPC agent for test_0 encountered error when reading incoming request from test_1: sendmsg: Broken pipe (this error originated at tensorpipe/common/socket.h:105)
[W tensorpipe_agent.cpp:682] RPC agent for test_0 encountered error when reading incoming request from test_3: sendmsg: Broken pipe (this error originated at tensorpipe/common/socket.h:105)
[W tensorpipe_agent.cpp:682] RPC agent for test_0 encountered error when reading incoming request from test_9: sendmsg: Broken pipe (this error originated at tensorpipe/common/socket.h:105)
[W tensorpipe_agent.cpp:682] RPC agent for test_0 encountered error when reading incoming request from test_11: sendmsg: Broken pipe (this error originated at tensorpipe/common/socket.h:105)
[W tensorpipe_agent.cpp:682] RPC agent for test_0 encountered error when reading incoming request from test_2: sendmsg: Broken pipe (this error originated at tensorpipe/common/socket.h:105)
[W tensorpipe_agent.cpp:682] RPC agent for test_0 encountered error when reading incoming request from test_18: sendmsg: Broken pipe (this error originated at tensorpipe/common/socket.h:105)
[W tensorpipe_agent.cpp:682] RPC agent for test_0 encountered error when reading incoming request from test_4: sendmsg: Broken pipe (this error originated at tensorpipe/common/socket.h:105)
[W tensorpipe_agent.cpp:682] RPC agent for test_0 encountered error when reading incoming request from test_17: sendmsg: Broken pipe (this error originated at tensorpipe/common/socket.h:105)
[W tensorpipe_agent.cpp:682] RPC agent for test_0 encountered error when reading incoming request from test_14: sendmsg: Broken pipe (this error originated at tensorpipe/common/socket.h:105)
[W tensorpipe_agent.cpp:682] RPC agent for test_0 encountered error when reading incoming request from test_8: sendmsg: Broken pipe (this error originated at tensorpipe/common/socket.h:105)
[W tensorpipe_agent.cpp:682] RPC agent for test_0 encountered error when reading incoming request from test_7: sendmsg: Broken pipe (this error originated at tensorpipe/common/socket.h:105)
[W tensorpipe_agent.cpp:682] RPC agent for test_0 encountered error when reading incoming request from test_12: sendmsg: Broken pipe (this error originated at tensorpipe/common/socket.h:105)
[W tensorpipe_agent.cpp:682] RPC agent for test_0 encountered error when reading incoming request from test_6: sendmsg: Broken pipe (this error originated at tensorpipe/common/socket.h:105)
[W tensorpipe_agent.cpp:682] RPC agent for test_0 encountered error when reading incoming request from test_5: sendmsg: Broken pipe (this error originated at tensorpipe/common/socket.h:105)
[W tensorpipe_agent.cpp:682] RPC agent for test_0 encountered error when reading incoming request from test_10: sendmsg: Broken pipe (this error originated at tensorpipe/common/socket.h:105)
[W tensorpipe_agent.cpp:682] RPC agent for test_0 encountered error when reading incoming request from test_15: sendmsg: Broken pipe (this error originated at tensorpipe/common/socket.h:105)
[W tensorpipe_agent.cpp:682] RPC agent for test_0 encountered error when reading incoming request from test_13: sendmsg: Broken pipe (this error originated at tensorpipe/common/socket.h:105)
[W tensorpipe_agent.cpp:863] RPC agent for test_6 encountered error when sending outgoing request #0 to test_0: async error on socket: Connection reset by peer (this error originated at tensorpipe/transport/shm/connection_impl.cc:187)
[W tensorpipe_agent.cpp:887] RPC agent for test_8 encountered error when reading incoming response from test_0: async error on socket: Connection reset by peer (this error originated at tensorpipe/transport/shm/connection_impl.cc:187)
[W tensorpipe_agent.cpp:887] RPC agent for test_11 encountered error when reading incoming response from test_0: async error on socket: Connection reset by peer (this error originated at tensorpipe/transport/shm/connection_impl.cc:187)
[W tensorpipe_agent.cpp:863] RPC agent for test_10 encountered error when sending outgoing request #0 to test_0: async error on socket: Connection reset by peer (this error originated at tensorpipe/transport/shm/connection_impl.cc:187)
[W tensorpipe_agent.cpp:887] RPC agent for test_14 encountered error when reading incoming response from test_0: async error on socket: Connection reset by peer (this error originated at tensorpipe/transport/shm/connection_impl.cc:187)
[W tensorpipe_agent.cpp:887] RPC agent for test_4 encountered error when reading incoming response from test_0: async error on socket: Connection reset by peer (this error originated at tensorpipe/transport/shm/connection_impl.cc:187)
[W tensorpipe_agent.cpp:887] RPC agent for test_7 encountered error when reading incoming response from test_0: async error on socket: Connection reset by peer (this error originated at tensorpipe/transport/shm/connection_impl.cc:187)
[W tensorpipe_agent.cpp:887] RPC agent for test_2 encountered error when reading incoming response from test_0: async error on socket: Connection reset by peer (this error originated at tensorpipe/transport/shm/connection_impl.cc:187)
[W tensorpipe_agent.cpp:887] RPC agent for test_17 encountered error when reading incoming response from test_0: async error on socket: Connection reset by peer (this error originated at tensorpipe/transport/shm/connection_impl.cc:187)
[W tensorpipe_agent.cpp:863] RPC agent for test_1 encountered error when sending outgoing request #0 to test_0: async error on socket: Connection reset by peer (this error originated at tensorpipe/transport/shm/connection_impl.cc:187)
[W tensorpipe_agent.cpp:887] RPC agent for test_18 encountered error when reading incoming response from test_0: async error on socket: Connection reset by peer (this error originated at tensorpipe/transport/shm/connection_impl.cc:187)
[W tensorpipe_agent.cpp:887] RPC agent for test_9 encountered error when reading incoming response from test_0: async error on socket: Connection reset by peer (this error originated at tensorpipe/transport/shm/connection_impl.cc:187)
Traceback (most recent call last):
  File "/home/ubuntu/myenv/experiment/ppo-new/rpc.py", line 30, in <module>
    main()
  File "/home/ubuntu/myenv/experiment/ppo-new/rpc.py", line 22, in main
    mp.spawn(
  File "/home/ubuntu/miniconda3/envs/myenv/lib/python3.10/site-packages/torch/multiprocessing/spawn.py", line 240, in spawn
    return start_processes(fn, args, nprocs, join, daemon, start_method='spawn')
  File "/home/ubuntu/miniconda3/envs/myenv/lib/python3.10/site-packages/torch/multiprocessing/spawn.py", line 198, in start_processes
    while not context.join():
  File "/home/ubuntu/miniconda3/envs/myenv/lib/python3.10/site-packages/torch/multiprocessing/spawn.py", line 160, in join
    raise ProcessRaisedException(msg, error_index, failed_process.pid)
torch.multiprocessing.spawn.ProcessRaisedException: 

-- Process 1 terminated with the following error:
Traceback (most recent call last):
  File "/home/ubuntu/miniconda3/envs/myenv/lib/python3.10/site-packages/torch/multiprocessing/spawn.py", line 69, in _wrap
    fn(i, *args)
  File "/home/ubuntu/myenv/experiment/ppo-new/rpc.py", line 11, in init_process
    rpc.init_rpc(
  File "/home/ubuntu/miniconda3/envs/myenv/lib/python3.10/site-packages/torch/distributed/rpc/__init__.py", line 190, in init_rpc
    _init_rpc_backend(backend, store, name, rank, world_size, rpc_backend_options)
  File "/home/ubuntu/miniconda3/envs/myenv/lib/python3.10/site-packages/torch/distributed/rpc/__init__.py", line 224, in _init_rpc_backend
    rpc_agent = backend_registry.init_backend(
  File "/home/ubuntu/miniconda3/envs/myenv/lib/python3.10/site-packages/torch/distributed/rpc/backend_registry.py", line 97, in init_backend
    return backend.value.init_backend_handler(*args, **kwargs)
  File "/home/ubuntu/miniconda3/envs/myenv/lib/python3.10/site-packages/torch/distributed/rpc/backend_registry.py", line 305, in _tensorpipe_init_backend_handler
    api._all_gather(None, timeout=rpc_backend_options.rpc_timeout)
  File "/home/ubuntu/miniconda3/envs/myenv/lib/python3.10/site-packages/torch/distributed/rpc/api.py", line 77, in wrapper
    return func(*args, **kwargs)
  File "/home/ubuntu/miniconda3/envs/myenv/lib/python3.10/site-packages/torch/distributed/rpc/api.py", line 204, in _all_gather
    rpc_sync(
  File "/home/ubuntu/miniconda3/envs/myenv/lib/python3.10/site-packages/torch/distributed/rpc/api.py", line 77, in wrapper
    return func(*args, **kwargs)
  File "/home/ubuntu/miniconda3/envs/myenv/lib/python3.10/site-packages/torch/distributed/rpc/api.py", line 767, in rpc_sync
    return fut.wait()
RuntimeError: async error on socket: Connection reset by peer (this error originated at tensorpipe/transport/shm/connection_impl.cc:187)
```




### Versions

```shell
PyTorch version: 1.11.0
Is debug build: False
CUDA used to build PyTorch: 11.3
ROCM used to build PyTorch: N/A

OS: Ubuntu 20.04.2 LTS (x86_64)
GCC version: (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0
Clang version: Could not collect
CMake version: version 3.16.3
Libc version: glibc-2.31

Python version: 3.10.4 (main, Mar 31 2022, 08:41:55) [GCC 7.5.0] (64-bit runtime)
Python platform: Linux-5.4.0-1045-aws-x86_64-with-glibc2.31
Is CUDA available: True
CUDA runtime version: Could not collect
GPU models and configuration: 
GPU 0: Tesla V100-SXM2-16GB
GPU 1: Tesla V100-SXM2-16GB
GPU 2: Tesla V100-SXM2-16GB
GPU 3: Tesla V100-SXM2-16GB

Nvidia driver version: 460.91.03
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.8.0.5
/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.0.5
/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.0.5
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.0.5
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.0.5
/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.0.5
/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.0.5
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] mypy==0.910
[pip3] mypy-extensions==0.4.3
[pip3] numpy==1.22.3
[pip3] torch==1.11.0
[conda] blas                      1.0                         mkl  
[conda] cudatoolkit               11.3.1               h2bc3f7f_2  
[conda] mkl                       2021.4.0           h06a4308_640  
[conda] mkl-service               2.4.0           py310h7f8727e_0  
[conda] mkl_fft                   1.3.1           py310hd6ae3a3_0  
[conda] mkl_random                1.2.2           py310h00e6091_0  
[conda] numpy                     1.22.3          py310hfa59a62_0  
[conda] numpy-base                1.22.3          py310h9585f30_0  
[conda] pytorch                   1.11.0          py3.10_cuda11.3_cudnn8.2.0_0    pytorch
[conda] pytorch-mutex             1.0                        cuda    pytorch
```

cc @pietern @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @SciPioneer @H-Huang @kwen2501 @jjlilley @mrzzd