PyTorch RPC crashed when using IB 
Labels: oncall: distributed, module: rpc

### 🐛 Describe the bug

Hi. I am using PyTorch's RPC with tensorpipe backend for my project and encounter a weird bug. Bascially, there are two workers on two machines. Worker A invoked rpc on worker B, and all of a sudden, worker B just crashed. 

I compile the PyTorch from scratch to dig into the position where it crashed with gdb. The figure below shows that it crashes due to deference a nullptr pointer. It seems related to IB. But I have no idea why it happened.  

However, when I run my program in machines without IB, it works well. 

Can you give some clues about this issue? Thanks! 

![image](https://user-images.githubusercontent.com/25879526/195796554-6b98a645-e296-4c61-aa8c-c072b3632982.png)


### Versions


```sh
(jasper) ➜  ~ python collect_env.py
Collecting environment information...
PyTorch version: 1.12.0a0+git664058f
Is debug build: False
CUDA used to build PyTorch: 11.3
ROCM used to build PyTorch: N/A

OS: Ubuntu 20.04.4 LTS (x86_64)
GCC version: (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0
Clang version: Could not collect
CMake version: version 3.22.1
Libc version: glibc-2.31

Python version: 3.8.12 (default, Oct 12 2021, 13:49:34)  [GCC 7.5.0] (64-bit runtime)
Python platform: Linux-5.4.0-110-generic-x86_64-with-glibc2.17
Is CUDA available: True
CUDA runtime version: 11.3.109
CUDA_MODULE_LOADING set to:
GPU models and configuration:
GPU 0: Tesla P100-PCIE-12GB
GPU 1: Tesla P100-PCIE-12GB

Nvidia driver version: 515.43.04
cuDNN version: Probably one of the following:
/usr/local/cuda-11.3/targets/x86_64-linux/lib/libcudnn.so.8.4.0
/usr/local/cuda-11.3/targets/x86_64-linux/lib/libcudnn_adv_infer.so.8.4.0
/usr/local/cuda-11.3/targets/x86_64-linux/lib/libcudnn_adv_train.so.8.4.0
/usr/local/cuda-11.3/targets/x86_64-linux/lib/libcudnn_cnn_infer.so.8.4.0
/usr/local/cuda-11.3/targets/x86_64-linux/lib/libcudnn_cnn_train.so.8.4.0
/usr/local/cuda-11.3/targets/x86_64-linux/lib/libcudnn_ops_infer.so.8.4.0
/usr/local/cuda-11.3/targets/x86_64-linux/lib/libcudnn_ops_train.so.8.4.0
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] numpy==1.22.3
[pip3] numpydoc==1.3.1
[pip3] torch==1.12.0a0+git664058f
[pip3] torch-scatter==2.0.9
[pip3] torchaudio==0.12.1+cu113
[pip3] torchvision==0.13.1+cu113
[conda] blas                      1.0                         mkl
[conda] magma-cuda113             2.5.2                         1    pytorch
[conda] mkl                       2021.4.0           h06a4308_640
[conda] mkl-include               2022.1.0           h06a4308_224
[conda] mkl-service               2.4.0            py38h7f8727e_0
[conda] mkl_fft                   1.3.1            py38hd3c417c_0
[conda] mkl_random                1.2.2            py38h51133e4_0
[conda] numpy                     1.22.3                   pypi_0    pypi
[conda] numpy-base                1.21.5           py38hf524024_2
[conda] numpydoc                  1.3.1                    pypi_0    pypi
[conda] torch                     1.12.0a0+git664058f           dev_0    <develop>
[conda] torch-scatter             2.0.9                     dev_0    <develop>
[conda] torchaudio                0.12.1+cu113             pypi_0    pypi
[conda] torchvision               0.13.1+cu113             pypi_0    pypi
```

cc @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @H-Huang @kwen2501 @awgu @pietern @jjlilley @mrzzd