RPC Tutorial can not profile the  rpc operations communication between workers
Labels: oncall: distributed, triaged, module: rpc, oncall: profiler

### 🐛 Describe the bug

I followed the RPC profiling tutorial [PROFILING PYTORCH RPC-BASED WORKLOADS](https://pytorch.org/tutorials/prototype/distributed_rpc_profiling.html), trying to produce a clear rpc operation process between two workers like below.

```python
# Some columns are omitted for brevity, exact output subject to randomness
----------------------------------------------------------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------
Name                                                              Self CPU total %  Self CPU total   CPU total %      CPU total        CPU time avg     Number of Calls  Node ID
----------------------------------------------------------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------
rpc_async#aten::add(worker0 -> worker1)                           0.00%            0.000us          0                20.462ms         20.462ms         1                0
rpc_async#aten::mul(worker0 -> worker1)                           0.00%            0.000us          0                5.712ms          5.712ms          1                0
rpc_async#aten::mul(worker0 -> worker1)#remote_op: mul            1.84%            206.864us        2.69%            302.162us        151.081us        2                1
rpc_async#aten::add(worker0 -> worker1)#remote_op: add            1.41%            158.501us        1.57%            176.924us        176.924us        1                1
rpc_async#aten::mul(worker0 -> worker1)#remote_op: output_nr      0.04%            4.980us          0.04%            4.980us          2.490us          2                1
rpc_async#aten::mul(worker0 -> worker1)#remote_op: is_leaf        0.07%            7.806us          0.07%            7.806us          1.952us          4                1
rpc_async#aten::add(worker0 -> worker1)#remote_op: empty          0.16%            18.423us         0.16%            18.423us         18.423us         1                1
rpc_async#aten::mul(worker0 -> worker1)#remote_op: empty          0.14%            15.712us         0.14%            15.712us         15.712us         1                1
----------------------------------------------------------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------
Self CPU time total: 11.237ms
```

## Test1: Same as the Official Code

```python
def worker(rank, world_size):
    # Above code omitted...
    if rank == 0:
        dst_worker_rank = (rank + 1) % world_size
        dst_worker_name = f"worker{dst_worker_rank}"
        t1, t2 = random_tensor(), random_tensor()
        # Send and wait RPC completion under profiling scope.
        with profiler.profile() as prof:
            fut1 = rpc.rpc_async(dst_worker_name, torch.add, args=(t1, t2))
            fut2 = rpc.rpc_async(dst_worker_name, torch.mul, args=(t1, t2))
            # RPCs must be awaited within profiling scope.
            fut1.wait()
            fut2.wait()

        print(prof.key_averages().table())
```

Profiling Logs show nothing!
```bash
DEBUG:root:worker1 successfully initialized RPC.
DEBUG:root:Rank 1 waiting for workers and shutting down RPC
DEBUG:root:worker0 successfully initialized RPC.
STAGE:2023-03-20 23:10:09 58750:58750 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
[W utils.cpp:173] Warning: Profiling a distributed call with the Kineto profiler will profile the caller, but not the worker. (function operator())
STAGE:2023-03-20 23:10:09 58750:58750 ActivityProfilerController.cpp:300] Completed Stage: Collection

DEBUG:root:Rank 0 waiting for workers and shutting down RPC
DEBUG:root:Rank 1 shutdown RPC
DEBUG:root:Rank 0 shutdown RPC
```


## Test2: Add print to show fut1/fu2 values()

```python
def worker(rank, world_size):
    # Above code omitted...
    if rank == 0:
        dst_worker_rank = (rank + 1) % world_size
        dst_worker_name = f"worker{dst_worker_rank}"
        t1, t2 = random_tensor(), random_tensor()
        # Send and wait RPC completion under profiling scope.
        with profiler.profile() as prof:
            fut1 = rpc.rpc_async(dst_worker_name, torch.add, args=(t1, t2))
            fut2 = rpc.rpc_async(dst_worker_name, torch.mul, args=(t1, t2))
            # RPCs must be awaited within profiling scope.
            fut1.wait()
            fut2.wait()
            # print vals
            print(fut1.value())
            print(fut2.value())

        print(prof.key_averages().table())
```

Profiling Logs show operations but not have rpc prefix!
```bash
DEBUG:root:worker1 successfully initialized RPC.
DEBUG:root:worker0 successfully initialized RPC.
DEBUG:root:Rank 1 waiting for workers and shutting down RPC
STAGE:2023-03-20 23:11:39 60149:60149 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
[W utils.cpp:173] Warning: Profiling a distributed call with the Kineto profiler will profile the caller, but not the worker. (function operator())
tensor([[0.7637, 0.8160, 1.0736],
        [0.8793, 1.3461, 0.9853],
        [1.2276, 0.7834, 1.1987]], requires_grad=True)
tensor([[0.1369, 0.1664, 0.2245],
        [0.0073, 0.3516, 0.1521],
        [0.3742, 0.1182, 0.2851]], requires_grad=True)
STAGE:2023-03-20 23:11:39 60149:60149 ActivityProfilerController.cpp:300] Completed Stage: Collection
-----------------------------  ------------  ------------  ------------  ------------  ------------  ------------  
                         Name    Self CPU %      Self CPU   CPU total %     CPU total  CPU time avg    # of Calls  
-----------------------------  ------------  ------------  ------------  ------------  ------------  ------------  
                aten::reshape         3.42%      35.000us         5.08%      52.000us      26.000us             2  
         aten::_reshape_alias         1.66%      17.000us         1.66%      17.000us       8.500us             2  
               aten::isfinite         7.91%      81.000us        30.86%     316.000us     158.000us             2  
                    aten::abs         8.01%      82.000us        12.60%     129.000us      16.125us             8  
                  aten::empty         1.27%      13.000us         1.27%      13.000us       1.300us            10  
                     aten::ne         9.38%      96.000us        16.89%     173.000us      28.833us             6  
                     aten::to         2.54%      26.000us        12.11%     124.000us      10.333us            12  
               aten::_to_copy         4.59%      47.000us         9.57%      98.000us      12.250us             8  
          aten::empty_strided         1.56%      16.000us         1.56%      16.000us       2.000us             8  
                  aten::copy_         3.42%      35.000us         3.42%      35.000us       4.375us             8  
                     aten::eq         2.15%      22.000us         2.15%      22.000us      11.000us             2  
                    aten::mul         1.46%      15.000us         1.46%      15.000us       7.500us             2  
                aten::__and__         0.59%       6.000us         3.12%      32.000us      16.000us             2  
            aten::bitwise_and         2.54%      26.000us         2.54%      26.000us      13.000us             2  
          aten::masked_select         4.88%      50.000us        15.72%     161.000us      80.500us             2  
                    aten::sum         6.93%      71.000us        10.06%     103.000us      51.500us             2  
             aten::as_strided         1.46%      15.000us         1.46%      15.000us       0.326us            46  
                  aten::fill_         0.10%       1.000us         0.10%       1.000us       0.167us             6  
                   aten::item         3.03%      31.000us         3.42%      35.000us       1.250us            28  
    aten::_local_scalar_dense         0.39%       4.000us         0.39%       4.000us       0.143us            28  
                    aten::min         3.22%      33.000us         3.61%      37.000us      18.500us             2  
                    aten::max         2.15%      22.000us         2.25%      23.000us      11.500us             2  
                 aten::unbind         9.96%     102.000us        16.31%     167.000us      41.750us             4  
                 aten::select         7.71%      79.000us         8.79%      90.000us       2.143us            42  
                   aten::ceil         3.81%      39.000us         3.81%      39.000us      19.500us             2  
             aten::is_nonzero         0.98%      10.000us         1.95%      20.000us       2.500us             8  
                    aten::div         1.76%      18.000us         1.76%      18.000us       9.000us             2  
                     aten::gt         1.76%      18.000us         1.76%      18.000us       4.500us             4  
                     aten::lt         1.37%      14.000us         1.37%      14.000us       7.000us             2  
           aten::resolve_conj         0.00%       0.000us         0.00%       0.000us       0.000us             6  
            aten::resolve_neg         0.00%       0.000us         0.00%       0.000us       0.000us             6  
-----------------------------  ------------  ------------  ------------  ------------  ------------  ------------  
Self CPU time total: 1.024ms

DEBUG:root:Rank 0 waiting for workers and shutting down RPC
DEBUG:root:Rank 1 shutdown RPC
DEBUG:root:Rank 0 shutdown RPC
```


## Conclusion

After reading `torch.autograd.profiler` and `torch.autograd.profiler_util`, I guess that the rpc profiling feature may have been discarded but the tutorial did not change.

- https://github.com/pytorch/pytorch/blob/master/torch/autograd/profiler.py#L362, kineto_event.name() doesn't have works like `rpc_async# .. (worker0 -> worker1)#remote_op: mul` as prefix or suffix
- https://github.com/pytorch/pytorch/blob/master/torch/autograd/profiler.py#L382, node_id is not passed to FunctionEvent which makes _build_table() not show Node ID column. 
- https://github.com/pytorch/pytorch/blob/master/torch/autograd/profiler_util.py#L756, Only append Node ID if any event has a valid (>= 0) Node ID

I'll be appropriate for the early replay.



### Versions

Env

```bash
Collecting environment information...
PyTorch version: 1.13.1+cu116
Is debug build: False
CUDA used to build PyTorch: 11.6
ROCM used to build PyTorch: N/A

OS: CentOS Linux 8 (Core) (x86_64)
GCC version: (GCC) 8.5.0 20210514 (Red Hat 8.5.0-4)
Clang version: Could not collect
CMake version: Could not collect
Libc version: glibc-2.28

Python version: 3.8.16 | packaged by conda-forge | (default, Feb  1 2023, 16:01:55)  [GCC 11.3.0] (64-bit runtime)
Python platform: Linux-4.18.0-193.el8.jd_025.x86_64-x86_64-with-glibc2.10
Is CUDA available: True
CUDA runtime version: 11.2.152
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: 
GPU 0: Tesla V100-PCIE-16GB
GPU 1: Tesla V100-PCIE-16GB
GPU 2: Tesla V100-PCIE-16GB
GPU 3: Tesla V100-PCIE-16GB

Nvidia driver version: 460.32.03
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture:        x86_64
CPU op-mode(s):      32-bit, 64-bit
Byte Order:          Little Endian
CPU(s):              40
On-line CPU(s) list: 0-39
Thread(s) per core:  2
Core(s) per socket:  10
Socket(s):           2
NUMA node(s):        1
Vendor ID:           GenuineIntel
CPU family:          6
Model:               79
Model name:          Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz
Stepping:            1
CPU MHz:             2199.998
BogoMIPS:            4399.99
Hypervisor vendor:   KVM
Virtualization type: full
L1d cache:           32K
L1i cache:           32K
L2 cache:            4096K
L3 cache:            16384K
NUMA node0 CPU(s):   0-39
Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon rep_good nopl xtopology cpuid tsc_known_freq pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single ibrs ibpb fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm rdseed adx smap xsaveopt arat

Versions of relevant libraries:
[pip3] mypy-extensions==1.0.0
[pip3] numpy==1.24.2
[pip3] torch==1.13.1+cu116
[pip3] torchvision==0.14.1+cu116
[conda] numpy                     1.24.2                   pypi_0    pypi
[conda] torch                     1.13.1+cu116             pypi_0    pypi
[conda] torchvision               0.14.1+cu116             pypi_0    pypi
```

cc @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @H-Huang @kwen2501 @awgu @pietern @jjlilley @mrzzd @robieta @chaekit @aaronenyeshi @ngimel @nbcsm @guotuofeng @guyang3532 @gaoteng-git @tiffzhaofb @dzhulgakov @davidberard98