`tag` parameter is ignored from NCCL P2P isend/irecv pair
Labels: oncall: distributed, module: nccl

### 🐛 Describe the bug

While making a multi-GPU utility with NCCL P2P communication, I found bugs that NCCL backend ignores tag parameter, and also the first irecv/isend pair does not work asynchronously.

The below code sends and receives two tensors between two processes.
1. Main proc: send `ones` (**tag 1**) -> sleep 3 secs -> send `zeros` (**tag 2**)
2. Worker proc: receive **tag 2** -> receive **tag 1** -> wait **tag 1** -> wait **tag 2**

Ideally, the worker should receive `ones` from tag 1, and `zeros` from tag 2.
However, the worker receives `zeros` from tag 1, and `ones` from tag 2, with NCCL backend.

**Test code**
```python
import os
import time
from argparse import ArgumentParser

import torch
import torch.distributed as dist
import torch.multiprocessing as mp


SIZE = 10000


def run_worker(rank, use_nccl):
    os.environ["MASTER_ADDR"] = "localhost"
    os.environ["MASTER_PORT"] = "29500"

    if use_nccl:
        dist.init_process_group("nccl", world_size=2, rank=rank)
        dev0 = "cuda:0"
        dev1 = "cuda:1"
    else:
        dist.init_process_group("gloo", world_size=2, rank=rank)
        dev0 = "cpu"
        dev1 = "cpu"

    if rank == 0:
        ones = torch.ones(SIZE, SIZE, device=dev0)
        zeros = torch.zeros(SIZE, SIZE, device=dev0)

        print(f"[Main] tag 1: {ones.sum()}, tag 2: {zeros.sum()}")

        start_time = time.time()

        print(f"[Main] first send ones (tag 1) at {time.time() - start_time:5.3f} sec")
        f1 = dist.isend(ones, 1, tag=1)

        print(f"[Main] sleeps 3 sec at {time.time() - start_time:5.3f} sec")
        time.sleep(3)

        print(f"[Main] second send zeros (tag 2) at {time.time() - start_time:5.3f} sec")
        f2 = dist.isend(zeros, 1, tag=2)

        print(f"[Main] waiting at {time.time() - start_time:5.3f} sec")
        f2.wait()
        f1.wait()

    else:
        buf1 = torch.rand(SIZE, SIZE, device=dev1)
        buf2 = torch.rand(SIZE, SIZE, device=dev1)

        start_time = time.time()

        print(f"[Worker] first recv (tag 2) at {time.time() - start_time:5.3f} sec")
        f2 = dist.irecv(buf2, 0, tag=2)

        print(f"[Worker] second recv (tag 1) at {time.time() - start_time:5.3f} sec")
        f1 = dist.irecv(buf1, 0, tag=1)

        print(f"[Worker] waiting at {time.time() - start_time:5.3f} sec")

        f1.wait()
        print(f"[Worker] got tag 1: {buf1.sum()} at {time.time() - start_time:5.3f} sec")

        f2.wait()
        print(f"[Worker] got tag 2: {buf2.sum()} at {time.time() - start_time:5.3f} sec")


if __name__ == "__main__":
    parser = ArgumentParser()
    parser.add_argument("--nccl", action="store_true")
    args = parser.parse_args()

    print(f"Use NCCL: {args.nccl}")

    mp.spawn(run_worker, nprocs=2, join=True, args=(args.nccl,))
```

**Result (GLOO)**
```
$ torchrun test.py
Use NCCL: False
[Main] tag 1: 100000000.0, tag 2: 0.0
[Main] first send ones (tag 1) at 0.000 sec
[Main] sleeps 3 sec at 0.000 sec
[Worker] first recv (tag 2) at 0.000 sec
[Worker] second recv (tag 1) at 0.000 sec
[Worker] waiting at 0.000 sec
[Worker] got tag 1: 100000000.0 at 0.112 sec
[Main] second send zeros (tag 2) at 3.003 sec
[Main] waiting at 3.126 sec
[Worker] got tag 2: 0.0 at 2.140 sec
```

**Result (NCCL)**
```
$ torchrun test.py --nccl
Use NCCL: True
[Worker] first recv (tag 2) at 0.000 sec
[Main] tag 1: 100000000.0, tag 2: 0.0
[Main] first send ones (tag 1) at 0.000 sec
[Main] sleeps 3 sec at 0.052 sec
[Worker] second recv (tag 1) at 0.056 sec
[Worker] waiting at 0.056 sec
[Main] second send zeros (tag 2) at 3.055 sec
[Main] waiting at 3.055 sec
[Worker] got tag 1: 0.0 at 3.068 sec
[Worker] got tag 2: 100000000.0 at 3.068 sec
```

These results show two problems:
1. The received values from the worker are skewed (see the values and times from `[Worker] got tag`)
2. The first isend/irecv calls do not return any waitable object, thus work as blocking APIs. (see the times from `[Main] sleeps` and `[Worker] second recv`)

I've tested this behavior from several PyTorch runtimes and machines. 
Please notice me whether I misused distributed APIs.




### Versions

2 GPU server (stable build w/ A4500 x 2)
```
Collecting environment information...
PyTorch version: 1.13.1
Is debug build: False
CUDA used to build PyTorch: 11.6
ROCM used to build PyTorch: N/A

OS: Ubuntu 22.04 LTS (x86_64)
GCC version: (Ubuntu 11.3.0-1ubuntu1~22.04) 11.3.0
Clang version: 14.0.0-1ubuntu1
CMake version: version 3.22.1
Libc version: glibc-2.35

Python version: 3.10.6 (main, Oct  7 2022, 20:19:58) [GCC 11.2.0] (64-bit runtime)
Python platform: Linux-5.15.0-48-generic-x86_64-with-glibc2.35
Is CUDA available: True
CUDA runtime version: 11.6.124
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration:
GPU 0: NVIDIA RTX A4500
GPU 1: NVIDIA RTX A4500

Nvidia driver version: 510.47.03
cuDNN version: Probably one of the following:
/usr/local/cuda-11.6/targets/x86_64-linux/lib/libcudnn.so.8.6.0
/usr/local/cuda-11.6/targets/x86_64-linux/lib/libcudnn_adv_infer.so.8.6.0
/usr/local/cuda-11.6/targets/x86_64-linux/lib/libcudnn_adv_train.so.8.6.0
/usr/local/cuda-11.6/targets/x86_64-linux/lib/libcudnn_cnn_infer.so.8.6.0
/usr/local/cuda-11.6/targets/x86_64-linux/lib/libcudnn_cnn_train.so.8.6.0
/usr/local/cuda-11.6/targets/x86_64-linux/lib/libcudnn_ops_infer.so.8.6.0
/usr/local/cuda-11.6/targets/x86_64-linux/lib/libcudnn_ops_train.so.8.6.0
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture:                    x86_64
CPU op-mode(s):                  32-bit, 64-bit
Address sizes:                   46 bits physical, 48 bits virtual
Byte Order:                      Little Endian
CPU(s):                          16
On-line CPU(s) list:             0-15
Vendor ID:                       GenuineIntel
Model name:                      Intel(R) Xeon(R) W-3223 CPU @ 3.50GHz
CPU family:                      6
Model:                           85
Thread(s) per core:              2
Core(s) per socket:              8
Socket(s):                       1
Stepping:                        7
CPU max MHz:                     4200.0000
CPU min MHz:                     1200.0000
BogoMIPS:                        7000.00
Flags:                           fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cdp_l3 invpcid_single intel_ppin ssbd mba ibrs ibpb stibp ibrs_enhanced tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb intel_pt avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts hwp hwp_act_window hwp_epp hwp_pkg_req pku ospke avx512_vnni md_clear flush_l1d arch_capabilities
Virtualization:                  VT-x
L1d cache:                       256 KiB (8 instances)
L1i cache:                       256 KiB (8 instances)
L2 cache:                        8 MiB (8 instances)
L3 cache:                        16.5 MiB (1 instance)
NUMA node(s):                    1
NUMA node0 CPU(s):               0-15
Vulnerability Itlb multihit:     KVM: Mitigation: VMX disabled
Vulnerability L1tf:              Not affected
Vulnerability Mds:               Not affected
Vulnerability Meltdown:          Not affected
Vulnerability Mmio stale data:   Mitigation; Clear CPU buffers; SMT vulnerable
Vulnerability Retbleed:          Mitigation; Enhanced IBRS
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp
Vulnerability Spectre v1:        Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:        Mitigation; Enhanced IBRS, IBPB conditional, RSB filling
Vulnerability Srbds:             Not affected
Vulnerability Tsx async abort:   Mitigation; TSX disabled

Versions of relevant libraries:
[pip3] numpy==1.23.1
[pip3] pytorch-sphinx-theme==0.0.24
[pip3] torch==1.13.1
[pip3] torch-tb-profiler==0.4.0
[pip3] torchaudio==0.13.1
[pip3] torchvision==0.14.1
[conda] blas                      1.0                         mkl
[conda] cudatoolkit               11.6.0              hecad31d_10    conda-forge
[conda] ffmpeg                    4.3                  hf484d3e_0    pytorch
[conda] mkl                       2021.4.0           h06a4308_640
[conda] mkl-service               2.4.0           py310ha2c4b55_0    conda-forge
[conda] mkl_fft                   1.3.1           py310h2b4bcf5_1    conda-forge
[conda] mkl_random                1.2.2           py310h00e6091_0
[conda] numpy                     1.23.1          py310h1794996_0
[conda] numpy-base                1.23.1          py310hcba007f_0
[conda] pytorch                   1.13.1          py3.10_cuda11.6_cudnn8.3.2_0    pytorch
[conda] pytorch-cuda              11.6                 h867d48c_1    pytorch
[conda] pytorch-mutex             1.0                        cuda    pytorch
[conda] pytorch-sphinx-theme      0.0.24                    dev_0    <develop>
[conda] torch-tb-profiler         0.4.0                    pypi_0    pypi
[conda] torchaudio                0.13.1              py310_cu116    pytorch
[conda] torchvision               0.14.1              py310_cu116    pytorch
```

8 GPU server (nightly build w/ RTX3090 x 8)
```
Collecting environment information...
PyTorch version: 2.0.0.dev20230213
Is debug build: False
CUDA used to build PyTorch: 11.8
ROCM used to build PyTorch: N/A

OS: Ubuntu 20.04.4 LTS (x86_64)
GCC version: (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0
Clang version: 10.0.0-4ubuntu1 
CMake version: version 3.16.3
Libc version: glibc-2.31

Python version: 3.10.9 (main, Jan 11 2023, 15:21:40) [GCC 11.2.0] (64-bit runtime)
Python platform: Linux-5.4.0-126-generic-x86_64-with-glibc2.31
Is CUDA available: True
CUDA runtime version: 11.6.124
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: 
GPU 0: NVIDIA GeForce RTX 3090
GPU 1: NVIDIA GeForce RTX 3090
GPU 2: NVIDIA GeForce RTX 3090
GPU 3: NVIDIA GeForce RTX 3090
GPU 4: NVIDIA GeForce RTX 3090
GPU 5: NVIDIA GeForce RTX 3090
GPU 6: NVIDIA GeForce RTX 3090
GPU 7: NVIDIA GeForce RTX 3090

Nvidia driver version: 510.47.03
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.8.5.0
/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.5.0
/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.5.0
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.5.0
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.5.0
/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.5.0
/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.5.0
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture:                    x86_64
CPU op-mode(s):                  32-bit, 64-bit
Byte Order:                      Little Endian
Address sizes:                   43 bits physical, 48 bits virtual
CPU(s):                          64
On-line CPU(s) list:             0-31
Off-line CPU(s) list:            32-63
Thread(s) per core:              1
Core(s) per socket:              16
Socket(s):                       2
NUMA node(s):                    2
Vendor ID:                       AuthenticAMD
CPU family:                      23
Model:                           49
Model name:                      AMD EPYC 7F52 16-Core Processor
Stepping:                        0
Frequency boost:                 enabled
CPU MHz:                         1797.658
CPU max MHz:                     3500.0000
CPU min MHz:                     2500.0000
BogoMIPS:                        6999.88
Virtualization:                  AMD-V
L1d cache:                       1 MiB
L1i cache:                       1 MiB
L2 cache:                        16 MiB
L3 cache:                        512 MiB
NUMA node0 CPU(s):               0-15
NUMA node1 CPU(s):               16-31
Vulnerability Itlb multihit:     Not affected
Vulnerability L1tf:              Not affected
Vulnerability Mds:               Not affected
Vulnerability Meltdown:          Not affected
Vulnerability Mmio stale data:   Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp
Vulnerability Spectre v1:        Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:        Mitigation; Retpolines, IBPB conditional, IBRS_FW, RSB filling
Vulnerability Srbds:             Not affected
Vulnerability Tsx async abort:   Not affected
Flags:                           fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid aperfmperf pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 hw_pstate sme ssbd mba sev ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 cqm rdt_a rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local clzero irperf xsaveerptr wbnoinvd arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif umip rdpid overflow_recov succor smca

Versions of relevant libraries:
[pip3] bert-pytorch==0.0.1a4
[pip3] clip-anytorch==2.5.0
[pip3] CoCa-pytorch==0.0.7
[pip3] dalle2-pytorch==1.10.5
[pip3] ema-pytorch==0.1.4
[pip3] functorch==1.14.0a0+408bcf1
[pip3] mypy-extensions==0.4.3
[pip3] numpy==1.23.5
[pip3] pytorch-transformers==1.2.0
[pip3] pytorch-warmup==0.1.1
[pip3] rotary-embedding-torch==0.2.1
[pip3] torch==2.0.0.dev20230213
[pip3] torch-fidelity==0.3.0
[pip3] torch-struct==0.5
[pip3] torchaudio==2.0.0.dev20230213
[pip3] torchmetrics==0.11.1
[pip3] torchrec-nightly==2023.1.30
[pip3] torchvision==0.15.0.dev20230213
[pip3] vector-quantize-pytorch==0.10.15
[conda] bert-pytorch              0.0.1a4                   dev_0    <develop>
[conda] blas                      1.0                         mkl  
[conda] clip-anytorch             2.5.0                    pypi_0    pypi
[conda] coca-pytorch              0.0.7                    pypi_0    pypi
[conda] dalle2-pytorch            1.10.5                   pypi_0    pypi
[conda] ema-pytorch               0.1.4                    pypi_0    pypi
[conda] functorch                 1.14.0a0+408bcf1          pypi_0    pypi
[conda] magma-cuda116             2.6.1                         1    pytorch
[conda] mkl                       2021.4.0           h06a4308_640  
[conda] mkl-service               2.4.0           py310h7f8727e_0  
[conda] mkl_fft                   1.3.1           py310hd6ae3a3_0  
[conda] mkl_random                1.2.2           py310h00e6091_0  
[conda] numpy                     1.21.2                   pypi_0    pypi
[conda] numpy-base                1.23.5          py310h8e6c178_0  
[conda] pytorch                   2.0.0.dev20230213 py3.10_cuda11.8_cudnn8.7.0_0    pytorch-nightly
[conda] pytorch-cuda              11.8                 h7e8668a_3    pytorch-nightly
[conda] pytorch-mutex             1.0                        cuda    pytorch-nightly
[conda] pytorch-transformers      1.2.0                    pypi_0    pypi
[conda] pytorch-warmup            0.1.1                    pypi_0    pypi
[conda] rotary-embedding-torch    0.2.1                    pypi_0    pypi
[conda] torch-fidelity            0.3.0                    pypi_0    pypi
[conda] torch-struct              0.5                      pypi_0    pypi
[conda] torchaudio                2.0.0.dev20230213     py310_cu118    pytorch-nightly
[conda] torchmetrics              0.11.1                   pypi_0    pypi
[conda] torchrec-nightly          2023.1.30                pypi_0    pypi
[conda] torchtriton               2.0.0+0d7e753227           py310    pytorch-nightly
[conda] torchvision               0.15.0.dev20230213     py310_cu118    pytorch-nightly
[conda] vector-quantize-pytorch   0.10.15                  pypi_0    pypi
```

cc @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @H-Huang @kwen2501 @awgu