Got wrong tensor when using dist.send to send tensors

Maxwell_Albert: <p>Hi, I am using <code>torch.linalg.qr</code> to produce some tensors and send then by using <code>dist.send</code>.<br>
But the sequence of the element is not the same as before, why?<br>
Here is the code</p>
<pre><code class="lang-auto">import torch
import torch.multiprocessing as mp
import torch.distributed as dist

def main_worker(rank, world_size, args):
    dist.init_process_group(
        backend="nccl",
        init_method="tcp://127.0.0.1:9001",
        world_size=world_size,
        rank=rank,
    )
    print("process begin", rank)

    if rank == 0:

        q_buffer = torch.rand([4,4]).to(0)      
        Q,R = torch.linalg.qr(q_buffer)
        print("local before q_buffer",Q)
        dist.send(Q,1)


                    


    elif rank == 1:

        q_buffer = torch.rand([4,4]).to(1)

        dist.recv(q_buffer,0)

        print("recv",q_buffer,q_buffer.shape)


def main():
    mp.spawn(main_worker, nprocs=2, args=(2, 2))


if __name__ == "__main__":
    main()

</code></pre>
<p>The result is</p>
<pre><code class="lang-auto">local before q_buffer tensor([[-0.4076,  0.5155,  0.4838, -0.5780],
        [-0.4182, -0.6807, -0.2695, -0.5378],
        [-0.4326,  0.4825, -0.7545,  0.1039],
        [-0.6869, -0.1954,  0.3522,  0.6049]], device='cuda:0')
recv tensor([[-0.4076, -0.4182, -0.4326, -0.6869],
        [ 0.5155, -0.6807,  0.4825, -0.1954],
        [ 0.4838, -0.2695, -0.7545,  0.3522],
        [-0.5780, -0.5378,  0.1039,  0.6049]], device='cuda:1') torch.Size([4, 4])
</code></pre>

qsj287068067: <p>You need to specify the <code>dtype</code> of the container tensor (<code>q_buffer</code> in your case) to the same with <code>Q</code>.</p>
<p>Please check the following issues.</p><aside class="onebox githubissue" data-onebox-src="https://github.com/pytorch/pytorch/issues/81545#issuecomment-1186409030">
  <header class="source">

      <a href="https://github.com/pytorch/pytorch/issues/81545#issuecomment-1186409030" target="_blank" rel="noopener nofollow ugc">github.com/pytorch/pytorch</a>
  </header>

  <article class="onebox-body">
    <div class="github-row">
  <div class="github-icon-container" title="Issue">
	  <svg width="60" height="60" class="github-icon" viewbox="0 0 14 16" aria-hidden="true"><path d="M7 2.3c3.14 0 5.7 2.56 5.7 5.7s-2.56 5.7-5.7 5.7A5.71 5.71 0 0 1 1.3 8c0-3.14 2.56-5.7 5.7-5.7zM7 1C3.14 1 0 4.14 0 8s3.14 7 7 7 7-3.14 7-7-3.14-7-7-7zm1 3H6v5h2V4zm0 6H6v2h2v-2z"></path></svg>
  </div>

  <div class="github-info-container">
    <h4>
      <a href="https://github.com/pytorch/pytorch/issues/81545#issuecomment-1186409030" target="_blank" rel="noopener nofollow ugc">Precision error from torch.distributed.send() to recv()</a>
    </h4>

    <div class="github-info">
      <div class="date">
        opened <span class="discourse-local-date" data-format="ll" data-date="2022-07-15" data-time="05:45:01" data-timezone="UTC">05:45AM - 15 Jul 22 UTC</span>
      </div>


      <div class="user">
        <a href="https://github.com/AdorableJiang" target="_blank" rel="noopener nofollow ugc">
          <img alt="AdorableJiang" src="https://avatars.githubusercontent.com/u/51251025?v=4" class="onebox-avatar-inline" width="20" height="20">
          AdorableJiang
        </a>
      </div>
    </div>

    <div class="labels">
        <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">
          oncall: distributed
        </span>
    </div>
  </div>
</div>

  <div class="github-row">
    <p class="github-body-container">### 🐛 Describe the bug

There is probably a precision error when using `torch.<span class="show-more-container"><a href="" rel="noopener" class="show-more">…</a></span><span class="excerpt hidden">distributed.send()` and `torch.distributed.recv()` pairs.
`torch.distributed.recv()` can receive a tensor correctly only when that tensor is sent of type `torch.float32`. The other float types `torch.float16` and `torch.float64` leads to wrong tensor values in receiver side.

## Reproduce

(on a dual GPU platform)

```python
import torch
import torch.multiprocessing as mp
import torch.distributed as dist
import time

def main_worker(rank, world_size, args):
    dist.init_process_group(
        backend="nccl",
        init_method="tcp://127.0.0.1:9001",
        world_size=world_size,
        rank=rank,
    )
    print("process begin", rank)

    for datatype in [None,torch.float,torch.float16,torch.float32,torch.float64]:
        if rank == 0:
            print(f"Current datatype: {datatype}.")
            t = torch.rand([4,4],dtype=datatype).to(torch.device('cuda',rank))      
            print(f"Generate tensor{t}")
            dist.send(t,1)

        elif rank == 1:
            r = torch.rand([4,4]).to(torch.device('cuda',rank)) 
            dist.recv(r,0)
            print("recv",r)
        print()
        time.sleep(1)

def main():
    mp.spawn(main_worker, nprocs=2, args=(2, 2))

if __name__ == "__main__":
    main()
```

## Output:

```python
process begin 0
Current datatype: None.
process begin 1
Generate tensortensor([[0.9230, 0.2856, 0.9419, 0.2844],
        [0.9732, 0.7029, 0.0026, 0.9697],
        [0.2188, 0.4143, 0.5163, 0.9863],
        [0.1562, 0.3484, 0.1138, 0.3271]], device='cuda:0')

recv tensor([[0.9230, 0.2856, 0.9419, 0.2844],
        [0.9732, 0.7029, 0.0026, 0.9697],
        [0.2188, 0.4143, 0.5163, 0.9863],
        [0.1562, 0.3484, 0.1138, 0.3271]], device='cuda:1')

Current datatype: torch.float32.
Generate tensortensor([[0.6158, 0.9911, 0.0677, 0.2109],
        [0.0591, 0.5609, 0.4182, 0.4432],
        [0.9296, 0.2350, 0.1028, 0.7265],
        [0.1949, 0.0324, 0.4484, 0.8104]], device='cuda:0')

recv tensor([[0.6158, 0.9911, 0.0677, 0.2109],
        [0.0591, 0.5609, 0.4182, 0.4432],
        [0.9296, 0.2350, 0.1028, 0.7265],
        [0.1949, 0.0324, 0.4484, 0.8104]], device='cuda:1')

Current datatype: torch.float16.
Generate tensortensor([[0.7212, 0.8945, 0.3042, 0.3184],
        [0.3804, 0.9648, 0.8076, 0.9756],
        [0.3862, 0.7358, 0.6611, 0.2539],
        [0.4365, 0.9434, 0.7075, 0.6084]], device='cuda:0',
       dtype=torch.float16)

recv tensor([[2.5669e-03, 5.6701e-07, 5.6217e-03, 6.2936e-03],
        [4.3337e-04, 1.3432e-07, 4.2790e-03, 1.0597e-04],
        [0.0000e+00, 0.0000e+00, 0.0000e+00, 0.0000e+00],
        [0.0000e+00, 0.0000e+00, 0.0000e+00, 0.0000e+00]], device='cuda:1')

Current datatype: torch.float32.
Generate tensortensor([[0.7289, 0.0532, 0.6294, 0.5030],
        [0.1043, 0.3015, 0.2626, 0.2357],
        [0.8202, 0.1919, 0.3556, 0.2653],
        [0.9763, 0.3292, 0.9931, 0.8236]], device='cuda:0')

recv tensor([[0.7289, 0.0532, 0.6294, 0.5030],
        [0.1043, 0.3015, 0.2626, 0.2357],
        [0.8202, 0.1919, 0.3556, 0.2653],
        [0.9763, 0.3292, 0.9931, 0.8236]], device='cuda:1')

Current datatype: torch.float64.
Generate tensortensor([[0.1401, 0.5205, 0.3881, 0.1536],
        [0.4686, 0.3280, 0.0725, 0.7440],
        [0.5029, 0.2960, 0.5149, 0.2452],
        [0.2024, 0.5243, 0.8930, 0.2613]], device='cuda:0',
       dtype=torch.float64)

recv tensor([[-9.8724e-14,  1.5151e+00,  1.6172e-05,  1.7551e+00],
        [ 1.6390e+35,  1.6941e+00, -1.9036e+12,  1.5286e+00],
        [ 6.4192e-38,  1.7343e+00,  2.3965e+07,  1.6640e+00],
        [-1.7412e+19,  1.3951e+00, -6.5071e+35,  1.8110e+00]], device='cuda:1')
```

### Versions

I tested and confirmed this phenomenon on two dual-GPU PCs and two versions of `PyTorch`.

#### Configure0: PyTorch `1.12.0`+`TITAN RTX`*2

```python
Collecting environment information...
PyTorch version: 1.12.0
Is debug build: False
CUDA used to build PyTorch: 11.6
ROCM used to build PyTorch: N/A

OS: Ubuntu 20.04.4 LTS (x86_64)
GCC version: (Ubuntu 7.5.0-6ubuntu2) 7.5.0
Clang version: Could not collect
CMake version: version 3.16.3
Libc version: glibc-2.31

Python version: 3.9.12 (main, Jun  1 2022, 11:38:51)  [GCC 7.5.0] (64-bit runtime)
Python platform: Linux-5.13.0-52-generic-x86_64-with-glibc2.31
Is CUDA available: True
CUDA runtime version: 11.7.64
GPU models and configuration:
GPU 0: NVIDIA TITAN RTX
GPU 1: NVIDIA TITAN RTX

Nvidia driver version: 510.73.05
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] info-nce-pytorch==0.1.4
[pip3] numpy==1.22.3
[pip3] torch==1.12.0
[pip3] torch-tb-profiler==0.4.0
[pip3] torchaudio==0.12.0
[pip3] torchinfo==1.6.5
[pip3] torchvision==0.13.0
[conda] blas                      1.0                         mkl
[conda] cudatoolkit               11.6.0              hecad31d_10    conda-forge
[conda] info-nce-pytorch          0.1.4                    pypi_0    pypi
[conda] libblas                   3.9.0            12_linux64_mkl    conda-forge
[conda] libcblas                  3.9.0            12_linux64_mkl    conda-forge
[conda] mkl                       2021.4.0           h06a4308_640
[conda] mkl-service               2.4.0            py39h7f8727e_0
[conda] mkl_fft                   1.3.1            py39hd3c417c_0
[conda] mkl_random                1.2.2            py39h51133e4_0
[conda] numpy                     1.22.3           py39he7a7128_0
[conda] numpy-base                1.22.3           py39hf524024_0
[conda] pytorch                   1.12.0          py3.9_cuda11.6_cudnn8.3.2_0    pytorch
[conda] pytorch-mutex             1.0                        cuda    pytorch
[conda] torch-tb-profiler         0.4.0                    pypi_0    pypi
[conda] torchaudio                0.12.0               py39_cu116    pytorch
[conda] torchinfo                 1.6.5              pyhd8ed1ab_0    conda-forge
[conda] torchvision               0.13.0               py39_cu116    pytorch
```

#### Configure1: PyTorch `1.8.2 LTS`+`TITAN RTX`*2

```python
Collecting environment information...
PyTorch version: 1.8.2
Is debug build: False
CUDA used to build PyTorch: 11.1
ROCM used to build PyTorch: N/A

OS: Ubuntu 20.04.4 LTS (x86_64)
GCC version: (Ubuntu 7.5.0-6ubuntu2) 7.5.0
Clang version: Could not collect
CMake version: version 3.16.3
Libc version: glibc-2.31

Python version: 3.8.13 (default, Mar 28 2022, 11:38:47)  [GCC 7.5.0] (64-bit runtime)
Python platform: Linux-5.13.0-52-generic-x86_64-with-glibc2.17
Is CUDA available: True
CUDA runtime version: 11.7.64
GPU models and configuration:
GPU 0: NVIDIA TITAN RTX
GPU 1: NVIDIA TITAN RTX

Nvidia driver version: 510.73.05
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] info-nce-pytorch==0.1.4
[pip3] numpy==1.22.3
[pip3] torch==1.8.2
[pip3] torchaudio==0.8.2
[pip3] torchinfo==1.6.5
[pip3] torchvision==0.9.2
[conda] blas                      1.0                         mkl
[conda] cudatoolkit               11.1.74              h6bb024c_0    nvidia
[conda] info-nce-pytorch          0.1.4                    pypi_0    pypi
[conda] mkl                       2021.4.0           h06a4308_640
[conda] mkl-service               2.4.0            py38h7f8727e_0
[conda] mkl_fft                   1.3.1            py38hd3c417c_0
[conda] mkl_random                1.2.2            py38h51133e4_0
[conda] numpy                     1.21.5           py38he7a7128_2
[conda] numpy-base                1.21.5           py38hf524024_2
[conda] pytorch                   1.8.2           py3.8_cuda11.1_cudnn8.0.5_0    pytorch-lts
[conda] torchaudio                0.8.2                      py38    pytorch-lts
[conda] torchinfo                 1.6.5              pyhd8ed1ab_0    conda-forge
[conda] torchvision               0.9.2                py38_cu111    pytorch-lts
```

#### Configure2: PyTorch `1.12.0`+`RTX3090`*2

```python
Collecting environment information...
PyTorch version: 1.12.0
Is debug build: False
CUDA used to build PyTorch: 11.6
ROCM used to build PyTorch: N/A

OS: Ubuntu 20.04.4 LTS (x86_64)
GCC version: (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0
Clang version: Could not collect
CMake version: version 3.22.0-rc2
Libc version: glibc-2.31

Python version: 3.10.4 (main, Mar 31 2022, 08:41:55) [GCC 7.5.0] (64-bit runtime)
Python platform: Linux-5.13.0-52-generic-x86_64-with-glibc2.31
Is CUDA available: True
CUDA runtime version: 11.7.64
GPU models and configuration:
GPU 0: NVIDIA GeForce RTX 3090
GPU 1: NVIDIA GeForce RTX 3090

Nvidia driver version: 510.73.05
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.8.4.1
/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.4.1
/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.4.1
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.4.1
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.4.1
/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.4.1
/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.4.1
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] info-nce-pytorch==0.1.4
[pip3] numpy==1.22.3
[pip3] torch==1.12.0
[pip3] torchaudio==0.12.0
[pip3] torchvision==0.13.0
[conda] blas                      1.0                         mkl
[conda] cudatoolkit               11.6.0               habf752d_9    nvidia
[conda] ffmpeg                    4.3                  hf484d3e_0    pytorch
[conda] info-nce-pytorch          0.1.4                    pypi_0    pypi
[conda] libblas                   3.9.0            12_linux64_mkl    conda-forge
[conda] libcblas                  3.9.0            12_linux64_mkl    conda-forge
[conda] liblapack                 3.9.0            12_linux64_mkl    conda-forge
[conda] mkl                       2021.4.0           h06a4308_640
[conda] mkl-service               2.4.0           py310ha2c4b55_0    conda-forge
[conda] mkl_fft                   1.3.1           py310h2b4bcf5_1    conda-forge
[conda] mkl_random                1.2.2           py310h00e6091_0
[conda] numpy                     1.22.3          py310hfa59a62_0
[conda] numpy-base                1.22.3          py310h9585f30_0
[conda] pytorch                   1.12.0          py3.10_cuda11.6_cudnn8.3.2_0    pytorch
[conda] pytorch-mutex             1.0                        cuda    pytorch
[conda] torchaudio                0.12.0              py310_cu116    pytorch
[conda] torchvision               0.13.0              py310_cu116    pytorch
```

#### Configure3: PyTorch `1.8.2 LTS`+`RTX3090`*2

```python
Collecting environment information...
PyTorch version: 1.8.2
Is debug build: False
CUDA used to build PyTorch: 11.1
ROCM used to build PyTorch: N/A

OS: Ubuntu 20.04.4 LTS (x86_64)
GCC version: (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0
Clang version: Could not collect
CMake version: version 3.22.0-rc2
Libc version: glibc-2.31

Python version: 3.8.13 (default, Mar 28 2022, 11:38:47)  [GCC 7.5.0] (64-bit runtime)
Python platform: Linux-5.13.0-52-generic-x86_64-with-glibc2.17
Is CUDA available: True
CUDA runtime version: 11.7.64
GPU models and configuration:
GPU 0: NVIDIA GeForce RTX 3090
GPU 1: NVIDIA GeForce RTX 3090

Nvidia driver version: 510.73.05
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.8.4.1
/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.4.1
/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.4.1
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.4.1
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.4.1
/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.4.1
/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.4.1
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] info-nce-pytorch==0.1.4
[pip3] numpy==1.21.5
[pip3] torch==1.8.2
[pip3] torch-tb-profiler==0.3.1
[pip3] torchaudio==0.8.2
[pip3] torchinfo==1.7.0
[pip3] torchvision==0.9.2
[conda] blas                      1.0                         mkl
[conda] cudatoolkit               11.1.74              h6bb024c_0    nvidia
[conda] mkl                       2021.4.0           h06a4308_640
[conda] mkl-service               2.4.0            py38h7f8727e_0
[conda] mkl_fft                   1.3.1            py38hd3c417c_0
[conda] mkl_random                1.2.2            py38h51133e4_0
[conda] numpy                     1.21.5           py38he7a7128_2
[conda] numpy-base                1.21.5           py38hf524024_2
[conda] pytorch                   1.8.2           py3.8_cuda11.1_cudnn8.0.5_0    pytorch-lts
[conda] torchaudio                0.8.2                      py38    pytorch-lts
[conda] torchinfo                 1.7.0              pyhd8ed1ab_0    conda-forge
[conda] torchvision               0.9.2                py38_cu111    pytorch-lts
```

cc @pietern @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @SciPioneer @H-Huang @kwen2501</span></p>
  </div>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
<aside class="onebox githubissue" data-onebox-src="https://github.com/NVIDIA/nccl/issues/703">
  <header class="source">

      <a href="https://github.com/NVIDIA/nccl/issues/703" target="_blank" rel="noopener nofollow ugc">github.com/NVIDIA/nccl</a>
  </header>

  <article class="onebox-body">
    <div class="github-row">
  <div class="github-icon-container" title="Issue">
	  <svg width="60" height="60" class="github-icon" viewbox="0 0 14 16" aria-hidden="true"><path d="M7 2.3c3.14 0 5.7 2.56 5.7 5.7s-2.56 5.7-5.7 5.7A5.71 5.71 0 0 1 1.3 8c0-3.14 2.56-5.7 5.7-5.7zM7 1C3.14 1 0 4.14 0 8s3.14 7 7 7 7-3.14 7-7-3.14-7-7-7zm1 3H6v5h2V4zm0 6H6v2h2v-2z"></path></svg>
  </div>

  <div class="github-info-container">
    <h4>
      <a href="https://github.com/NVIDIA/nccl/issues/703" target="_blank" rel="noopener nofollow ugc">Self-adaptive datatype in `ncclrecv()`</a>
    </h4>

    <div class="github-info">
      <div class="date">
        opened <span class="discourse-local-date" data-format="ll" data-date="2022-07-17" data-time="09:09:25" data-timezone="UTC">09:09AM - 17 Jul 22 UTC</span>
      </div>


      <div class="user">
        <a href="https://github.com/AdorableJiang" target="_blank" rel="noopener nofollow ugc">
          <img alt="AdorableJiang" src="https://avatars.githubusercontent.com/u/51251025?v=4" class="onebox-avatar-inline" width="20" height="20">
          AdorableJiang
        </a>
      </div>
    </div>

    <div class="labels">
    </div>
  </div>
</div>

  <div class="github-row">
    <p class="github-body-container">## Description
In the last few days I suffered from `torch.distributed.recv()` <span class="show-more-container"><a href="" rel="noopener" class="show-more">…</a></span><span class="excerpt hidden">getting the wrong tensor values from `torch.distributed.send()`. And it turns out to be caused by the mismatch tensor `dtype` between `send()` and `recv()`. Please see [#pytorch/pytorch/issues/81545](https://github.com/pytorch/pytorch/issues/81545). Although `nccl` has introduced this feature in its [doc](https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/usage/p2p.html#point-to-point-communication):

&gt; Any point-to-point communication needs two NCCL calls : a call to [ncclSend()](https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/api/p2p.html#c.ncclSend) on one rank and a corresponding [ncclRecv()](https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/api/p2p.html#c.ncclRecv) on the other rank, ___with the same count and data type___.

`PyTorch` says nothing about this in its [doc](https://pytorch.org/docs/stable/distributed.html#torch.distributed.send).

Besides, I am not the only one who encontered this problem.

&gt; Got wrong tensor when using dist.send to send tensors - distributed - PyTorch Forums
&gt; https://discuss.pytorch.org/t/got-wrong-tensor-when-using-dist-send-to-send-tensors/151812

## Expectation

In practice, the users may be aware of that they are going to tansfer a e.g. `float` type data, but can be hard or just inconvenience to know exactly the data type is `float16`, `float32`, or `float64`. 

Is it possible to implement an overloading of
```c++
ncclRecv(recvbuff, recvcount, recvtype, peer, comm, stream)
```
that looks like 
```c++
ncclRecv(recvbuff, recvcount, peer, comm, stream)
```
which does not require `recvtype` but acquire it from `sendtype`. And cast `recvbuff` to `sendtype` automatically.</span></p>
  </div>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>