When variables are transferred between GPUs, their values change

username: Antonio_Lopez
cooked: <p>When I was using pytorch2.1, I found the following weird situation. Is this a bug?</p>
<pre><code class="lang-auto">Python 3.9.13 | packaged by conda-forge | (main, May 27 2022, 16:58:50) 
[GCC 10.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.cuda.device_count()
8
&gt;&gt;&gt; torch.cuda.is_available()
True
&gt;&gt;&gt; torch.__version__
'2.1.0+cu121'
&gt;&gt;&gt; temp = torch.tensor([[33, 34, 34, 34, 34, 35, 35, 35, 35, 35, 35, 35, 35, 36, 36, 36, 36, 36, 36, 36, 36, 37, 37, 37, 38, 38]], device='cuda:0')
&gt;&gt;&gt; temp
tensor([[33, 34, 34, 34, 34, 35, 35, 35, 35, 35, 35, 35, 35, 36, 36, 36, 36, 36,
         36, 36, 36, 37, 37, 37, 38, 38]], device='cuda:0')
&gt;&gt;&gt; temp.to('cuda:1')
tensor([[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
         0, 0]], device='cuda:1')
&gt;&gt;&gt; temp.to('cpu' )
tensor([[33, 34, 34, 34, 34, 35, 35, 35, 35, 35, 35, 35, 35, 36, 36, 36, 36, 36,
         36, 36, 36, 37, 37, 37, 38, 38]])
&gt;&gt;&gt; temp.to('cuda:1' )
tensor([[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
         0, 0]], device='cuda:1')
&gt;&gt;&gt; temp.to('cuda:2' )
tensor([[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
         0, 0]], device='cuda:2')
&gt;&gt;&gt; temp.to('cuda:3' )
tensor([[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
         0, 0]], device='cuda:3')
&gt;&gt;&gt; temp.to('cuda:4' )
tensor([[33, 34, 34, 34, 34, 35, 35, 35, 35, 35, 35, 35, 35, 36, 36, 36, 36, 36,
         36, 36, 36, 37, 37, 37, 38, 38]], device='cuda:4')
&gt;&gt;&gt; temp.to('cuda:5' )
tensor([[33, 34, 34, 34, 34, 35, 35, 35, 35, 35, 35, 35, 35, 36, 36, 36, 36, 36,
         36, 36, 36, 37, 37, 37, 38, 38]], device='cuda:5')
&gt;&gt;&gt; temp.to('cuda:6' )
tensor([[33, 34, 34, 34, 34, 35, 35, 35, 35, 35, 35, 35, 35, 36, 36, 36, 36, 36,
         36, 36, 36, 37, 37, 37, 38, 38]], device='cuda:6')
&gt;&gt;&gt; temp.to('cuda:7' )
tensor([[33, 34, 34, 34, 34, 35, 35, 35, 35, 35, 35, 35, 35, 36, 36, 36, 36, 36,
         36, 36, 36, 37, 37, 37, 38, 38]], device='cuda:7')
&gt;&gt;&gt;
</code></pre>
<p>I measured it under different versions of torch (such as torch2.0.1+cuda118) and different servers A6000 (A6000 Ada), and I found this problem always exists. The test script is as follows:</p>
<pre><code class="lang-auto">import torch


gpus = ['cuda:' + str(i) for i in range(8)]

for i in range(8):

    cur_device = gpus[i]

    temp = torch.tensor([[33, 34, 34, 34, 34, 35, 35, 35, 35, 35, 35, 35, 35, 36, 36, 36, 36, 36, 36, 36, 36, 37, 37, 37, 38, 38]], device=cur_device)

    print('&gt;&gt;&gt;&gt;&gt;&gt;&gt;      ', temp, '      &lt;&lt;&lt;&lt;&lt;&lt;&lt;')

    for j in range(8):

        if i == j:
            continue

        to_device = gpus[j]

        _temp = temp.cpu().to(device=to_device)

        print(cur_device, ' -&gt; ', to_device)
        print(_temp)
</code></pre>

username: ptrblck
cooked: <p>Make sure your multi-GPU setup is properly working e.g. by running <a href="https://github.com/NVIDIA/nccl-tests"><code>nccl-test</code></a>s. If these tests also show data corruption, check the <a href="https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/troubleshooting.html#pci-access-control-services-acs">NCCL FAQ</a> as e.g. you might want to disable IOMMU if that’s active.</p>

username: fduwjj
cooked: <p>Can you share how you run the script? Did you just run it with python or torchrun?</p>

username: ptrblck
cooked: <p><code>nccl-test</code> is a binary and the repository shows its <a href="https://github.com/NVIDIA/nccl-tests?tab=readme-ov-file#usage">usage</a>. PyTorch is not used and irrelevant in these tests.</p>

username: Antonio_Lopez
cooked: <p>Thanks! I will check it.</p>

username: Antonio_Lopez
cooked: <p>Yes, I just run it with python, like python main.py.</p>

username: fduwjj
cooked: <p>I tested on a machine with 4 H100 and cannot repro it. And looks like this is not distributed related can you fire a github issue in pytorch repo? I guess maybe this is is cuda version specific.</p>