Tensor totally changes when allocating moving from gpu to gpu

JuanFMontesinos: <p>Hi,<br>
I was training a network using a single gpu, alright.<br>
As the gpu utilization was bit low I decided to do the preprocessing in a second gpu allocating tensors in dataset’s ‘<strong>getitem</strong>’ and working on the main thread.<br>
Evberything ok.<br>
Then I realized that when I move my ground-truth from cuda:1 to cuda:0 the tensor totally changes to a completely different one.</p>
<p>(Trying to reproduce but console crashed not freezing gpu memory… <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=9" title=":confused:" class="emoji" alt=":confused:"> )<br>
Any idea?</p>

ptrblck: <p>I don’t have an idea yet, but it sounds like a bug so we would need to dig into it.<br>
Could you explain your use case a bit so that we might try to work on a reproduction as well?</p>

JuanFMontesinos: <p>Well it’s nothing out of the box.<br>
I could reproduce something similar just by using the console.</p>
<pre><code class="lang-auto">import torch
a=(torch.rand(70,1,256,256)&gt;0.5).float().cuda(1)
b=a.cuda() # Same effect with a.to('cuda:0')
</code></pre>
<p>After restarting the computer and running the aforementioned code  B is all zeros.</p>
<p>Originally I discovered the issue using following a "complex pipeline which involves STFT, functional.grid_sample and einsum but I think that’s irrelevant.</p>
<p>Originally the tensor was stored in a dictionary</p>
<pre><code class="lang-auto">vars['gt']=vars['gt'].to(torch.device(0))
</code></pre>
<p>Afterwards vars[‘gt’] became a tensor bounded between -1 and 5 at gpu 0.<br>
Even trying</p>
<pre><code class="lang-auto">import torch
a=(torch.rand(70,1,256,256)&gt;0.5).float().cuda(1)
b=a.cuda() # Same effect with a.to('cuda:0')
c=b.cpu() 
</code></pre>
<p>just to force it to be on cpu (avoiding sync problems). Neither using torch.cuda.synchronize() between commands.<br>
Torch version is 1.2.0 according to torch.<strong>version</strong><br>
cuda version is 10.0.130<br>
ipython 3.6.8, default version<br>
Nvidia-smi 410.48<br>
Cuda0: Quadro P6000<br>
Cuda1: Gefore GTX 1080 Ti</p>

ptrblck: <p>Could you update PyTorch to the latest version, please, and retry the code?</p>
<p>If I’m not mistaken, we’ve seen a similar issue some time ago, which boiled down to a hardware issue, but I can’t find the post.<br>
Maybe <a class="mention" href="/u/alband">@albanD</a> remembers it.</p>

albanD: <p>I don’t remember exactly but I would bet on hardware issue as well. Can’t find the thread either.</p>

JuanFMontesinos: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a>, <a class="mention" href="/u/alband">@albanD</a><br>
It keeps happening with pytorch 1.4 and driver 440.<br>
I discovered something iteresting.<br>
It does happed going from GTX1080Ti to P6000 but not the other way around.</p>
<pre><code class="lang-auto">import torch
a=(torch.rand(70,1,256,256)&gt;0.5).float().cuda(1)
b=a.cuda() 
</code></pre>
<p>So this fails</p>
<pre><code class="lang-auto">import torch
a=(torch.rand(70,1,256,256)&gt;0.5).float().cuda()
b=a.cuda(1)
</code></pre>
<p>But this works</p>

ptrblck: <p>Could you try to disable P2P access via <a href="https://docs.nvidia.com/deeplearning/sdk/nccl-developer-guide/docs/env.html#nccl-p2p-level">these env variables</a> and check the behavior?</p>

JuanFMontesinos: <p>Hi,<br>
Exactly the same behaviour.</p>

JuanFMontesinos: <p>Is there any new about this?</p>

ptrblck: <p>Unfortunately, no updates as I cannot reproduce this issue and haven’t seen it before.<br>
I would recommend to:</p>
<ul>
<li>update PyTorch, the NVIDIA driver, etc. to the latest version</li>
<li>try different environments, such as the official PyTorch docker container or the <a href="https://ngc.nvidia.com/catalog/containers/nvidia:pytorch">NVIDIA NGC one</a>, which uses a newer NCCL version</li>
<li>run the PyTorch <a href="https://github.com/pytorch/pytorch/tree/master/test">tests</a> (in particular the CUDA tests) on both GPUs and check, if you are seeing some numerical issues</li>
</ul>
<p>If that doesn’t help, I wouldn’t exclude a hardware defect.</p>

JuanFMontesinos: <p>Hi,<br>
So I was running the tests</p>
<pre><code class="lang-auto">-i test_cuda -i test_cuda_primary_ctx -i test_torch -i test_expecttest -i test_foreach
</code></pre>
<p>and<br>
raiases this</p>
<pre><code class="lang-auto">Fail to import hypothesis in common_utils, tests are not derandomized
</code></pre>
<p>test_foreach raises</p>
<pre><code class="lang-auto">AttributeError: module 'torch' has no attribute '_foreach_add'
</code></pre>
<p>ant test_torch</p>
<pre><code class="lang-auto">  File "test_torch.py", line 29, in &lt;module&gt;
    from torch.testing._internal.common_utils import TestCase, iter_indices, TEST_NUMPY, TEST_SCIPY, \
ImportError: cannot import name 'wrapDeterministicFlagAPITest'
</code></pre>

albanD: <p>Hi,</p>
<p>This kind of error <code>AttributeError: module 'torch' has no attribute '_foreach_add'</code> where you’re missing a cpp API usually means that your python code for your install is not the same version as the bianry install.</p>
<p>That happens if you do <code>setup.py develop</code> and then update your local repo.<br>
You might want to clean up your install here.</p>

JuanFMontesinos: <p>Soo if my binary install is 1.6.0<br>
I understand that I should use the tests  under the tag<br>
<code>1.6.fix4</code> for example?<br>
There are several 1.6 tags in GitHub</p>

albanD: <p>If you’re using a binary install, that means that you either have both a binary and develop install at the same time that conflict. Or you have a folder called <code>torch</code> in your current directory.</p>

baffo32: <p>I’m experiencing this now on torch 1.13.1 on a K80. (the newer driver for torch 2 doesn’t support the K80 (edit: i guess this was a quirk of the image with newer nccl, maybe i will try torch 2).)</p>
<p>I have the tests running. It’s a little confusing because pytorch docker images can come with torch installed via conda, which can clash with a source install.</p>
<p>It seems ideally there’d be a way to log the cuda calls and narrow the problem down further.</p>
<p>I have another K80 I can swap in to try, too, although they’ve both weathered the same storms and could be broken similarly if there’s damage.</p>

whywww: <p>I encountered a similar issue when I was using a machine with multiple RTX4090 GPUs.<br>
It can be reproduced simply by running this snippet on the console:</p>
<pre><code class="lang-auto">&gt;&gt;&gt; l = torch.tensor(1, device='cuda:0')
&gt;&gt;&gt; l.to('cuda:1')
tensor(0, device='cuda:1')
</code></pre>
<p>It happens for both pytorch 2.0.0 and 1.10.0 on two RTX4090s, but it does NOT happen on two RTX3090 GPUs (pytorch 1.10.0).</p>
<p>The issue can temporarily be solved by moving to CPU first and then to the second GPU:</p>
<pre><code class="lang-auto">&gt;&gt;&gt; l = torch.tensor(1, device='cuda:0')
&gt;&gt;&gt; l.to('cpu').to('cuda:1')
tensor(1, device='cuda:1')
</code></pre>
<p>Any thoughts?</p>

JuanFMontesinos: <p>I’m afraid I couldn’t find a solution. I think I did some experiments and it was machine-related but cannot really ensure.</p>
<p>Best<br>
Juan</p>

ptrblck: <p>Disable P2P access between the 4090s via <code>NCCL_P2P_DISABLE=1</code> or update to the latest driver.</p>

whywww: <p>Thanks. I tried <code>NCCL_P2P_DISABLE=1</code> but it doesn’t work. I don’t have access to the driver update so I cannot try this way either.</p>