Bizarre CUDA behavior with nn.Linear

sjasonw: <p>I am encountering some particularly strange behavior with very simple usage of <code>nn.Linear</code> when using CUDA. I wanted to post and see if others have encountered similar behavior. I’ve posted some information about my environment below the example.</p>
<p><strong>Example:</strong></p>
<p>I start with a trivial <code>nn.Linear</code> and a <code>Tensor</code>, both on CUDA.</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn

linear = nn.Linear(1, 1, bias=False).to('cuda:0')
x = torch.ones(3, 1, device='cuda:0')
</code></pre>
<p>There’s only one parameter for <code>linear</code>:</p>
<pre><code class="lang-auto">&gt;&gt;&gt; linear.state_dict()
OrderedDict([('weight', tensor([[0.3293]], device='cuda:0'))])
</code></pre>
<p>This default parameter is fine for the example. The crazy behavior appears when we use <code>linear.forward</code>:</p>
<pre><code class="lang-auto">&gt;&gt;&gt; linear(x)  # This call returns nonsense
tensor([[0.0000],
        [1.8750],
        [1.0000]], device='cuda:0', grad_fn=&lt;MmBackward&gt;)
</code></pre>
<p>(Side note: if I repeatedly call <code>linear(x</code>), the result can fluctuate dramatically.)</p>
<p>Just as a reality-check, when running on CPU, the result comes out correctly:</p>
<pre><code class="lang-auto">&gt;&gt;&gt; linear.to('cpu')
&gt;&gt;&gt; x = x.cpu()
&gt;&gt;&gt; linear(x)
tensor([[0.3293],
        [0.3293],
        [0.3293]], grad_fn=&lt;MmBackward&gt;)
</code></pre>
<p><strong>Environment</strong><br>
I am using an EC2 with instance type p2.xlarge. I haven’t made any changes to the environment–I’m it’s a newly spun-up instance and I’m using a default virtual environment with <code>source activate pytorch_latest_p37</code>. The GPU is a Tesla K80. Here is the output from <code>nvidia-smi</code>:</p>
<pre><code class="lang-auto">$ nvidia-smi
Fri Jul  2 16:04:45 2021       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 450.119.03   Driver Version: 450.119.03   CUDA Version: 11.0     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  Tesla K80           On   | 00000000:00:1E.0 Off |                    0 |
| N/A   38C    P0    54W / 149W |    500MiB / 11441MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A     12914      C   ...rch_latest_p37/bin/python      497MiB |
+-----------------------------------------------------------------------------+

</code></pre>
<p>Thanks in advance for any suggestions about what could be going on here, and sorry if I’m making some silly obvious mistake.</p>

eqy: <p>I can’t seem to reproduce this on my end with</p>
<pre><code class="lang-auto">import torch
import subprocess

linearcuda = torch.nn.Linear(1,1,bias=False,device='cuda')
linearcpu = torch.nn.Linear(1,1,bias=False)
onescuda = torch.ones(3,1,device='cuda')
onescpu = torch.ones(3,1)
print(linearcuda(onescuda))
print(linearcpu(onescpu))
</code></pre>
<pre><code class="lang-auto">tensor([[-0.6604],
        [-0.6604],
        [-0.6604]], device='cuda:0', grad_fn=&lt;MmBackward&gt;)
tensor([[-0.6166],
        [-0.6166],
        [-0.6166]], grad_fn=&lt;MmBackward&gt;)
</code></pre>
<p>Can you give some more details about this setup e.g.,</p>
<pre><code class="lang-auto">print(torch.__version__)
print(torch.version.cuda)
</code></pre>

sjasonw: <p>Thanks very much for looking at this. Can’t say I’m surprised that it isn’t reproducible for you.</p>
<aside class="quote no-group" data-username="eqy" data-post="2" data-topic="125664">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/eqy/48/46233_2.png" class="avatar"> eqy:</div>
<blockquote>
<p>Can you give some more details about this setup e.g.,</p>
</blockquote>
</aside>
<pre><code class="lang-auto">&gt;&gt;&gt; print(torch.__version__)
1.8.1+cu111
&gt;&gt;&gt; print(torch.version.cuda)
11.1
</code></pre>
<p>Is there any other information I can provide?</p>

fabio: <p>The issue seems to be related to the version.</p>
<p>For the same one used above,</p>
<pre><code class="lang-auto">&gt;&gt;&gt; print(torch.__version__)
1.8.1+cu111
&gt;&gt;&gt; print(torch.version.cuda)
11.1
</code></pre>
<p>This is the result:</p>
<pre><code class="lang-auto">&gt;&gt;&gt; linearcuda = torch.nn.Linear(1,1,bias=False).to('cuda:0')
&gt;&gt;&gt; linearcpu = torch.nn.Linear(1,1,bias=False)
&gt;&gt;&gt; onescuda = torch.ones(3,1,device='cuda:0')
&gt;&gt;&gt; onescpu = torch.ones(3,1)
&gt;&gt;&gt; print(linearcuda(onescuda))
tensor([[1.],
        [1.],
        [1.]], device='cuda:0', grad_fn=&lt;MmBackward&gt;)
&gt;&gt;&gt; print(linearcpu(onescpu))
tensor([[0.2037],
        [0.2037],
        [0.2037]], grad_fn=&lt;MmBackward&gt;)
&gt;&gt;&gt; print(linearcuda.state_dict())
OrderedDict([('weight', tensor([[0.0904]], device='cuda:0'))])
</code></pre>
<p>While with an older version:</p>
<pre><code class="lang-auto">&gt;&gt;&gt; print(torch.__version__)
1.4.0
&gt;&gt;&gt; print(torch.version.cuda)
10.1
&gt;&gt;&gt; linearcuda = torch.nn.Linear(1,1,bias=False).to('cuda:0')
&gt;&gt;&gt; linearcpu = torch.nn.Linear(1,1,bias=False)
&gt;&gt;&gt; onescuda = torch.ones(3,1,device='cuda:0')
&gt;&gt;&gt; onescpu = torch.ones(3,1)
&gt;&gt;&gt; print(linearcuda(onescuda))
tensor([[0.2613],
        [0.2613],
        [0.2613]], device='cuda:0', grad_fn=&lt;MmBackward&gt;)
&gt;&gt;&gt; print(linearcpu(onescpu))
tensor([[0.8475],
        [0.8475],
        [0.8475]], grad_fn=&lt;MmBackward&gt;)
</code></pre>

ptrblck: <p>Could you update to <code>1.9.0</code> and rerun the code, please?<br>
We’ve seen some issues in the pip wheels in 1.8.0 and 1.8.1 in particular for sm_61 by leaking cublas symbols, which might be also visible on the K80.</p>

sjasonw: <p>You’re right, <code>pip install -U torch</code> seems to have fixed the problem. Thanks! Odd that this is a default configuration for the instance with such an issue…</p>