Something wrong with pytorch 1.7.1 and cuda 11.0

user_123454321: <p>I am getting weird results when I use pytorch 1.7.1 with cuda 11.0 on P100. Steps to reproduce:</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
torch.manual_seed(42)
l = [3, 32, 64, 128, 256, 512, 256, 128, 64, 32, 1]
model = nn.Sequential(
	*[nn.Conv2d(l[i], l[i+1], 3, padding=1) for i in range(len(l)-1)],
	nn.AdaptiveAvgPool2d((1, 1))
	).to('cuda:0')
torch.manual_seed(42)
a = torch.randn(16, 3, 224, 224)
b = model(a.to('cuda:0')).squeeze()
b
</code></pre>
<p>When I run this on 1.7.1 with cuda 11.0, I get</p>
<pre><code class="lang-auto">&gt;&gt;&gt; import torch
&gt;&gt;&gt; import torch.nn as nn
&gt;&gt;&gt; torch.manual_seed(42)
&lt;torch._C.Generator object at 0x7f56c3859290&gt;
&gt;&gt;&gt; l = [3, 32, 64, 128, 256, 512, 256, 128, 64, 32, 1]
&gt;&gt;&gt; model = nn.Sequential(
...     *[nn.Conv2d(l[i], l[i+1], 3, padding=1) for i in range(len(l)-1)],
...     nn.AdaptiveAvgPool2d((1, 1))
...     ).to('cuda:0')
&gt;&gt;&gt; torch.manual_seed(42)
&lt;torch._C.Generator object at 0x7f56c3859290&gt;
&gt;&gt;&gt; a = torch.randn(16, 3, 224, 224)
&gt;&gt;&gt; b = model(a.to('cuda:0')).squeeze()
&gt;&gt;&gt; torch.__version__
'1.7.1'
&gt;&gt;&gt; torch.version.cuda
'11.0'
&gt;&gt;&gt; b
tensor([0.0105, 0.0105, 0.0105, 0.0105, 0.0105, 0.0105, 0.0105, 0.0105, 0.0105,
        0.0105, 0.0105, 0.0105, 0.0105, 0.0105, 0.0105, 0.0105],
       device='cuda:0', grad_fn=&lt;SqueezeBackward0&gt;)
&gt;&gt;&gt;
</code></pre>
<p>But when I run on 1.7.0 with cuda 11.0, I get</p>
<pre><code class="lang-auto">&gt;&gt;&gt; import torch
&gt;&gt;&gt; import torch.nn as nn
&gt;&gt;&gt; torch.manual_seed(42)
&lt;torch._C.Generator object at 0x7f88ae14b050&gt;
&gt;&gt;&gt; l = [3, 32, 64, 128, 256, 512, 256, 128, 64, 32, 1]
&gt;&gt;&gt; model = nn.Sequential(
...     *[nn.Conv2d(l[i], l[i+1], 3, padding=1) for i in range(len(l)-1)],
...     nn.AdaptiveAvgPool2d((1, 1))
...     ).to('cuda:0')
&gt;&gt;&gt; torch.manual_seed(42)
&lt;torch._C.Generator object at 0x7f88ae14b050&gt;
&gt;&gt;&gt; a = torch.randn(16, 3, 224, 224)
&gt;&gt;&gt; b = model(a.to('cuda:0')).squeeze()
&gt;&gt;&gt; torch.__version__
'1.7.0'
&gt;&gt;&gt; torch.version.cuda
'11.0'
&gt;&gt;&gt; b
tensor([0.0109, 0.0109, 0.0109, 0.0109, 0.0109, 0.0109, 0.0109, 0.0109, 0.0109,
        0.0109, 0.0109, 0.0109, 0.0109, 0.0108, 0.0108, 0.0109],
       device='cuda:0', grad_fn=&lt;SqueezeBackward0&gt;)
</code></pre>
<p>Then I tried with pytorch 1.7.1 and 10.2 and I get the same</p>
<pre><code class="lang-auto">&gt;&gt;&gt; import torch
&gt;&gt;&gt; import torch.nn as nn
&gt;&gt;&gt; torch.manual_seed(42)
&lt;torch._C.Generator object at 0x7f87f1b50270&gt;
&gt;&gt;&gt; l = [3, 32, 64, 128, 256, 512, 256, 128, 64, 32, 1]
&gt;&gt;&gt; model = nn.Sequential(
...     *[nn.Conv2d(l[i], l[i+1], 3, padding=1) for i in range(len(l)-1)],
...     nn.AdaptiveAvgPool2d((1, 1))
...     ).to('cuda:0')
&gt;&gt;&gt; torch.manual_seed(42)
&lt;torch._C.Generator object at 0x7f87f1b50270&gt;
&gt;&gt;&gt; a = torch.randn(16, 3, 224, 224)
&gt;&gt;&gt; b = model(a.to('cuda:0')).squeeze()
&gt;&gt;&gt; torch.__version__
'1.7.1'
&gt;&gt;&gt; torch.version.cuda
'10.2'
&gt;&gt;&gt; b
tensor([0.0109, 0.0109, 0.0109, 0.0109, 0.0109, 0.0109, 0.0109, 0.0109, 0.0109,
        0.0109, 0.0109, 0.0109, 0.0109, 0.0108, 0.0108, 0.0109],
       device='cuda:0', grad_fn=&lt;SqueezeBackward0&gt;)
&gt;&gt;&gt;
</code></pre>
<p>Then I tried on cpu…</p>
<pre><code class="lang-auto">&gt;&gt;&gt; import torch
&gt;&gt;&gt; import torch.nn as nn
&gt;&gt;&gt; torch.manual_seed(42)
&lt;torch._C.Generator object at 0x7f56c3859290&gt;
&gt;&gt;&gt; l = [3, 32, 64, 128, 256, 512, 256, 128, 64, 32, 1]
&gt;&gt;&gt; model = nn.Sequential(
...     *[nn.Conv2d(l[i], l[i+1], 3, padding=1) for i in range(len(l)-1)],
...     nn.AdaptiveAvgPool2d((1, 1))
...     )
&gt;&gt;&gt; torch.manual_seed(42)
&lt;torch._C.Generator object at 0x7f56c3859290&gt;
&gt;&gt;&gt; a = torch.randn(16, 3, 224, 224)
&gt;&gt;&gt; b = model(a).squeeze()
&gt;&gt;&gt; torch.__version__
'1.7.1'
&gt;&gt;&gt; torch.version.cuda
'11.0'
&gt;&gt;&gt; b
tensor([0.0109, 0.0109, 0.0109, 0.0109, 0.0109, 0.0109, 0.0109, 0.0109, 0.0109,
        0.0109, 0.0109, 0.0109, 0.0109, 0.0108, 0.0108, 0.0109],
       grad_fn=&lt;SqueezeBackward0&gt;)
</code></pre>
<p>As we can see, just on 1.7.1 with cuda 11.0 we are getting a different result. This is really weird and as far as I have seen it only occurs when we have high number of filters and batch size (I think it happens when there is some kind of memory issue, but I am not sure).<br>
I encountered this, when I was training my regular UNet models in 1.7.1 with 10.2 and when I moved to 11.0, suddenly my loss jumped to nan pretty soon. It happened to multiple models with different datasets! Then I tried to train on 10.2 again and it did not occur.</p>
<p><a class="mention" href="/u/ptrblck">@ptrblck</a>, can you please look into this.</p>

ptrblck: <p>I cannot reproduce this issue on a recent source build using CUDA11.2 and an internal cudnn version as well as the PyTorch 1.7.1 conda binaries with CUDA11.0 and get always:</p>
<pre><code class="lang-python">tensor([0.0109, 0.0109, 0.0109, 0.0109, 0.0109, 0.0109, 0.0109, 0.0109, 0.0109,
        0.0109, 0.0109, 0.0109, 0.0109, 0.0108, 0.0108, 0.0109],
</code></pre>
<p>on a node with P100-SXM2 16GB GPUs.</p>
<p>EDIT: could you store the <code>state_dict</code> as well as the input from the CUDA10.2 run, load it in the CUDA11.0 environment, and check the outputs again?</p>

user_123454321: <p>Hi, thank you so much for checking this. I tried on 2 instances and it happened, so I guessed it would be reproducible. I saved model and ‘a’ in 10.2 using</p>
<pre><code class="lang-auto">&gt;&gt;&gt; import torch
&gt;&gt;&gt; import torch.nn as nn
&gt;&gt;&gt; torch.__version__
'1.7.1'
&gt;&gt;&gt; torch.version.cuda
'10.2'
&gt;&gt;&gt; torch.manual_seed(42)
&lt;torch._C.Generator object at 0x7f02f91602b0&gt;
&gt;&gt;&gt; l = [3, 32, 64, 128, 256, 512, 256, 128, 64, 32, 1]
&gt;&gt;&gt; model = nn.Sequential(
...     *[nn.Conv2d(l[i], l[i+1], 3, padding=1) for i in range(len(l)-1)],
...     nn.AdaptiveAvgPool2d((1, 1))
...     ).to('cuda:0')
&gt;&gt;&gt; torch.manual_seed(42)
&lt;torch._C.Generator object at 0x7f02f91602b0&gt;
&gt;&gt;&gt; a = torch.randn(16, 3, 224, 224)
&gt;&gt;&gt; b = model(a.to('cuda:0')).squeeze()
&gt;&gt;&gt; b
tensor([0.0109, 0.0109, 0.0109, 0.0109, 0.0109, 0.0109, 0.0109, 0.0109, 0.0109,
        0.0109, 0.0109, 0.0109, 0.0109, 0.0108, 0.0108, 0.0109],
       device='cuda:0', grad_fn=&lt;SqueezeBackward0&gt;)
&gt;&gt;&gt; torch.save({'a': a}, 'a_10_2.pth')
&gt;&gt;&gt; torch.save(model.state_dict(), 'model_10_2.pth')
</code></pre>
<p>and then load into model in cuda 11 as</p>
<pre><code class="lang-auto">&gt;&gt;&gt; import torch
&gt;&gt;&gt; import torch.nn as nn
&gt;&gt;&gt; torch.__version__
'1.7.1'
&gt;&gt;&gt; torch.version.cuda
'11.0'
&gt;&gt;&gt; l = [3, 32, 64, 128, 256, 512, 256, 128, 64, 32, 1]
&gt;&gt;&gt; model = nn.Sequential(
...     *[nn.Conv2d(l[i], l[i+1], 3, padding=1) for i in range(len(l)-1)],
...     nn.AdaptiveAvgPool2d((1, 1))
...     ).to('cuda:0')
&gt;&gt;&gt; model.load_state_dict(torch.load('model_10_2.pth'))
&lt;All keys matched successfully&gt;
&gt;&gt;&gt; a = torch.load('a_10_2.pth')['a']
&gt;&gt;&gt; b = model(a.to('cuda:0')).squeeze()
&gt;&gt;&gt; b
tensor([0.0105, 0.0105, 0.0105, 0.0105, 0.0105, 0.0105, 0.0105, 0.0105, 0.0105,
        0.0105, 0.0105, 0.0105, 0.0105, 0.0105, 0.0105, 0.0105],
       device='cuda:0', grad_fn=&lt;SqueezeBackward0&gt;)
</code></pre>

user_123454321: <p>Note, this does not happen when the batch size is small (actually this occurred when I was using batch size of 2 in segmentation, so I think it is more to do with how much memory is being used rather than just batch size).<br>
On 10.2</p>
<pre><code class="lang-auto">&gt;&gt;&gt; import torch
&gt;&gt;&gt; import torch.nn as nn
&gt;&gt;&gt; torch.__version__
'1.7.1'
&gt;&gt;&gt; torch.version.cuda
'10.2'
&gt;&gt;&gt; torch.manual_seed(42)
&lt;torch._C.Generator object at 0x7f32829412b0&gt;
&gt;&gt;&gt; l = [3, 32, 64, 128, 256, 512, 256, 128, 64, 32, 1]
&gt;&gt;&gt; model = nn.Sequential(
...     *[nn.Conv2d(l[i], l[i+1], 3, padding=1) for i in range(len(l)-1)],
...     nn.AdaptiveAvgPool2d((1, 1))
...     ).to('cuda:0')
&gt;&gt;&gt; torch.manual_seed(42)
&lt;torch._C.Generator object at 0x7f32829412b0&gt;
&gt;&gt;&gt; a = torch.randn(4, 3, 224, 224)
&gt;&gt;&gt; b = model(a.to('cuda:0')).squeeze()
&gt;&gt;&gt; b
tensor([0.0109, 0.0109, 0.0109, 0.0109], device='cuda:0',
       grad_fn=&lt;SqueezeBackward0&gt;)
</code></pre>
<p>On 11.0</p>
<pre><code class="lang-auto">&gt;&gt;&gt; import torch
&gt;&gt;&gt; import torch.nn as nn
&gt;&gt;&gt; torch.__version__
'1.7.1'
&gt;&gt;&gt; torch.version.cuda
'11.0'
&gt;&gt;&gt; torch.manual_seed(42)
&lt;torch._C.Generator object at 0x7f1da0aaa290&gt;
&gt;&gt;&gt; l = [3, 32, 64, 128, 256, 512, 256, 128, 64, 32, 1]
&gt;&gt;&gt; model = nn.Sequential(
...     *[nn.Conv2d(l[i], l[i+1], 3, padding=1) for i in range(len(l)-1)],
...     nn.AdaptiveAvgPool2d((1, 1))
...     ).to('cuda:0')
&gt;&gt;&gt; torch.manual_seed(42)
&lt;torch._C.Generator object at 0x7f1da0aaa290&gt;
&gt;&gt;&gt; a = torch.randn(4, 3, 224, 224)
&gt;&gt;&gt; b = model(a.to('cuda:0')).squeeze()
&gt;&gt;&gt; b
tensor([0.0109, 0.0109, 0.0109, 0.0109], device='cuda:0',
       grad_fn=&lt;SqueezeBackward0&gt;)
</code></pre>