When does nan get turned into inf?

username: Sam_Lerman
cooked: <p>I am running two Conv2d  layers on a tensor of nans and getting -infs as output.</p>
<p>Two questions:</p>
<ol>
<li>why?</li>
<li>is there a way to prevent this and keep them as nans?</li>
</ol>

username: KFrank
cooked: <p>Hi Sam!</p>
<aside class="quote no-group" data-username="Sam_Lerman" data-post="1" data-topic="142191" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/sam_lerman/48/37165_2.png" class="avatar"> Sam_Lerman:</div>
<blockquote>
<p>I am running two Conv2d  layers on a tensor of nans and getting -infs as output.</p>
</blockquote>
</aside>
<p>This is strange, and not something I would expect to happen.  After all,<br>
“nan” is the “One Floating-Point Number to rule them all.”</p>
<p>“nan” should infect all arithmetic and turn the results into “nan”:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.__version__
'1.9.0'
&gt;&gt;&gt; tnan = torch.tensor ([float ('nan')])
&gt;&gt;&gt; tinf = torch.tensor ([float ('inf')])
&gt;&gt;&gt; tnan
tensor([nan])
&gt;&gt;&gt; tnan + tinf
tensor([nan])
&gt;&gt;&gt; tnan * tinf
tensor([nan])
&gt;&gt;&gt; tnan / 0.0
tensor([nan])
</code></pre>
<p>As expected, when I feed “nan” into a <code>Conv2d</code>, I get “nan” out:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; conv = torch.nn.Conv2d (1, 1, 3)
&gt;&gt;&gt; t = torch.randn (1, 1, 10, 10)
&gt;&gt;&gt; t[0, 0, 4, 4] = tnan
&gt;&gt;&gt; conv (t)
tensor([[[[ 0.0393,  0.0779,  0.0731,  0.8591, -0.8162,  0.6578, -0.8222,
            0.8873],
          [-1.3210,  1.4260,  0.0526, -0.5711,  1.2245,  0.7201, -0.3848,
           -0.2762],
          [ 0.1857, -0.7266,     nan,     nan,     nan,  0.8714,  0.4864,
            0.2397],
          [-0.7764,  1.5293,     nan,     nan,     nan, -0.0663,  0.2233,
           -0.5896],
          [ 0.2750, -0.0982,     nan,     nan,     nan, -0.2574, -0.1529,
            0.5295],
          [ 0.7697,  1.0994, -0.3693, -0.5683, -0.4822,  0.9385, -0.7202,
            0.7361],
          [-1.3883,  0.6983,  0.3545,  0.8573, -0.0080, -0.2240,  0.2517,
           -0.4848],
          [-0.5190,  0.1873,  0.3782,  0.3108, -0.1297, -0.5012,  1.1124,
           -0.7626]]]], grad_fn=&lt;ThnnConv2DBackward&gt;)
</code></pre>
<p>Can you reproduce this with short, self-contained, runnable script?</p>
<p>Best.</p>
<p>K. Frank</p>

username: smth
cooked: <p>Are you also running a max-pooling layer? I can see how a max-pooling implementation somewhere might start with</p>
<pre><code class="lang-auto">max = -inf
</code></pre>
<p>and then try to find the max of a window, but comparing anything to <code>nan</code> is futile, and maybe the <code>max</code> remains to be <code>-inf</code></p>

username: KFrank
cooked: <p>Hi smth!</p>
<aside class="quote no-group" data-username="smth" data-post="3" data-topic="142191" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/smth/48/13_2.png" class="avatar"> smth:</div>
<blockquote>
<p>but comparing anything to <code>nan</code> is futile, and maybe the <code>max</code> remains to be <code>-inf</code></p>
</blockquote>
</aside>
<p>Pytorch’s <code>min()</code> and <code>max()</code> appear to “do the right thing” with <code>nan</code>:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.__version__
'1.9.0'
&gt;&gt;&gt; tnan = torch.tensor ([float ('nan')])
&gt;&gt;&gt; tinf = torch.tensor ([float ('inf')])
&gt;&gt;&gt; torch.min (tinf, tnan)
tensor([nan])
&gt;&gt;&gt; torch.max (tinf, tnan)
tensor([nan])
</code></pre>
<p>Also, <code>MaxPool2d</code> appears to work as “expected”:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; maxp = torch.nn.MaxPool2d (3)
&gt;&gt;&gt; t = torch.randn (1, 1, 10, 10)
&gt;&gt;&gt; t[0, 0, 4, 4] = tnan
&gt;&gt;&gt; maxp (t)
tensor([[[[1.1035, 1.3305, 3.2121],
          [2.3521,    nan, 1.2039],
          [0.7372, 0.4652, 2.2204]]]])
</code></pre>
<p>Best.</p>
<p>K. Frank</p>

username: Sam_Lerman
cooked: <p>Okay, I have a short reproduced script. It returns:</p>
<pre><code class="lang-auto">tensor(True, device='cuda:0')
tensor(False, device='cuda:0')
</code></pre>
<p>on cuda, however not on CPU:</p>
<pre><code class="lang-auto">import torch
from torch import nn

class CNN(nn.Module):
    def __init__(self):
        super().__init__()
        self.CNN = nn.Sequential(nn.Conv2d(1, 32, 3, stride=2))

    def forward(self, x):
        return self.CNN(x)

cnn = CNN().to('cuda')
x = torch.full([128, 1, 28, 28], float('nan')).to('cuda')
print(torch.isnan(x).all())
y = cnn(x)
print(torch.isnan(y).all())
</code></pre>
<p>On CPU, it returns</p>
<pre><code class="lang-auto">tensor(True)
tensor(True)
</code></pre>
<p>as expected.</p>

username: Sam_Lerman
cooked: <p>An even shorter version:</p>
<pre><code class="lang-auto">import torch

cnn = torch.nn.Conv2d(1, 32, 3, stride=2).to('cuda')
x = torch.full([128, 1, 28, 28], float('nan')).to('cuda')
print(torch.isnan(x).all())
y = cnn(x)
print(torch.isnan(y).all())
</code></pre>
<p>Prints:</p>
<pre><code class="lang-auto">tensor(True, device='cuda:0')
tensor(False, device='cuda:0')
</code></pre>

username: KFrank
cooked: <p>Hi Sam!</p>
<aside class="quote no-group" data-username="Sam_Lerman" data-post="6" data-topic="142191" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/sam_lerman/48/37165_2.png" class="avatar"> Sam_Lerman:</div>
<blockquote>
<pre><code class="lang-auto">cnn = torch.nn.Conv2d(1, 32, 3, stride=2).to('cuda')
x = torch.full([128, 1, 28, 28], float('nan')).to('cuda')
print(torch.isnan(x).all())
y = cnn(x)
</code></pre>
</blockquote>
</aside>
<p>Okay, this is fun …</p>
<p>My take:</p>
<pre data-code-wrap="python"><code class="lang-python">import torch
print (torch.__version__)
print (torch.version.cuda)
print (torch.cuda.get_device_name())

kernel = 1
stride = 1
H = 1
W = 1
res = torch.nn.Conv2d (1, 1, kernel_size = kernel, stride = stride).cuda() (torch.full ((1, 1, H, W), float ('nan'), device = 'cuda'))
print ('kernel:', kernel, ', stride:', stride, ', H:', H, ', W:', W)
print ('all nans:', torch.isnan (res).all().item(), '  all infs:', torch.isinf (res).all().item())

kernel = 3
H = 3
W = 74
res = torch.nn.Conv2d (1, 1, kernel_size = kernel, stride = stride).cuda() (torch.full ((1, 1, H, W), float ('nan'), device = 'cuda'))
print ('kernel:', kernel, ', stride:', stride, ', H:', H, ', W:', W)
print ('all nans:', torch.isnan (res).all().item(), '  all infs:', torch.isinf (res).all().item())
W = 75
res = torch.nn.Conv2d (1, 1, kernel_size = kernel, stride = stride).cuda() (torch.full ((1, 1, H, W), float ('nan'), device = 'cuda'))
print ('kernel:', kernel, ', stride:', stride, ', H:', H, ', W:', W)
print ('all nans:', torch.isnan (res).all().item(), '  all infs:', torch.isinf (res).all().item())

H = 2048
W = 74
res = torch.nn.Conv2d (1, 1, kernel_size = kernel, stride = stride).cuda() (torch.full ((1, 1, H, W), float ('nan'), device = 'cuda'))
print ('kernel:', kernel, ', stride:', stride, ', H:', H, ', W:', W)
print ('all nans:', torch.isnan (res).all().item(), '  all infs:', torch.isinf (res).all().item())
W = 75
res = torch.nn.Conv2d (1, 1, kernel_size = kernel, stride = stride).cuda() (torch.full ((1, 1, H, W), float ('nan'), device = 'cuda'))
print ('kernel:', kernel, ', stride:', stride, ', H:', H, ', W:', W)
print ('all nans:', torch.isnan (res).all().item(), '  all infs:', torch.isinf (res).all().item())

kernel = 1
stride = 2
H = 1
W = 1
res = torch.nn.Conv2d (1, 1, kernel_size = kernel, stride = stride).cuda() (torch.full ((1, 1, H, W), float ('nan'), device = 'cuda'))
print ('kernel:', kernel, ', stride:', stride, ', H:', H, ', W:', W)
print ('all nans:', torch.isnan (res).all().item(), '  all infs:', torch.isinf (res).all().item())
H = 2
W = 1
res = torch.nn.Conv2d (1, 1, kernel_size = kernel, stride = stride).cuda() (torch.full ((1, 1, H, W), float ('nan'), device = 'cuda'))
print ('kernel:', kernel, ', stride:', stride, ', H:', H, ', W:', W)
print ('all nans:', torch.isnan (res).all().item(), '  all infs:', torch.isinf (res).all().item())

H = 1
W = 2048
res = torch.nn.Conv2d (1, 1, kernel_size = kernel, stride = stride).cuda() (torch.full ((1, 1, H, W), float ('nan'), device = 'cuda'))
print ('kernel:', kernel, ', stride:', stride, ', H:', H, ', W:', W)
print ('all nans:', torch.isnan (res).all().item(), '  all infs:', torch.isinf (res).all().item())
H = 2
W = 2048
res = torch.nn.Conv2d (1, 1, kernel_size = kernel, stride = stride).cuda() (torch.full ((1, 1, H, W), float ('nan'), device = 'cuda'))
print ('kernel:', kernel, ', stride:', stride, ', H:', H, ', W:', W)
print ('all nans:', torch.isnan (res).all().item(), '  all infs:', torch.isinf (res).all().item())

kernel = 3
stride = 2
H = 10
W = 3
res = torch.nn.Conv2d (1, 1, kernel_size = kernel, stride = stride).cuda() (torch.full ((1, 1, H, W), float ('nan'), device = 'cuda'))
print ('kernel:', kernel, ', stride:', stride, ', H:', H, ', W:', W)
print ('all nans:', torch.isnan (res).all().item(), '  all infs:', torch.isinf (res).all().item())
H = 11
res = torch.nn.Conv2d (1, 1, kernel_size = kernel, stride = stride).cuda() (torch.full ((1, 1, H, W), float ('nan'), device = 'cuda'))
print ('kernel:', kernel, ', stride:', stride, ', H:', H, ', W:', W)
print ('all nans:', torch.isnan (res).all().item(), '  all infs:', torch.isinf (res).all().item())

H = 3
W = 14
res = torch.nn.Conv2d (1, 1, kernel_size = kernel, stride = stride).cuda() (torch.full ((1, 1, H, W), float ('nan'), device = 'cuda'))
print ('kernel:', kernel, ', stride:', stride, ', H:', H, ', W:', W)
print ('all nans:', torch.isnan (res).all().item(), '  all infs:', torch.isinf (res).all().item())
W = 15
res = torch.nn.Conv2d (1, 1, kernel_size = kernel, stride = stride).cuda() (torch.full ((1, 1, H, W), float ('nan'), device = 'cuda'))
print ('kernel:', kernel, ', stride:', stride, ', H:', H, ', W:', W)
print ('all nans:', torch.isnan (res).all().item(), '  all infs:', torch.isinf (res).all().item())
</code></pre>
<p>And its output:</p>
<pre><code class="lang-plaintext">1.10.0
10.2
GeForce GTX 1050 Ti
kernel: 1 , stride: 1 , H: 1 , W: 1
all nans: False   all infs: True
kernel: 3 , stride: 1 , H: 3 , W: 74
all nans: True   all infs: False
kernel: 3 , stride: 1 , H: 3 , W: 75
all nans: False   all infs: True
kernel: 3 , stride: 1 , H: 2048 , W: 74
all nans: True   all infs: False
kernel: 3 , stride: 1 , H: 2048 , W: 75
all nans: False   all infs: True
kernel: 1 , stride: 2 , H: 1 , W: 1
all nans: True   all infs: False
kernel: 1 , stride: 2 , H: 2 , W: 1
all nans: False   all infs: True
kernel: 1 , stride: 2 , H: 1 , W: 2048
all nans: True   all infs: False
kernel: 1 , stride: 2 , H: 2 , W: 2048
all nans: False   all infs: True
kernel: 3 , stride: 2 , H: 10 , W: 3
all nans: True   all infs: False
kernel: 3 , stride: 2 , H: 11 , W: 3
all nans: False   all infs: True
kernel: 3 , stride: 2 , H: 3 , W: 14
all nans: True   all infs: False
kernel: 3 , stride: 2 , H: 3 , W: 15
all nans: False   all infs: True
</code></pre>
<p>Technically speaking, I would say that this is a bug.  <a class="mention" href="/u/ptrblck">@ptrblck</a>?</p>
<p>Best.</p>
<p>K. Frank</p>

username: ptrblck
cooked: <p>It could be a known issue with e.g. older cuDNN versions.<br>
Since you are using the CUDA 10.2 binaries, your cuDNN version would be 7.6.5, so could you update to the latest CUDA11 binaries?<br>
With CUDA 11.5 and cuDNN 8.3.2 I get:</p>
<pre><code class="lang-python">1.11.0.dev20220108+cu115
11.5
NVIDIA GeForce RTX 3090
kernel: 1 , stride: 1 , H: 1 , W: 1
all nans: True   all infs: False
kernel: 3 , stride: 1 , H: 3 , W: 74
all nans: True   all infs: False
kernel: 3 , stride: 1 , H: 3 , W: 75
all nans: True   all infs: False
kernel: 3 , stride: 1 , H: 2048 , W: 74
all nans: True   all infs: False
kernel: 3 , stride: 1 , H: 2048 , W: 75
all nans: True   all infs: False
kernel: 1 , stride: 2 , H: 1 , W: 1
all nans: True   all infs: False
kernel: 1 , stride: 2 , H: 2 , W: 1
all nans: True   all infs: False
kernel: 1 , stride: 2 , H: 1 , W: 2048
all nans: True   all infs: False
kernel: 1 , stride: 2 , H: 2 , W: 2048
all nans: True   all infs: False
kernel: 3 , stride: 2 , H: 10 , W: 3
all nans: True   all infs: False
kernel: 3 , stride: 2 , H: 11 , W: 3
all nans: True   all infs: False
kernel: 3 , stride: 2 , H: 3 , W: 14
all nans: True   all infs: False
kernel: 3 , stride: 2 , H: 3 , W: 15
all nans: True   all infs: False
</code></pre>

username: KFrank
cooked: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a>!</p>
<aside class="quote no-group" data-username="ptrblck" data-post="8" data-topic="142191" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>Since you are using the CUDA 10.2 binaries, your cuDNN version would be 7.6.5, so could you update to the latest CUDA11 binaries?</p>
</blockquote>
</aside>
<p>Yes, I can confirm that upgrading to cuda 11 makes the <code>inf</code>s go away.<br>
After updating my nightly installation to “cudatoolkit=11.3,” I get:</p>
<pre><code class="lang-plaintext">1.11.0.dev20220124
11.3
GeForce GTX 1050 Ti
kernel: 1 , stride: 1 , H: 1 , W: 1
all nans: True   all infs: False
kernel: 3 , stride: 1 , H: 3 , W: 74
all nans: True   all infs: False
kernel: 3 , stride: 1 , H: 3 , W: 75
all nans: True   all infs: False
kernel: 3 , stride: 1 , H: 2048 , W: 74
all nans: True   all infs: False
kernel: 3 , stride: 1 , H: 2048 , W: 75
all nans: True   all infs: False
kernel: 1 , stride: 2 , H: 1 , W: 1
all nans: True   all infs: False
kernel: 1 , stride: 2 , H: 2 , W: 1
all nans: True   all infs: False
kernel: 1 , stride: 2 , H: 1 , W: 2048
all nans: True   all infs: False
kernel: 1 , stride: 2 , H: 2 , W: 2048
all nans: True   all infs: False
kernel: 3 , stride: 2 , H: 10 , W: 3
all nans: True   all infs: False
kernel: 3 , stride: 2 , H: 11 , W: 3
all nans: True   all infs: False
kernel: 3 , stride: 2 , H: 3 , W: 14
all nans: True   all infs: False
kernel: 3 , stride: 2 , H: 3 , W: 15
all nans: True   all infs: False
</code></pre>
<p>Best.</p>
<p>K. Frank</p>

username: Sam_Lerman
cooked: <p>My school’s Nvidia driver might not support CUDA &gt;10.2 <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=12" title=":confused:" class="emoji" alt=":confused:"></p>