Beginner question: model returns NAN

flyingdutchman: <p>Hi, my model returns a NAN, i’m using the torchvision datasets api to get the MNIST dataset.<br>
Then I expend the 28x28 images in an extra dimension with</p>
<pre><code class="lang-auto">torch.einsum('i,jkl-&gt;jikl', vector, img)
</code></pre>
<p>where <code>vector</code> is array of ones with zeros on the front and back<br>
the resulting shape is <code>(B, C, D, H, W) = (1, 1, 28, 28, 28)</code> and looks like:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/d/9dde32b42418933955e7a380ab00843f26b4a536.png" data-download-href="https://discuss.pytorch.org/uploads/default/9dde32b42418933955e7a380ab00843f26b4a536" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/9/d/9dde32b42418933955e7a380ab00843f26b4a536.png" alt="image" data-base62-sha1="mwz1eVdmhz8NOIXnVEejXVoTqVo" width="345" height="223" data-dominant-color="1E1E1E"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">703×455 14 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>The model i’m using is defined as:</p>
<pre data-code-wrap="Python"><code class="lang-Python">class ConvAutoEncoder4(torch.nn.Module):
    def __init__(self):
        super(ConvAutoEncoder4, self).__init__()
        channels = 4
        self.fcn3 = nn.Conv3d(in_channels=1, out_channels=channels, kernel_size=4, stride=4)
        self.fcn4 = nn.Conv3d(in_channels=channels, out_channels=channels, kernel_size=1)
        self.fcn5 = nn.Conv3d(in_channels=channels, out_channels=channels, kernel_size=1)
        self.fcn6 = nn.ConvTranspose3d(in_channels=channels, out_channels=1, kernel_size=4, stride=4)
        
    def forward(self, tensor):
        
#         for slc in tensor[0][0]:
#             print(slc)
        
#         print(tensor.shape)
        tensor = self.fcn3(tensor)
        tensor = torch.relu(tensor)
        print(tensor.shape)
        for slc in tensor[0][0]:
            print(slc)
        tensor = self.fcn4(tensor) 
###        NOW IT CONTAINS NANS???
        print(tensor.shape)
        for slc in tensor[0][0]:
            print(slc)
        tensor = torch.relu(tensor)
        tensor = self.fcn5(tensor)
        tensor = torch.relu(tensor)
#         print(tensor.shape)
        tensor = self.fcn6(tensor)
        tensor = torch.relu(tensor)
#         print(tensor.shape)
#         for slc in tensor[0][0]:
#             print(slc)
        return tensor
</code></pre>
<p>Now the NAN’s are arising in the second image alread usually around the second layer.<br>
I don’t understand why, the 1x1x1 kernel should be ok right?<br>
I tried this on the original 2D dataset before and that worked fine.</p>

flyingdutchman: <p>When i test this with</p>
<pre><code class="lang-auto">it = iter(train_loader)
imgs = [next(it)[0] for i in range(100)]
print(len(imgs))

print(imgs[0].shape)

mod = ConvAutoEncoder4()
crtrn = nn.MSELoss()
ptmzr = optimizer = optim.Adam(mod.parameters(), lr=1e-3)

torch.autograd.set_detect_anomaly(False)

ptmzr.zero_grad()
res = mod(imgs[0])
lss = crtrn(res, imgs[0])
lss.backward()
ptmzr.step()
print(lss.item())

ptmzr.zero_grad()
res = mod(imgs[1])
lss = crtrn(res, imgs[1])
lss.backward()
ptmzr.step()
print(lss.item())
</code></pre>
<p>The ouput is</p>
<pre><code class="lang-auto">100
torch.Size([1, 1, 28, 28, 28])
torch.Size([1, 4, 7, 7, 7])
tensor([[0., 0., 0., 0., 0., 0., 0.],
        [0., 0., 0., 0., 0., 0., 0.],
        [0., 0., 0., 0., 0., 0., 0.],
        [0., 0., 0., 0., 0., 0., 0.],
        [0., 0., 0., 0., 0., 0., 0.],
        [0., 0., 0., 0., 0., 0., 0.],
        [0., 0., 0., 0., 0., 0., 0.]], grad_fn=&lt;UnbindBackward&gt;)
tensor([[0., 0., 0., 0., 0., 0., 0.],
        [0., 0., 0., 0., 0., 0., 0.],
        [0., 0., 0., 0., 0., 0., 0.],
        [0., 0., 0., 0., 0., 0., 0.],
        [0., 0., 0., 0., 0., 0., 0.],
        [0., 0., 0., 0., 0., 0., 0.],
        [0., 0., 0., 0., 0., 0., 0.]], grad_fn=&lt;UnbindBackward&gt;)
tensor([[0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000],
        [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000],
        [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000],
        [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000],
        [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0209, 0.0000],
        [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000],
        [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000]],
       grad_fn=&lt;UnbindBackward&gt;)
tensor([[0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000],
        [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000],
        [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000],
        [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000],
        [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0074, 0.0000],
        [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000],
        [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000]],
       grad_fn=&lt;UnbindBackward&gt;)
tensor([[0., 0., 0., 0., 0., 0., 0.],
        [0., 0., 0., 0., 0., 0., 0.],
        [0., 0., 0., 0., 0., 0., 0.],
        [0., 0., 0., 0., 0., 0., 0.],
        [0., 0., 0., 0., 0., 0., 0.],
        [0., 0., 0., 0., 0., 0., 0.],
        [0., 0., 0., 0., 0., 0., 0.]], grad_fn=&lt;UnbindBackward&gt;)
tensor([[0., 0., 0., 0., 0., 0., 0.],
        [0., 0., 0., 0., 0., 0., 0.],
        [0., 0., 0., 0., 0., 0., 0.],
        [0., 0., 0., 0., 0., 0., 0.],
        [0., 0., 0., 0., 0., 0., 0.],
        [0., 0., 0., 0., 0., 0., 0.],
        [0., 0., 0., 0., 0., 0., 0.]], grad_fn=&lt;UnbindBackward&gt;)
tensor([[0., 0., 0., 0., 0., 0., 0.],
        [0., 0., 0., 0., 0., 0., 0.],
        [0., 0., 0., 0., 0., 0., 0.],
        [0., 0., 0., 0., 0., 0., 0.],
        [0., 0., 0., 0., 0., 0., 0.],
        [0., 0., 0., 0., 0., 0., 0.],
        [0., 0., 0., 0., 0., 0., 0.]], grad_fn=&lt;UnbindBackward&gt;)
torch.Size([1, 4, 7, 7, 7])
tensor([[-0.4506, -0.4506, -0.4506, -0.4506, -0.4506, -0.4506, -0.4506],
        [-0.4506, -0.4506, -0.4506, -0.4506, -0.4506, -0.4506, -0.4506],
        [-0.4506, -0.4506, -0.4506, -0.4506, -0.4506, -0.4506, -0.4506],
        [-0.4506, -0.4506, -0.4506, -0.4506, -0.4506, -0.4506, -0.4506],
        [-0.4506, -0.4506, -0.4506, -0.4506, -0.4506, -0.4506, -0.4506],
        [-0.4506, -0.4506, -0.4506, -0.4506, -0.4506, -0.4506, -0.4506],
        [-0.4506, -0.4506, -0.4506, -0.4506, -0.4506, -0.4506, -0.4506]],
       grad_fn=&lt;UnbindBackward&gt;)
tensor([[-0.4506, -0.4506, -0.4506, -0.4506, -0.4506, -0.4506, -0.4506],
        [-0.4506, -0.4506, -0.4506, -0.4506, -0.4506, -0.4506, -0.4506],
        [-0.4506, -0.4506, -0.4506, -0.4506, -0.4506, -0.4506, -0.4506],
        [-0.4506, -0.4506, -0.4506, -0.4506, -0.4506, -0.4506, -0.4506],
        [-0.4506, -0.4506, -0.4506, -0.4506, -0.4506, -0.4506, -0.4506],
        [-0.4506, -0.4506, -0.4506, -0.4506, -0.4506, -0.4506, -0.4506],
        [-0.4506, -0.4506, -0.4506, -0.4506, -0.4506, -0.4506, -0.4506]],
       grad_fn=&lt;UnbindBackward&gt;)
tensor([[-0.4506, -0.4506, -0.4506, -0.4506, -0.4506, -0.4506, -0.4506],
        [-0.4506, -0.4506, -0.4506, -0.5038, -0.4605, -0.4621, -0.4506],
        [-0.4506, -0.4506, -0.4553, -0.4688, -0.4506, -0.4613, -0.4506],
        [-0.4506, -0.4518, -0.5405, -0.4506, -0.4505, -0.4559, -0.4506],
        [-0.4506, -0.4724, -0.4928, -0.4506, -0.4550, -0.4908, -0.4506],
        [-0.4506, -0.4837, -0.4551, -0.4715, -0.4912, -0.4506, -0.4506],
        [-0.4506, -0.4506, -0.4506, -0.4506, -0.4506, -0.4506, -0.4506]],
       grad_fn=&lt;UnbindBackward&gt;)
tensor([[-0.4506, -0.4506, -0.4506, -0.4506, -0.4506, -0.4506, -0.4506],
        [-0.4506, -0.4506, -0.4506, -0.4650, -0.5366, -0.4510, -0.4506],
        [-0.4506, -0.4506, -0.4545, -0.5428, -0.4506, -0.4499, -0.4506],
        [-0.4506, -0.4520, -0.5970, -0.4506, -0.4505, -0.4499, -0.4506],
        [-0.4506, -0.5225, -0.5748, -0.4506, -0.4538, -0.5510, -0.4506],
        [-0.4506, -0.5077, -0.4728, -0.4642, -0.5557, -0.4506, -0.4506],
        [-0.4506, -0.4506, -0.4506, -0.4506, -0.4506, -0.4506, -0.4506]],
       grad_fn=&lt;UnbindBackward&gt;)
tensor([[-0.4506, -0.4506, -0.4506, -0.4506, -0.4506, -0.4506, -0.4506],
        [-0.4506, -0.4506, -0.4506, -0.4657, -0.5845, -0.4610, -0.4506],
        [-0.4506, -0.4506, -0.4750, -0.5655, -0.4506, -0.4499, -0.4506],
        [-0.4506, -0.4508, -0.5578, -0.4506, -0.4506, -0.4499, -0.4506],
        [-0.4506, -0.4942, -0.6176, -0.4506, -0.4762, -0.5629, -0.4506],
        [-0.4506, -0.4777, -0.5257, -0.5045, -0.5382, -0.4506, -0.4506],
        [-0.4506, -0.4506, -0.4506, -0.4506, -0.4506, -0.4506, -0.4506]],
       grad_fn=&lt;UnbindBackward&gt;)
tensor([[-0.4506, -0.4506, -0.4506, -0.4506, -0.4506, -0.4506, -0.4506],
        [-0.4506, -0.4506, -0.4506, -0.4506, -0.4506, -0.4506, -0.4506],
        [-0.4506, -0.4506, -0.4506, -0.4506, -0.4506, -0.4506, -0.4506],
        [-0.4506, -0.4506, -0.4506, -0.4506, -0.4506, -0.4506, -0.4506],
        [-0.4506, -0.4506, -0.4506, -0.4506, -0.4506, -0.4506, -0.4506],
        [-0.4506, -0.4506, -0.4506, -0.4506, -0.4506, -0.4506, -0.4506],
        [-0.4506, -0.4506, -0.4506, -0.4506, -0.4506, -0.4506, -0.4506]],
       grad_fn=&lt;UnbindBackward&gt;)
tensor([[-0.4506, -0.4506, -0.4506, -0.4506, -0.4506, -0.4506, -0.4506],
        [-0.4506, -0.4506, -0.4506, -0.4506, -0.4506, -0.4506, -0.4506],
        [-0.4506, -0.4506, -0.4506, -0.4506, -0.4506, -0.4506, -0.4506],
        [-0.4506, -0.4506, -0.4506, -0.4506, -0.4506, -0.4506, -0.4506],
        [-0.4506, -0.4506, -0.4506, -0.4506, -0.4506, -0.4506, -0.4506],
        [-0.4506, -0.4506, -0.4506, -0.4506, -0.4506, -0.4506, -0.4506],
        [-0.4506, -0.4506, -0.4506, -0.4506, -0.4506, -0.4506, -0.4506]],
       grad_fn=&lt;UnbindBackward&gt;)
0.03493741527199745
torch.Size([1, 4, 7, 7, 7])
tensor([[0., 0., 0., 0., 0., 0., 0.],
        [0., 0., 0., 0., 0., 0., 0.],
        [0., 0., 0., 0., 0., 0., 0.],
        [0., 0., 0., 0., 0., 0., 0.],
        [0., 0., 0., 0., 0., 0., 0.],
        [0., 0., 0., 0., 0., 0., 0.],
        [0., 0., 0., 0., 0., 0., 0.]], grad_fn=&lt;UnbindBackward&gt;)
tensor([[0., 0., 0., 0., 0., 0., 0.],
        [0., 0., 0., 0., 0., 0., 0.],
        [0., 0., 0., 0., 0., 0., 0.],
        [0., 0., 0., 0., 0., 0., 0.],
        [0., 0., 0., 0., 0., 0., 0.],
        [0., 0., 0., 0., 0., 0., 0.],
        [0., 0., 0., 0., 0., 0., 0.]], grad_fn=&lt;UnbindBackward&gt;)
tensor([[0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000],
        [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000],
        [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000],
        [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000],
        [0.0000, 0.0000, 0.0000, 0.0000, 0.0405, 0.0000, 0.0000],
        [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000],
        [0.0000, 0.0000, 0.0000, 0.0020, 0.0000, 0.0000, 0.0000]],
       grad_fn=&lt;UnbindBackward&gt;)
tensor([[0., 0., 0., 0., 0., 0., 0.],
        [0., 0., 0., 0., 0., 0., 0.],
        [0., 0., 0., 0., 0., 0., 0.],
        [0., 0., 0., 0., 0., 0., 0.],
        [0., 0., 0., 0., 0., 0., 0.],
        [0., 0., 0., 0., 0., 0., 0.],
        [0., 0., 0., 0., 0., 0., 0.]], grad_fn=&lt;UnbindBackward&gt;)
tensor([[0., 0., 0., 0., 0., 0., 0.],
        [0., 0., 0., 0., 0., 0., 0.],
        [0., 0., 0., 0., 0., 0., 0.],
        [0., 0., 0., 0., 0., 0., 0.],
        [0., 0., 0., 0., 0., 0., 0.],
        [0., 0., 0., 0., 0., 0., 0.],
        [0., 0., 0., 0., 0., 0., 0.]], grad_fn=&lt;UnbindBackward&gt;)
tensor([[0., 0., 0., 0., 0., 0., 0.],
        [0., 0., 0., 0., 0., 0., 0.],
        [0., 0., 0., 0., 0., 0., 0.],
        [0., 0., 0., 0., 0., 0., 0.],
        [0., 0., 0., 0., 0., 0., 0.],
        [0., 0., 0., 0., 0., 0., 0.],
        [0., 0., 0., 0., 0., 0., 0.]], grad_fn=&lt;UnbindBackward&gt;)
tensor([[0., 0., 0., 0., 0., 0., 0.],
        [0., 0., 0., 0., 0., 0., 0.],
        [0., 0., 0., 0., 0., 0., 0.],
        [0., 0., 0., 0., 0., 0., 0.],
        [0., 0., 0., 0., 0., 0., 0.],
        [0., 0., 0., 0., 0., 0., 0.],
        [0., 0., 0., 0., 0., 0., 0.]], grad_fn=&lt;UnbindBackward&gt;)
torch.Size([1, 4, 7, 7, 7])
tensor([[nan, nan, nan, nan, nan, nan, nan],
        [nan, nan, nan, nan, nan, nan, nan],
        [nan, nan, nan, nan, nan, nan, nan],
        [nan, nan, nan, nan, nan, nan, nan],
        [nan, nan, nan, nan, nan, nan, nan],
        [nan, nan, nan, nan, nan, nan, nan],
        [nan, nan, nan, nan, nan, nan, nan]], grad_fn=&lt;UnbindBackward&gt;)
tensor([[nan, nan, nan, nan, nan, nan, nan],
        [nan, nan, nan, nan, nan, nan, nan],
        [nan, nan, nan, nan, nan, nan, nan],
        [nan, nan, nan, nan, nan, nan, nan],
        [nan, nan, nan, nan, nan, nan, nan],
        [nan, nan, nan, nan, nan, nan, nan],
        [nan, nan, nan, nan, nan, nan, nan]], grad_fn=&lt;UnbindBackward&gt;)
tensor([[nan, nan, nan, nan, nan, nan, nan],
        [nan, nan, nan, nan, nan, nan, nan],
        [nan, nan, nan, nan, nan, nan, nan],
        [nan, nan, nan, nan, nan, nan, nan],
        [nan, nan, nan, nan, nan, nan, nan],
        [nan, nan, nan, nan, nan, nan, nan],
        [nan, nan, nan, nan, nan, nan, nan]], grad_fn=&lt;UnbindBackward&gt;)
tensor([[nan, nan, nan, nan, nan, nan, nan],
        [nan, nan, nan, nan, nan, nan, nan],
        [nan, nan, nan, nan, nan, nan, nan],
        [nan, nan, nan, nan, nan, nan, nan],
        [nan, nan, nan, nan, nan, nan, nan],
        [nan, nan, nan, nan, nan, nan, nan],
        [nan, nan, nan, nan, nan, nan, nan]], grad_fn=&lt;UnbindBackward&gt;)
tensor([[nan, nan, nan, nan, nan, nan, nan],
        [nan, nan, nan, nan, nan, nan, nan],
        [nan, nan, nan, nan, nan, nan, nan],
        [nan, nan, nan, nan, nan, nan, nan],
        [nan, nan, nan, nan, nan, nan, nan],
        [nan, nan, nan, nan, nan, nan, nan],
        [nan, nan, nan, nan, nan, nan, nan]], grad_fn=&lt;UnbindBackward&gt;)
tensor([[nan, nan, nan, nan, nan, nan, nan],
        [nan, nan, nan, nan, nan, nan, nan],
        [nan, nan, nan, nan, nan, nan, nan],
        [nan, nan, nan, nan, nan, nan, nan],
        [nan, nan, nan, nan, nan, nan, nan],
        [nan, nan, nan, nan, nan, nan, nan],
        [nan, nan, nan, nan, nan, nan, nan]], grad_fn=&lt;UnbindBackward&gt;)
tensor([[nan, nan, nan, nan, nan, nan, nan],
        [nan, nan, nan, nan, nan, nan, nan],
        [nan, nan, nan, nan, nan, nan, nan],
        [nan, nan, nan, nan, nan, nan, nan],
        [nan, nan, nan, nan, nan, nan, nan],
        [nan, nan, nan, nan, nan, nan, nan],
        [nan, nan, nan, nan, nan, nan, nan]], grad_fn=&lt;UnbindBackward&gt;)
nan
</code></pre>
<p>The same happens when I use a dataloader</p>

flyingdutchman: <p>So from a different question I found<br>
<code>torch.autograd.set_detect_anomaly(True)</code><br>
the resulting stacktrace is<br>
<code>RuntimeError: Function 'SlowConv3DBackward' returned nan values in its 1th output.</code></p>

flyingdutchman: <p>using</p>
<pre><code class="lang-auto">for name, param in model.named_parameters():
    print(name, torch.isfinite(param.grad).all())
</code></pre>
<p>gives stacktrace</p>
<pre><code class="lang-auto">---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
&lt;ipython-input-105-bd15723eb4e7&gt; in &lt;module&gt;
     12 
     13 ptmzr.zero_grad()
---&gt; 14 res = mod(imgs[0])
     15 lss = crtrn(res, imgs[0])
     16 lss.backward()

/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    725             result = self._slow_forward(*input, **kwargs)
    726         else:
--&gt; 727             result = self.forward(*input, **kwargs)
    728         for hook in itertools.chain(
    729                 _global_forward_hooks.values(),

&lt;ipython-input-103-cb1d63b32440&gt; in forward(self, tensor)
     20             print(slc)
     21         for name, param in self.named_parameters():
---&gt; 22             print(name, torch.isfinite(param.grad).all())
     23         tensor = self.fcn4(tensor)
     24         for name, param in self.named_parameters():

TypeError: isfinite(): argument 'input' (position 1) must be Tensor, not NoneType
</code></pre>
<p>So the output is none of the previous layer?</p>

ptrblck: <p>The last error is raised, since the <code>.grad</code> attribute is still set to <code>None</code>.<br>
This might be the case, if you are trying to check the gradients before they were calculated in the first <code>backward()</code> operation.</p>

flyingdutchman: <p>Thanks for the reply!<br>
I deleted parts of the network and only inputted 1 image and still it gave me the same error. I created a <a href="https://discuss.pytorch.org/t/non-deterministic-nan-error-in-slowconv3dbackward-with-seed/110802">second question</a> with the reduced code.<br>
The error was appearing stachostically, even with seeding. I’m faily certain the code does not use the gradients before calling <code>backward()</code> the first time. (The results are deterministic if it does not crash)</p>

J_Johnson: <p>You might have an issue in the data being fed in. For instance, if you took the images and preprocess them in a spreadsheet and exported to csv, then it’s possible one of the images got saved with quotes around it’s values. How are you loading the data?</p>