Getting NaN in the softmax Layer

dhirajsuvarna: <p>Hi,</p>
<p>I am trying to train an existing neural network from a published paper, using custom dataset.</p>
<p>However, why trainng this I am getting NAN as my predictions even before completeing the first batch of training (batch size = 32).</p>
<p>I tried to google out the error and came across multiple post from this forum and tried few things -</p>
<ol>
<li>Reducing the learning rate (default was 0.001, reduced it to 0.0001)</li>
<li>Reducing batch size from 32 to 10</li>
<li>Using <code>torch.autograd.detect_anomaly()</code>
</li>
</ol>
<p>by using <code>torch.autograd.detect_anomaly()</code> I got the following error</p>
<pre><code class="lang-auto">Warning: Traceback of forward call that caused the error:
  File "train_classification.py", line 126, in &lt;module&gt;
    pred, trans, trans_feat = classifier(points)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/content/mydrive/My Drive/Projects/pointnet/pointnet.pytorch/pointnet/model.py", line 147, in forward
    return F.log_softmax(x, dim=1), trans, trans_feat
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/functional.py", line 1317, in log_softmax
    ret = input.log_softmax(dim)
 (print_stack at /pytorch/torch/csrc/autograd/python_anomaly_mode.cpp:57)
Traceback (most recent call last):
  File "train_classification.py", line 133, in &lt;module&gt;
    loss.backward()
  File "/usr/local/lib/python3.6/dist-packages/torch/tensor.py", line 195, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "/usr/local/lib/python3.6/dist-packages/torch/autograd/__init__.py", line 99, in backward
    allow_unreachable=True)  # allow_unreachable flag
RuntimeError: Function 'LogSoftmaxBackward' returned nan values in its 0th output.
</code></pre>
<p>If I am interpreting this correctly, the error seems to be from the softmax layer -  <code>F.log_softmax()</code></p>
<p>Now I am stuck here, do not know how to proceed, can someone help me in this regards?</p>

ptrblck: <p>Could you check the min and max values of <code>x</code> before feeding it to <code>F.log_softmax</code>?</p>

dhirajsuvarna: <p>The <code>forward()</code> function of the model is as follows -</p>
<pre><code class="lang-auto">    def forward(self, x):
        x, trans, trans_feat = self.feat(x)
        x = F.relu(self.bn1(self.fc1(x)))
        x = F.relu(self.bn2(self.dropout(self.fc2(x))))
        x = self.fc3(x)
        return F.log_softmax(x, dim=1), trans, trans_feat
</code></pre>
<p>I printed out the <code>max</code> and <code>min</code> for the weights of <code>f3</code> layer.<br>
Here is the output dump for the last few iterations -</p>
<pre><code class="lang-auto">[0: 40/47] test loss: 3.190464 accuracy: 0.250000
f3 Weights Min: -0.06608034670352936
f3 Weights Max: 0.06457919627428055
f3 Gradients Min: -0.18587647378444672
f3 Gradients Max: 0.06978222727775574
[0: 41/47] train loss: 2.302364 accuracy: 0.593750
f3 Weights Min: -0.066163070499897
f3 Weights Max: 0.06460865586996078
f3 Gradients Min: -0.19260874390602112
f3 Gradients Max: 0.060340650379657745
[0: 42/47] train loss: 2.837160 accuracy: 0.281250
f3 Weights Min: -0.0662456750869751
f3 Weights Max: 0.06465635448694229
f3 Gradients Min: -0.23064641654491425
f3 Gradients Max: 0.03692895919084549
[0: 43/47] train loss: 2.571262 accuracy: 0.406250
f3 Weights Min: -0.06632933020591736
f3 Weights Max: 0.06470420211553574
f3 Gradients Min: -0.2090958207845688
f3 Gradients Max: 0.071017786860466
[0: 44/47] train loss: 2.492180 accuracy: 0.375000
f3 Weights Min: -0.06641349196434021
f3 Weights Max: 0.06475285440683365
Warning: Traceback of forward call that caused the error:
  File "train_classification.py", line 126, in &lt;module&gt;
    pred, trans, trans_feat = classifier(points)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/content/mydrive/My Drive/Projects/pointnet/pointnet.pytorch/pointnet/model.py", line 147, in forward
    return F.log_softmax(x, dim=1), trans, trans_feat
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/functional.py", line 1317, in log_softmax
    ret = input.log_softmax(dim)
 (print_stack at /pytorch/torch/csrc/autograd/python_anomaly_mode.cpp:57)
Traceback (most recent call last):
  File "train_classification.py", line 133, in &lt;module&gt;
    loss.backward()
  File "/usr/local/lib/python3.6/dist-packages/torch/tensor.py", line 195, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "/usr/local/lib/python3.6/dist-packages/torch/autograd/__init__.py", line 99, in backward
    allow_unreachable=True)  # allow_unreachable flag
RuntimeError: Function 'LogSoftmaxBackward' returned nan values in its 0th output.
</code></pre>
<p>I don’t see the gradients exploding but i think the gradients are vanishing, am I right?</p>

SandPhoenix: <p>I have the same problem since two days ago and I do not find the cause of the problem. I even deleted the logsoftmax from my code and still the same…<br>
Please tell me if you find a solution and I will do the same if I do<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/9/5993e051884cbccb8ae4267e3e8df6f8e60e994d.png" data-download-href="https://discuss.pytorch.org/uploads/default/5993e051884cbccb8ae4267e3e8df6f8e60e994d" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/9/5993e051884cbccb8ae4267e3e8df6f8e60e994d_2_690x70.png" alt="image" data-base62-sha1="cMrjJoGuQF0boStsd96SUa5nsqV" width="690" height="70" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/9/5993e051884cbccb8ae4267e3e8df6f8e60e994d_2_690x70.png, https://discuss.pytorch.org/uploads/default/original/3X/5/9/5993e051884cbccb8ae4267e3e8df6f8e60e994d.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/5/9/5993e051884cbccb8ae4267e3e8df6f8e60e994d.png 2x" data-dominant-color="16191E"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1013×104 42.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

dhirajsuvarna: <p>Sure, I will, I am actively pursuing this. I broke my head over this for two days, I will try to update this post as soon as I get help from any forum.</p>
<p>In the meantime, did you try to print out the weights as suggested by <a class="mention" href="/u/ptrblck">@ptrblck</a>.</p>

SandPhoenix: <p>I did and they seeme pretty normal to me. when I remove the detect_anomaly(), it starts working for like a sec and then from nowhere the nan values come out. It was the first thing I did actually even before using the detect_anomaly</p>

ptrblck: <p>Is this only visible on the CPU, GPU or both devices?<br>
Could you try to update numpy as suggested <a href="https://discuss.pytorch.org/t/well-formed-input-into-a-simple-linear-layer-output-nan/74720/7">here</a>?</p>
<p>CC <a class="mention" href="/u/dhirajsuvarna">@dhirajsuvarna</a></p>

SandPhoenix: <p>For me, I am using the CPU. I can’t try using the GPU currently.</p>

ptrblck: <p>Does updating numpy work for you?</p>

SandPhoenix: <p>Sadly, it is already the latest version.</p>

ptrblck: <p>Could you try to store the tensors, which create the NaN output so that we could further debug?<br>
Alternatively, could you post the shapes and stats of the tensors, so that we could run it with random values and try to reproduce the failure?</p>

SandPhoenix: <p>def convNxN(in_p,out_p,N):<br>
return nn.Conv2d(in_p,ou_p,kernel_size=(N,N),stride=(1,1),padding=(1,1),bias=False)<br>
class Modelname(nn.Module):<br>
def <strong>init</strong>(self):<br>
self.Conv1 = convNxN(3,64,3)<br>
<span class="hashtag">#other</span> Conv2d calls of the same way.</p>
<p>def forward(self,x):<br>
out=self.Conv1(x)<br>
# When I print it here, it already starts giving nans</p>
<p>By the way, I am using CIFAR-10 as dataset.</p>

ptrblck: <p>How is <code>convNxN</code> defined?</p>

SandPhoenix: <p>def convNxN(in_p,out_p,N):<br>
return nn.Conv2d(in_p,ou_p,kernel_size=(N,N),stride=(1,1),padding=(1,1),bias=False)</p>

ptrblck: <p>Thanks, could you post all arguments to create an instance of this conv layer as well as the input shape and the stats of the input?</p>

SandPhoenix: <p>I didn’t quite understand what you are asking exactly. Do you want me to give a screenshot of input sample? Because I thought I gave how I am calling for the Convolution I created isn’t that it? (I am sorry but I get things and I am trying to handle everything but I am not quite familliar with the vocabulary)<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/a/da17a9389b38eb21c3aa8b0d45802111cc33cfec.png" data-download-href="https://discuss.pytorch.org/uploads/default/da17a9389b38eb21c3aa8b0d45802111cc33cfec" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/d/a/da17a9389b38eb21c3aa8b0d45802111cc33cfec.png" alt="image" data-base62-sha1="v7kMueEZ3QBpn2irj8kP5GINMyU" width="690" height="352" data-dominant-color="080808"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">745×381 3.78 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>No, I just asked for the arguments to create an instance of your convolution, i.e. <code>in_channels</code>, <code>out_channels</code>, <code>kernel_size</code>, etc.<br>
Also, could you post the shape of your input, please?</p>

SandPhoenix: <p>Hello,<br>
<code>self.Conv1 = nn.Conv2d(3,64,(3,3),(1,1),(1,1),False)</code><br>
the shape of my input is:<br>
<code>torch.size([2, 3, 32, 32])</code></p>

dhirajsuvarna: <p><a class="mention" href="/u/ptrblck">@ptrblck</a>, <a class="mention" href="/u/sandphoenix">@SandPhoenix</a></p>
<p>Update from my end.<br>
The input <code>x</code> had a NAN value in it, which was the root cause of the problem.</p>
<p>This NAN was not present in the input as I had double checked it, but got introduced during the Normalization process.</p>
<p>Right now, I have figured out the input causing this NAN and removed it input dataset.<br>
Things are working now.</p>
<p><a class="mention" href="/u/sandphoenix">@SandPhoenix</a>: do please check your input for NAN’s before passing it to the Network Layers.</p>

SandPhoenix: <p>Happy to hear that you solved your problem.<br>
It is not the same thing for me but thank you for the information.<br>
I tried <code>print((x!=x).any())</code> and it gave me <code>tensor(False)</code><br>
I tried everything xD I need to think more.<br>
<a class="mention" href="/u/ptrblck">@ptrblck</a> Do you have any further advices for me ? I would be grateful, I already am for your help.</p>