Loss outputting nan

Ghostv: <p>Hi,</p>
<p>I’m trying to build a simple NN for a categorical classification problem, however, i’m not able to get any value out of the losses. I’m not sure if something is wrong with my layers or it’s just some syntax issue.</p>
<p>My dataset is a 5518x512 tensor (5518 observations, 512 features per observation) and my labels is a categorical 5518x1 tensor, which i converted into a 5518x5 one hot encoded tensor (5 classes).</p>
<p>My model is as follows:</p>
<pre><code class="lang-auto">class NN(tt.nn.Module):
    def __init__(self):
        super().__init__()
        self.dense = tt.nn.Linear(in_features = 512, out_features = 5)
    def forward(self,x):
        y = self.dense(x)
        return y
</code></pre>
<p>My optimizer and loss fxns:</p>
<pre><code class="lang-auto">criterion = tt.nn.CrossEntropyLoss()
optimizer = tt.optim.SGD(model.parameters(), lr=0.0001)
</code></pre>
<p>Now, when i initialize the model and do a forward pass, it works perfect, but, when i calculate the losses, i get nan, every time, even in the first iteration.</p>
<p>Example, one forward pass gives me:</p>
<pre><code class="lang-auto">&gt;&gt;&gt; yp
tensor([[-0.0309, -0.0312, -0.0166, -0.0349,  0.0427],
        [-0.0257, -0.0310, -0.0115, -0.0375,  0.0446],
        [-0.0281, -0.0321, -0.0115, -0.0370,  0.0461],
        ...,
        [-0.0266, -0.0240, -0.0497, -0.0416,  0.0226],
        [-0.0145, -0.0241, -0.0463, -0.0558,  0.0208],
        [-0.0247, -0.0249, -0.0480, -0.0471,  0.0220]], device='cuda:0', grad_fn=&lt;AddmmBackward0&gt;)
</code></pre>
<p>And my training labels are:</p>
<pre><code class="lang-auto">&gt;&gt;&gt; y
tensor([[1., 0., 0., 0., 0.],
        [1., 0., 0., 0., 0.],
        [1., 0., 0., 0., 0.],
        ...,
        [0., 0., 0., 0., 1.],
        [0., 0., 0., 0., 1.],
        [0., 0., 0., 0., 1.]], device='cuda:0')
</code></pre>
<p>And after calculating the losses (criterion(yp,y)), i always get:</p>
<p><code>tensor(nan, device='cuda:0', grad_fn &lt;DivBackward1&gt;)</code></p>
<p>Any idea what could it be?</p>
<p>Thanks in advance!</p>
<p>PS: I’m still not sure if this is the best way to do a categorical classification (encoding, type of nn, etc), i’ve found many very different solutions and i’m not sure which one would be the best one (any advice is more than welcome)</p>

dscarmo: <p>Your model is very simple but should work as demo.</p>
<p>Your inputs (x) probably need to be normalized. If you have a large intensity range it can cause numerical instability.</p>
<p>Look at min-max normalization or z-score normalization.</p>

Ghostv: <p>Indeed i didn’t normalize them, i’ll try that in a while, however, all the values are small (basically all the features for an observation add up to 1).</p>
<p>I understand that can cause a numerical instability and convergence issues, but why would it be outputting nan in the losses?</p>
<p>EDIT: This is from the first iteration, before doing anything to the grads, like, literally the example i wrote.</p>

KFrank: <p>Hi Ghost!</p>
<aside class="quote no-group quote-modified" data-username="Ghostv" data-post="1" data-topic="204808" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/g/7ea924/48.png" class="avatar"> Ghostv:</div>
<blockquote>
<p>when i calculate the losses, i get nan, every time<br>
…<br>
Any idea what could it be?</p>
</blockquote>
</aside>
<p>Most likely, you have a <code>nan</code> in your data somewhere.  First check<br>
whether <code>yp</code> or <code>y</code> have <code>nan</code>s or <code>inf</code>s in them, and, if so, work backwards<br>
to find out what causes them.</p>
<p>Note that the tensors you posted are elided for display purposes.  But<br>
using your elided tensors, you can see the <code>CrossEntropyLoss</code> works<br>
just fine with them:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; print (torch.__version__)
2.3.1
&gt;&gt;&gt;
&gt;&gt;&gt; # your elided tensors
&gt;&gt;&gt;
&gt;&gt;&gt; yp = torch.tensor ([[-0.0309, -0.0312, -0.0166, -0.0349,  0.0427],
...                     [-0.0257, -0.0310, -0.0115, -0.0375,  0.0446],
...                     [-0.0281, -0.0321, -0.0115, -0.0370,  0.0461],
...                     [-0.0266, -0.0240, -0.0497, -0.0416,  0.0226],
...                     [-0.0145, -0.0241, -0.0463, -0.0558,  0.0208],
...                     [-0.0247, -0.0249, -0.0480, -0.0471,  0.0220]]
... )
&gt;&gt;&gt;
&gt;&gt;&gt; y  = torch.tensor ([[1., 0., 0., 0., 0.],
...                     [1., 0., 0., 0., 0.],
...                     [1., 0., 0., 0., 0.],
...                     [0., 0., 0., 0., 1.],
...                     [0., 0., 0., 0., 1.],
...                     [0., 0., 0., 0., 1.]]
... )
&gt;&gt;&gt;
&gt;&gt;&gt; torch.nn.CrossEntropyLoss() (yp, y)
tensor(1.5945)
</code></pre>
<p>If you still have trouble after searching for a <code>nan</code> somewhere in your<br>
input, please post a super-simplified (no tensor dimensions of size<br>
5518 or 512, please), fully-self-contained, runnable script that illustrates<br>
your issue, together with the output you get when you run it.</p>
<aside class="quote no-group">
<blockquote>
<p>PS: I’m still not sure if this is the best way to do a categorical classification (encoding, type of nn, etc)</p>
</blockquote>
</aside>
<p>First, your neural network consists of a single <code>Linear</code> layer with no<br>
non-linearities.  As such, it won’t really be able to learn anything (except<br>
when working with contrived linear problems).  Try using at least two<br>
<code>Linear</code> layers (perhaps more), separated by non-linear “activations”<br>
such as <code>ReLU</code> or <code>Tanh</code>.</p>
<p>Second, use integer categorical labels (rather than converting them to<br>
one-hot format).  Your prediction (<code>yp</code>) will have shape <code>[nBatch, nClass]</code><br>
and your target (<code>y</code>) will have shape <code>[nBatch]</code> (with no class dimension)<br>
and should be of type <code>LongTensor</code>.</p>
<p>(You can use a floating-point, one-hot target of shape <code>[nBatch, nClass]</code>,<br>
but it’s extra work and there’s no point to it.)</p>
<p>Best.</p>
<p>K. Frank</p>

Ghostv: <p>Thanks! Indeed, i was testing some operations on the input tensor and was getting nan also, so i was indeed suspecting nan’s or some issue with the input type. I’m gonna test tomorrow and report back!</p>
<p>Thank again!</p>

Ghostv: <p>Yep, it was that indeed… i had two rows full of nans in the dataset; tracked the issue back and it was some problem with the function i was using to generate the data.</p>
<p>Now, for the categorical labels instead of one-hot, in the output in this case, the predicted categories are just the torch.max(yp,dim=1), right? in that case, how do i set that in my training loop so the losses knows that it must get the max? Do i have to use a softmax layer?</p>

KFrank: <p>Hi Ghost!</p>
<aside class="quote no-group" data-username="Ghostv" data-post="6" data-topic="204808" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/g/7ea924/48.png" class="avatar"> Ghostv:</div>
<blockquote>
<p>Now, for the categorical labels instead of one-hot, in the output in this case, the predicted categories are just the torch.max(yp,dim=1), right?</p>
</blockquote>
</aside>
<p>If I understand what you are asking, no.</p>
<p>For <em>training,</em> you want your predictions (that you then feed into your<br>
<code>CrossEntropyLoss</code> loss function) to be unnormalized log-probabilities.<br>
These are what you naturally get out of a final <code>Linear</code> layer with<br>
<code>out_features = num_classes</code> (without any following <code>Softmax</code>).</p>
<aside class="quote no-group">
<blockquote>
<p>in that case, how do i set that in my training loop so the losses knows that it must get the max? Do i have to use a softmax layer?</p>
</blockquote>
</aside>
<p>Again, if I understand what you’re asking, you don’t want your <em>training</em><br>
loop to “get the max.”  Just use the raw output of your final <code>Linear</code> layer.<br>
(Note, <code>CrossEntropyLoss</code> has <code>log_softmax()</code> built into it.)</p>
<p>For various evaluation metrics, for example, the <em>accuracy</em> (the fraction<br>
of predictions your model gets right), you want to turn the “probabilistic”<br>
predictions produced by your final <code>Linear</code> layer into “hard,” single-class<br>
predictions.  To get these, compute the <code>argmax()</code> of the output of the<br>
final <code>Linear</code> layer.  Doing so gives you integer categorical class label<br>
predicted by your model (which you would then test for equality with your<br>
single-integer training labels (without any one-hot)).</p>
<p>Best.</p>
<p>K. Frank</p>

Ghostv: <p>Perfect, so, setting the last layer of my forward pass to an output dim of size nclasses and giving that along the labels (with dim = 1) should be interpreted correctly by the losses, right?.</p>