Nan loss in RNN model?

username: Response777
cooked: <p>Hi, I’m trying out the code from the awesome <a href="https://github.com/spro/practical-pytorch" rel="nofollow noopener">practical-python</a> codes. And I’m replacing the text with a slightly bigger one (originally 164KB, and mine is 966KB).</p>
<p>However, the loss becomes nan after several iterations.</p>
<p>the model by <a class="mention" href="/u/spro">@spro</a> is below.</p>
<pre><code class="lang-auto">import torch.nn as nn
from torch.autograd import Variable

class RNN(nn.Module):
    def __init__(self, input_size, hidden_size, output_size):
        super(RNN, self).__init__()
        
        self.hidden_size = hidden_size
        
        self.i2h = nn.Linear(input_size + hidden_size, hidden_size)
        self.i2o = nn.Linear(input_size + hidden_size, output_size)
        self.softmax = nn.LogSoftmax()
    
    def forward(self, input, hidden):
        combined = torch.cat((input, hidden), 1)
        hidden = self.i2h(combined)
        output = self.i2o(combined)
        output = self.softmax(output)
        return output, hidden

    def initHidden(self):
        return Variable(torch.zeros(1, self.hidden_size))
</code></pre>
<p>Then I replace the <code>LogSoftmax</code> with <code>softmax+log(output+eps)</code> like that</p>
<pre><code class="lang-auto">import torch.nn as nn
from torch.autograd import Variable

class RNN(nn.Module):
    def __init__(self, input_size, hidden_size, output_size):
        super(RNN, self).__init__()
        
        self.hidden_size = hidden_size
        
        self.i2h = nn.Linear(input_size + hidden_size, hidden_size)
        self.i2o = nn.Linear(input_size + hidden_size, output_size)
    
        self.softmax = nn.Softmax()
    
    def forward(self, input, hidden):
        combined = torch.cat((input, hidden), 1)
        hidden = self.i2h(combined)
        output = self.i2o(combined)
        
        output = self.softmax(output)
        output = output.add(1e-8)
        output = output.log()
        
        return output, hidden

    def initHidden(self):
        return Variable(torch.zeros(1, self.hidden_size).type(dtypeFloat))
</code></pre>
<p>The result is slightly better, but still end up nan</p>
<p>Is anybody could provide me with some suggestion on how such thing could happen?<br>
Thanks.</p>

username: Response777
cooked: <p>Besides, another problem is how to get nan in PyTorch.</p>
<p>Ok, this is solved, since the type(nan) shows that it’s float, which could be examined with <code>math.isnan(x)</code></p>
<p>But the NaN problem above is remained unsolved…</p>

username: apaszke
cooked: <p>The NaNs appear, because softmax + log separately can be a numerically unstable operation.</p>
<p>If you’re using <code>CrossEntropyLoss</code> for training, you could use the <code>F.log_softmax</code> function at the end of your model and use <code>NLLLoss</code>. The loss will be equivalent, but much more stable.</p>

username: Response777
cooked: <p>Thanks for reply. But I have added an <strong>epsilon</strong> to the log and my loss function was <strong>NLLLoss</strong>.</p>
<p>And I used <code>torch.nn.LogSoftmax</code> and <code>NLLLoss</code> at the beginning, its result was the worst one, and that is why I separated <code>softmax</code> and <code>log</code> operation…</p>

username: Response777
cooked: <pre><code class="lang-auto">def cross_entropy(input, target, weight=None, size_average=True):
    r"""This criterion combines `log_softmax` and `nll_loss` in one single class.
    See :class:`torch.nn.CrossEntropyLoss` for details.
    Args:
        input: Variable :math:`(N, C)` where `C = number of classes`
        target: Variable :math:`(N)` where each value is `0 &lt;= targets[i] &lt;= C-1`
        weight (Variable, optional): a manual rescaling weight given to each
                class. If given, has to be a Variable of size "nclasses"
        size_average (bool, optional): By default, the losses are averaged
                over observations for each minibatch. However, if the field
                sizeAverage is set to False, the losses are instead summed
                for each minibatch.
    """
    return nll_loss(log_softmax(input), target, weight, size_average)
</code></pre>
<p>Shouldn’t <code>CrossEntropyLoss</code> behave the same as <code>log_softmax+NLLLoss</code> ?</p>

username: splinter
cooked: <p>I have the same problem with you, and replace cross_entropy with log_softmax + nll_loss doesn’t work.</p>
<p>It seems to me that nan is more likely to happen when the network is big; when I try the same architecture on a much smaller scale the nan disappear</p>

username: Response777
cooked: <p>yep, I set eps to <code>1e-6</code> and I think it can not be explained with “numerically unstable”, since the numerically unstable may result in problematic problem which have difficult converge, but it cannot explain a negative output of <code>exp</code>, which is totally wrong.</p>

username: splinter
cooked: <p>I further checked my codes and find out that my problem is due to the gradient explosion of RNN. This code might help you if the cause is the same as mine, pay attention to the function ‘clip_gradient’</p>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/pytorch/examples/blob/master/word_language_model/main.py" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/examples/blob/master/word_language_model/main.py" target="_blank" rel="nofollow noopener">pytorch/examples/blob/master/word_language_model/main.py</a></h4>
<pre><code class="lang-py"># coding: utf-8
import argparse
import time
import math
import torch
import torch.nn as nn
from torch.autograd import Variable

import data
import model

parser = argparse.ArgumentParser(description='PyTorch Wikitext-2 RNN/LSTM Language Model')
parser.add_argument('--data', type=str, default='./data/wikitext-2',
                    help='location of the data corpus')
parser.add_argument('--model', type=str, default='LSTM',
                    help='type of recurrent net (RNN_TANH, RNN_RELU, LSTM, GRU)')
parser.add_argument('--emsize', type=int, default=200,
                    help='size of word embeddings')
parser.add_argument('--nhid', type=int, default=200,
                    help='number of hidden units per layer')
</code></pre>

  This file has been truncated. <a href="https://github.com/pytorch/examples/blob/master/word_language_model/main.py" target="_blank" rel="nofollow noopener">show original</a>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>


username: Response777
cooked: <p>Thanks, that is exactly what I’ve  missed!</p>

username: YisongMiao
cooked: <p>Hi all,</p>
<p>I would agree that it is very likely due to the gradient explosion.</p>
<p><a href="https://machinelearningmastery.com/exploding-gradients-in-neural-networks/" class="onebox" target="_blank" rel="nofollow noopener">https://machinelearningmastery.com/exploding-gradients-in-neural-networks/</a><br>
This URL is very helpful.<br>
Besides clipping, this URL also suggest using ReLU, LSTM, etc…</p>
<p>Wish it helps!<br>
Yisong</p>

username: Iridium_Blue
cooked: <p>Thank you <a class="mention" href="/u/splinter">@splinter</a> , this is very helpful and successfully rid my LSTM of NaNs.</p>
<p>To abbreviate <a class="mention" href="/u/splinter">@splinter</a>’s answer a little, the trick is to call <strong>torch.nn.utils.clip_grad_norm_</strong> just after <strong>loss.backward</strong> like this :</p>
<pre><code class="lang-auto">loss.backward()
torch.nn.utils.clip_grad_norm_(model.parameters(), 0.5)

</code></pre>

username: MrAdamJensen
cooked: <p>Thanks <a class="mention" href="/u/iridium_blue">@Iridium_Blue</a> <a class="mention" href="/u/splinter">@splinter</a> helped me a lot</p>

username: Iridium_Blue
cooked: <p>I notice that the clip_grad_norm trick appears frequently in Cutting edge solutions like fastai; aside from preventing NaNs it helps to correct a fundamental weakness of any RNN, that of vanishing and exploding gradients.  This can greatly improve the models performance.</p>

username: AbdulsalamBande
cooked: <p>Just as a suggestion for someone that have this problem, make sure you check your inputs thoroughly and make sure they do not contain NAN. I had this same problem, I spent a week trying  all different solutions such as gradient clipping just to figure out that my problem was that one of the input values is NAN which throws off my calculations leading to a loss thats NAN. <img src="https://discuss.pytorch.org/images/emoji/apple/grinning.png?v=9" title=":grinning:" class="emoji" alt=":grinning:"></p>