Nan loss in RNN model?

Response777: However, the loss becomes nan after several iterations. <SCODE>import torch.nn as nn
from torch.autograd import Variable

class RNN(nn.Module):
    def __init__(self, input_size, hidden_size, output_size):
        super(RNN, self).__init__()
        
        self.hidden_size = hidden_size
        
        self.i2h = nn.Linear(input_size + hidden_size, hidden_size)
        self.i2o = nn.Linear(input_size + hidden_size, output_size)
        self.softmax = nn.LogSoftmax()
    
    def forward(self, input, hidden):
        combined = torch.cat((input, hidden), 1)
        hidden = self.i2h(combined)
        output = self.i2o(combined)
        output = self.softmax(output)
        return output, hidden

    def initHidden(self):
        return Variable(torch.zeros(1, self.hidden_size))
<ECODE> <SCODE>import torch.nn as nn
from torch.autograd import Variable

class RNN(nn.Module):
    def __init__(self, input_size, hidden_size, output_size):
        super(RNN, self).__init__()
        
        self.hidden_size = hidden_size
        
        self.i2h = nn.Linear(input_size + hidden_size, hidden_size)
        self.i2o = nn.Linear(input_size + hidden_size, output_size)
    
        self.softmax = nn.Softmax()
    
    def forward(self, input, hidden):
        combined = torch.cat((input, hidden), 1)
        hidden = self.i2h(combined)
        output = self.i2o(combined)
        
        output = self.softmax(output)
        output = output.add(1e-8)
        output = output.log()
        
        return output, hidden

    def initHidden(self):
        return Variable(torch.zeros(1, self.hidden_size).type(dtypeFloat))
<ECODE> The result is slightly better, but still end up nan

Response777: Besides, another problem is how to get nan in PyTorch. But the NaN problem above is remained unsolved…

apaszke: The NaNs appear, because softmax + log separately can be a numerically unstable operation.

Response777: 

Response777: <SCODE>def cross_entropy(input, target, weight=None, size_average=True):
    r"""This criterion combines `log_softmax` and `nll_loss` in one single class.
    See :class:`torch.nn.CrossEntropyLoss` for details.
    Args:
        input: Variable :math:`(N, C)` where `C = number of classes`
        target: Variable :math:`(N)` where each value is `0 <= targets[i] <= C-1`
        weight (Variable, optional): a manual rescaling weight given to each
                class. If given, has to be a Variable of size "nclasses"
        size_average (bool, optional): By default, the losses are averaged
                over observations for each minibatch. However, if the field
                sizeAverage is set to False, the losses are instead summed
                for each minibatch.
    """
    return nll_loss(log_softmax(input), target, weight, size_average)
<ECODE>

splinter: I have the same problem with you, and replace cross_entropy with log_softmax + nll_loss doesn’t work. It seems to me that nan is more likely to happen when the network is big; when I try the same architecture on a much smaller scale the nan disappear

Response777: 

splinter: I further checked my codes and find out that my problem is due to the gradient explosion of RNN. This code might help you if the cause is the same as mine, pay attention to the function ‘clip_gradient’

Response777: Thanks, that is exactly what I’ve  missed!

YisongMiao: Hi all, I would agree that it is very likely due to the gradient explosion.

Iridium_Blue: <SCODE>loss.backward()
torch.nn.utils.clip_grad_norm_(model.parameters(), 0.5)

<ECODE>

MrAdamJensen: 

Iridium_Blue: I notice that the clip_grad_norm trick appears frequently in Cutting edge solutions like fastai; aside from preventing NaNs it helps to correct a fundamental weakness of any RNN, that of vanishing and exploding gradients.  This can greatly improve the models performance.

AbdulsalamBande: