NaN loss with linear regression

username: Saida2020
cooked: <p>Hello</p>
<p>I have classification problem. My input is sequence of length 341 and output one of three classes {0,1,2}, I want to train linear regression model using Pytorch, I have the following class but during the training, the loss values start to have numbers then inf then NAN. I do not know how to fix that . Also I tried to initialize the weights for linear model but it is the same thing. Also I tried different epochs 5, 20 ,100 all are same . Any suggestions.</p>
<pre><code class="lang-auto">class regression(nn.Module):
    def __init__(self, input_dim):
        super().__init__()
        self.input_dim = input_dim
        # One layer
        self.linear = nn.Linear(input_dim, 1)

    def forward(self, x):
        y_pred = self.linear(x)
        return y_pred

criterion = torch.nn.MSELoss()

def fit(model, data_loader, optim, epochs):
    for epoch in range(epochs):

        for i, (X, y) in enumerate(data_loader):
            X = X.float()
            y = y.unsqueeze(1).float()
            X = Variable(X, requires_grad=True)
            y = Variable(y, requires_grad=True)
            # Make a prediction for the input X
            pred = model(X)
            #loss = (y-pred).pow(2).mean()
            loss = criterion(y, pred)
            optim.zero_grad()
            loss.backward()
            optim.step()
            print(loss)
            print(type(loss))
        # Give some feedback after each 5th pass through the data
        if epoch % 5 == 0:
            print("Epoch", epoch, f"loss: {loss}")
    return None
regnet = regression(input_dim=341)
optim = SGD(regnet.parameters(), lr=0.01)
fit(regnet, data_loader, optim=optim, epochs=5)
pred = regnet(torch.Tensor(test_set.data_info).float())
pred = pred.detach().numpy()
</code></pre>

username: KFrank
cooked: <p>Hi Saida!</p>
<aside class="quote no-group" data-username="Saida2020" data-post="1" data-topic="139783" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/saida2020/48/53003_2.png" class="avatar"> Saida2020:</div>
<blockquote>
<p>during the training, the loss values start to have numbers then inf then NAN.</p>
</blockquote>
</aside>
<p>Because you are performing a regression with <code>MSELoss</code>, your<br>
model should not be <em>actively</em> unstable.  However, depending on<br>
data, your model could be underdetermined – especially if you<br>
have fewer samples than parameters – and therefore also might<br>
not be <em>actively</em> stable.</p>
<p>I would suggest you add <code>weight_decay</code> to your <code>SGD</code> optimizer.<br>
This is equivalent to adding an “L2” penalty to your loss function,<br>
and should keep any underdetermined parameters from running<br>
off to infinity.</p>
<p>Some side comments:</p>
<aside class="quote no-group">
<blockquote>
<p>I have classification problem. My input is sequence of length 341 and output one of three classes {0,1,2}</p>
</blockquote>
</aside>
<p>If you really have a classification problem, you should consider not<br>
trying to approach it as a regression.  Do your class labels really<br>
represent something numerical, such as $0, $1, $2, where “1” is<br>
numerically between “0” and “2” (and indeed, halfway between)?<br>
Or are your labels just arbitrary class labels, where, for example,<br>
“0” means “dog,” “1” means “cat,” and “2” means “horse?”</p>
<p>If your labels are not naturally numerical, you should treat your<br>
problem as a classification problem.  You would want to add more<br>
layers to your model, separated by non-linear “activations,” and use,<br>
most likely, <code>CrossEntropyLoss</code> as your loss function.</p>
<aside class="quote no-group">
<blockquote>
<pre><code class="lang-auto">            X = Variable(X, requires_grad=True)
            y = Variable(y, requires_grad=True)
</code></pre>
</blockquote>
</aside>
<p>There is no reason to set <code>requires_grad = True</code> for <code>X</code> and <code>y</code>.  Any<br>
gradients that you calculate for <code>X</code> and <code>y</code> just get ignored.  (<code>X</code> and <code>y</code><br>
haven’t been added to your optimizer – nor should they be.)</p>
<p>Also, <code>Variable</code> is deprecated – it doesn’t really do anything anymore.<br>
(To set <code>requires_grad = True</code> for <code>X</code> – but you don’t want to – just<br>
use <code>X.requires_grad = True</code>.)</p>
<p>Best.</p>
<p>K. Frank</p>