Outputing 'nan' as predictions in my neural network training loop

Brooklyn_Amaro: <p>For my neural network I noticed that my predictions were coming out to be ‘nan’ in my training loop. To overcome this problem I have tried downgrading my PyTorch from 11.8 to 11.7 but that only changed the device from using cpu to gpu. I don’t know if this is a bug with PyTorch or if my code is just not working. Any advice would help.</p>
<pre><code class="lang-auto">import torch
import numpy as np
from torch import nn
from torch import optim
from torch.utils.data import TensorDataset
from torch.utils.data import DataLoader
import pandas as pd

# Get data from tables
# Split data up into training and testing (80% Train, 20% Test)
pnt_cont_data = pd.read_excel('C:\\Users\\bamar\\Downloads\\Chile_Research\\Research_Resources\\DoE_point_contact.xlsm', sheet_name='Sample', index_col = 0, names=['#','E1','E2','v1','v2','Ap','rho0','mue0','u1','u2','R','Fn','hmin','hc','p'])
pnt_cont_data.drop(index = pnt_cont_data.index[0], axis = 0, inplace =True)
del pnt_cont_data['p']

# Turning data into numpy arrays
X = pnt_cont_data.to_numpy()[:,:-3]
split = int(0.8*len(X))
X_train_np, X_test_np = X[:split], X[split:]

y = pnt_cont_data.to_numpy()[:,11:]
y_train_np, y_test_np = y[:split], y[split:]

X_train_float = X_train_np.astype(np.float32)
X_train = torch.Tensor(X_train_float)


X_test_float = X_test_np.astype(np.float32)
X_test = torch.Tensor(X_test_float)

y_train_float = y_train_np.astype(np.float32)
y_train = torch.Tensor(y_train_float)

y_test_float = y_test_np.astype(np.float32)
y_test = torch.Tensor(y_test_float)

# Creating a train dataset and dataloader with batch_size = 40
train_dataset = TensorDataset(X_train, y_train)
train_dataloader = DataLoader(train_dataset, batch_size = 40)


# Creating a test dataset and dataloader with batch_size = 20
test_dataset = TensorDataset(X_test, y_test)
test_dataloader = DataLoader(test_dataset, batch_size = 10)


# Building Neural Network
#device = 'cuda' if torch.cuda.is_available() else 'cpu'
device = torch.device('cuda')

input_dim = 10
output_dim = 2

class NN(nn.Module):
    def __init__(self, input_dim, output_dim):
        super(NN, self).__init__()
        self.linear = nn.Linear(input_dim, output_dim)
        
        '''
        Add hidden layers here:
        '''
        
    def forward(self, x):
        y = self.linear(x)
        
        '''
        Add activation functions for hidden layers here:
        '''
        return y

model = NN(input_dim, output_dim).to(device)

lr = 0.1
optimizer = optim.Adam(model.parameters(), lr = lr)
criterion = nn.MSELoss()


LOSS = []

epochs = 100

for epoch in range(epochs):
    for X, y in train_dataloader:
        X, y = X.to(device), y.to(device)
        
        #X and y are printing as the data given
        #y_pred is printing as nan
        y_pred = model(X)
        
        
        loss = criterion(y_pred, y)
        LOSS.append(loss.item())
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()</code></pre>

KFrank: <p>Hi Brooklyn!</p>
<aside class="quote no-group" data-username="Brooklyn_Amaro" data-post="1" data-topic="183151" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/b/0ea827/48.png" class="avatar"> Brooklyn_Amaro:</div>
<blockquote>
<p>I noticed that my predictions were coming out to be ‘nan’ in my training loop.</p>
</blockquote>
</aside>
<p>Check that your input data is free of <code>nan</code>s.</p>
<p>Try a single forward pass.  Are the outputs of your network free of <code>nan</code>s?</p>
<p>Try a single backward pass.  Are your gradients free of <code>nan</code>s?</p>
<p>Try a single optimization step.  Are your model weights free of <code>nan</code>s?</p>
<p>If you pass these tests, your training is probably becoming unstable<br>
over time.</p>
<p>Try training with plain-vanilla <code>SGD</code> (no momentum nor weight decay).<br>
Start with a low learning rate.  Can you train successfully, even if slowly?<br>
If so, try increasing the learning rate and possibly turning on momentum.</p>
<aside class="quote no-group">
<blockquote>
<pre><code class="lang-auto">lr = 0.1
optimizer = optim.Adam(model.parameters(), lr = lr)
</code></pre>
</blockquote>
</aside>
<p>Although it often trains faster, <code>Adam</code> can be unstable sometimes.</p>
<p>Also, as a general rule, the learning rate with which <code>Adam</code> can train<br>
stably tends to be numerically significantly smaller than those that<br>
work with <code>SGD</code>.  I would suggest starting with <code>lr = 1.e-6</code> and increasing<br>
it until you either get successful, if slow, training, or unstable training.</p>
<p>Good luck!</p>
<p>K. Frank</p>

Brooklyn_Amaro: <p>Thank you so much for the response! I took out any rows in my data that contained ‘nan’, changed Adam to SGD, and changed my learning rate to be much lower and everything seems to work now! I appreciate the help!</p>