Confused about why model isn't predicting correctly

username: arundas
cooked: <p>I’m trying to write a model that will try to approximate the underlying function behind some data (I have a lot of x,y pairs).</p>
<p>Here is the model architecture:</p>
<pre><code class="lang-auto">class Net(torch.nn.Module):
	def __init__(self):
		super(Net, self).__init__()
		self.fc1 = Linear(1, 50)
		self.fc2 = Linear(50, 1)
		
	def forward(self, x):
		sigmoid = torch.nn.Sigmoid()
		x = self.fc1(x)
		x = sigmoid(x)
		x = self.fc2(x)
		return x
	
model = Net()
</code></pre>
<p>The X values are in the numpy array X = [[366922500], [696530521], …], the Y values are in a numpy array “res” formatted res = [[0.0], [-652.2099712329232],…].</p>
<p>And here is how I am attempting to training it:</p>
<pre><code class="lang-auto"># parameters
num_epochs = 200
learning_rate = 0.1

# define the loss function
critereon = L1Loss()

# define the optimizer
optimizer = Rprop(model.parameters(), lr=learning_rate)

predictions = []

# training loop
for epoch in range(num_epochs):
	epoch_loss = 0
	predictions = []
	
	for ix in range(len(X)):
		y_pred = model(Variable(Tensor(X[ix])))
		# print(X[ix], res[ix], y_pred.item())
		predictions.append(y_pred.item())
	
		loss = critereon(y_pred, Variable(Tensor(res[ix])))
		epoch_loss += loss.data.item()
	
		optimizer.zero_grad()
		loss.backward()
		optimizer.step()

	epoch_loss = epoch_loss/len(X)
	print("Epoch: {} Loss: {}".format(epoch, epoch_loss))

plt.plot(X,predictions)
plt.plot(X,res)
plt.show()
</code></pre>
<p>The final three lines of that snippet are so I can see how the predictions during the final training loop compare to the true Y values. Orange are the true values, blue are the predictions.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/6/640e07ba48a0e86ae292d7f568922712858ad067.png" data-download-href="https://discuss.pytorch.org/uploads/default/640e07ba48a0e86ae292d7f568922712858ad067" title="ResidualNN"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/6/640e07ba48a0e86ae292d7f568922712858ad067_2_689x482.png" alt="ResidualNN" data-base62-sha1="eh7O0uEz60YKTNtKtDCKiz8WsWr" width="689" height="482" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/6/640e07ba48a0e86ae292d7f568922712858ad067_2_689x482.png, https://discuss.pytorch.org/uploads/default/optimized/2X/6/640e07ba48a0e86ae292d7f568922712858ad067_2_1033x723.png 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/6/640e07ba48a0e86ae292d7f568922712858ad067.png 2x" data-dominant-color="FDFCFA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">ResidualNN</span><span class="informations">1308×914 76.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>The loss values decrease over the course of training, and the predictions do improve after each round of training.</p>
<p>Now, I am trying to predict using this model, doing the following:</p>
<pre><code class="lang-auto">preds = []
for idx in range(len(X)):
	pred = model(Variable(Tensor(X[idx])))
	print(X[idx], res[idx], pred.item())
	preds.append(pred.item())

plt.plot(X,preds)
plt.plot(X,res)
plt.show()
</code></pre>
<p>However, when I do this, the model just predicts one value for the entirety of the X values, resulting in this plot:</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/2X/a/a6e55ec64e85ca6bc05b6cfe2361dd7156e65fcd.png" alt="Figure_1" data-base62-sha1="nOqGnpEYj21ZwohI3GH4eNQVSpf" width="640" height="480"></p>
<p>Does anyone have any idea why this is happening? During training, the predictions seems to be correct, but after training, the model just predicts one value for all the inputs. These are the same inputs it was trained on, and the same inputs for the first plot, so I have no idea what is happening.</p>

username: ptrblck
cooked: <p>Both codes look pretty much equal, so it’s weird your model does not output the same values for the same inputs.<br>
Could you post the complete code so that we could have a look at some code bugs or other issues?</p>

username: arundas
cooked: <p>Of course. Here is the code:</p>
<pre><code class="lang-auto">import torch
from torch import Tensor
from torch.nn import Linear, L1Loss, MSELoss, functional as F
from torch.optim import SGD, Adam, RMSprop, Rprop
from torch.autograd import Variable
import numpy as np
import matplotlib.pyplot as plt

# parameters
num_epochs = 10
learning_rate = 0.1

# net architecture
class Net(torch.nn.Module):
	def __init__(self):
		super(Net, self).__init__()
		self.fc1 = Linear(1, 50)
		self.fc2 = Linear(50, 1)
		
	def forward(self, x):
		sigmoid = torch.nn.Sigmoid()
		x = self.fc1(x)
		x = sigmoid(x)
		x = self.fc2(x)
		return x
	
model = Net()

# define the loss function
critereon = L1Loss()

# define the optimizer
optimizer = Rprop(model.parameters(), lr=learning_rate)

# load data
data = np.loadtxt("/Users/arun/Desktop/JHU/sapling/rust/sa_ecoli.key.row.sample", usecols=(0,1),skiprows=0, delimiter=" ")
data = data.T

# turn into a ist of lists
X = np.array([[i]for i in data[0]])
y = np.array([[i]for i in data[1]])

# generate residual

# compute gradient
dX = X[len(X)-1][0] - X[0][0]
# print(dX)
dy = y[len(y)-1][0] - y[0][0]
# print(dy)
m = dy/dX
c = y[0][0] - X[0][0]*m

# compute residual values
res = []
for idx in range(len(X)):
	lineVal = X[idx][0] * m + c
	trueVal = y[idx][0]
	resVal = lineVal - trueVal
	res.append([resVal])

# scale data
res = (res - np.min(res))/np.ptp(res)

# plt.plot(X,y)
# plt.show()
# plt.plot(X, res)
# plt.show()

# store the predictions during training
predictions = []

# create our training loop
for epoch in range(num_epochs):
	epoch_loss = 0
	predictions = []
	
	for ix in range(len(X)):
		y_pred = model(Variable(Tensor(X[ix])))
		# print(X[ix], res[ix], y_pred.item())
		predictions.append(y_pred.item())
	
		loss = critereon(y_pred, Variable(Tensor(res[ix])))
		epoch_loss += loss.data.item()
	
		optimizer.zero_grad()
		loss.backward()
		optimizer.step()

	epoch_loss = epoch_loss/len(X)
	print("Epoch: {} Loss: {}".format(epoch, epoch_loss))
	# print("Epoch: {}".format(epoch))


plt.plot(X,predictions)
plt.plot(X,res)
plt.show()

model.eval()

preds = []
for idx in range(len(X)):
	pred = model(Variable(Tensor(X[idx])))
	# print(X[idx], res[idx], pred.item())
	preds.append(pred.item())

plt.plot(X,preds)
plt.plot(X,res)
plt.show()
</code></pre>
<p>The data is here:<br>
<a href="https://github.com/arun96/random/blob/master/sa_ecoli.key.row.sample" rel="nofollow noopener">https://github.com/arun96/random/blob/master/sa_ecoli.key.row.sample</a></p>
<p>Any help would be greatly appreciated - I’m at my wits end when it comes to understanding this.</p>

username: ptrblck
cooked: <p>Thanks for the code!</p>
<p>It looks like the model tends to predict a constant value for the data.<br>
Since you are feeding each sample one by one in the training loop and updating the parameters of the model for each sample, you might force the model to update the predictions for each sample.<br>
If you get rid of the for loop and feed the whole data as a batch during training, the predictions will also be constant.</p>
<p>Generally, your values are really high, so that normalizing them should yield better results.<br>
Since you are dealing with values &gt; 1e8, I would assume that the sigmoid output is basically saturated for each input, which is probably the reason for the constant output.</p>
<p>Let me know, if that helps somehow.</p>

username: arundas
cooked: <p>Thank you for the suggestions!</p>
<p>How do you feed the whole data in as a batch during training? I’m not entirely sure how to do that.</p>
<p>And I’ll look into normalizing! Right now I’ve been scaling the y-values, but I will normalize the x ones too.</p>
<p>Thank you!</p>

username: ptrblck
cooked: <p>You would just have to pass the whole batch into the model:</p>
<pre><code class="lang-python">y_pred = model(torch.tensor(X).float())
</code></pre>
<p>Also note, that <code>Variables</code> are deprecated, so you can just use tensors in PyTorch <code>&gt; 0.4.0</code>.</p>