PyTorch fails to (over)fit Boston housing dataset

username: Alaya-in-Matrix
cooked: <p>I am trying to use neural network to fit the boston housing dataset, as a starting point, I want to firstly overfit the training data.  This seems to be a trivial task, the below code is used</p>
<pre><code class="lang-auto">from sklearn.datasets import load_boston
from sklearn.model_selection import train_test_split
from sklearn.neural_network import MLPRegressor
import numpy  as np
import sklearn
import matplotlib.pyplot as plt

import torch
import torch.nn as nn

boston = load_boston()
X,y   = (boston.data, boston.target)
dim = X.shape[1]

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.1, random_state=0)
num_train = X_train.shape[0]

torch.set_default_dtype(torch.float64)
net = nn.Sequential(
    nn.Linear(dim, 50, bias = True), nn.ReLU(),
    nn.Linear(50,   50, bias = True), nn.ReLU(),
    nn.Linear(50,   50, bias = True), nn.ReLU(),
    nn.Linear(50,   1)
)
criterion = nn.MSELoss()
opt = torch.optim.Adam(net.parameters(), lr = 1e-6)

num_epochs = 1000
from torch.utils.data import TensorDataset, DataLoader
dataset = TensorDataset(torch.from_numpy(X_train).clone(), torch.from_numpy(y_train).clone())
loader = DataLoader(dataset=dataset, batch_size=128, shuffle=True)



for i in range(num_epochs):
    for x,y in loader:
        loss = criterion(y, net(x))
        loss.backward()
        opt.step()
    if i &gt; 0 and i % 100 == 0:
        print('Epoch %d, loss = %g' % (i, loss))
        
py = net(torch.DoubleTensor(X_train))
plt.plot(y_train, py.detach().numpy(), '+')
plt.xlabel('Actual value of training set')
plt.ylabel('Prediction')
</code></pre>
<p>However, the trained model completely underfits the data, the below figure is drawn:</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/2X/5/5375049f982b18a07825a7152425f43752b3d106.png" alt="image" data-base62-sha1="bUil5hMAXZkYNZYIU3nnFBfzqHY" width="396" height="266"></p>
<p>This seems to indicate some bugs in my code instead of low model capacity, I also tried the <code>MLPRegressor</code> in sklearn using the very same architecture, optimizer and hyperparameters:</p>
<pre><code class="lang-auto">model = MLPRegressor(
    hidden_layer_sizes=(50,50,50),
    alpha = 0,
    activation='relu',
    batch_size=128,
    learning_rate_init = 1e-3,
    solver = 'adam',
    learning_rate = 'constant',
    verbose = False, 
    n_iter_no_change = 1000,
    validation_fraction = 0.0,
    max_iter=1000)
model.fit(X_train, y_train)

py = model.predict(X_test)
err = y_test - py
mse = np.mean(err**2)
rmse = np.sqrt(mse)
print('rmse for test %g' % rmse)
plt.subplot(121)
plt.plot(y_test, py, '+')

err = y_train - model.predict(X_train)
mse = np.mean(err**2)
print('rmse for train: %g' % np.sqrt(mse))
plt.subplot(122)
plt.plot(y_train, model.predict(X_train), '+')
</code></pre>
<p>The sklearn code gives reasonable predictions, as shown below:</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/2X/e/ebf9f56513110c896825e119c11621cb5451b0bd.png" alt="image" data-base62-sha1="xFxJPYUkFjFd9EKu8RARPErVCNv" width="386" height="278"></p>

username: Alaya-in-Matrix
cooked: <p>I found the reason, it’s because the shape of <code>y</code> and <code>net(x)</code> don’t match, so the MSE loss is incorrect</p>

username: Jelee
cooked: <p>How to fix it??<br>
Could you explain more?<br>
I have similar problem.</p>

username: Alaya-in-Matrix
cooked: <p>In my problem, shape of  <code>train_y</code>  is  <code>(455,)</code> , while the shape of  <code>net(torch.from_numpy(X_train))</code>  is  <code>(455,1)</code> , the inconsistense of shapes would make the MSELoss incorrect.</p>
<p>For example, you can run the below code to see the difference:</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
xs   = torch.randn(100)
ys   = torch.randn(100)
crit = nn.MSELoss()
print('Loss1 = %g,  Loss2=%g' % (crit(xs, ys),  crit(xs.reshape(100, 1), ys)))
</code></pre>

username: SimonW
cooked: <p>Also you are missing the line that clears gradients, i.e., <code>opt.zero_grad()</code>.</p>

username: Jelee
cooked: <p>Thanks.</p>
<p>I have one more question!<br>
Which shape is correct for MSEloss (455,) or (455,1)??</p>

username: Alaya-in-Matrix
cooked: <p>I think both shapes are OK, as long as they are the same.</p>

username: Jelee
cooked: <p>Oh… It’s really interesting.<br>
Why MSEloss distinct those two tensor?..</p>
<p>Anyway Thanks <a class="mention" href="/u/alaya-in-matrix">@Alaya-in-Matrix</a></p>

username: Alaya-in-Matrix
cooked: <p>It seems to be the broadcast semantics, if you define <code>MSE(x,y)</code> as <code>((x - y)**2).mean()</code>, you can see similar things in numpy</p>

username: SimonW
cooked: <p>(455,) and (455,1) are different in broadcasting though, because broadcasting starts from the right end!</p>
<p>If you do (455,1) - (455,), you will get a (455, 455) tensor, while (455,1) - (455,1) gives you a (455,1) tensor. MSE, if it is implemented as (x - y).pow(2).mean(), will work differently.</p>

username: mshuaibi
cooked: <p>I’m facing similar issues with my model and noticed my targets are (400,) while outputs are (400,1). However, when changing the targets to (400,1) I still get the same MSELoss() that was calculated before. My predictions end up being a constant at the average of the targets. A similar implementation was done with SGD but with the same issue My code currently looks as such:</p>
<pre><code class="lang-auto">    for epoch in range(num_epochs):
        MSE=0.0
        for data_sample in dataloader:
            input_data=data_sample[0]
            target=data_sample[1]
            batch_size=len(target)
            target=target.reshape(400,1)

            input_data=input_data.to(device)
            target=target.to(device)

            def closure():
                optimizer.zero_grad()
                output=model(input_data)
                loss=criterion(output,target)
                loss.backward()
                return loss

            loss=optimizer.step(closure)
            MSE+=loss.item()*batch_size

        MSE=MSE/dataset_size
        RMSE=np.sqrt(MSE)
        epoch_loss=RMSE
        print epoch_loss
</code></pre>

username: talasinski
cooked: <p>This seems to work:<br>
#!/usr/bin/env python</p>
<h1>coding: utf-8</h1>
<p>from sklearn.datasets import load_boston<br>
from sklearn.model_selection import train_test_split<br>
from sklearn.neural_network import MLPRegressor<br>
import numpy  as np<br>
import sklearn<br>
import matplotlib.pyplot as plt</p>
<p>import torch<br>
import torch.nn as nn<br>
import pandas as pd</p>
<p>boston = load_boston()<br>
X,y   = (boston.data, boston.target)<br>
dim = X.shape[1]</p>
<p>X.shape<br>
house = pd.read_csv(‘BostonHousing.csv’)</p>
<p>print(house.head(10))</p>
<p>house.hist(column=‘medv’, bins=50)<br>
plt.show()</p>
<h1>In[13]:</h1>
<p>X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.9, random_state=0)<br>
num_train = X_train.shape[0]<br>
X_train</p>
<h1>In[23]:</h1>
<p>torch.set_default_dtype(torch.float64)<br>
net = nn.Sequential(<br>
nn.Linear(dim, 50, bias = True), nn.ELU(),<br>
nn.Linear(50,   50, bias = True), nn.ELU(),<br>
nn.Linear(50,   50, bias = True), nn.Sigmoid(),<br>
nn.Linear(50,   1)<br>
)<br>
criterion = nn.MSELoss()<br>
opt = torch.optim.Adam(net.parameters(), lr = .0005)</p>
<h1>In[24]:</h1>
<p>num_epochs = 8000<br>
<span class="hashtag">#from</span> torch.utils.data import TensorDataset, DataLoader<br>
y_train_t =torch.from_numpy(y_train).clone().reshape(-1, 1)<br>
x_train_t =torch.from_numpy(X_train).clone()<br>
<span class="hashtag">#dataset</span> = TensorDataset(torch.from_numpy(X_train).detach().clone(), torch.from_numpy(y_train).reshape(-1,1).detach().clone())<br>
<span class="hashtag">#loader</span> = DataLoader(dataset=dataset, batch_size=128, shuffle=True)<br>
losssave = []<br>
stepsave = []</p>
<p>for i in range(num_epochs):<br>
y_hat = net(x_train_t)<br>
loss = criterion(y_train_t,net(x_train_t))<br>
losssave.append(loss.item())<br>
stepsave.append(i)<br>
loss.backward()<br>
opt.step()<br>
opt.zero_grad()<br>
y_hat_class = (y_hat.detach().numpy())<br>
accuracy = np.sum(y_train.reshape(-1,1)== y_hat_class )/len(y_train)<br>
if i &gt; 0 and i % 100 == 0:<br>
print('Epoch %d, loss = %g acc = %g ’ % (i, loss,  accuracy))</p>
<p>ss=np.array(stepsave)<br>
ss.shape<br>
sl =np.array(losssave)<br>
sl.shape<br>
<span class="hashtag">#print</span> (y_hat_class)<br>
<span class="hashtag">#print</span>(y_train.reshape(-1,1))<br>
<span class="hashtag">#ss</span>.reshape(8000)<br>
<span class="hashtag">#sl</span>.reshape(8000)</p>
<h1>In[28]:</h1>
<p>py = net(torch.DoubleTensor(X_train))<br>
plt.plot(sl, ‘+’)<br>
plt.xlabel(‘Actual value of training set’)<br>
plt.ylabel(‘Prediction’)<br>
plt.show()</p>
<h1>In[29]:</h1>
<p>ypred = net(torch.from_numpy(X_test).detach())<br>
err = ypred.detach().numpy() - y_test<br>
mse = np.mean(err*err)<br>
print(np.sqrt(mse))<br>
plt.plot(ypred.detach().numpy(),y_test, ‘+’)<br>
plt.show()</p>
<h1>In[ ]:</h1>
<p>model = MLPRegressor(<br>
hidden_layer_sizes=(50,50,50),<br>
alpha = 0,<br>
activation=‘relu’,<br>
batch_size=128,<br>
learning_rate_init = 1e-3,<br>
solver = ‘adam’,<br>
learning_rate = ‘constant’,<br>
verbose = False,<br>
n_iter_no_change = 1000,<br>
validation_fraction = 0.0,<br>
max_iter=1000)<br>
model.fit(X_train, y_train)</p>
<p>py = model.predict(X_test)<br>
err = y_test - py<br>
mse = np.mean(err<strong>2)<br>
rmse = np.sqrt(mse)<br>
print(‘rmse for test %g’ % rmse)<br>
plt.subplot(121)<br>
plt.plot(y_test, py, ‘+’)<br>
plt.show()<br>
err = y_train - model.predict(X_train)<br>
mse = np.mean(err</strong>2)</p>
<h1>In[ ]:</h1>
<p>plt.plot(py)<br>
py.mean()</p>
<h1>In[ ]:</h1>

username: talasinski
cooked: <p>Based on Anaya-In-Matrix## Wenlong Lyu’s jupyter notebook. The opt.zero_grad() was essential. Note batches were not used…this is a small  problem.</p>
<p>Also the translation from/to numpy was a pain in the ass!</p>
<p>Base in part on <a href="https://gist.github.com/santi-pdp/d0e9002afe74db04aa5bbff6d076e8fe" rel="nofollow noopener">https://gist.github.com/santi-pdp/d0e9002afe74db04aa5bbff6d076e8fe</a></p>

username: unnir
cooked: <p>did you scale the data?</p>