Strange results measuring accuracy of NN

norhther: <p>I’m trying to make a model to predict prices using the Boston dataset from sklearn.datasets</p>
<pre><code class="lang-auto"># -*- coding: utf-8 -*-
from sklearn.datasets import load_boston
from torch.utils.data.dataset import Dataset
from torch.utils.data import DataLoader
import torch.nn.functional as F
import torch
import torch.nn as nn
import torch.optim as optim
import numpy as np

class Net(nn.Module):
    def __init__(self):
        super().__init__()
        self.fc1 = nn.Linear(13, 64)
        self.fc2 = nn.Linear(64, 32)
        self.fc3 = nn.Linear(32, 16)
        self.fc4 = nn.Linear(16,1)

    def forward(self, x):
        x = self.fc1(x)
        x = F.relu(x)
        x = self.fc2(x)
        x = F.relu(x)
        x = self.fc3(x)
        x = F.relu(x)
        return self.fc4(x)

net = Net()

class BostonData(Dataset):
    __xs = np.array(0)
    __ys = np.array(0)

    def __init__(self, train = True):
        df = load_boston()
        index = int(len(df["data"]) * 0.7)
        if train:
            self.__xs = np.array(df["data"][0:index])
            self.__ys = np.array(df["target"][0:index])
        else:
            self.__xs = np.array(df["data"][index:])
            self.__ys = np.array(df["target"][index:])

        self.__xs = (self.__xs - self.__xs.mean()) / self.__xs.std()

    def __getitem__(self, index):
        return self.__xs[index], self.__ys[index]

    def __len__(self):
        return len(self.__xs)


dset_train = BostonData(train = True)
dset_test = BostonData(train = False)
train_loader = DataLoader(dset_train, batch_size=10, shuffle=True)
test_loader = DataLoader(dset_train, batch_size=10, shuffle=True)
</code></pre>
<p>After training the model, the strange behavior that I’m talking is on the following part. I want to measure an error average between the true values and the predictions using the test set as follows</p>
<pre><code class="lang-auto">res = 0
with torch.no_grad():
    for testbatch in test_loader:
        X,y = testbatch
        output = net(X.float())
        for i,elem in enumerate(output):
            res = res + np.abs(y[i] - elem)
print(res/len(dset_train))
</code></pre>
<p>This gives me no error whatsoever, but I was curious about if the model was oferfitting, so I changed the <code>test_loader</code> for the <code>train_loader</code></p>
<pre><code class="lang-auto">res = 0
with torch.no_grad():
    for testbatch in train_loader:
        X,y = testbatch
        output = net(X.float())
        for i,elem in enumerate(output):
            res = res + np.abs(y[i] - elem)
print(res/len(dset_train))
</code></pre>
<p>Executing only these lines of code (I’m using Spyder as IDE) brings the same result for <code>res/len(dset_train)</code> as doing it for <code>test_loader</code> , which seems really strange. Is there any error that I didn’t notice?</p>

dejanbatanjac: <aside class="quote no-group" data-username="norhther" data-post="1" data-topic="58416">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/norhther/48/16747_2.png" class="avatar"> norhther:</div>
<blockquote>
<p>with torch.no_grad():</p>
</blockquote>
</aside>
<p>As a side note, <code>with torch.no_grad():</code>  should be for validation only. While training, you should not have the <code>with torch.no_grad():</code> block. This may result in problems you defined as <em>strange results</em>.</p>

norhther: <p>This is the code of the training process</p>
<pre><code class="lang-auto">optimizer = optim.Adam(net.parameters(), lr = 0.001)
criterion = torch.nn.MSELoss() 
EPOCHS = 500000

lloss = []

for epoch in range(EPOCHS):
    avg_train_loss = 0
    for trainbatch in train_loader:
        X,y = trainbatch
        net.zero_grad()
        output = net(X.float())
        loss = criterion(output[:, -1], y)
        loss.backward()
        optimizer.step()
        avg_train_loss += loss.item() / len(train_loader)
    lloss.append(avg_train_loss)
        
</code></pre>
<p>There is no no_grad() there, and I think it has nothing to do with the problem I presented</p>

alex.veuthey: <p>You could try printing several “intermediate” variables, like <code>X</code> and <code>y</code>, <code>output</code>, <code>res</code>, to make sure that you get different values between the train loader and test loader.</p>
<p>The most important thing is checking <code>y[i]</code> against <code>elem</code>, this way you will see if your model makes mistakes.</p>
<p>One thing I’ve noticed in your code (but it might be due to copy/pasting and not be present in your actual code) is in the first error measurement (on the test set, i.e. the 2nd code block in your message): you print <code>res/len(dset_train)</code>. Shouldn’t it be <code>res/len(dset_test)</code> ?</p>

norhther: <p>I saw the problem</p>
<p><code>train_loader = DataLoader(dset_train, batch_size=10, shuffle=True) test_loader = DataLoader(dset_train, batch_size=10, shuffle=True)</code></p>
<p>Something strange when I call DataLoader <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"><br>
Thanks for replying</p>