Simple ANN model not training

username: paragG
cooked: <p>Hi, I am new to PyTorch. I am trying to train a simple ANN model for regression. The issue is my loss doesn’t seem to be changing much over ~10 epochs. The exact same model/loss/optimizer, on the same data shows very different loss in another implementation. The two implementations primarily differ in data prep. The data is in table format - each row has a target col and a bunch of feature cols. I would greatly appreciate any help (been stuck for several days). Below is my code:<br>
First I read in the data in a pandas dataframes df and df_test. Next, I split df into train_data+val_data using stratified split. The rest of the code is below:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/c/4cc804f85706b724659a3c7977a093c6bc30ea78.png" data-download-href="https://discuss.pytorch.org/uploads/default/4cc804f85706b724659a3c7977a093c6bc30ea78" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/4/c/4cc804f85706b724659a3c7977a093c6bc30ea78_2_323x500.png" alt="image" data-base62-sha1="aXePiq9wDOQAuwpW6Sj7ZDZSKEg" width="323" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/4/c/4cc804f85706b724659a3c7977a093c6bc30ea78_2_323x500.png, https://discuss.pytorch.org/uploads/default/original/3X/4/c/4cc804f85706b724659a3c7977a093c6bc30ea78.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/4/c/4cc804f85706b724659a3c7977a093c6bc30ea78.png 2x" data-dominant-color="F3F3F3"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">480×741 123 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>The val_loss changes between 3.84-3.85 over 10 epochs. The other implementation starts with a val_loss &lt; 2 and by 6 epochs is &lt; 1 (same batch size as mine). Sorry for the lengthy post. I would appreciate any help (may be an obvious mistake I am overlooking?). Thanks.</p>

username: my3bikaht
cooked: <p>it’s hard to read on screenshot<br>
you can use google colab to share your notebook</p>

username: vfdev-5
cooked: <p><a class="mention" href="/u/paragg">@paragG</a> yes, an executable code would be nice to have.<br>
Do you see that the model is training, e.g. training batch loss is decreasing ?</p>

username: paragG
cooked: <pre><code class="lang-auto">import numpy as np
import pandas as pd
import os  

from PIL import Image  

import torch  
import torch.nn as nn
import torch.utils.data as data_utils
from torch.optim.lr_scheduler import ExponentialLR
import torch.nn.functional as F 

from torchvision import models, transforms as T

from tqdm import tqdm_notebook

from glob import glob
from sklearn.preprocessing import LabelEncoder
from sklearn.model_selection import train_test_split

import matplotlib.pyplot as plt
import seaborn as sns
sns.set(style = "darkgrid")
%matplotlib inline

import warnings
warnings.filterwarnings("ignore")

from ignite.engine import Events, create_supervised_evaluator, create_supervised_trainer
from ignite.metrics import Loss, Accuracy
from ignite.contrib.handlers.tqdm_logger import ProgressBar
from ignite.handlers import  EarlyStopping, ModelCheckpoint

target_col = ["target"]
features_col = ["feature1", "feature2", "feature3",...,"feature966"]

def df_to_tensor_w_target_and_features(df):
    target = torch.tensor(df[target_col].values, dtype=torch.float32)
    features = torch.tensor(df[features_col].values, dtype=torch.float32)
    combined = data_utils.TensorDataset(features, target)
    return combined

device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
BATCH_SIZE = 64

df = pd.read_csv("./train.dat", sep="\t")
df_test = pd.read_csv("./test.dat", sep="\t")

train_data, val_data = train_test_split(df, random_state = 42, test_size = 0.2, stratify = df["avg_grp_no"])

train = df_to_tensor_w_target_and_features(train_data)
val = df_to_tensor_w_target_and_features(val_data)
test = df_to_tensor_w_target_and_features(df_test)

train_loader = data_utils.DataLoader(train, batch_size=BATCH_SIZE, shuffle=True)
val_loader = data_utils.DataLoader(val, batch_size=BATCH_SIZE, shuffle=False)
test_loader = data_utils.DataLoader(test, batch_size=BATCH_SIZE, shuffle=True)

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        
        self.fc1 = nn.Linear(966, 1024)
        self.fc2 = nn.Linear(1024, 256)
        self.fc3 = nn.Linear(256, 10)
        self.fc4 = nn.Linear(10, 1)
        
        self.drpout = nn.Dropout(0.2)

    def forward(self, x):
        x = x.view(-1, 966)
        x = F.relu(self.fc1(x))
        x = self.drpout(x)
        x = F.relu(self.fc2(x))
        x = self.drpout(x)
        x = F.relu(self.fc3(x))
        x = self.fc4(x)
        return x  
    
model = Net()

criterion = nn.MSELoss()
optimizer = torch.optim.SGD(model.parameters(),lr = 0.001)

metrics = { "loss" : Loss(criterion)}
trainer = create_supervised_trainer(model, optimizer, criterion, device = device)
val_eval = create_supervised_evaluator(model, metrics = metrics, device = device)

@trainer.on(Events.EPOCH_COMPLETED)
def compute_and_display(engine):
    epoch = engine.state.epoch
    metrics = val_eval.run(val_loader).metrics
    print("Validation Results - Epoch: {}  Average Loss: {:.4f} ".format(engine.state.epoch, metrics['loss']))
    
handler = EarlyStopping(patience = 40, score_function = lambda engine : engine.state.metrics['loss'], trainer = trainer)
val_eval.add_event_handler(Events.COMPLETED, handler)

checkpoints = ModelCheckpoint("models_1", "Model_1", n_saved = 3, create_dir = True, require_empty=False)
trainer.add_event_handler(Events.EPOCH_COMPLETED, checkpoints, {"ANN" : model})

pbar = ProgressBar(bar_format = '')
pbar.attach(trainer, output_transform = lambda x : {"loss" : x})

trainer.run(train_loader, max_epochs = 4)
</code></pre>

username: paragG
cooked: <p><a class="mention" href="/u/my3bikaht">@my3bikaht</a> , <a class="mention" href="/u/vfdev-5">@vfdev-5</a> , I am going to try posting on google colab. I have the code in Kaggle and can share the link if that helps. I apologize for the poor readability. Since my issue is not resolved, I would try to post it in a format that is more readable. Meanwhile, thanks for looking into it.</p>

username: paragG
cooked: <p>I found the bug in the code above. I needed to unsqueeze the input target tensors by the adding the line: target = target.unsqueeze(1) This changed the shape of the target tensor from torch.Size([num_of_rows]) to torch.Size([num_of_rows], <strong>1</strong>) . The model trains fine now:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/d/f/dfe33db663c9c0bc52ab5f8451eb38cc816101c2.png" alt="image" data-base62-sha1="vWBkVUgaJEvHmK4yttF78OJQCYO" width="433" height="305"><br>
Thanks to those who looked to help.</p>