Training loss is not decreasing below a specific value

Harsh_Chaudhary: <p>I am trying to solve a problem that I found in deep learning with pytorch course on Udacity: <strong>“Predict whether a student will get selected or rejected by the university”</strong>.</p>
<p>But i am getting the training loss ~0.2000 every time. The loss is not falling below this value. Could you please go through the code to give me feedback?</p>
<pre><code class="lang-auto">
import pandas as pd
import numpy as np
data = pd.read_csv('student_data.csv')


import matplotlib.pyplot as plt

def plot_points(data):
    X = np.array(data[['gre', 'gpa']])
    Y = np.array(data['admit'])
    admitted = X[np.argwhere(Y==1)]
    rejected = X[np.argwhere(Y==0)]
    plt.scatter([s[0][0] for s in rejected], [s[0][1] for s in rejected], color='red')
    plt.scatter([s[0][0] for s in admitted], [s[0][1] for s in admitted], color='cyan')
    plt.xlabel('Test (GRE)')
    plt.ylabel('Grades (GPA)')

plot_points(data)

data = data.drop('rank', axis=1)

# Normalizing the values
data['gre'] /= max(data['gre'])
data['gpa'] /= max(data['gpa'])



# splitting in train and test set
sample = np.random.choice(data.index, size=int(len(data)*0.9), replace=False)
train_data, test_data = data.iloc[sample], data.drop(sample)

train_features = train_data.drop('admit', axis=1)
train_labels = train_data['admit']

import torch
from torch import nn, optim
import torch.nn.functional as F 

# converting dataFrames into tensors
input_tensor = torch.from_numpy(train_features.values).type(torch.FloatTensor)
label_tensor = torch.from_numpy(train_labels.values).type(torch.FloatTensor)


class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        
        self.fc1 = nn.Linear(2, 4)
        self.fc2 = nn.Linear(4, 1)
   
    def forward(self, x):
        
        x = F.relu(self.fc1(x))
        x = (self.fc2(x))
        return x

model = Net()

criteron = nn.MSELoss()
optimizer = optim.Adam(model.parameters(), lr=0.003)

losses=[]
EPOCHS = 501
for e in range(EPOCHS):
    train_loss = 0
    
    output = model(input_tensor)
    loss = criteron(output, label_tensor.view(360, 1))
    
    train_loss += loss.item()
    losses.append(train_loss)
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()
    
    if e%100==0:
        print('epoch {}: Training loss: {:.4f}'.format(e+1, train_loss))

plt.plot(losses)  
</code></pre>
<p><a href="https://github.com/udacity/deep-learning-v2-pytorch/blob/master/keras/student-admissions-keras/student_data.csv" rel="nofollow noopener">here</a> is how the data looks like.</p>

abodh_ltd: <p>Just some random suggestions:</p>
<ol>
<li>can you put  <code>optimizer.zero_grad()</code> before <code>loss = criteron(output, label_tensor.view(360, 1))</code>.</li>
<li>set <code>model.train()</code> before you start the training process</li>
</ol>
<p>let me know if this does not work.</p>

Harsh_Chaudhary: <p>thank you for your reply mate!</p>
<p>But this won’t help because optimizer.zero_grad() is required before the gradient calculation i.e loss.backward(). So it wont make any difference.</p>
<p>setting model.train() also wont help because models are by default into train mode until we explicitly put it into evaluation mode.</p>