Loss is not decreasing over the different iterations

Mohamed_Nabih: <p>First, this is my Dataloader</p>
<pre><code>X_train, X_test, y_train, y_test = train_test_split(feat, labels, test_size=0.2, random_state=1)

X_train, X_val, y_train, y_val = train_test_split(X_train, y_train, test_size=0.2, random_state=1)

train = data_utils.TensorDataset(X_train, y_train)

train_loader = data_utils.DataLoader(train, batch_size=1000, shuffle=True)

test = data_utils.TensorDataset(X_test, y_test)

test_loader = data_utils.DataLoader(test, batch_size=1000, shuffle=False)

input_size = 13
hidden1_size = 13
hidden2_size = 64
hidden3_size = 128
hidden4_size = 256
hidden5_size = 1024
output_size = 3989

class DNN(nn.Module):

def __init__(self, input_size, hidden1_size, hidden2_size, hidden3_size, hidden4_size, hidden5_size, output_size):
    super(DNN, self).__init__()
    self.fc1 = nn.Linear(input_size, hidden1_size)
    self.drp1 = nn.Dropout(p=0.2, inplace=False)
    self.relu1 = nn.ReLU()
    self.tan1 = nn.Tanh()
    self.fc2 = nn.Linear(hidden1_size, hidden2_size)
    self.drp2 = nn.Dropout(p=0.2, inplace=False)
    self.relu2 = nn.ReLU()
    self.tan2 = nn.Tanh()
    self.fc3 = nn.Linear(hidden2_size, hidden3_size)
    self.drp3 = nn.Dropout(p=0.2, inplace=False)
    self.relu3 = nn.ReLU()
    self.tan3 = nn.Tanh()
    self.fc4 = nn.Linear(hidden3_size, hidden4_size)
    self.drp4 = nn.Dropout(p=0.2, inplace=False)
    self.relu4 = nn.ReLU()
    self.tan4 = nn.Tanh()
    self.fc5 = nn.Linear(hidden4_size, hidden5_size)
    self.drp5 = nn.Dropout(p=0.2, inplace=False)
    self.relu5 = nn.ReLU()
    self.tan5 = nn.Tanh()
    self.fc6 = nn.Linear(hidden5_size, output_size)
    self.tan6 = nn.Tanh()

def forward(self, x):
    out = self.fc1(x)
    out = self.drp1(out)
    out = self.relu1(out)
    out = self.tan1(out)
    out = self.fc2(out)
    out = self.drp2(out)
    out = self.relu2(out)
    out = self.tan2(out)
    out = self.fc3(out)
    out = self.drp3(out)
    out = self.relu3(out)
    out = self.tan3(out)
    out = self.fc4(out)
    out = self.drp4(out)
    out = self.relu4(out)
    out = self.tan4(out)
    out = self.fc5(out)
    out = self.drp5(out)
    out = self.relu5(out)
    out = self.tan5(out)
    out = self.fc6(out)
    out = self.tan6(out)
    return out
  batch_size = 10
  n_iterations = 50
  no_eps = n_iterations / (13 / batch_size)
  no_epochs = int(no_eps)
  model = DNN(input_size, hidden1_size, hidden2_size, hidden3_size, hidden4_size, hidden5_size, output_size)

 criterion = nn.CrossEntropyLoss()
 learning_rate = 0.0001
 optimizer = torch.optim.SGD(model.parameters(), lr=learning_rate)
  iter = 0
 for epoch in range(no_epochs):
for i, (X_train, y_train) in enumerate(train_loader):
    optimizer.zero_grad()
    outputs = model(Variable(X_train))
    loss = criterion(outputs, Variable(y_train))
    print('Iter %d --&gt; loss %f' % (i, loss.item()))
    loss.backward()
    optimizer.step()
correct = 0
total = 0
print('test')
for X_test, y_test in test_loader:
    outputs = model(Variable(X_test))
    pred = outputs.argmax(dim=1, keepdim=True)
    total += y_test.size(0)
    correct += (pred.squeeze() == y_test).sum()  # pred.eq(y_test.view_as(pre  d)).sum().item()

accuracy = 100 * correct / total

print('Iteration: {}.  Accuracy: {}'.format(epoch, accuracy))
</code></pre>
<pre><code class="lang-auto"></code></pre>

simaiden: <p>you shouldn’t use two activation functions (here ReLU ad Tanh) , also if is a classifier you may use Sigmoid in final layer instead of tanh. And <code>Variable</code> is deprecated, pass the tensor directly instead</p>

klory: <p><a href="https://pytorch.org/docs/stable/nn.html#torch.nn.CrossEntropyLoss" rel="nofollow noopener">CrossEntropyLoss</a> means you do not need an activation layer at the end of your network, also try Adam optimizer if SGD is not working, and as <a class="mention" href="/u/simaiden">@simaiden</a> mentioned, two act is not needed.</p>

Mohamed_Nabih: <p>Hello <a class="mention" href="/u/klory">@klory</a> <a class="mention" href="/u/simaiden">@simaiden</a><br>
I did all of this but the loss didn’t improve</p>