Neural Network is barely learning anything

SibYz: <p>Greetings,<br>
I have made a neural network for a classification problem (horse health classification) from kaggle, But the model is barely learning anything. I have tried setting the epoch to 20k but still almost 50-60% answers are predicted wrong by the model. I have preprocessed the data and used over sampling, my sklearn models did great in the predictions.<br>
here is my code <strong>Neural network</strong>,</p>
<pre><code class="lang-auto">import torch.nn as nn

class NNModel(nn.Module):
    def __init__(self, inp_size, out_size, hidden):
        super(NNModel, self).__init__()
        
        self.input_layer = nn.Linear(in_features=inp_size, out_features=hidden)
        self.act1 = nn.LeakyReLU()
        self.hidden_layer = nn.Linear(in_features=hidden, out_features=hidden)
        self.act2 = nn.LeakyReLU()
        self.output_layer = nn.Linear(in_features=hidden, out_features=out_size)
        self.dropout = nn.Dropout(0.1)
        self.softmax = nn.Softmax()
    
    def forward(self, input_tensor):
        x = self.input_layer(input_tensor)
        x = self.act1(x)
        x = self.hidden_layer(x)
        x = self.act2(x)
        x = self.output_layer(x)
        x = self.dropout(x)
        x = self.softmax(x)
        return x 
</code></pre>
<p>here is the loss function and optimizer,</p>
<pre><code class="lang-auto">model = NNModel(train_2.shape[1]-1, 3, 512).to('cuda')
optim = torch.optim.Adagrad(model.parameters(), lr=0.01)
criterion = nn.CrossEntropyLoss()

</code></pre>
<p>here is the training function,</p>
<pre><code class="lang-auto">def trainning(dataLoader, epochs, optimizer, criterion):
    model.train()
    loss_h = []
    for i in range(epochs):
        for batch, (X, y) in enumerate(dataLoader):
            y_hat = model(X)
            loss = criterion(y_hat, y)
            loss_h.append(loss.item())
            loss.backward()
            optimizer.step()
            optimizer.zero_grad()
            if i%100 == 0:
                print(f'epoch: {i}, batch: {batch}, loss: {loss}', y_hat[0], y[0]
</code></pre>
<p>Can anyone please help me out a little? I am very new to pytorch.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/a/9aa8f88b9c8383c94bda8d57528edc7a9b9ccce6.png" data-download-href="https://discuss.pytorch.org/uploads/default/9aa8f88b9c8383c94bda8d57528edc7a9b9ccce6" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/9/a/9aa8f88b9c8383c94bda8d57528edc7a9b9ccce6_2_689x183.png" alt="image" data-base62-sha1="m4by1VZK0ReZqzgv3kXVvWrpwmG" width="689" height="183" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/9/a/9aa8f88b9c8383c94bda8d57528edc7a9b9ccce6_2_689x183.png, https://discuss.pytorch.org/uploads/default/optimized/3X/9/a/9aa8f88b9c8383c94bda8d57528edc7a9b9ccce6_2_1033x274.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/9/a/9aa8f88b9c8383c94bda8d57528edc7a9b9ccce6.png 2x" data-dominant-color="2B2C2F"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1240×330 73.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p><code>nn.CrossEntropyLoss</code> expects raw logits as the model output, so remove the <code>nn.Softmax</code> layer at the end of your forward pass. I would probably also remove the last <code>nn.Dropout</code> layer as I don’t know if masking logits would be helpful.</p>

SibYz: <p>Thanks for replying.<br>
I tried by removing the softmax and the dropout earlier, the result was pretty much the same, by same i meant it would predict almost identical to the screenshot i gave but in 0.xx or -x.yy, which after using softmax results in similar values.</p>
<p>that’s why i thought of adding soft max.</p>
<p>The model could guess 40% of the data correctly. I trained the model for 1k, 5k, 10k, 20k epoch with various combinations of layers/activation functions/loss,</p>
<p>Tried different preprocessing techniques but to no avail.</p>