Simple Binary Classification NN doesn't converge?

SlimShadys: <p>Hi everyone. I’m building a simple model for a binary classification task on the <a href="https://www.tensorflow.org/datasets/catalog/german_credit_numeric" rel="noopener nofollow ugc">German Credit Numeric</a> dataset.<br>
I have a numpy array of inputs (24 features), followed by a numpy array of outputs {0,1} as follows:</p>
<p>X:</p>
<pre><code class="lang-auto">X[0]: [ 3  6  4 13  2  5  1  4  3 28  3  2  2  2  1  1  0  1  0  0  1  0  0  1]
X shape: (750, 24)
</code></pre>
<p>Y:</p>
<pre><code class="lang-auto">y[0]: 1
Y shape: (750,)
</code></pre>
<p>My model is a simple FC model with ReLU on top:</p>
<pre><code class="lang-auto">class LinearModel(nn.Module):
    def __init__(self):
        super(LinearModel, self).__init__()
        
        # 24 Features
        self.fc1 = nn.Linear(24, 32, device=device)
        self.fc2 = nn.Linear(32, 16, device=device)
        self.fc3 = nn.Linear(16, 1, device=device)
        
    def forward(self, x):
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x[0] # We want the single value only, not a list with a single value
</code></pre>
<p>I decided to use <code>BCEWithLogitsLoss()</code>, since I did not put the <code>sigmoid</code> function on the last layer of my model, along with the <code>Adam()</code> optimizer:</p>
<pre><code class="lang-auto">loss = nn.BCEWithLogitsLoss()
cnn = LinearModel()
opt = torch.optim.Adam(cnn.parameters(), lr = 1e-3)
epochs = 100
</code></pre>
<p>and my train function as follows:</p>
<pre><code class="lang-auto">num_correct_train = 0
num_samples_train = 0
num_correct_val = 0
num_samples_val = 0

valAccuracies = []
trainAccuracies = []

trainLosses = []
valLosses = []

avgTrainLosses = []
avgValLosses = []

for epoch in range(1, epochs+1):
  
  # We put the CNN in training mode
  cnn.train()

  # We iterate over the train set, taking batches of xb, yb.
  for xb, yb in zip(Xtrain, ytrain):
    
      xb = torch.from_numpy(xb).float()
      yb = torch.tensor(yb).float()
      
      xb, yb = xb.to(device), yb.to(device) # Move them to the GPU
      
      opt.zero_grad() # We empty the gradients

      ypred = cnn(xb) # Actual prediction of our model
      lTrain = loss(ypred, yb) # We perform a Binary Classification Entropy Loss with Logits between the real value and our prediction
      
      lTrain.backward() # We compute the gradients
      opt.step() # Parameters updated -&gt; Single step optimization 
         
      # We round the prediction and:
      # if == ground truth -&gt; Correct prediction
      # if != ground truth -&gt; Wrong prediction
      ypred_tag = torch.round(torch.sigmoid(ypred))

      if(ypred_tag == yb):
        num_correct_train += 1
      
      num_samples_train += 1
      
      trainLosses.append(lTrain.item())


  with torch.no_grad():
    cnn.eval() # We put our model in evaluation mode for testing the accuracy over the test set
    for xb, yb in zip(Xtest, ytest):

      xb = torch.from_numpy(xb).float()
      yb = torch.tensor(yb).float()
      
      xb, yb = xb.to(device), yb.to(device) # Move them to the GPU
      
      ypred = cnn(xb)      
      
      lVal = loss(ypred, yb) # We perform a Binary Classification Entropy Loss with Logits between the real value and our prediction
      
      ypred_tag = torch.round(torch.sigmoid(ypred))
      
      if(ypred_tag == yb):
        num_correct_val += 1
      
      num_samples_val += 1
      
      valLosses.append(lVal.item())

  avgTrainLosses.append(np.average(trainLosses))
  avgValLosses.append(np.average(valLosses))

  train_acc = float(num_correct_train)/float(num_samples_train)
  trainAccuracies.append(train_acc)
  val_acc = float(num_correct_val)/float(num_samples_val)
  valAccuracies.append(val_acc)
  
  if (epoch % 5 == 0):
    print(f'Epoch {epoch+0:03}:\nTrain Loss: {np.average(trainLosses):.5f} | Train Accuracy: {train_acc:.3f} /----/ Val Loss: {np.average(valLosses):.5f} | Val Accuracy: {val_acc:.3f}')
    print("-------------------------------")
</code></pre>
<p>But after training the network for almost 30-40 epochs, it seems like the validation loss and validation accuracy tend to get “flattened” and always keep values around 0.55 (loss) and 0.73 (accuracy).</p>
<p>As for the training loss and accuracy, after 30-40 epochs, they tend to go from 0.47 (Loss) to 0.45 and 0.77 (Accuracy) to 0.78 which is not converging that fast neither.</p>
<p>I don’t get why it doesn’t properly converge faster in the very beginning of the epochs.</p>
<p>Feels like I could train this network for hours and still get a value of accuracy oscillating in the range of 0.73.</p>
<p>Am I doing something wrong?</p>
<p>Thanks!</p>

KFrank: <p>Hi Slim!</p>
<aside class="quote no-group quote-modified" data-username="SlimShadys" data-post="1" data-topic="166103" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/slimshadys/48/54890_2.png" class="avatar"> SlimShadys:</div>
<blockquote>
<pre><code class="lang-auto">  # We iterate over the train set, taking batches of xb, yb.
  for xb, yb in zip(Xtrain, ytrain):
</code></pre>
<p>As for the training loss and accuracy, after 30-40 epochs, they tend to go from 0.47 (Loss) to 0.45 and 0.77 (Accuracy) to 0.78 which is not converging that fast neither.</p>
</blockquote>
</aside>
<p>If I understand correctly what you are doing, you have a training dataset<br>
that consists of 750 individual samples and you are using a batch size of<br>
one sample – that is, you take optimization steps based on the gradients<br>
from single individual samples.</p>
<p>It’s conceivable that the single-sample gradients are simply too noisy for<br>
the optimization to make any systematic progress (without using a very<br>
small learning rate).</p>
<p>Try training with significantly larger batches.  For example, you could use<br>
a batch size of 15, so that an epoch would consist of 50 optimization steps<br>
(each based on 15 individual samples).  Or you could try a batch size of<br>
75.  Or maybe even a batch size of 750 where an epoch would consist of<br>
only one optimization step, but using gradients averaged over your entire<br>
training dataset.</p>
<p>Whether you will get good <em>validation</em> results will depend on the character<br>
of your data.  But I would certainly expect you to be able to overfit your<br>
training dataset so as to get a low loss and a high accuracy.</p>
<p>Note, your model is not particularly complicated, but it still consists of a<br>
reasonably large number of parameters, so you may need <em>significantly</em><br>
more than 40 epochs to train / overfit effectively.</p>
<p>Also, there is some lore that training can get stuck when you use <code>ReLU</code>,<br>
as its gradients can vanish.  If the input to <code>ReLU</code> is negative, the gradient<br>
is exactly zero, and the optimizer has no idea which way to move to get<br>
out of the zero-gradient region.  So you might try some other activation,<br>
such as <a href="https://pytorch.org/docs/1.13/generated/torch.nn.LeakyReLU.html" rel="noopener nofollow ugc">LeakyReLU,</a> that is designed to avoid this problem.</p>
<p>Best.</p>
<p>K. Frank</p>

SlimShadys: <p>Hello Frank and thank you so much for answering.</p>
<p>You opened my thoughts about the batch size (I even thought about it, honestly).</p>
<p>I did convert my numpy arrays to two dataloaders as follows:</p>
<pre><code class="lang-auto">TrainTensor = TensorDataset(torch.Tensor(Xtrain),torch.Tensor(ytrain))
TrainDataLoader = DataLoader(TrainTensor, batch_size=750, shuffle=True)

TestTensor = TensorDataset(torch.Tensor(Xtest),torch.Tensor(ytest))
TestDataLoader = DataLoader(TestTensor, batch_size=750, shuffle=False)
</code></pre>
<p>and my training schedule now takes 250 samples of X (with 24 features) and 250 labels, since I’m iterating like this:<br>
<code>for xb, yb in TrainDataLoader:</code></p>
<p>and I’m checking for the accuracy as follows:</p>
<pre><code class="lang-auto">ypred_tag = torch.round(torch.sigmoid(ypred))
num_correct_train = (ypred_tag == yb).sum().float()
train_acc = torch.round(num_correct_train/yb.shape[0] * 100)
</code></pre>
<p>where <code>ypred</code> is the output of my model (250x1) and <code>train_acc</code> is the average between the correct ones and the total number of predictions. I do the same also for the test set.</p>
<p>I did change my <code>ReLU</code> function with <code>LeakyReLU</code> over all the hidden layers in my model.</p>
<p>I ran 300 epochs and the final result is as follows:</p>
<pre><code class="lang-auto">Epoch 300: Train Loss: 0.40169 | Train Accuracy: 81.000 /----/ Val Loss: 24.78004 | Val Accuracy: 35.000
</code></pre>
<p>I really don’t know why my validation loss is so high, I haven’t changed a lot of things except for reshaping my numpy arrays to be dataloaders.</p>
<p>Please note that the train/val accuracy/loss is that specific accuracy/loss for that epoch (in that case 300).</p>
<p>Overall it started with these values (I’m printing every 50 epochs):</p>
<pre><code class="lang-auto">Epoch 50: Train Loss: 0.59462 | Train Accuracy: 71.000 /----/ Val Loss: 2.69253 | Val Accuracy: 68.000
</code></pre>
<p>I tested it on 1000 epochs and the training accuracy reached 97% and loss 0.1, while validation is still the opposite (32% and 280 loss - which is absurd).<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/f/3/f3a39a9a0930e2dadc596dafc669efbe651326f7.png" alt="Accuracies" data-base62-sha1="yLkxNjeRMO7OW0pepuSvB3Iv7Mj" width="628" height="498"><br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/9/1/9170481d663037dbd738be9440dbbb06404385f6.png" alt="Loss" data-base62-sha1="kKBNkssgh5MblUZdJQxu2C1loTs" width="628" height="498"></p>

KFrank: <p>Hi Slim!</p>
<aside class="quote no-group" data-username="SlimShadys" data-post="3" data-topic="166103" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/slimshadys/48/54890_2.png" class="avatar"> SlimShadys:</div>
<blockquote>
<p>I tested it on 1000 epochs and the training accuracy reached 97% and loss 0.1, while validation is still the opposite (32% and 280 loss - which is absurd).</p>
</blockquote>
</aside>
<p>You are most likely <em>overfitting</em> your training data.  This means that your<br>
network is, in effect, “memorizing” your individual training samples without<br>
actually learning to perform your general classification task.</p>
<p>The good news is that you are successfully training your network (to overfit).<br>
The bad news is that your network won’t successfully classify data samples<br>
that are not in your training set.</p>
<p>(You should always be open to the possibility that you simply have a bug<br>
somewhere, so check your code.)</p>
<p>The best solution to overfitting is to train with (much) more training data<br>
(if you have it).</p>
<p>A related approach is to <em>augment</em> your training data (although I don’t<br>
know off-hand whether there is a sensible way to augment your “Credit<br>
Numeric” data).</p>
<p>You might also consider adding <a href="https://pytorch.org/docs/1.13/generated/torch.nn.Dropout.html" rel="noopener nofollow ugc">Dropout</a> layers to your network.</p>
<p>One last question:  Is your validation data of the same character as your<br>
training data?  Does it come from the same source?  For example, do<br>
you have a larger dataset that you <em>randomly</em> split into your training and<br>
validation datasets so that they are statistically identical?</p>
<p>If your training and validation datasets are rather different in character<br>
it becomes harder to train your model to perform well on your validation<br>
data – if you can do so at all – and is likely to require more training data<br>
in order to do so.</p>
<p>Best.</p>
<p>K. Frank</p>

SlimShadys: <aside class="quote no-group" data-username="KFrank" data-post="4" data-topic="166103" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar"> KFrank:</div>
<blockquote>
<p>(You should always be open to the possibility that you simply have a bug<br>
somewhere, so check your code.)</p>
</blockquote>
</aside>
<p>I just checked my code and there was just one bug where I did scale my train data, but not my test data. So the features in the test data were a bit different from the train one. With this being said:</p>
<aside class="quote no-group" data-username="KFrank" data-post="4" data-topic="166103" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar"> KFrank:</div>
<blockquote>
<p>A related approach is to <em>augment</em> your training data (although I don’t<br>
know off-hand whether there is a sensible way to augment your “Credit<br>
Numeric” data).</p>
</blockquote>
</aside>
<p>I’m thinking hard on how to augment my data apart from scaling feature values between 0 and 1.</p>
<aside class="quote no-group" data-username="KFrank" data-post="4" data-topic="166103" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar"> KFrank:</div>
<blockquote>
<p>One last question:  Is your validation data of the same character as your<br>
training data?  Does it come from the same source?  For example, do<br>
you have a larger dataset that you <em>randomly</em> split into your training and<br>
validation datasets so that they are statistically identical?</p>
</blockquote>
</aside>
<p>The German Credit dataset has been given by my professor and can be found <a href="https://archive.ics.uci.edu/ml/machine-learning-databases/statlog/german/german.data-numeric" rel="noopener nofollow ugc">here</a> where the first 23 columns represent a potential client (Salary, Job, Age, etc.), while last column represent the label 0 (Bad) and 1 (Good).<br>
This dataset it’s composed by 1000 samples and does not have a test set itself, so the professor splitted it 75% train and 25% test (750 samples for train and 250 samples for test).</p>
<aside class="quote no-group" data-username="KFrank" data-post="4" data-topic="166103" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar"> KFrank:</div>
<blockquote>
<p>You might also consider adding <a href="https://pytorch.org/docs/1.13/generated/torch.nn.Dropout.html" rel="noopener nofollow ugc">Dropout</a> layers to your network.</p>
</blockquote>
</aside>
<p>By adding dropout with probability 0.5 at each output of <code>LeakyReLU</code>, now my model performs this way.</p>
<pre><code class="lang-auto">Epoch 1000:
	- Training accuracy: 90.000 (avg. 82.341)
	- Training loss: 0.268 (avg. 0.386)
	- Validation accuracy: 71.000 (avg. 70.331)
	- Validation loss: 0.768 (avg. 0.641)
</code></pre>
<p>I think my model it’s still attracted to train data and still overfits, as the validation loss still rises up.</p>
<p>I’ll leave some screenshots below.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/3/6/3644e552c4651f6bc9778d3e6d36aeae3a9be81d.png" alt="Accuracies" data-base62-sha1="7K5oiHDJAhMqAI46YWPoFM8qmAt" width="618" height="498"><br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/b/9/b94fd9a83cefcfff801f16d5b38ed6eeba93e61d.png" alt="Loss" data-base62-sha1="qrlprnkfzyB6YvjRl1SiN3FqKQJ" width="633" height="498"></p>
<p>Anyways thank you for your kind help, I do really appreciate it! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
<p>-Slim</p>

KFrank: <p>Hi Slim!</p>
<aside class="quote no-group" data-username="SlimShadys" data-post="5" data-topic="166103" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/slimshadys/48/54890_2.png" class="avatar"> SlimShadys:</div>
<blockquote>
<p>I did scale my train data, but not my test data.</p>
</blockquote>
</aside>
<p>This could be a significant issue.  Scaling your training and validation data<br>
differently could certainly degrade inference on your validation data or cause<br>
it not to work at all.</p>
<aside class="quote no-group">
<blockquote>
<p>I’m thinking hard on how to augment my data</p>
</blockquote>
</aside>
<p>Input values that are naturally continuous (such as Salary and Age) could<br>
reasonably have some “noise” added to them.  So, for example, a sample<br>
for which Salary = €100,000 could be modified to make an augmented<br>
sample by changing the Salary to €107,000.</p>
<p>The idea is that the model can’t simply “memorize” that the one sample<br>
that happens to have Salary exactly equal to €100,000 is “bad”.  It has<br>
to learn that a sample with such-and-such a Job and such-and-such an<br>
Age with a Salary that is <em>about</em> €100,000 is likely to be bad.</p>
<p>(It’s not so clear to me that you could sensibly augment your data by<br>
modifying a discrete label such as Job.)</p>
<aside class="quote no-group">
<blockquote>
<p>By adding dropout with probability 0.5 at each output of <code>LeakyReLU</code>, now my model performs this way.</p>
<pre><code class="lang-auto">Epoch 1000:
	- Training accuracy: 90.000 (avg. 82.341)
	- Training loss: 0.268 (avg. 0.386)
	- Validation accuracy: 71.000 (avg. 70.331)
	- Validation loss: 0.768 (avg. 0.641)
</code></pre>
</blockquote>
</aside>
<p>This looks like significant progress.  It does look like your training starts<br>
to overfit somewhere around epoch 150, but, still, you’ve brought your<br>
validation down significantly and your accuracy up.</p>
<p>It might be worth trying some experiments to see whether this improvement<br>
is due to consistent scaling of your training and validation data or whether<br>
it’s due to the addition of <code>Dropout</code>s (or maybe both).</p>
<p>Last, you can make your model less likely to overfit by making it smaller,<br>
either “narrower” or “shallower” or both.  Your model is already quite simple,<br>
so it’s not clear how much room for improvement you would have here, but<br>
you could try reducing your internal numbers of features from 32 and 16 to,<br>
say, 16 and 8.  (Of course reducing the size of your model could also make<br>
it so small that it can’t effectively learn your actual classification problem.)</p>
<p>Best.</p>
<p>K. Frank</p>

SlimShadys: <p>Hi Frank!<br>
I did try to modify my hidden layers from 24-&gt;12-&gt;6-&gt;1 and this significantly brought down my validation loss (less overfit).<br>
It looks almost stable now. I played with <code>dropout</code> as well, where higher value of p brings more stability to the validation loss curve (even tho I could see some starting overfitting process around epoch 400/500).</p>
<p>I did not try to augment my data further, but I left my data as it was originally + scaling values between 0 and 1.</p>
<p>This is my overall final results:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/4/e/4eb0004a234dd020383704ab1233dc9a01701535.png" alt="Loss" data-base62-sha1="be6k34qxnkJ7dwv8DdnhsXlLCDP" width="642" height="498"><br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/b/d/bd54a075c4759daef253ebce925dd41d2108851f.png" alt="output" data-base62-sha1="r0TylxClxHwda9uIhfLAiRVlY5N" width="618" height="498"></p>
<p>And I think this is my final result actually, since checking the <a href="https://archive-beta.ics.uci.edu/dataset/144/statlog+german+credit+data" rel="noopener nofollow ugc">official</a> doc of German Credit Dataset, it shows (in the scheme at the bottom) that this dataset performs on an average of 64% of accuracy and maximum of 70% in a Neural Network. It performs better on ML algorithms.</p>
<p>With the plots above I did reach this result:</p>
<pre><code class="lang-auto">Epoch 1000:
	- Training accuracy: 78.000 (avg. 69.438)
	- Training loss: 0.470 (avg. 0.577)
	- Validation accuracy: 74.000 (avg. 69.822)
	- Validation loss: 0.577 (avg. 0.599)
</code></pre>
<p>which seems really consistent and stable.</p>
<p>Thank you for your time Frank, you really helped me out a lot. Wish we would connect for some other tasks! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>