Loss not converging on a binary classification problem

rgbarriada: <p>Hi there,</p>
<p>I’m trying to implement a very simple model (multi layer perceptron) to tackle a binary classification problem but the loss function does not decrease and is saw-shaped.</p>
<p>I do have very few labelled samples: (train (60) | test (15)). The input data is tabular from 7 different data types, which is then normalized (max-min):</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/1/31aab53a6bbbcad7999d2423659b055f0bea478a.png" data-download-href="https://discuss.pytorch.org/uploads/default/31aab53a6bbbcad7999d2423659b055f0bea478a" title="imagen"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/3/1/31aab53a6bbbcad7999d2423659b055f0bea478a_2_345x236.png" alt="imagen" data-base62-sha1="75n8r7XfZwgEkYKkhrw5YpPwnzA" width="345" height="236" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/3/1/31aab53a6bbbcad7999d2423659b055f0bea478a_2_345x236.png, https://discuss.pytorch.org/uploads/default/optimized/3X/3/1/31aab53a6bbbcad7999d2423659b055f0bea478a_2_517x354.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/3/1/31aab53a6bbbcad7999d2423659b055f0bea478a_2_690x472.png 2x" data-dominant-color="E5E5E5"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">imagen</span><span class="informations">760×521 42.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Each sample belong to class 0 or 1</p>
<p>The model:</p>
<pre><code class="lang-auto">class CustomModel(nn.Module):
     def __init__(self, ):
         torch.manual_seed(3)
         super(CustomModel, self).__init__()
         self.fc1 = nn.Linear(7, 60)
         self.fc2 = nn.Linear(60, 100)
         self.fc3 = nn.Linear(100, 30)
         self.fc4 = nn.Linear(30, 2)
         self.apply(self._init_weights)

    def _init_weights(self, module):
        if isinstance(module, nn.Linear):
            torch.nn.init.xavier_uniform_(module.weight.data)
            torch.nn.init.constant_(module.bias.data, 0)

    def forward(self, data):
        x = data
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = F.relu(self.fc3(x))
        x = self.fc4(x)
        return x
</code></pre>
<p>The training process:</p>
<pre><code class="lang-auto">    # model instance
    model = CustomModel()
    device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')

    # Optimizer and criterion
    optimizer = optim.SGD(model.parameters(), lr=0.0001)
    criterion = nn.CrossEntropyLoss()

    model.train(True)
    for i in range(1000):
        for i, batch in enumerate(train_loader):
            sample, ground= batch
            sample = sample.to(device=device, dtype=torch.float32)
            ground = ground.to(device=device, dtype=torch.long)

            optimizer.zero_grad()

            prediction = model(sample)
            loss = criterion(prediction, ground)
            loss.backward()
            optimizer.step()
</code></pre>
<p>And the obtained loss function:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/2/f2b804215b8557a81dbc10e60263a44458e8c72a.png" data-download-href="https://discuss.pytorch.org/uploads/default/f2b804215b8557a81dbc10e60263a44458e8c72a" title="imagen"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/f/2/f2b804215b8557a81dbc10e60263a44458e8c72a_2_690x276.png" alt="imagen" data-base62-sha1="yDbNDuNnH5v6qlQST33fQeMdyGu" width="690" height="276" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/f/2/f2b804215b8557a81dbc10e60263a44458e8c72a_2_690x276.png, https://discuss.pytorch.org/uploads/default/optimized/3X/f/2/f2b804215b8557a81dbc10e60263a44458e8c72a_2_1035x414.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/f/2/f2b804215b8557a81dbc10e60263a44458e8c72a.png 2x" data-dominant-color="D3E3EF"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">imagen</span><span class="informations">1124×450 78.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I don´t know where the problem might be: data processing, model definition or the training process itself. Batch size is set to 8. I have checked out the model parameters are updating (slightly but not much).</p>
<p>Looking to the gradients at last layer at some point the are few values to 0.0 so, I don´t know if this could be the problem.</p>
<pre><code class="lang-auto">print(list(model.parameters())[6].grad)
</code></pre>
<blockquote>
<p>tensor([[ 0.0032, -0.0025,  0.0000,  0.0000,  0.0025, -0.0045,  0.0214,  0.0219,<br>
0.0102,  0.0052, -0.0004, -0.0027,  0.0000, -0.0109,  0.0008,  0.0020,<br>
0.0000,  0.0024,  0.0000,  0.0161,  0.0000,  0.0041,  0.0000, -0.0058,<br>
0.0078, -0.0006,  0.0000,  0.0060,  0.0049,  0.0027],<br>
[-0.0032,  0.0025,  0.0000,  0.0000, -0.0025,  0.0045, -0.0214, -0.0219,<br>
-0.0102, -0.0052,  0.0004,  0.0027,  0.0000,  0.0109, -0.0008, -0.0020,<br>
0.0000, -0.0024,  0.0000, -0.0161,  0.0000, -0.0041,  0.0000,  0.0058,<br>
-0.0078,  0.0006,  0.0000, -0.0060, -0.0049, -0.0027]])</p>
</blockquote>
<p>Thanks in advance.</p>

ptrblck: <p>Your model is able to overfit random samples:</p>
<pre><code class="lang-python">class CustomModel(nn.Module):
    def __init__(self, ):
        torch.manual_seed(3)
        super(CustomModel, self).__init__()
        self.fc1 = nn.Linear(7, 60)
        self.fc2 = nn.Linear(60, 100)
        self.fc3 = nn.Linear(100, 30)
        self.fc4 = nn.Linear(30, 2)
        self.apply(self._init_weights)

    def _init_weights(self, module):
        if isinstance(module, nn.Linear):
            torch.nn.init.xavier_uniform_(module.weight.data)
            torch.nn.init.constant_(module.bias.data, 0)

    def forward(self, data):
        x = data
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = F.relu(self.fc3(x))
        x = self.fc4(x)
        return x
    
model = CustomModel()
optimizer = optim.SGD(model.parameters(), lr=0.0001)
criterion = nn.CrossEntropyLoss()

x = torch.randn(64, 7)
target = torch.randint(0, 2, (64,))

for epoch in range(100000):
    optimizer.zero_grad()
    out = model(x)
    loss = criterion(out, target)
    loss.backward()
    optimizer.step()
    print(f"epoc: {epoch}, loss: {loss:.5f}")
</code></pre>
<p>but it quite slow as the learning rate seems to be low.<br>
I would also expect this behavior since your have a total of <code>60 * 7 = 420</code> input values with:</p>
<pre><code class="lang-python">sum([p.nelement() for p in model.parameters()])
# 9672
</code></pre>
<p>trainable parameters.<br>
You could use my code and should see a constant decrease in the loss value and eventually a 100% accuracy on this random training set.<br>
With that being said, the number of samples is quite low and even if you can overfit the training set it might be tricky to generalize to the validation/test set.</p>

rgbarriada: <p>Thanks <a class="mention" href="/u/ptrblck">@ptrblck</a>!<br>
It was indeed a matter of iterations (now 80000) for the loss to start to converge:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/f/4f36735e821cdc842299b50d2788dc381b089730.png" data-download-href="https://discuss.pytorch.org/uploads/default/4f36735e821cdc842299b50d2788dc381b089730" title="imagen"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/4/f/4f36735e821cdc842299b50d2788dc381b089730_2_689x229.png" alt="imagen" data-base62-sha1="biKnzmKbMxRG9us9K5oKQlWDZqE" width="689" height="229" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/4/f/4f36735e821cdc842299b50d2788dc381b089730_2_689x229.png, https://discuss.pytorch.org/uploads/default/optimized/3X/4/f/4f36735e821cdc842299b50d2788dc381b089730_2_1033x343.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/4/f/4f36735e821cdc842299b50d2788dc381b089730.png 2x" data-dominant-color="E7F0F6"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">imagen</span><span class="informations">1374×458 87.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I dont know yet why this “spike” shape, but the training tendency is decreasing. And the train/test curves make sense with the model over fitting the train data.</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/7/0/70b643d4a4f5e75276ed0be69abde75d30fc9399.png" alt="imagen" data-base62-sha1="g55YBN8A9D0CMEVucimb4R74mFz" width="494" height="362"></p>