Binary classification model not training

atl: <p>Hi Pytorch community,</p>
<p>I’m new to Pytorch and relatively new to neural networks.<br>
What I want to build is a network simulating a human learning task, where a stimulus of 2 dimensions with different SNRs maps onto a binary response. I have thus created my binary target vector (y) and an input vector (x) with the mean shifted positive/negative depending on the target response.</p>
<p>The network doesn’t seem to learn - the accuracy stays at 50% and the loss also only decreases marginally. I have played with the parameters (learning rate, weight initialisation etc), but nothing changed. I’ve now been stuck at this point for days and couldn’t find any help in the discussion forum so far, so I’d really appreciate any advice on what I’m doing wrong!</p>
<p>Thanks a lot.</p>
<p>Here’s my code:</p>
<p>y = torch.empty(10000,1, dtype=torch.float).random_(2)</p>
<p>batchlen = torch.Tensor.nelement(y)</p>
<p>x = torch.empty(10000,1, dtype=torch.float)</p>
<p>for t in range(batchlen):<br>
if y[t] == 0:<br>
x[t] = torch.randn(1)*(-1)<br>
elif y[t] == 1:<br>
x[t] = torch.randn(1)</p>
<p>sig_m = 1.3<br>
sig_c = .1</p>
<p>x_m = x<em>sig_m<br>
x_c = x</em>sig_c</p>
<p>x = torch.cat((x_m,x_c),1)<br>
x.requires_grad=True</p>
<p>class Network(nn.Module):<br>
def <strong>init</strong>(self):<br>
super().<strong>init</strong>()</p>
<pre><code>    self.hidden = nn.Linear(2,100)
    self.output = nn.Linear(100,1)
    
    self.relu = nn.ReLU()
    self.softmax = nn.Sigmoid()
    
def weights_init(self):
    for module in self.modules():
        if isinstance(module, nn.Linear):
            nn.init.normal_(module.weight, mean = 0, std = 0.1)

def forward(self, x):
    x = self.hidden(x)
    x = self.relu(x)
    x = self.output(x)
    x = self.softmax(x)
    
    return(x)
</code></pre>
<p>model = Network()<br>
model.weights_init()</p>
<p>criterion = nn.BCELoss()<br>
optimizer = optim.SGD(model.parameters(), lr=0.0001)</p>
<p>def binary_acc(pred, target):<br>
pred_tag = torch.round(torch.sigmoid(pred))</p>
<pre><code>correct_results_sum = (pred_tag == target).sum().float()
acc = correct_results_sum/target.shape[0]
acc = torch.round(acc * 100)

return acc
</code></pre>
<p>loss_all = np.zeros((batchlen))<br>
acc_all = np.zeros((batchlen))</p>
<p>for e in range(batchlen):<br>
running_loss = 0<br>
running_acc = 0</p>
<pre><code>out = model(x)

loss = criterion(out,y)
acc = binary_acc(out,y)

optimizer.zero_grad()

loss.backward()
optimizer.step()

running_loss += loss.item()
running_acc += acc.item()

acc_all[e] = acc.item()
loss_all[e] = loss.item()
</code></pre>
<p>else:<br>
print(f"Training loss: {running_loss}")<br>
print(f"Training accuracy: {running_acc}")</p>

KFrank: <p>Hi atl!</p>
<aside class="quote no-group quote-modified" data-username="atl" data-post="1" data-topic="75136" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/eb8c5e/48.png" class="avatar"> atl:</div>
<blockquote>
<p>The network doesn’t seem to learn</p>
<pre data-code-wrap="python"><code class="lang-python">for t in range(batchlen):
    if y[t] == 0:
        x[t] = torch.randn(1)*(-1)
    elif y[t] == 1:
        x[t] = torch.randn(1)
</code></pre>
</blockquote>
</aside>
<p>The <code>x</code> you generate for <code>y = 0</code> isn’t really different than that for <code> y = 1</code>.</p>
<p><code>torch.randn()</code> draws samples from the normal distribution, and<br>
this distribution is symmetrical about zero.  Therefore <code>randn()</code> and<br>
<code>randn() * (-1)</code> are statistically the same.</p>
<p>That is, your inputs for <code>y = 0</code> and <code>y = 1</code> aren’t actually any different,<br>
so your network <em>can’t</em> learn.</p>
<p>As a second issue, you aren’t calculating your accuracy correctly.</p>
<aside class="quote no-group">
<blockquote>
<pre data-code-wrap="python"><code class="lang-python">def binary_acc(pred, target):
    pred_tag = torch.round(torch.sigmoid(pred))

    correct_results_sum = (pred_tag == target).sum().float()
    acc = correct_results_sum/target.shape[0]
    acc = torch.round(acc * 100)
    
    return acc
</code></pre>
</blockquote>
</aside>
<p>Here, you are applying <code>sigmoid()</code> twice in your accuracy calculation,<br>
once in the last “layer” of your model:</p>
<aside class="quote no-group">
<blockquote>
<pre data-code-wrap="python"><code class="lang-python">    self.softmax = nn.Sigmoid()
...
    x = self.softmax(x)
</code></pre>
</blockquote>
</aside>
<p>(<em>Calling</em> your last layer <code>softmax</code> when it’s really <code>nn.Sigmoid()</code> is a<br>
confusing naming choice.)</p>
<p>and then again in your <code>binary_acc()</code> function.</p>
<p>So the first <code>sigmoid()</code> (in <code>model</code>) takes the output of<br>
<code>self.output = nn.Linear(100,1)</code> that ranges from <code>-inf</code> to <code>inf</code><br>
and maps it to <code>[0.0, 1.0]</code>.  The second <code>sigmoid()</code> (in <code>binary_acc()</code>)<br>
maps this to <code>[0.5, 1.0]</code>.  This is then <em>always</em> rounded to 1, so you<br>
compare 100% 1’s in <code>pred_tag</code> to 50% 1’s in <code>target</code> and return<br>
50% for your accuracy, <code>acc</code>.</p>
<p>(As an aside, best practices would recommend, for reasons of<br>
numerical stability, that you get rid of the final <code>Sigmoid</code> in <code>model</code><br>
and use BCEWithLogitsLoss<code>(rather than</code>BCELoss`) as your<br>
loss function.)</p>
<p>Good luck.</p>
<p>K. Frank</p>

atl: <p>Dear K.Frank,</p>
<p>thank you so much for this extensive reply and all the explanations. That was massive help.</p>
<p>I have now fixed the accuracy calculation and changed my loss function to BCEWithLogitsLoss and got rid of the sigmoid in my network.</p>
<p>I’m still a bit confused though about how I should initialise my inputs, so that the normal distributions are not the same for y=0 and y=1.</p>
<p>This is what I did now, but the accuracy drops from 50% to 3% during the course of training:</p>
<p>x_m = torch.empty(10000,1, dtype=torch.float)<br>
x_c = torch.empty(10000,1, dtype=torch.float)</p>
<p>for t in range(batchlen):<br>
if y[t] == 0:<br>
x_m[t] = torch.normal(-1, 1.3, size=(1,1))<br>
x_c[t] = torch.normal(-1, 0.1, size=(1,1))<br>
elif y[t] == 1:<br>
x_m[t] = torch.normal(1, 1.3, size=(1,1))<br>
x_c[t] = torch.normal(1, 0.1, size=(1,1))</p>
<p>Thank you so much! Really appreciate your help.</p>

KFrank: <p>Hi atl!</p>
<aside class="quote no-group" data-username="atl" data-post="3" data-topic="75136" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/eb8c5e/48.png" class="avatar"> atl:</div>
<blockquote>
<p>the accuracy drops from 50% to 3% during the course of training:</p>
</blockquote>
</aside>
<p>It sounds like you are calculating your accuracy backwards.</p>
<p>If you’re getting an accuracy of 3% on a binary problem, then you<br>
could just as well be making predictions with 97% accuracy by<br>
swapping your “0” predictions for “1” and vice versa.</p>
<p>Is your loss decreasing sensibly as you train?</p>
<aside class="quote no-group">
<blockquote>
<p>I’m still a bit confused though about how I should initialise my inputs, so that the normal distributions are not the same for y=0 and y=1.</p>
</blockquote>
</aside>
<p>The way you construct <code>x</code> should work, but it would be a lot cleaner<br>
(and more efficient) if you would get rid of the <code>for</code> and <code>if</code> statements.</p>
<p>You can do it all with pytorch tensor operations, which, in general, is<br>
to be preferred.</p>
<p>I think I would do something like:</p>
<pre data-code-wrap="python"><code class="lang-python">x_m = (1 - y) * torch.normal (-1.0, 1.3, size = (10000, 1)) + y * torch.normal (1.0, 1.3, size = (10000, 1))
x_c = (1 - y) * torch.normal (-1.0, 0.1, size = (10000, 1)) + y * torch.normal (1.0, 0.1, size = (10000, 1))
x = torch.cat ((x_m, x_c), 1)
</code></pre>
<p>This has the minor disadvantage that you generate twice as many<br>
random samples as you need, throwing half away, but it looks simpler<br>
and easier to read to me than the fancier (and more efficient)<br>
alternatives.</p>
<p>As an aside, notice that for the second of your two input values<br>
(the <code>x_c</code> piece), the means of your two distributions (that is, for<br>
<code>y = 0</code> and <code>y = 1</code>) differ by about <em>fifteen</em> standard deviations,<br>
having in practice <em>no</em> overlap.  Therefore it should be very easy<br>
to construct a perfect classifier.  After you get this working, it might<br>
be fun to make the standard deviation of your <code>x_c</code> distributions<br>
something like <code>1.0</code> so that the <code>y = 0</code> and <code>y = 1</code> distributions do,<br>
in fact,  overlap, and see how large an accuracy you can achieve<br>
in practice with your network vs. the maximum theoretical accuracy.</p>
<p>Best.</p>
<p>K. Frank</p>

atl: <p>Hi K.Frank!</p>
<p>Thank you so much again for your help.</p>
<p>I implemented your version of creating the input nodes without the loops, works like a charm. Thank you again also for the long explanation - this is super helpful for learning!</p>
<p>Regarding the accuracy: yes, that is also what I thought. And the loss decreases from 0.7 to 0.001 during the course of training, so I figured there must be a mistake somewhere in my accuracy calculation.</p>
<h1>define accuracy function</h1>
<p>def binary_acc(pred, target):<br>
pred_tag = torch.round(pred)</p>
<pre><code>correct_results_sum = (pred_tag == target).sum().float()
acc = correct_results_sum/target.shape[0]
acc = acc*100

return acc
</code></pre>
<p>This is the function I use, I compare just generally where the (rounded) output corresponds to the target. I tried it on two other toy tensors and it worked sensibly.<br>
Can you spot my mistake?</p>
<p>Thank you so much!</p>
<p>Best,<br>
atl</p>

KFrank: <p>Hello atl!</p>
<aside class="quote no-group quote-modified" data-username="atl" data-post="5" data-topic="75136" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/eb8c5e/48.png" class="avatar"> atl:</div>
<blockquote>
<p>I figured there must be a mistake somewhere in my accuracy calculation.</p>
<pre data-code-wrap="python"><code class="lang-python"># define accuracy function
def binary_acc(pred, target):
    pred_tag = torch.round(pred)

    correct_results_sum = (pred_tag == target).sum().float()
    acc = correct_results_sum/target.shape[0]
    acc = acc*100
    
    return acc
</code></pre>
</blockquote>
</aside>
<p>Based on what you have said, you have taken the <code>sigmoid()</code><br>
out of your network.  (Correct, because you changed your loss<br>
function to <code>BCEWithLogitsLoss()</code>.)  But you have <em>also</em> taken<br>
the <code>sigmoid()</code> out of your accuracy calculation.</p>
<p>Without the network <code>sigmoid()</code> the tensor <code>pred</code> will contain<br>
values that can range from <code>-inf</code> to <code>inf</code>.  (We generally call<br>
these kinds of predictions <em>logits</em>.)  So <code>pred_tag</code> will be integers<br>
that also range, in principle, from <code>-inf</code> to <code>inf</code>, and will rarely<br>
match your <code>target</code> of 0 or 1.</p>
<p>You could put the <code>sigmoid()</code> back in your accuracy calculation.<br>
Or, probably a little bit nicer, compare</p>
<pre data-code-wrap="python"><code class="lang-python">(pred &gt; 0.0).long() == target
</code></pre>
<p>to test for correct predictions.  (And make sure you understand<br>
what this is doing, and why it works.)</p>
<p>Good luck.</p>
<p>K. Frank</p>

atl: <p>Hi K.Frank,</p>
<p>thank you so much for this hint. Of course it couldn’t work without the sigmoid() in my accuracy function. Added it back in and everything works perfectly now.</p>
<p>Thank you su much for your comments &amp; feedback! That was great help and saved me days of debugging <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>