Output of the binary classification model

shirui-japina: <p>I’m using PyTorch to build a model for <strong>binary classification</strong>.</p>
<pre><code class="lang-auto">class XceptionLike(nn.Module):
    def __init__(self):
        super().__init__()
        # CNN part
        ...
        # final
        self.fc_last_1 = nn.Linear(384, 64)
        self.fc_last_2 = nn.Linear(64, 1)

    def forward(self, input_16, input_32, input_48):
        # CNN part
        ...
        out = torch.cat([out_16, out_32, out_48], dim=1)
        out = F.relu(self.fc_last_1(out))
        out = F.relu(self.fc_last_2(out))
        return out

# the model output part
output = model(batch_z_16, batch_z_32, batch_z_48) # model prediction
output = torch.sigmoid(output)
loss = criterion(output, batch_label)
</code></pre>
<p>What confuses me is that can this model used for <strong>binary classification</strong> really?<br>
In my understanding, for <strong>binary classification</strong></p>
<ul>
<li>
<p>output of model [0, 0.5] means prediction for one class.</p>
</li>
<li>
<p>output of model [0.5, 1] means prediction for the other one.</p>
</li>
</ul>
<p>But ReLU function returns [0, positive infinity],<br>
and when sigmoid function gets the output of the model,<br>
it returns [0.5, 1], so the model cant’t return [0, 0.5], which means it can’t predict the class which is belong to [0, 0.5].</p>
<p>What is wrong with my understanding?<br>
How can I deal with it?</p>

shirui-japina: <p>At the last layer, I should <strong>not</strong> use <code>out = F.relu(out)</code> but <code>out = torch.sigmoid(out)</code>, then the model can output [0, 1]. (so can predict class [0, 0.5] and [0.5, 1])</p>
<pre><code class="lang-auto">    def forward(self, input_16, input_32, input_48):
        # CNN part
        ...
        out = torch.cat([out_16, out_32, out_48], dim=1)
        out = F.relu(self.fc_last_1(out))
        out = self.fc_last_2(out) 
        out = torch.sigmoid(out)
        return out

# the model output part
output = model(batch_z_16, batch_z_32, batch_z_48) # model prediction
loss = criterion(output, batch_label)
</code></pre>
<p>Is this the correct answer?<img src="https://discuss.pytorch.org/images/emoji/apple/thinking.png?v=9" title=":thinking:" class="emoji" alt=":thinking:"></p>

dejanbatanjac: <p>Yes, you should use sigmoid function.</p>
<pre><code class="lang-auto">def sigmoid(x): return 1/(1 + (-x).exp())
</code></pre>
<p>It will convert the space of [-inf, inf] into a probability [0,1].<br>
Note this sigmoid works on a tensor. So it will do that for all your activations.</p>
<p>What ever goes to the sigmoid you can call “logit”, even though this is not a mathematical logit function.</p>
<p>After that you will use bce, which works on a batch.</p>
<pre><code class="lang-auto">def binary_cross_entropy(p, y): return -(p.log()*y + (1-y)*(1-p).log()).mean()
</code></pre>
<p>Note that sigmoid used exponential function to grab the probabilities and now we use log of these probabilities.</p>

shirui-japina: <p>I got it <img src="https://discuss.pytorch.org/images/emoji/apple/blush.png?v=9" title=":blush:" class="emoji" alt=":blush:"><br>
Appreciate your detailed explanation.<img src="https://discuss.pytorch.org/images/emoji/apple/grinning.png?v=9" title=":grinning:" class="emoji" alt=":grinning:"></p>