Proper way of doing binary classification with one probability output ? (what loss function/activation function to use and how to compute accuracy ? )

Richard_S: <p>Hello everyone, sorry for rookie question I’m starting to learn pytoch</p>
<p>this is my simple 1 layer linear classifier :</p>
<pre><code class="lang-auto">class Classifier(nn.Module):
    def __init__(self, in_dim ):
        super(Classifier, self).__init__()
        self.classify = nn.Linear(in_dim  , 1 )

    def forward(self, features ):

        final = torch.sigmoid ( self.classify(features) )
        return  final
</code></pre>
<p>I want the output to be probability, so ~1 means class 1 and ~0 means class 0</p>
<p>but I don’t know which loss function to use and how to calculate the accuracy in each epoch when I’m using batching ?</p>
<p>This is my current training loop but the loss is not correct, i feel like i need to change the code because this code is written for multi class classification, not a single output classification :</p>
<pre><code class="lang-auto">  

criterion = nn.CrossEntropyLoss()
optimizer = optim.Adam(model.parameters(), lr=0.001)
scheduler = optim.lr_scheduler.StepLR(optimizer, step_size=50, gamma=0.5)

net.train()


running_loss = 0
total_iters = len(trainloader)

for pos, (train_samples, labels) in zip(bar, trainloader):

    outputs = net(train_samples)

    loss = criterion(outputs, labels.float() )
    running_loss += loss.item()

    optimizer.zero_grad()
    loss.backward()
    optimizer.step()


running_loss = running_loss / total_iters

return running_loss
</code></pre>

pchandrasekaran: <p>For the loss function, switch out CrossEntropyLoss for BCELoss. I usually like to write a separate function that computes the accuracy (over the whole set) and use that within my training loop. The function takes in as arguments the model and the train dataloader.</p>

KFrank: <p>Hi Richard!</p>
<aside class="quote no-group quote-modified" data-username="Richard_S" data-post="1" data-topic="101895" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/r/3be4f8/48.png" class="avatar"> Richard_S:</div>
<blockquote>
<pre><code class="lang-auto">class Classifier(nn.Module):
    def __init__(self, in_dim ):
        super(Classifier, self).__init__()
        self.classify = nn.Linear(in_dim  , 1 )

    def forward(self, features ):

        final = torch.sigmoid ( self.classify(features) )
        return  final
</code></pre>
<p>I want the output to be probability, so ~1 means class 1 and ~0 means class 0</p>
<pre><code class="lang-auto">criterion = nn.CrossEntropyLoss()
optimizer = optim.Adam(model.parameters(), lr=0.001)
scheduler = optim.lr_scheduler.StepLR(optimizer, step_size=50, gamma=0.5)
</code></pre>
</blockquote>
</aside>
<p>As Prashanth notes, you could use <code>BCELoss</code> in place of<br>
<code>CrossEntropyLoss</code>.</p>
<p>However, you’ll be better off removing the <code>torch.sigmoid()</code><br>
and using <code>BCEWithLogitsLoss</code>.  Doing so will be mathematically<br>
the same, but numerically more stable.</p>
<p>Thus:</p>
<pre data-code-wrap="python"><code class="lang-python">class Classifier(nn.Module):
    def __init__(self, in_dim ):
        super(Classifier, self).__init__()
        self.classify = nn.Linear(in_dim  , 1 )

    def forward(self, features ):

        final = self.classify(features) 
        return  final
</code></pre>
<p>and:</p>
<pre data-code-wrap="python"><code class="lang-python">criterion = nn.BCEWithLogitsLoss()
</code></pre>
<p>Your <code>Classifier</code> will now output raw-score <em>logits</em> that range from<br>
<code>-inf</code> to <code>inf</code> instead of probabilities.  Should you need probabilities<br>
for subsequent processing, you can always pass the logits through<br>
<code>sigmoid()</code>.  Note, you don’t need probabilities to make hard 0-1<br>
predictions: <code>prediction = 1 if logit &gt; 0.0</code> is the same as<br>
<code>prediction = 1 if probability &gt; 0.5</code>.</p>
<p>Two side comments:</p>
<p>As written, you never call <code>scheduler.step()</code> so <code>scheduler</code> doesn’t<br>
do anything.</p>
<p>For getting started with the code, one <code>Linear</code> layer is fine, but it<br>
won’t be much of a classifier for anything but special toy problems.<br>
Leaving aside the <code>sigmoid()</code>, your single output is just a linear<br>
function of your <code>in_dim</code> inputs.  Things already get much more<br>
interesting (and useful) if you add a single “hidden” layer:</p>
<pre data-code-wrap="python"><code class="lang-python">class Classifier (nn.Module):
    def __init__ (self, in_dim, hidden_dim):
        super (Classifier, self).__init__()
        self.fc1 = nn.Linear (in_dim, hidden_dim)
        self.activation = nn.ReLU()   # for example
        self.fc2 = nn.Linear (hidden_dim, 1)
    def forward (self, features):
        x = self.fc1 (features)
        x = self.activation (x)
        x = self.fc2 (x)
        return x
</code></pre>
<p>For more interesting classification tasks, the non-linear activation<br>
(for example, <code>ReLU</code>) between <code>fc1</code> and <code>fc2</code> is the “secret sauce.”</p>
<p>Best.</p>
<p>K. Frank</p>