Confused about binary classification with Pytorch

alx: <p>I have 5 classes and would like to use binary classification on one of them.</p>
<p>This is my model:</p>
<pre><code class="lang-auto">    model = models.resnet50(pretrained=pretrain_status)
    num_ftrs = model.fc.in_features
    model.fc = nn.Sequential(
      nn.Dropout(dropout_rate),
      nn.Linear(num_ftrs, 2))
</code></pre>
<p>I then split my dataset into two folders. The one I want to predict (1) and the rest (0,2,3,4).</p>
<p>However, this setup does two predictions and, as I understand it, binary classification is one prediction (True, False).</p>
<p>My question is how do you properly setup a binary classification model in Pytorch?</p>

Nikronic: <p>Hi,</p>
<aside class="quote no-group" data-username="alx" data-post="1" data-topic="83759">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alx/48/17306_2.png" class="avatar"> alx:</div>
<blockquote>
<p><code>nn.Linear(num_ftrs, 2))</code></p>
</blockquote>
</aside>
<p>For binary classification, you need only one logit so, a linear layer that maps its input to a single neuron is adequate. Also, you need to put a threshold on the logit output by linear layer. But an activation layer as the last layer is more rational, something like sigmoid.</p>

alx: <p>Thanks! Would you mind giving me an example with code? I’ve never done it.</p>
<p>What loss do you recommend? I’m currently using CrossEntropy.</p>
<p>I have one concern regarding the use of a single output, I’m worried that if the model only sees the class I’m trying to predict, it will predict class 0,2,3,4 as class 1 later on.</p>

Nikronic: <p>If you want to define your model from scratch, use this <a href="https://pytorch.org/tutorials/beginner/blitz/cifar10_tutorial.html#sphx-glr-beginner-blitz-cifar10-tutorial-py" rel="noopener nofollow ugc">tutorial</a>. But based on your sample code, it seems you are using transfer learning. Here is the <a href="https://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html" rel="noopener nofollow ugc">tutorial for transfer learning</a>.</p>
<p>For case of binary, <a href="https://pytorch.org/docs/master/generated/torch.nn.BCELoss.html" rel="noopener nofollow ugc">BCELoss</a> is a good choice.</p>
<aside class="quote no-group" data-username="alx" data-post="3" data-topic="83759">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alx/48/17306_2.png" class="avatar"> alx:</div>
<blockquote>
<p>if the model only sees the class I’m trying to predict,</p>
</blockquote>
</aside>
<p>Why should model see only class 1 samples? In the first post, you have mentioned that you are using two separate folders, one for class 1 and the latter for class 0, 2, 3, 4. So, the model will see all samples and learns class 1 as 1 and 0, 2, 3, 4 as class not 1 or zero.<br>
You can define binary models to learn each separate class and combine them. This approach called OneVsAll.</p>

KFrank: <p>Hello Alex and Doosti!</p>
<aside class="quote no-group quote-modified" data-username="Nikronic" data-post="2" data-topic="83759" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/nikronic/48/71873_2.png" class="avatar"> Nikronic:</div>
<blockquote>
<p>Hi,</p>
<aside class="quote no-group" data-username="alx" data-post="1" data-topic="83759">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alx/48/17306_2.png" class="avatar"> alx:</div>
<blockquote>
<p><code>nn.Linear(num_ftrs, 2))</code></p>
</blockquote>
</aside>
<p>For binary classification, you need only one logit so, a linear layer that maps its input to a single neuron is adequate. Also, you need to put a threshold on the logit output by linear layer. But an activation layer as the last layer is more rational, something like sigmoid.</p>
</blockquote>
</aside>
<aside class="quote no-group" data-username="Nikronic" data-post="4" data-topic="83759">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/nikronic/48/71873_2.png" class="avatar"> Nikronic:</div>
<blockquote>
<p>For case of binary, <a href="https://pytorch.org/docs/master/generated/torch.nn.BCELoss.html" rel="noopener nofollow ugc">BCELoss</a> is a good choice.</p>
</blockquote>
</aside>
<p>Just to clarify something, for a binary-classification problem, you<br>
are best off using the logits that come out of a final <code>Linear</code> layer,<br>
with no threshold or <code>Sigmoid</code> activation, and feed them into<br>
<code>BCEWithLogitsLoss</code>.  (Using <code>Sigmoid</code> and <code>BCELoss</code> is less<br>
numerically stable.)</p>
<p>And, as Doosti recommended, your last layer should have a single<br>
output, rather than 2.  Thus:</p>
<pre><code class="lang-auto">nn.Linear(num_ftrs, 1))
</code></pre>
<p>Best.</p>
<p>K. Frank</p>

alx: <p>Thanks!</p>
<p>What do you use instead of argmax?</p>

KFrank: <p>Hi Alex!</p>
<aside class="quote no-group" data-username="alx" data-post="7" data-topic="83759" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alx/48/17306_2.png" class="avatar"> alx:</div>
<blockquote>
<p>What do you use instead of argmax?</p>
</blockquote>
</aside>
<p>The short answer is that you threshold your single logit output<br>
against <code>0.0</code>, rather than running a set of <code>nClass</code> outputs through<br>
<code>argmax()</code>.</p>
<p>Let me confirm what I think you are asking:</p>
<p>In addition to calculating your loss function (used for training), you<br>
often also want to calculate the accuracy of your predictions.</p>
<p>For a multi-class classification problem, you typically pass a set<br>
of <code>nClass</code> predicted logits (or predicted probabilities) though<br>
<code>argmax()</code> to get the single predicted integer class label (that<br>
you then compare with your kown class label).  I assume that<br>
this is the “argmax” you are talking about.</p>
<p>For a binary problem, your last <code>Linear</code> layer will output a single<br>
predicted logit for the sample being in class-“1” (as opposed to<br>
being in class-“0”).  (Or, if you pass this logit through a <code>sigmoid()</code>,<br>
you will get the predicted probability of the sample being in class-“1”.)</p>
<p>In this case you threshold the output to get a binary prediction:<br>
<code>logit &gt; 0.0 == True</code> means you predict that the sample is<br>
in class-“1” (and <code>logit &gt; 0.0 == False</code> means class-“0”).  (If<br>
you are working with probabilities, then <code>prob &gt; 0.5 == True</code><br>
means class-“1”.)  You then compare this prediction with the known<br>
class-“0” / class-“1” binary label for the sample in question.</p>
<p>Good luck</p>
<p>K. Frank</p>

alx: <p>Thanks for that detailed answer! Its exactly what I’m trying to do. Here’s my code:</p>
<pre><code class="lang-auto">criterion = nn.BCEWithLogitsLoss()

for image_batch, label_batch in val_dl:
  image_batch, label_batch = image_batch.to(DEVICE), label_batch.to(DEVICE)

  pred_batch = model(image_batch)
  label_batch = label_batch.view(-1, 1)
  batch_loss = criterion(pred_batch.double(), label_batch.double())
  total_val_loss += batch_loss.item()
</code></pre>
<p>Here the pred_batch output:<br>
<code>[ [-0.7402], [-1.0285], [-0.7212], [-0.6358], [-0.4438], [-0.8488]...]</code></p>
<p>From here do I have to use sigmoid to get 0/1?</p>

alx: <p>In fact, I just tried:</p>
<pre><code class="lang-auto">      pred_batch[pred_batch &gt;= 0] = 1
      pred_batch[pred_batch &lt; 0] = 0
</code></pre>
<p>and</p>
<pre><code class="lang-auto">      pred_batch = torch.sigmoid(pred_batch)
      pred_batch[pred_batch &gt;= 0.5] = 1
      pred_batch[pred_batch &lt; 0.5] = 0
</code></pre>
<p>but in both cases the model is not learning</p>