What Loss function to use in Binary CNN Classification problem

username: auyer
cooked: <p>I am running a Transfer Learning scenario with a ResNet model. The original work was a classifier with hundreds of classes, and it used the CrossEntropyLoss function <code>nn.CrossEntropyLoss()</code>.<br>
A <a href="https://discuss.pytorch.org/t/is-it-ok-to-use-nn-crossentropyloss-even-for-binary-classification/14660/2">thread</a>  here suggest BCELoss, but there is BCEWithLogitsLoss that also seems fit.</p>
<p>In a confusion matrix, I want to optimize for the least amount of False Positives, even if it hurts my True Positive score.<br>
What (of all) loss functions should I use ? And why ?</p>

username: KFrank
cooked: <p>Hi Rafael!</p>
<aside class="quote no-group" data-username="auyer" data-post="1" data-topic="47980" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/7ba0ec/48.png" class="avatar"> auyer:</div>
<blockquote>
<p>I am running a Transfer Learning scenario with a ResNet model. The original work was a classifier with hundreds of classes, and it used the CrossEntropyLoss function <code>nn.CrossEntropyLoss()</code>.<br>
A <a href="https://discuss.pytorch.org/t/is-it-ok-to-use-nn-crossentropyloss-even-for-binary-classification/14660/2">thread</a>  here suggest BCELoss, but there is BCEWithLogitsLoss that also seems fit.</p>
</blockquote>
</aside>
<p>For a binary classification problem, <code>BCEWithLogitsLoss</code><br>
should be your go-to loss function.  (You would only want to<br>
use <code>BCELoss</code> if your network naturally emits probabilities,<br>
which it almost certainly doesn’t.)</p>
<aside class="quote no-group">
<blockquote>
<p>In a confusion matrix, I want to optimize for the least amount of False Positives, even if it hurts my True Positive score.<br>
What (of all) loss functions should I use ? And why ?</p>
</blockquote>
</aside>
<p>Consider carefully the trade-off that you are implying here.  As<br>
an extreme, you could simply classify everything as “negative”.<br>
You now achieve the “least amount of False Positives” (none<br>
at all), but, of course your True Positive score is also really<br>
bad, that is, zero.</p>
<p>So I assume your real trade-off is that you are willing to reduce<br>
your True Positives some, but not too much,  if you can get a<br>
substantial reduction in your False Positives.</p>
<p>A sensible approach to achieve this would be to weight your<br>
“negative” samples more heavily than your “positive” samples.<br>
<code>BCEWithLogitsLoss</code> has a <code>weight</code> argument for this purpose.</p>
<p>The relative weight between your “negative” and “positive” samples<br>
will determine how much you train your network to reduce your<br>
False Positives at the cost of reducing your True Positives.</p>
<p>Good luck.</p>
<p>K. Frank</p>

username: auyer
cooked: <p>Thanks ! Very informative.</p>
<p>I’m just trying to figure out what changes when using <code>CrossEntropyLoss().cuda()</code> and <code>BCEWithLogitsLoss().cuda()</code></p>
<p>Just placing the BCE in place of the CE trows me this error:<br>
<code>ValueError: Target size (torch.Size([64])) must be the same as input size (torch.Size([64, 2]))</code></p>
<p>This is a sniplet of the training step:</p>
<pre><code class="lang-auto">    for i, (input, target) in enumerate(val_loader):
        target = target.cuda(async=True)
        input_var = torch.autograd.Variable(input, volatile=True)
        target_var = torch.autograd.Variable(target, volatile=True)

        # compute output
        output = model(input_var)
        loss = criterion(output, target_var) # &lt;- error here!
</code></pre>
<p>Edit: More Information</p>
<p>Here are the two variables passed into criterion() :</p>
<pre><code class="lang-auto">(Pdb) output
tensor([[-0.2657,  0.1728],
        [ 0.3407, -0.6961],
        [ 0.8020, -0.8201],
        [ 0.1457,  0.0311],
        [-0.2517,  0.0223],
        [-0.1266, -0.3978],
        [ 0.4527, -0.6096],
        [ 0.2077, -0.1428],
        [-0.1205, -0.5252],
        [ 0.5462, -0.3988],
        [-0.1215, -0.1321],
        [ 0.3062, -0.5417],
        [ 0.0723, -0.0537],
        [-0.5435, -1.1898],
        [ 0.0718, -0.0986],
        [ 0.0118, -0.0860],
        [-0.0998, -0.8494],
        [-0.2591, -0.4207],
        [ 0.2687, -0.6160],
        [-0.2336, -0.4814],
        [-0.1896, -0.1463],
        [ 0.4623, -0.5179],
        [-0.3181, -0.3042],
        [-0.2550, -0.1824],
        [-0.6250, -0.1293],
        [-0.8920,  0.1077],
        [ 0.0013, -0.1081],
        [-0.2565, -0.0777],
        [-0.2360, -0.3112],
        [ 0.0615, -0.3419],
        [-0.4794, -0.1323],
        [-0.0624,  0.1003],
        [ 0.1803, -0.2833],
        [-0.0859,  0.0516],
        [-0.0256, -0.4226],
        [-0.6047, -0.3403],
        [ 0.2778, -0.6168],
        [ 0.0973, -0.3736],
        [-0.2165, -0.2941],
        [ 0.0252, -0.2497],
        [-0.1285, -0.3079],
        [-0.3292, -0.5657],
        [ 0.1660, -0.5869],
        [-0.1829, -0.3313],
        [-0.5305,  0.0671],
        [ 0.2120, -0.5442],
        [-0.1197, -0.0711],
        [ 0.2132, -0.5229],
        [-0.0977, -0.3243],
        [ 0.1694, -0.2342],
        [ 0.0137, -0.3607],
        [-0.3495, -0.2702],
        [ 0.3058, -0.8327],
        [ 0.4417, -0.7817],
        [-0.7523, -0.5299],
        [ 0.0826, -0.3280],
        [-0.4834, -0.4926],
        [-0.5763,  0.0012],
        [ 0.0992, -0.8658],
        [-0.1066,  0.4763],
        [-0.4472,  0.2544],
        [-0.3449, -0.1687],
        [-0.1852,  0.1073],
        [-0.0782, -0.5123]], device='cuda:0', grad_fn=&lt;AddmmBackward&gt;)
</code></pre>
<pre><code class="lang-auto">(Pdb) target
tensor([0, 1, 1, 1, 1, 0, 0, 0, 0, 1, 0, 0, 1, 1, 1, 0, 1, 1, 0, 1, 0, 0, 1, 1,
        0, 0, 1, 0, 0, 1, 0, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1,
        0, 1, 0, 0, 1, 1, 1, 1, 1, 0, 1, 1, 1, 0, 1, 1], device='cuda:0')
</code></pre>

username: KFrank
cooked: <p>Hello Rafael!</p>
<aside class="quote no-group quote-modified" data-username="auyer" data-post="3" data-topic="47980" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/7ba0ec/48.png" class="avatar"> auyer:</div>
<blockquote>
<p>…<br>
Just placing the BCE in place of the CE trows me this error:<br>
<code>ValueError: Target size (torch.Size([64])) must be the same as input size (torch.Size([64, 2]))</code><br>
…</p>
</blockquote>
</aside>
<p>Without seeing the rest of your code – your model, in particular –<br>
I’m guessing somewhat, but I believe that your problem could be<br>
the following:</p>
<p>If you have a multi-class problem (where “multi” implies more<br>
than two, and two classes is what we call a “binary” problem)<br>
with nClass classes, then the output of your model should be<br>
nClass logits.  (This is what <code>CrossEntropyLoss</code> expects.)</p>
<p>In contrast, for a binary problem, the output of your model should<br>
be a single logit (not two), conventionally taken to be the logit for<br>
your “positive” class.  This is what <code>BCEWithLogitsLoss</code> expects.</p>
<p>If you build your binary-problem model as a two-class multi-class<br>
model, then you will (redundantly) have two logits as your output<br>
(one for your “negative” class, as well as for your “positive” class).<br>
This won’t match what <code>BCEWithLogitsLoss</code> expects, so I would<br>
think you would get the error you report.</p>
<p>(You can build your binary model as a two-class multi-class<br>
model, but then you should feed the model’s output into<br>
<code>CrossEntropyLoss</code>.  If you match everything up right, you<br>
should get the same results as you would with a conventional<br>
binary model feeding <code>BCEWithLogitsLoss</code>, but you probably<br>
lose a more or less insignificant bit of efficiency.)</p>
<p>Best.</p>
<p>K. Frank</p>