Problem with fine-tuning for binary classification

TTeuZ: <p>Hello guys!</p>
<p>I’m working on fine-tuning a MobileNetV3 for binary classification. My first attempt was to change the classification layer to output one feature and use BCEWithLogisticLoss as my loss function and torch.sigmoid for prediction.</p>
<p>Now, I want to check if the result would be better if I change to output 2 features and use the BCELoss as my loss function and here is where I’m stuck.</p>
<p>Firstly, I only changed the code to set 2 output features, and with that I got this error:</p>
<pre><code class="lang-auto">Using a target size (torch.Size([64])) that is different to the input size (torch.Size([64, 2])) is deprecated. Please ensure they have the same size.
</code></pre>
<p>After researching a little, I discovered that the right way is to use one_hot labels to perform the loss function, and with that I changed the implementation to one_hot labels</p>
<p>the implementation is this:</p>
<pre><code class="lang-auto">labels_one_hot = torch.zeros(labels.size(0), 2).to(device)
labels_one_hot.scatter_(1, labels.unsqueeze(1), 1)
</code></pre>
<p>With this change, I got this error:</p>
<pre><code class="lang-auto">t./aten/src/ATen/native/cuda/Loss.cu:94: operator(): block: [0,0,0], thread: [2,0,0] Assertion `input_val &gt;= zero &amp;&amp; input_val &lt;= one` failed.
</code></pre>
<p>Searching again, I found that this error is related to out of range exceptions, and that probably the range of my labels tensor is not between 0 and num_classes-1.</p>
<p>So, to check that I printed both my labels and one_hot_labels tensors and got this:</p>
<pre><code class="lang-auto">labels: tensor([0, 0, 0, 0, 0, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 1, 0, 1, 1, 1, 0, 0, 1, 1,
        0, 1, 0, 1, 1, 0, 1, 1, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 0, 1, 0, 1, 1, 0,
        1, 0, 0, 0, 0, 0, 1, 1, 0, 1, 0, 0, 0, 1, 0, 0], device='cuda:0')

</code></pre>
<pre><code class="lang-auto">one hot: tensor([[1., 0.], [1., 0.], [1., 0.], [1., 0.], [1., 0.], [1., 0.], [0., 1.], [1., 0.], [0., 1.], [1., 0.], [0., 1.], [1., 0.], [0., 1.], [1., 0.], [0., 1.], [0., 1.], [1., 0.], [0., 1.], [0., 1.], [0., 1.], [1., 0.], [1., 0.], [0., 1.], [0., 1... [1., 0.], [0., 1.], [1., 0.], [1., 0.], [1., 0.], [0., 1.], [1., 0.], [1., 0.], [0., 1.], [0., 1.], [1., 0.], [0., 1.], [1., 0.], [1., 0.], [1., 0.], [0., 1.], [1., 0.], [1., 0.]], device='cuda:0')

</code></pre>
<p>So, it seems that the range is right (between 0 and 1) the only difference is that the one_hot is float and the labels are int. With that in mind, I decide to force the one_hot_labels to int type and now the “out of range” error disappeared but I got this error:</p>
<pre><code class="lang-auto">Found dtype Int but expected Float
</code></pre>
<p>Long story short, if I use float I get an “out of range error”, but if I change to int this error disappears and I get one “I need float” error.</p>
<p>Do you guys know what I could be doing wrong?</p>

KFrank: <p>Hi Paulo!</p>
<aside class="quote no-group" data-username="TTeuZ" data-post="1" data-topic="198373" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/tteuz/48/67911_2.png" class="avatar"> TTeuZ:</div>
<blockquote>
<p>My first attempt was to change the classification layer to output one feature and use BCEWithLogisticLoss as my loss function and torch.sigmoid for prediction.</p>
</blockquote>
</aside>
<p>As an aside, just to be clear, you do <em>not</em> want to pass the output of your<br>
classification layer through <code>sigmoid()</code> and then to <code>BCEWithLogitsLoss</code>.</p>
<aside class="quote no-group">
<blockquote>
<p>Now, I want to check if the result would be better if I change to output 2 features and use the BCELoss as my loss function and here is where I’m stuck.</p>
</blockquote>
</aside>
<p>You do not want use <code>BCELoss</code> with a two-output classification layer.  It is<br>
true that by using a two-output layer you are logically performing a binary<br>
classification, but it is set up as a multi-class classification that happens to<br>
have two (therefore binary) classes.</p>
<p>For a multi-class problem you would want to use <code>CrossEntropyLoss</code>.</p>
<p>It is reasonable perform binary classification as a two-class multi-class<br>
classification using a two-output classification layer and <code>CrossEntropyLoss</code>.<br>
The two approaches are very similar, and in my experience, it doesn’t really<br>
matter which you use.</p>
<p>(My preference is to use a single-output layer with <code>BCEWithLogitsLoss</code><br>
as theoretically this should be ever so slightly more efficient.)</p>
<aside class="quote no-group">
<blockquote>
<p>I got this error:</p>
</blockquote>
</aside>
<p>Before we delve into your specific errors, first sort out whether you should<br>
be using <code>CrossEntropyLoss</code> for your two-output approach.  (Please post<br>
follow-up questions if you still get errors.)</p>
<p>Best.</p>
<p>K. Frank</p>

TTeuZ: <p>Hey K. Frank, thanks for your answer!</p>
<blockquote>
<p>As an aside, just to be clear, you do <em>not</em> want to pass the output of your<br>
classification layer through <code>sigmoid()</code> and then to <code>BCEWithLogitsLoss</code> .</p>
</blockquote>
<p>Just to clarify, I’m not passing the results to a sigmoid and then passing them to BCEWithLogisticLoss. Maybe my English wasn’t the best when I wrote this topic haha.</p>
<p>I already fine-tuned the MobileNetV3 with a one-output classification layer and got my results, and now I decided to check if the result is different if I use a two-output classification layer.</p>
<p>I decided to use BCELoss (Binary Cross Entropy Loss) as my loss function in the training process because it seemed to make sense, but if it’s not right I can change it.</p>
<p>The issue that I’m facing is that, if I use the BCELoss, I need to convert my one-size labels tensor to a two-size labels tensor, and the indicated way that I found to do that is use one-hot-encoded labels.</p>
<pre><code class="lang-auto">Using a target size (torch.Size([64])) that is different to the input size (torch.Size([64, 2])) is deprecated. Please ensure they have the same size.
</code></pre>
<p>In my way to create my one-hot-encoded labels tensor I found this implementation</p>
<pre><code class="lang-auto">labels_one_hot = torch.zeros(labels.size(0), 2).to(device)
labels_one_hot.scatter_(1, labels.unsqueeze(1), 1) 
</code></pre>
<p>But with this implementation I got this error</p>
<pre><code class="lang-auto">../aten/src/ATen/native/cuda/Loss.cu:94: operator(): block: [0,0,0], thread: [127,0,0] Assertion `input_val &gt;= zero &amp;&amp; input_val &lt;= one` failed.
</code></pre>
<p>It tells me that my labels aren’t between 0, 1(num_classes-1), but when I print the tensor I can check that my labels are between 0, 1 and it’s here that I’m stuck.</p>
<pre><code class="lang-auto">one hot: tensor([[1., 0.], [1., 0.], [1., 0.], [1., 0.], [1., 0.], [1., 0.], [0., 1.], [1., 0.], [0., 1.], [1., 0.],
[0., 1.], [1., 0.], [0., 1.], [1., 0.], [0., 1.], [0., 1.], [1., 0.], [0., 1.], [0., 1.], [0., 1.], [1., 0.], [1., 0.],
[0., 1.], [0., 1... [1., 0.], [0., 1.], [1., 0.], [1., 0.], [1., 0.], [0., 1.], [1., 0.], [1., 0.], [0., 1.], [0., 1.],
[1., 0.], [0., 1.], [1., 0.], [1., 0.], [1., 0.], [0., 1.], [1., 0.], [1., 0.]], device='cuda:0')
</code></pre>
<p>I checked and if I use the CrossEntropyLoss it works perfectly, but I still want to try the BCELoss if it makes sense.</p>

ptrblck: <aside class="quote no-group" data-username="TTeuZ" data-post="3" data-topic="198373">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/tteuz/48/67911_2.png" class="avatar"> TTeuZ:</div>
<blockquote>
<p>but I still want to try the BCELoss if it makes sense.</p>
</blockquote>
</aside>
<p>No, it doesn’t since you are creating a multi-label classification as was already explained. <code>nn.BCELoss</code> and <code>nn.BCEWithLogitsLoss</code> are used for the same use cases (binary or multi-label) and the main difference between them is that the former requires a probability input tensor (usually acquired via <code>sigmoid</code>) and is numerically less stable.<br>
<code>nn.CrossEntropyLoss</code> with two output logits is the correct alternative.</p>

TTeuZ: <p>Hi guys, sorry for the late response</p>
<p>Thank you so much for the help, I got why it isn’t a good idea to use BCELoss. I did the modifications and now I’m using <code>CrossEntropyLoss</code> as mentioned by  KFrank and you.</p>
<p>I will mark this post as solved =D.</p>