Using sigmoid output with cross entropy loss

thao: <p>Hi.<br>
I’m trying to modify Yolo v1 to work with my task which each object has only 1 class. (e.g: an obj cannot be both cat and dog)<br>
Due to the architecture (other outputs like localization prediction must be used regression) so <strong>sigmoid was applied to the last output of the model</strong> (f.sigmoid(nearly_last_output)). And for classification, yolo 1 also use MSE as loss. But as far as I know that MSE sometimes not going well compared to cross entropy for one-hot like what I want.<br>
And specific: GT like this:                      0 0 0 0 1 (let say we have only 5 classes in total, each only has 1 class so only one number 1 in them)<br>
and output model at classification part: 0.1 0.1 0.9 0.2 0.1</p>
<p>I found some suggestion use nn.BCE / nn.BCEWithLogitsLoss but I think I should ask here for more correct since I’m not good at math and maybe I’m wrong somewhere so just ask to learn more and for sure what should I use.</p>

KFrank: <p>Hello Thao!</p>
<aside class="quote no-group" data-username="thao" data-post="1" data-topic="96439" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/thao/48/26535_2.png" class="avatar"> thao:</div>
<blockquote>
<p>Due to the architecture (other outputs like localization prediction must be used regression) so <strong>sigmoid was applied to the last output of the model</strong> (f.sigmoid(nearly_last_output)). And for classification, yolo 1 also use MSE as loss. But as far as I know that MSE sometimes not going well compared to cross entropy for one-hot like what I want.<br>
And specific: GT like this:                      0 0 0 0 1 (let say we have only 5 classes in total, each only has 1 class so only one number 1 in them)<br>
and output model at classification part: 0.1 0.1 0.9 0.2 0.1</p>
</blockquote>
</aside>
<p>I don’t fully understand your use case, but I would proceed as follows:</p>
<p>Use <code>CrossEntropyLoss</code> for the classification part of your model.  To<br>
do this, your model should output raw-score <em>logits,</em>, not probabilities,<br>
so the last layer of your model should most likely be a <code>Linear</code> layer<br>
with <code>nClass</code> outputs.  Don’t feed this through a <code>sigmoid()</code> (nor a<br>
<code>softmax()</code>).  Your <code>target</code> (ground truth) should not be in one-hot<br>
format, but rather integer class labels that run from <code>0</code> to <code>nClass - 1</code>.</p>
<p>(You could try to undo the <code>sigmoid()</code> that is being applied to your<br>
“nearly-last” layer, but this risks becoming numerically unstable.)</p>
<p>But if you need the <code>sigmoid()</code> for other parts of your loss function (You<br>
talk about localization, regression, and MSE.), you should incorporate<br>
the <code>sigmoid()</code> into that part of your loss function, rather than into your<br>
model.  (It is numerically better to apply the <code>sigmoid()</code> to your logits<br>
in the MSE part of your loss, that to try to undo a <code>sigmoid()</code> in the<br>
<code>CrossEntropyLoss</code> part of your loss.)</p>
<p>(<code>BCELoss</code> is not appropriate for the classification part of your model<br>
as you are working on a <em>single-label,</em> multi-class  problem, not on<br>
a so-called <em>multi-label,</em> multi-class problem.)</p>
<p>Good luck.</p>
<p>K. Frank</p>