Multi-label: Don't calculate BCEWithLogitsLoss for row in batch when confident enough

username: NumesSanguis
cooked: <h1>Current situation</h1>
<p>I have a multi-label classification problem for which being overconfident is a problem in the end application. The data is labeled with 1 or more from <code>[A, B, C, D, E]</code> , but in reality e.g. label B should not be treated as 1 or 0, but e.g. 0.7 (unfortunately unattainable).</p>
<h2>Normal training</h2>
<p>If I would use BCEWithLogitsLoss as normal on data like this:</p>
<pre><code class="lang-python">import torch

loss_func = torch.nn.BCEWithLogitsLoss()

pred = torch.tensor([[0.1, 0.1, .7, 0.2, .7],
                     [.8, .5, 0.1, 0.1, 0.2]])
target = torch.tensor([[0, 0, 1, 0, 1],
                       [1, 1, 0, 0, 0]])

loss_func(pred, target.type(torch.FloatTensor))
# tensor(0.6225)
</code></pre>
<p>I can successfully train a model. The problem is that the confidence values are like <code>[0.0, 0.0, 0.99, 0.0, 0.98]</code>.</p>
<h1>Goal</h1>
<p>I want to say to the the loss function: “If confidence values of correct labels are above &gt;=0.6, and wrong labels below &lt;0.6, don’t calculate a loss”.</p>
<h2>Attempt</h2>
<p>Set the rows which are correct to binary format, so the loss is 0 for this row.</p>
<pre><code class="lang-python">pred_binary = torch.where(pred &gt;= 0.6, torch.tensor(1), torch.tensor(0))
compare = torch.where(pred_binary == target, torch.tensor(1), torch.tensor(0))
# tensor([[1, 1, 1, 1, 1],
#         [1, 0, 1, 1, 1]])
compare_row = compare.type(torch.FloatTensor).mean(axis=1)
# tensor([1.0000, 0.8000])
select_row = torch.where(compare_row &gt;= 1, torch.tensor([1]), torch.tensor([0]))
# tensor([1, 0])
select_row = select_row.type(torch.bool)
# tensor([ True, False])

pred[select_row, :] = pred_binary[select_row].type(torch.FloatTensor)
# tensor([[0.0000, 0.0000, 1.0000, 0.0000, 1.0000],
#         [0.8000, 0.5000, 0.1000, 0.1000, 0.2000]])

loss_func(pred, target.type(torch.FloatTensor))
# tensor(0.5838)
</code></pre>
<p>The loss is indeed lower (<code>0.6225</code> versus <code>0.5838</code>).</p>
<h1>Questions</h1>
<ol>
<li>Is there a smarter way to do what I want?</li>
<li>Or more efficient code?</li>
<li>Any feedback on what I should watch out for doing this?</li>
</ol>

username: KFrank
cooked: <p>Hi Numes!</p>
<aside class="quote no-group" data-username="NumesSanguis" data-post="1" data-topic="96470" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/numessanguis/48/29777_2.png" class="avatar"> NumesSanguis:</div>
<blockquote>
<p>The data is labeled with 1 or more from <code>[A, B, C, D, E]</code> , but in reality e.g. label B should not be treated as 1 or 0, but e.g. 0.7 (unfortunately unattainable).</p>
</blockquote>
</aside>
<p>The short answer is to use a “less-certain” <code>target</code>:</p>
<pre data-code-wrap="python"><code class="lang-python">target = torch.tensor([[0.3, 0.3, 0.7, 0.3, 0.7],
                       [0.3, 0.7, 0.3, 0.3, 0.3]])
</code></pre>
<p>As I understand it, the “ground-truth” labels you are given are all<br>
exactly <code>0.0</code> or <code>1.0</code>.  But (because you understand the problem)<br>
you realize that the ground-truth training data really shouldn’t be<br>
that certain, and therefore you don’t want to train your model to<br>
be that certain.</p>
<p>As a general rule, you should train your model to do what you want<br>
it to do, rather than try to fix it up somehow after the fact.</p>
<p>So tell your model that you want less-certain, lower-confidence<br>
predictions by training it with less-certain, lower-confidence <code>target</code>s,<br>
as above.</p>
<p>Note, <code>BCEWithLogitsLoss</code> does take probabilities between <code>0.0</code> and<br>
<code>1.0</code> for its <code>target</code> values – they are not restricted to be exactly <code>0.0</code><br>
or <code>1.0</code>.</p>
<p>Best.</p>
<p>K. Frank</p>

username: NumesSanguis
cooked: <p>Hi K. Frank,</p>
<p>Sorry for my late reply!</p>
<p>I actually expected the model confidence to hover around the uncertainty value, but with 0.1 and 0.9 the confidence values were covering the full range between [0-1].</p>
<p>Thank you!</p>