How to calculate accuracy multi-label

username: netaglazer
cooked: <p>hi, i have a multi label problem<br>
but i minimize the loss even when the model was right in only one class</p>
<p>i mean, if the model was right in one class:<br>
label = [1,1,0,0,1]<br>
predication = [1,0,0,0,0]<br>
this is a success (loss=0)</p>
<p>im not sure how to calculate the accuracy in that case<br>
when i should count a predication as +1 for acuuracy?</p>
<p>i know the metric of sklearn for multi label, but it dos not feet for my case</p>

username: KFrank
cooked: <p>Hello Neta!</p>
<aside class="quote no-group" data-username="netaglazer" data-post="1" data-topic="73883" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/n/f1d935/48.png" class="avatar"> netaglazer:</div>
<blockquote>
<p>hi, i have a multi label problem</p>
</blockquote>
</aside>
<p>One good way to think about a multi-label classification problem<br>
is to understand it as a set of binary classification problems (in<br>
your case, five binary classification problems) that all use the same<br>
input run through the same network.  That is, for a given input, each<br>
of your five labels is either absent or present, so your network is<br>
making five yes-or-no (binary) predictions.</p>
<aside class="quote no-group">
<blockquote>
<p>but i minimize the loss even when the model was right in only one class</p>
</blockquote>
</aside>
<p>Understanding your problem as a set of binary classifications, we<br>
see that the appropriate loss function is <code>BCEWithLogitsLoss</code> (or,<br>
less optimally, <code>BCELoss</code>).  These loss functions have support for<br>
multi-label problems built in.</p>
<aside class="quote no-group">
<blockquote>
<p>i mean, if the model was right in one class:<br>
label = [1,1,0,0,1]<br>
predication = [1,0,0,0,0]</p>
</blockquote>
</aside>
<p>In this case I would say that three of your five predictions (one<br>
“1” and two "0"s) were  correct, and two (two "0’s that should have<br>
been "1"s) were wrong.</p>
<aside class="quote no-group">
<blockquote>
<p>this is a success (loss=0)</p>
</blockquote>
</aside>
<p>This is a partial success (three out of five) but not perfect.  So your<br>
loss should not be 0 (assuming that 0 is the minimum value of your<br>
loss function).</p>
<p><code>BCEWithLogitsLoss</code> will, indeed, partially penalize this kind of<br>
three-out-of-five prediction.</p>
<aside class="quote no-group">
<blockquote>
<p>im not sure how to calculate the accuracy in that case<br>
when i should count a predication as +1 for acuuracy?</p>
</blockquote>
</aside>
<p>(Just to be clear, loss and accuracy are different things.)</p>
<p>I would count each correct prediction towards your overall<br>
accuracy.  If it turned out that all of your samples produced<br>
three-out-of-five-correct predictions (an unlikely artificial<br>
assumption), then your accuracy would be 60%.</p>
<p>Just a quick note about the specifics of <code>BCEWithLogitsLoss</code>:</p>
<p>Your labels (<em>targets</em>) are appropriately 0-or-1 binary class<br>
labels.  (These can be understood as probabilities – 0% chance<br>
of the label being present vs. 100% chance.  Furthermore,<br>
<code>BCEWithLogitsLoss</code> will actually accept probabilities between<br>
0 and 1 for its targets – but you don’t have to use it this way.)</p>
<p>Your predictions, however, should <em>not</em> be 0-or-1 class labels<br>
(nor probabilities), but rather <em>logits</em> – that is “raw scores” that<br>
range from <code>-inf</code> to <code>+inf</code>.  You would normally get these from<br>
the last linear layer of your network with (in your case) five<br>
outputs.</p>
<p>(You could use class labels / probabilities with <code>BCELoss</code>, but<br>
using logits with <code>BCEWithLogitsLoss</code> is better.)</p>
<p>Best.</p>
<p>K. Frank</p>

username: netaglazer
cooked: <p>thank you for your answer, but i believe you didnt anderstand me</p>
<p>my specific problem is a bit different from a classic multi-label problem<br>
i <strong>want</strong> to minimize my loss when the prediction is correct in only one class (or more)<br>
and this is what i’m doing. a have a costume loss for my problem:</p>
<pre><code class="lang-auto">def new_loss_function(p, y, b):
    eps = 0.000000000000000000000000001
    losses = 0
    k = len(y[0])
    ones = torch.ones(1, 1).expand(b, k).cuda()
    loss1 = -((ones-y)*(((ones-p)+(eps)).log())).sum(dim=1)
    prod = (ones-y)*k - y*((p+ eps).log())
    loss2 = torch.min(prod, dim=1)[0]
    losses = (loss1 + loss2).sum()
    return losses / b
</code></pre>
<p>but my problem is, that in that case i dont know what is the best way to calculate the accuracy</p>