How to calculate accuracy for multi label classification?

Hyo: <p>I have 100 classes and I am using  <code>BCEWithLogitsLoss</code> how do I calculate the accuracy?<br>
Labels : <code> torch.tensor([0,1,0,1,0.......,1])</code></p>

RaLo4: <aside class="quote no-group" data-username="Hyo" data-post="1" data-topic="94906">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/h/c4cdca/48.png" class="avatar"> Hyo:</div>
<blockquote>
<p>I have 100 classes and I am using <code>BCEWithLogitsLoss</code></p>
</blockquote>
</aside>
<p>What do you mean you have 100 classes?</p>
<aside class="quote no-group" data-username="Hyo" data-post="1" data-topic="94906">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/h/c4cdca/48.png" class="avatar"> Hyo:</div>
<blockquote>
<p>Labels : <code> torch.tensor([0,1,0,1,0.......,1])</code></p>
</blockquote>
</aside>
<p>You probably meant, you have 2 classes (or one, depends on how you look at it) 0 and 1.</p>
<p>One way to calculate accuracy would be to round your outputs. This would make 0.5 the classification border.</p>
<pre><code class="lang-auto">correct = 0.
total = 0.
with torch.no_grad():
    #get testing data from data_loader
    for data in test_loader:
        #get images and labels
        images, labels = data
        #move data to gpu
        images = images.to(device)
        #send data through the network and save outputs
        outputs = net(images)
        #map outputs to range of 0-1
        outputs = torch.sigmoid(outputs).cpu()        #&lt;--- since you use BCEWithLogitsLoss
        #round up and down to either 1 or 0
        predicted = np.round(outputs) #keep in mind that np.round() is a round to even function
        total += labels.size(0)
        #calculate how many images were correctly classified
        correct += (predicted == labels).sum().item()
accuracy = 100 * correct / total
print("Accuracy: {}%".format(accuracy))
</code></pre>
<p>Since you are using <code>BCEWithLogitsLoss</code> and not <code>BCELoss</code> I am assuming you do not have a sigmoid layer in your net. This is why I put a sigmoid function in there.</p>

Mughees: <p>For multi-label classification you can sk-learn librarys <a href="https://scikit-learn.org/stable/modules/generated/sklearn.metrics.accuracy_score.html#:~:text=accuracy_score,-sklearn.metrics.&amp;text=Accuracy%20classification%20score.,set%20of%20labels%20in%20y_true." rel="nofollow noopener">accuracy score</a> function.</p>

Hyo: <p>For every observation I have 4-5  categories and total number of categories are 100.<br>
For 1 observation the target labels are <code>[1,3,56,71] </code> I have converted it into one hot vector representation</p>

Hyo: <p>In the accuracy_score I need to round of the values of the output to 1 and 0 how do I take the threshold?</p>

Mughees: <p>you can use <code>np.round()</code> function.</p>
<pre><code class="lang-auto">def accuracy(gt_S,pred_S):       
    gt_S  =np.asarray(gt_S) 
    pred_S=np.round(pred_S)      #will round to the nearest even number
    acc =  accuracy_score(gt_S,pred_S)
    f1m = f1_score(gt_S,pred_S,average = 'macro', zero_division=1)
    f1mi = f1_score(gt_S,pred_S,average = 'micro', zero_division=1)
    print('f1_Macro_Score{}'.format(f1m))
    print('f1_Micro_Score{}'.format(f1mi))
    print('Accuracy{}'.format(acc))
</code></pre>

Hyo: <p><code>np.round()</code> function rounds off to nearest value what if I get different values in the output tensor like <code>tensor([-3.44,-2.678,-0.65,0.96])</code><br>
then after rounding I get <code>array([-3,-2,-0,1])</code> but for <code>accuracy_score</code> the values should be in <code>0</code> and <code>1</code></p>

Mughees: <p>please try to understand the code provided by <a class="mention" href="/u/ralo4">@RaLo4</a>. he explained in detail that you need to pass your logits from sigmoid function. This will convert raw logits to probabilities which you can use for <code>round()</code> function.</p>
<aside class="quote no-group" data-username="RaLo4" data-post="2" data-topic="94906">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/r/65b543/48.png" class="avatar"> RaLo4:</div>
<blockquote>
<p><code>outputs = torch.sigmoid(outputs).cpu()  </code></p>
</blockquote>
</aside>

Hyo: <p>Okay so for calculating the loss I need to pass the logits but to calculate accuracy I need to pass the probabilities</p>

RaLo4: <p>yes. this is because the <code>BCEWithLogitsLoss</code> you are using has a build in sigmoid layer.</p>
<blockquote>
<p>This loss combines a Sigmoid layer and the BCELoss in one single class.</p>
</blockquote>
<p>See <a href="https://pytorch.org/docs/stable/generated/torch.nn.BCEWithLogitsLoss.html#bcewithlogitsloss" rel="noopener nofollow ugc">here</a>.</p>

Hyo: <p>It worked thanks. Training accuracy is increasing as well as the validation is increasing and loss is also at minimum but in the test set the output after applying the sigmoid the values are all zeros none is <code>1</code></p>

RaLo4: <aside class="quote no-group" data-username="Hyo" data-post="11" data-topic="94906">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/h/c4cdca/48.png" class="avatar"> Hyo:</div>
<blockquote>
<p>but in the test set the output after applying the sigmoid the values are all zeros none is <code>1</code></p>
</blockquote>
</aside>
<p>By “zeroes” do you mean 0.something? Cause this would be the expected behavior. Remember, 0.5 is your threshold.</p>
<p>After the sigmoid your values should be in a <strong>range</strong> between 0 and 1 (so not exceeding 1.0).<br>
After np.round they should be either 0 or 1 (everything from 0.0 to 0.5 will become 0 and everything from &gt;0.5 to 1.0 will become 1. So 0.5 is your threshold here).</p>

Hyo: <p>Yeah <code>0.0</code> if I get any value as <code>1</code> then that will be my predicted label right but all the values are <code>0</code>. So I need to change the threshold to some value lower than <code>0.5</code></p>

RaLo4: <p>I have no idea what you are trying to say here.<br>
Are all your results 0 after rounding?</p>
<p>This would mean, that they are between 0.0 and 0.5 after the sigmoid.<br>
Keep in mind, that the output of sigmoid represents a probability.<br>
Which would mean, that your network is never more than 50% sure that a given input belongs to the class.</p>
<p>If that is indeed the case, then lowering your threshold is probably not the right thing to do. Since this would suggests, that there might be a problem in your network. Like a heavily imbalanced dataset for example.</p>
<p>If you still want to lower your threshold, you could do this by comparing the output of the sigmoid to the threshold and setting the value either 0 or 1 accordingly.</p>

KFrank: <p>Hello Hyo and RaLo!</p>
<aside class="quote no-group" data-username="Hyo" data-post="4" data-topic="94906" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/h/c4cdca/48.png" class="avatar"> Hyo:</div>
<blockquote>
<p>For every observation I have 4-5  categories and total number of categories are 100.<br>
For 1 observation the target labels are <code>[1,3,56,71] </code> I have converted it into one hot vector representation</p>
</blockquote>
</aside>
<aside class="quote no-group" data-username="RaLo4" data-post="14" data-topic="94906">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/r/65b543/48.png" class="avatar"> RaLo4:</div>
<blockquote>
<p>Like a heavily imbalanced dataset for example.</p>
</blockquote>
</aside>
<p>Yes, from Hyo’s post, this should be understood as a imbalanced<br>
dataset.  This can be addressed with <code>BCEWithLogitsLoss</code>’s<br>
<code>pos_weight</code> constructor argument.</p>
<p>This is not necessarily imbalanced in the sense of, say, class 7 vs.<br>
class 23 (might be, might not be – from what Hyo has said, we don’t<br>
know yet), but it is imbalanced in the sense of the <em>presence</em>, say, of<br>
class 7 vs. the <em>absence</em> of class 7.</p>
<p>Let me give a few words of explanation:</p>
<p>This <em>multi-label</em>, 100-class classification problem should be<br>
understood as 100 binary classification problems (run through the<br>
same network “in parallel”).  For each of the classes, say class 7, and<br>
each sample, you make the binary prediction as to whether that class<br>
is present in that sample.</p>
<p>Your class-present / class-absent binary-choice imbalance is (averaged<br>
over classes) something like 5% class-present vs. 95% class-absent.<br>
This is imbalanced enough that your network is likely being trained<br>
to predict any one specific class being present with low probability.<br>
It sounds like this is what your are seeing.</p>
<p>(The “standard” approach for using <code>pos_weight</code> would be to calculate<br>
for each class <code>c</code> the fraction of times, <code>f_c</code>, that class <code>c</code> is present<br>
in your samples (regardless of which other classes are present or<br>
absent), and the calculate the weight <code>w_c = (1 - f_c) / f_c</code>.  You<br>
then pass the one-dimensional tensor <code>[w_0, w_1, ..., w_99]</code> into<br>
<code>BCEWithLogitsLoss</code>’s constructor as its <code>pos_weight</code> argument.)</p>
<aside class="quote no-group" data-username="Hyo" data-post="5" data-topic="94906" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/h/c4cdca/48.png" class="avatar"> Hyo:</div>
<blockquote>
<p>In the accuracy_score I need to round of the values of the output to 1 and 0 how do I take the threshold?</p>
</blockquote>
</aside>
<p>A second comment:</p>
<p>The most straightforward way to convert your network output to<br>
<code>0</code> vs. <code>1</code> predictions is to threshold the output <em>logits</em> against<br>
<code>0.0</code>.  You are certainly allowed to convert the logits to probabilities,<br>
and then threshold against <code>0.5</code> (or, equivalently, round), but doing<br>
so is not necessary.  More detail is given in this post:</p>
<aside class="quote quote-modified" data-post="2" data-topic="59293">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/bcewithlogitsloss-and-model-accuracy-calculation/59293/2">BCEWithLogitsLoss and model accuracy calculation</a> 
  </div>
  <blockquote>
    Hi Mehdi! 

You should use 0.0 as your threshold for class “1”. 
Here are the details: 
A logit is a sort of score that runs from -infinity to +infinity. 
When you run it through the sigmoid function, you get a 
probability that runs from 0 to 1. (BCEWithLogitsLoss has, 
in effect, a sigmoid function inside of it.)  We interpret this 
probability as being the probability of class “1”.   So we (usually) 
convert such a probability to a yes-no prediction by saying if 
the probability of being clas…
  </blockquote>
</aside>

<p>Good luck.</p>
<p>K. Frank</p>

Hyo: <p>I have included the <code>pos_weights</code> in loss function</p>
<pre><code class="lang-auto">train_loss:1.2012622356414795  accuracy:0.4861111111111111 
train_loss:1.312620997428894  accuracy:0.4876543209876543 
train_loss:1.292380928993225  accuracy:0.4868827160493827 
train_loss:1.3908039331436157  accuracy:0.4691358024691358 
train_loss:1.4518522024154663  accuracy:0.4729938271604938 
</code></pre>
<p>train _loss is in between 1.5-1.2 and is not decreasing<br>
I have tried different learning rates</p>