Calculating Precision, Recall and F1 score in case of multi label classification

somnath: <p>I have the Tensor containing the ground truth labels that are one hot encoded. My predicted tensor has the probabilities for each class. In this case, how can I calculate the precision, recall and F1 score in case of multi label classification in PyTorch?</p>

ptrblck: <p>Precision, recall and F1 score are defined for a binary classification task.<br>
Usually you would have to treat your data as a collection of multiple binary problems to calculate these metrics.</p>
<p>The multi label metric will be calculated using an average strategy, e.g. macro/micro averaging.<br>
You could use the <a href="http://scikit-learn.org/stable/modules/model_evaluation.html#from-binary-to-multiclass-and-multilabel">scikit-learn metrics</a> to calculate these metrics.</p>

somnath: <p>I am using scikit learn metrics for this and used this code:</p>
<pre><code class="lang-auto">print('F1: {}'.format(f1_score(outGT, outPRED, average="samples")))
print('Precision: {}'.format(precision_score(outGT, outPRED, average="samples")))
print('Recall: {}'.format(recall_score(outGT, outPRED, average="samples")))
</code></pre>
<p>This is throwing this error:</p>
<pre><code class="lang-auto">ValueError: Classification metrics can't handle a mix of multilabel-indicator and continuous-multioutput targets
</code></pre>
<p>The output of</p>
<pre><code class="lang-auto">print('Ground Truth: {}'.format(outGT))
print('Predicted Truth: {}'.format(outPRED))
</code></pre>
<p>is as below:</p>
<pre><code class="lang-auto">Ground Truth: 
    0     0     0  ...      0     0     0
    0     0     0  ...      0     0     0
    0     0     0  ...      0     0     0
       ...          ⋱          ...       
    1     0     0  ...      0     0     0
    1     0     0  ...      0     0     0
    0     0     0  ...      0     0     0
[torch.cuda.FloatTensor of size 22433x14 (GPU 0)]

Predicted Truth: 
 0.0901  0.0916  0.0389  ...   0.0021  0.0078  0.0016
 0.0424  0.0084  0.0111  ...   0.0053  0.0079  0.0025
 0.0611  0.0205  0.0206  ...   0.0024  0.0074  0.0018
          ...             ⋱             ...          
 0.3588  0.0223  0.1421  ...   0.0036  0.0094  0.0035
 0.1782  0.0226  0.2275  ...   0.0033  0.0129  0.0016
 0.2574  0.0176  0.2255  ...   0.0034  0.0118  0.0023
[torch.cuda.FloatTensor of size 22433x14 (GPU 0)]

</code></pre>

ptrblck: <p>Try to use a threshold on your predictions, so that they indicate a predicted label.<br>
This should work:</p>
<pre><code class="lang-python">f1_score(outGT, outPRED &gt; 0.5, average="samples")
</code></pre>
<p>EDIT: Also, you might want to push the tensors to CPU first.</p>

somnath: <p>Thanks a lot. The solution makes a lot of sense. I have to perform masking before trying to calculate the score.</p>

Surekha_Gaikwad: <p><a class="mention" href="/u/ptrblck">@ptrblck</a><br>
I am also working on multi label classification task where I have ground truth labels as one hot encoded. I got predicted values for the sample and also getting loss properly. But when I am trying to compute accuracy as you suggested in the post I am still getting error as “ValueError: Classification metrics can’t handle a mix of unknown and multilabel-indicator targets”.</p>
<pre><code class="lang-auto">print('F1: {}'.format(f1_score(labels.data.to('cpu'), outputs.data.to('cpu') &gt; 0.5, average="samples")))
</code></pre>

shinyeyes: <p>I wrote the function in PyTorch in an attempt to train with F1 loss. <a href="https://gist.github.com/SuperShinyEyes/dcc68a08ff8b615442e3bc6a9b55a354" rel="nofollow noopener">https://gist.github.com/SuperShinyEyes/dcc68a08ff8b615442e3bc6a9b55a354</a></p>

debalb: <p>Usually this has worked for me:</p>
<blockquote>
<p>def precision(outputs, labels):<br>
op = outputs.cpu()<br>
la = labels.cpu()<br>
_, preds = torch.max(op, dim=1)<br>
return torch.tensor(precision_score(la,preds, average=‘weighted’))</p>
</blockquote>

gmanoj: <p>Why we have to push the tensors to CPU ? Can’t I do calculation on GPU ?</p>

ptrblck: <p><code>scikit-learn</code> metrics use numpy under the hood which does not support tensors stored on the GPU. You should also note the post was created 4 years ago and nowadays you might want to check e.g. <code>torchmetrics</code>.</p>