F1 Score for Multi-label Classification

username: se7en.dreams
cooked: <p>Hi,</p>
<p>I am trying to calculate F1 score (and accuracy) for my multi-label classification problem. Could you please provide feedback on my method, if I’m calculating it correctly. Note that I’m calculating <code>IOU</code>  (intersection over union) when model predicts an object as <code>1</code>, and mark it as <code>TP</code> only if <code>IOU</code> is greater than or equal to <code>0.5</code>.</p>
<p>GT labels: <code>14 x 10 x 128</code><br>
Output: <code>14 x 10 x 128</code></p>
<p>where <code>14</code> is the <code>batch_size</code>, <code>10</code> is the <code>sequence_length</code>, and <code>128</code> is the object vector (i.e., <code>1</code> if the object at an index belongs to the sequence and <code>0</code> otherwise).</p>
<pre><code class="lang-python">for epoch in range(10):
    TP = FP = TN = FN = EPOCH_PRECISION = EPOCH_RECALL = EPOCH_F1 = 0.
    for inputs, gt_labels in tr_dl:
        out = model(inputs) # out shape: (14, 10, 128)

        # loop through batch samples
        for batch_idx, batch_output in enumerate(out):
            
            # loop through sequence elements
            # batch_output shape: (10, 128)
            for seq_idx, seq_output in enumerate(batch_output):
                # pred_labels shape: (128,)
                pred_labels = (torch.sigmoid(seq_output) &gt;= 0.5).float().type(torch.int64) # consider all predictions above 0.5 as 1, rest 0
                
                # loop through predicted objects for a sequence element    
                for object_idx, pred_label in enumerate(pred_labels):
                    if pred_label == 1 and gt_labels[batch_idx, seq_idx, object_idx] == 1:
                        # calculate IOU (overlap of prediction and gt bounding box)
                        iou = 0.78 # assume we get this iou value for objects at idx
                        if iou &gt;= 0.5:
                            TP += 1
                        else:
                            FP += 1
                    elif pred_label == 1 and gt_labels[batch_idx, seq_idx, object_idx] == 0:
                        FP += 1
                    elif pred_label == 0 and gt_labels[batch_idx, seq_idx, object_idx] == 1:
                        FN += 1
                    else:
                        TN += 1
          
    EPOCH_ACC = (TP + TN) / (TP + TN + FP + FN)
     
    if TP + FP &gt; 0:
        EPOCH_PRECISION = TP / (TP + FP)
    if TP + FN &gt; 0:
        EPOCH_RECALL = TP / (TP + FN)

    EPOCH_F1 = (2 * EPOCH_PRECISION * EPOCH_RECALL) / (EPOCH_PRECISION + EPOCH_RECALL)

</code></pre>

username: ParGG
cooked: <p>Hi,</p>
<p>I looked at your code snippet and I believe that there might be some confusion. I have two comments for you:</p>
<ol>
<li>The implementation is really inefficient. You can compute the statistics without looping over the dimensions. All the operations you need are sum, element-wise multiplication and negation. I leave it here as a tip, if you feel like you need help, I can write some code for you.</li>
<li>The IOU is supposed to be computed over higher dimensional data, this means starting from 1D to ND tensors. Since you are computing it point wise, it loses the meaning of IOU, and it actually falls back to the simpler statistics that you have correctly indicated in your control flow.</li>
</ol>
<p>In short, compute all the statistics using tensor operations and you are good <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

username: se7en.dreams
cooked: <p>Thank you for your response. So basically, I haven’t provided any details about how I’m computing IOU but I’m doing it over the bounding boxes (8 corners) for GT and predicted objects.</p>
<p>And the reason for inefficiency (i.e., looping over dimensions) is because I’m dealing with sequences within the batch and not all sequences are relevant. For example, the first element in the batch (<code>14 x 10 x 128</code>) might only have <code>3 x 128</code>, while the the rest <code>7 x 128</code> is just padding. I do have a mask tensor that tells me about the “relevant” sequence elements.</p>
<p>Could you please provide an example of doing this efficiently and how I can compute IOU and then go about computing accuracy and F1? Essentially, I’m looking to calculate <code>F1@0.5IOU</code> (similar to average precision at IOU).</p>

username: se7en.dreams
cooked: <p>I was able to make my implementation much more efficient. Am I doing it correctly now?</p>
<pre><code class="lang-python">def calculate_performance_metrics(total_padded_elements, gt_labels, predicted_labels):
        # check if TP pred objects overlap with TP gt objects
        TP_INDICES = (torch.logical_and(predicted_labels == 1, gt_labels == 1)).nonzero() # we only want the batch and object indices, i.e. the 0 and 2 indices
        TP = calculate_tp_with_iou() # details of this don't matter for now
        FP = torch.sum(torch.logical_and(predicted_labels == 1, gt_labels == 0)).item()
        TN = torch.sum(torch.logical_and(predicted_labels == 0, gt_labels == 0)).item()
        FN = torch.sum(torch.logical_and(predicted_labels == 0, gt_labels == 1)).item()
        return float(TP), float(FP), float(TN - total_padded_elements), float(FN)


for epoch in range(10):
    TP = FP = TN = FN = EPOCH_PRECISION = EPOCH_RECALL = EPOCH_F1 = 0.
    for inputs, gt_labels, masks in tr_dl:
        outputs = model(inputs) # out shape: (14, 10, 128)

        # mask shape: (14, 10). So need to expand it to the shape of output
        masks = masks[:, :, None].expand_as(outputs)
        
        pred_labels = (torch.sigmoid(outputs) &gt;= 0.5).float().type(torch.int64) # consider all predictions above 0.5 as 1, rest 0
        pred_labels = pred_labels * masks
        gt_labels = (gt_labels * masks).type(torch.int64)
        total_padded_elements = masks.numel() - masks.sum() # need this to get accurate true negatives

        batch_tp, batch_fp, batch_tn, batch_fn = calculate_performance_metrics(gt_labels, pred_labels, total_padded_elements)
        EPOCH_TP += batch_tp
        EPOCH_FP += batch_fp
        EPOCH_TN += batch_tn
        EPOCH_FN += batch_fn

        EPOCH_ACCURACY = (EPOCH_TP + EPOCH_TN) / (EPOCH_TP + EPOCH_TN + EPOCH_FP + EPOCH_FN)
        
        if EPOCH_TP + EPOCH_FP &gt; 0:
            EPOCH_PRECISION = EPOCH_TP / (EPOCH_TP + EPOCH_FP)

        if EPOCH_TP + EPOCH_FN &gt; 0:
            EPOCH_RECALL = EPOCH_TP / (EPOCH_TP + EPOCH_FN)

        EPOCH_F1 = (2 * EPOCH_PRECISION * EPOCH_RECALL) / (EPOCH_PRECISION + EPOCH_RECALL)

</code></pre>
<p><a class="mention" href="/u/ptrblck">@ptrblck</a> <a class="mention" href="/u/pargg">@ParGG</a></p>

username: ParGG
cooked: <p>Much better <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
<p>Although I think you are still leaving some performance on the table. You don’t need to perform the comparisons in the <code>logical_and</code> (you already have 0s and 1s in the tensors), in general comparisons (from what I have seen during profiling) are expensive. Instead you can negate the values by performing <code>1 - gt_labels</code> or <code>1 - predicted_labels</code>. Also, you can exploit the fact that <code>TP + FN = GT_P</code>, <code>FP + TN = GT_N</code>, <code>TP + FP = PRED_P</code> and <code>FN + TN = PRED_N</code> to avoid some computations.</p>