Top_k accuracy for multilabel classification

jpainam: <p>Hi.<br>
Top-K Metrics are widely used in assessing the quality of <code>Multi-Label classification.</code><br>
I didn’t find metrics on pytorch that can be used for monitoring multi-label classification training out of the box. I tried using <a href="https://github.com/bearpaw/pytorch-classification/blob/24f1c456f48c78133088c4eefd182ca9e6199b03/utils/eval.py#L5" rel="noopener nofollow ugc">this</a> (Edited following <a class="mention" href="/u/eta_c">@Eta_C</a> comment <img src="https://discuss.pytorch.org/images/emoji/apple/hugs.png?v=9" title=":hugs:" class="emoji" alt=":hugs:">).<br>
But got some error</p>
<pre><code class="lang-python">output = torch.randn(64, 134)
target = torch.randn(64, 134)

def accuracy(output, target, topk=(1,)):
    maxk = max(topk)
    batch_size = target.size(0)

    _, pred = output.topk(maxk, dim=1, largest=True, sorted=True)
    pred = pred.t()
    correct = pred.eq(target.view(1, -1).expand_as(pred))

    ret = []
    for k in topk:
        correct_k = correct[:k].contiguous().view(-1).float().sum(dim=0, keepdim=True)
        ret.append(correct_k.mul_(1. / batch_size))
    return ret
## using
acc = accuracy(output, target, topk=(1,2,5))
</code></pre>
<p>An error</p>
<pre><code class="lang-python">Traceback (most recent call last):
  File "main.py", line 77, in &lt;module&gt;
    trainer.train(epoch)
  File "/home/xx/xxx/trainer.py", line 152, in train
    prec = accuracy(pred_attrs.data, attrs.data, topk=(1, 2, 5))
  File "/home/xx/xxx/evaluation/classification.py", line 23, in accuracy
    correct = pred.eq(target.view(1, -1).expand_as(pred))
RuntimeError: The expanded size of the tensor (64) must match the existing size (8576) at non-singleton dimension 1.  Target sizes: [5, 64].  Tensor sizes: [1, 8576]
</code></pre>
<p>Any suggestions how to go about this implementation?<br>
Thank you</p>

Eta_C: <p>Hey Bro, Maybe your code comes from</p><aside class="onebox githubblob" data-onebox-src="https://github.com/bearpaw/pytorch-classification/blob/cc9106d598ff1fe375cc030873ceacfea0499d77/utils/eval.py">
  <header class="source">

      <a href="https://github.com/bearpaw/pytorch-classification/blob/cc9106d598ff1fe375cc030873ceacfea0499d77/utils/eval.py" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/bearpaw/pytorch-classification/blob/cc9106d598ff1fe375cc030873ceacfea0499d77/utils/eval.py" target="_blank" rel="noopener nofollow ugc">bearpaw/pytorch-classification/blob/cc9106d598ff1fe375cc030873ceacfea0499d77/utils/eval.py</a></h4>


      <pre><code class="lang-py">from __future__ import print_function, absolute_import

__all__ = ['accuracy']

def accuracy(output, target, topk=(1,)):
    """Computes the precision@k for the specified values of k"""
    maxk = max(topk)
    batch_size = target.size(0)

    _, pred = output.topk(maxk, 1, True, True)
    pred = pred.t()
    correct = pred.eq(target.view(1, -1).expand_as(pred))

    res = []
    for k in topk:
        correct_k = correct[:k].view(-1).float().sum(0)
        res.append(correct_k.mul_(100.0 / batch_size))
    return res</code></pre>




  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>See Here,</p><aside class="onebox stackexchange" data-onebox-src="https://stats.stackexchange.com/questions/95391/what-is-the-definition-of-top-n-accuracy/331508#331508">
  <header class="source">

      <a href="https://stats.stackexchange.com/questions/95391/what-is-the-definition-of-top-n-accuracy/331508#331508" target="_blank" rel="noopener nofollow ugc">stats.stackexchange.com</a>
  </header>

  <article class="onebox-body">
      <a href="https://stats.stackexchange.com/users/197343/rocksyne" target="_blank" rel="noopener nofollow ugc">
    <img alt="rocksyne" src="https://www.gravatar.com/avatar/06299bf231b24efd74aadd41e1b47149?s=256&amp;d=identicon&amp;r=PG&amp;f=y&amp;so-version=2" class="thumbnail onebox-avatar" width="256" height="256">
  </a>

<h4>
  <a href="https://stats.stackexchange.com/questions/95391/what-is-the-definition-of-top-n-accuracy/331508#331508" target="_blank" rel="noopener nofollow ugc">What is the definition of Top-n accuracy?</a>
</h4>

<div class="tags">
  <strong>machine-learning</strong>
</div>

<div class="date">
  
  answered by
  <a href="https://stats.stackexchange.com/users/197343/rocksyne" target="_blank" rel="noopener nofollow ugc">
    rocksyne
  </a>
  on <a href="https://stats.stackexchange.com/questions/95391/what-is-the-definition-of-top-n-accuracy/331508#331508" target="_blank" rel="noopener nofollow ugc">05:00AM - 03 Mar 18 UTC</a>
</div>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<blockquote>
<p>If we take the top-3 accuracy for this, the correct class only needs to be in the top three predicted classes to count.</p>
</blockquote>
<p>Assume that you have 64 samples, it should be</p>
<pre data-code-wrap="python"><code class="lang-python">output = torch.randn(64, 134)
target = torch.randn(64)
</code></pre>

jpainam: <p>I used this code a while ago for a classification problem. Don’t really remember on which StackOverflow thread. But thank you for googling and pointing out the github repo (it may not be the original repo).<br>
But my problem still remains. The implementation works for classification (binary of multi class), not for multi-label classification. In multi-label classification, a sample can have more than one category.<br>
For instance, for <code>5</code> classes, a target for a sample <code>x</code> could be</p>
<pre><code class="lang-python">target_x = [1, 0, 1, 0, 0]
# then for 64 samples, the targets are [64, 5] not [64] 
# I'm using  134 categories
</code></pre>
<p>Multi-label classification is mostly used in attribute classification where a given image can have more than one attribute. Each attribute is treated as binary classification problem. Hope you understand now</p>

Eta_C: <p>So <code>target</code> is a boolean tensor. I think, maybe…</p>
<pre><code class="lang-python">output = torch.randn(64, 134)
target = torch.randn(64, 134) &gt; 0.5  # multi hot

def accuracy(output, target, topk=(1,)):
    maxk = max(topk)
    batch_size = target.size(0)

    _, pred = output.topk(maxk, dim=1, largest=True, sorted=True)
    print(pred.shape)
    ret = []
    for k in topk:
        correct = (target * torch.zeros_like(target).scatter(1, pred[:, :k], 1)).float()
        ret.append(correct.sum() / target.sum())
    return ret
## using
acc = accuracy(output, target, topk=(1,2,5))
print(acc)
</code></pre>

jpainam: <p>yes, <code>target</code> is a boolean tensor. Thank you. it’s working as expected.<br>
Thanks</p>