Implementation of the multiclass case for precision metric

jamba_juice: <p>Hi, I wanted to try the example of the multiclass case for precision metric from the ignite library with Average=True. I used the exact same values and code but still got an error. I was interested to test out different cases and binary &amp; multilabel seem to work, but the multiclass case is not working as it should work like in the docs.</p>
<pre><code class="lang-auto">import torch
from ignite.engine import Engine
from ignite.metrics import Precision
def process_function(engine, data):
  return data[0][0], data[0][1]
default_evaluator = Engine(process_function)

metric = Precision(average=True)
metric.attach(default_evaluator, "precision")
y_true = torch.Tensor([2, 0, 2, 1, 0, 1]).long()
y_pred = torch.Tensor([
    [0.0266, 0.1719, 0.3055],
    [0.6886, 0.3978, 0.8176],
    [0.9230, 0.0197, 0.8395],
    [0.1785, 0.2670, 0.6084],
    [0.8448, 0.7177, 0.7288],
    [0.7748, 0.9542, 0.8573],
])
state = default_evaluator.run([[y_pred, y_true]])
print(state.metrics["precision"])
</code></pre>
<p>This is the error I get:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/f/2fdcb4c85ba5986d5b114c130c2a67008fc69fbb.png" data-download-href="https://discuss.pytorch.org/uploads/default/2fdcb4c85ba5986d5b114c130c2a67008fc69fbb" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/2/f/2fdcb4c85ba5986d5b114c130c2a67008fc69fbb.png" alt="image" data-base62-sha1="6PpiIBlCqoUOseLqlPRil8hbemT" width="690" height="299" data-dominant-color="393A3B"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1386×601 35.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p><strong>In the docs  link below, you can find the same code block for the multiclass case, and they get the precision as 0.6111…</strong></p>
<aside class="onebox allowlistedgeneric" data-onebox-src="https://pytorch.org/ignite/master/generated/ignite.metrics.precision.Precision.html">
  <header class="source">
      <img src="https://pytorch.org/ignite/master/_static/ignite_logomark.svg" class="site-icon" width="250" height="250">

      <a href="https://pytorch.org/ignite/master/generated/ignite.metrics.precision.Precision.html" target="_blank" rel="noopener nofollow ugc">PyTorch-Ignite</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/344;"><img src="https://raw.githubusercontent.com/pytorch/ignite/master/assets/logo/ignite_logo.png" class="thumbnail" width="690" height="344"></div>

<h3><a href="https://pytorch.org/ignite/master/generated/ignite.metrics.precision.Precision.html" target="_blank" rel="noopener nofollow ugc">Precision — PyTorch-Ignite v0.4.13 Documentation</a></h3>

  <p>High-level library to help with training and evaluating neural networks in PyTorch flexibly and transparently.</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>I would be very grateful if someone could point out where I was going wrong, or if I need to do something else to test the multiclass case for precision.</p>
<p>Thank you!</p>

vfdev-5: <p>Thanks for pointing out potential issues with our documentation and doctests!<br>
In your example, the definition of <code>default_evaluator</code> should be different. For our doctests it is defined here:<br>
<a href="https://github.com/pytorch/ignite/blob/5c5837ac2528e9e9f1451dfbceacfc0deb86fe56/docs/source/conf.py#L349-L352" rel="noopener nofollow ugc">https://github.com/pytorch/ignite/blob/5c5837ac2528e9e9f1451dfbceacfc0deb86fe56/docs/source/conf.py#L349-L352</a></p>
<pre><code class="lang-python">def eval_step(engine, batch):
    return batch

default_evaluator = Engine(eval_step)
</code></pre>
<p>Complete example is the following:</p>
<pre><code class="lang-python">import torch
from ignite.metrics import Precision
from ignite.engine import Engine

def eval_step(engine, batch):
    return batch

default_evaluator = Engine(eval_step)

metric = Precision(average=True)
metric.attach(default_evaluator, "precision")
y_true = torch.Tensor([2, 0, 2, 1, 0, 1]).long()
y_pred = torch.Tensor([
[0.0266, 0.1719, 0.3055],
[0.6886, 0.3978, 0.8176],
[0.9230, 0.0197, 0.8395],
[0.1785, 0.2670, 0.6084],
[0.8448, 0.7177, 0.7288],
[0.7748, 0.9542, 0.8573],
])
state = default_evaluator.run([[y_pred, y_true]])
print(state.metrics["precision"])
&gt; 0.6111111

</code></pre>
<p>Hope this helps</p>

sdesrozis: <p>Hi, thanks for the question.</p>
<p>The error comes from the <code>process_function</code> function you used in the <code>Engine</code>. In this very simple example, the output of this function feeds directly into the metric as a demonstration. Note that <code>data[0]</code> is <code>y_pred</code>. It means that <code>data[0][0]</code> is <code>[0.0266, 0.1719, 0.3055]</code> and <code>data[0][0]</code> is <code>[0.6886, 0.3978, 0.8176]</code>. This can’t be correctly used in the metric.</p>

jamba_juice: <p>Thank you so much for the reply!</p>

jamba_juice: <p>Thank you! I understand why it was giving an error.</p>