How access inputs in custom Ignite Metric?

odats: <p>I have implemented a custom Ignite Metric based on this <a href="https://pytorch.org/ignite/metrics.html#how-to-create-a-custom-metric" rel="nofollow noopener">tutorial</a>.</p>
<pre><code class="lang-auto">def update(self, output):
        y_pred, y = output
</code></pre>
<p>How to access inputs (x values) in the update function?</p>

vfdev-5: <p>The simplest way to add <code>x</code> is to pass it into the output as <code>(y_pred, y, x)</code>.</p>
<p>Another way (more clearly implemented) is to reimplement <code>iteration_completed()</code> method : <a href="https://github.com/pytorch/ignite/blob/6faa6ac1e3a46c79e0dfcfd976439b86329717b0/ignite/metrics/metric.py#L198" rel="nofollow noopener">https://github.com/pytorch/ignite/blob/6faa6ac1e3a46c79e0dfcfd976439b86329717b0/ignite/metrics/metric.py#L198</a></p>
<p>and pass to <code>update</code> method everything it needs without “hiding” things: output, input, etc…</p>
<p>Let me know if it answers your question. Thanks</p>

odats: <p>how to pass <code>x</code> into the output? And not to break other metrics.</p>
<pre><code class="lang-auto">evaluator = create_supervised_evaluator(
        model, metrics={
            "loss": Loss(criterion),
            "accuracy": Accuracy(), 
            "accuracy_pix2pix": PixelToPixelAccuracy()}, # custom metric to extend
    )
</code></pre>

vfdev-5: <p>Pass all necessary data y, predictions, x to all metrics and use <code>output_transform</code> to filter out required args to existing metrics.</p>
<p>Probably, more elegant way to do this is :</p>
<ul>
<li>return output as a dictionary with keys like “y” - target, “y_pred” - predictions, “x” for input x.</li>
<li>override <code>_required_output_keys</code> for your custom metric as a tuple <code>("y", "y_pred", "x")</code>
</li>
</ul>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/pytorch/ignite/blob/6faa6ac1e3a46c79e0dfcfd976439b86329717b0/ignite/metrics/metric.py#L129" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/ignite/blob/6faa6ac1e3a46c79e0dfcfd976439b86329717b0/ignite/metrics/metric.py#L129" target="_blank" rel="nofollow noopener">pytorch/ignite/blob/6faa6ac1e3a46c79e0dfcfd976439b86329717b0/ignite/metrics/metric.py#L129</a></h4>
<pre class="onebox"><code class="lang-py"><ol class="start lines" start="119" style="counter-reset: li-counter 118 ;">
<li>Args:</li>
<li>    output_transform (callable, optional): a callable that is used to transform the</li>
<li>        :class:`~ignite.engine.engine.Engine`'s ``process_function``'s output into the</li>
<li>        form expected by the metric. This can be useful if, for example, you have a multi-output model and</li>
<li>        you want to compute the metric with respect to one of the outputs.</li>
<li>        By default, metrics require the output as ``(y_pred, y)`` or ``{'y_pred': y_pred, 'y': y}``.</li>
<li>    device (str of torch.device, optional): optional device specification for internal storage.</li>
<li>
</li><li>"""</li>
<li>
</li><li class="selected">_required_output_keys = ("y_pred", "y")</li>
<li>
</li><li>def __init__(</li>
<li>    self, output_transform: Callable = lambda x: x, device: Optional[Union[str, torch.device]] = None,</li>
<li>):</li>
<li>    self._output_transform = output_transform</li>
<li>
</li><li>    # Check device if distributed is initialized:</li>
<li>    if idist.get_world_size() &gt; 1:</li>
<li>
</li><li>        # check if reset and update methods are decorated. Compute may not be decorated</li>
</ol></code></pre>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>Thus, I think it can be possible to fetch needed args without using output transform…</p>

odats: <p>Can you please provide short code example?</p>

vfdev-5: <p>Here is a colab with an example : <a href="https://colab.research.google.com/drive/1-EL_YGLPzEPIw_6jU-tWRRLXnx0lN106?usp=sharing" rel="nofollow noopener">https://colab.research.google.com/drive/1-EL_YGLPzEPIw_6jU-tWRRLXnx0lN106?usp=sharing</a></p>
<pre><code class="lang-python">import torch
import torch.nn as nn

from ignite.metrics import Metric, Accuracy
from ignite.engine import create_supervised_evaluator


class CustomMetric(Metric):

    _required_output_keys = ("y_pred", "y", "x")

    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
    
    def update(self, output):
        print("CustomMetric: output=")
        for i, o in enumerate(output):
            print(i, o.shape)

    def reset(self):
        pass

    def compute(self):
        return 0.0



model = nn.Linear(10, 3)

metrics = {
    "Accuracy": Accuracy(),
    "CustomMetric": CustomMetric()
}

evaluator = create_supervised_evaluator(
    model, 
    metrics=metrics, 
    output_transform=lambda x, y, y_pred: {"x": x, "y": y, "y_pred": y_pred}
)

data = [
    (torch.rand(4, 10), torch.randint(0, 3, size=(4, ))),
    (torch.rand(4, 10), torch.randint(0, 3, size=(4, ))),
    (torch.rand(4, 10), torch.randint(0, 3, size=(4, )))
]
res = evaluator.run(data)
</code></pre>
<p>Output:</p>
<pre><code class="lang-auto">CustomMetric: output=
0 torch.Size([4, 3])
1 torch.Size([4])
2 torch.Size([4, 10])
CustomMetric: output=
0 torch.Size([4, 3])
1 torch.Size([4])
2 torch.Size([4, 10])
CustomMetric: output=
0 torch.Size([4, 3])
1 torch.Size([4])
2 torch.Size([4, 10])
</code></pre>

vfdev-5: <p><a class="mention" href="/u/odats">@odats</a> since v0.4.2 which will be released Sep 22-25th, please consider using <code>required_output_keys</code> as a public class attribute instead of a private one <code>_required_output_keys</code>.</p>