Ignite - dataloader with more than input-target pair and custom evaluator

xen0f0n: <p>My dataloader returns image, target and a weight map for my loss function. For this, I’ve written an update function for the trainer engine (because I have to pass that weight map to the loss function). But, during evaluation, I don’t know how (or if) it’s possible to include this weight map and attach loss as a metric to the evaluator. <code>prepare_batch</code> expects an input-target pair (I cant’ return more than 2 items) and I don’t think I can pass the weight map through <code>output_transform</code>…</p>
<p>Any ideas?</p>

vfdev-5: <p>Hi <a class="mention" href="/u/xen0f0n">@xen0f0n</a><br>
So, output transform argument of Loss metric : <a href="https://pytorch.org/ignite/metrics.html#ignite.metrics.Loss" rel="nofollow noopener">https://pytorch.org/ignite/metrics.html#ignite.metrics.Loss</a><br>
does not work in your case ?</p>
<p>If it’s the case, please, provide a minimal code snippet to understand the issue. Thanks!</p>

xen0f0n: <p><a class="mention" href="/u/vfdev-5">@vfdev-5</a> I’m not sure I can use Loss from metrics, no.<br>
So, my update_model function for the trainer engine is</p>
<pre><code class="lang-auto">def update_model(engine, batch):
        inputs, targets, weight_map = batch
        inputs = inputs.to(device)
        targets = targets.to(device)
        weight_map = weight_map.to(device)
        optimizer.zero_grad()
        outputs = net(inputs)
        loss = criterion(outputs, targets, weight_map)
        loss.backward()
        optimizer.step()
        return loss.item()
</code></pre>
<p>I need to unpack the batch, that has 3 items, and follow the same scheme for evaluation. If I could unpack the batch inside <code>loss_fn</code> (in metrics.Loss), or using <code>prepare_batch</code> in <code>create_supervised_evaluator</code> that would do the trick.</p>

vfdev-5: <p>If I understand your case correctly, the following would work</p>
<pre><code class="lang-python">import torch
import torch.nn as nn

from ignite.engine import Engine, create_supervised_evaluator, Events
from ignite.metrics import Loss

data = [
    # inputs, targets, weight_map
    [torch.rand(1), torch.rand(1), torch.rand(2)],
    [torch.rand(1), torch.rand(1), torch.rand(2)],
    [torch.rand(1), torch.rand(1), torch.rand(2)],
    [torch.rand(1), torch.rand(1), torch.rand(2)]
]

net = nn.Linear(1, 1)
optimizer = torch.optim.SGD(net.parameters(), lr=0.001)

class WMapLoss(nn.Module):
    
    def forward(self, y_pred, y, weight_map):
        return (y_pred - y).sum() + weight_map.sum()


criterion = WMapLoss()


def update_model(engine, batch):
    inputs, targets, weight_map = batch
    optimizer.zero_grad()
    outputs = net(inputs)
    loss = criterion(outputs, targets, weight_map)
    loss.backward()
    optimizer.step()
    return loss.item()


def eval_fn(engine, batch):
    net.eval()
    with torch.no_grad():
        inputs, targets, weight_map = batch
        outputs = net(inputs)
        return outputs, targets, {"weight_map": weight_map}


trainer = Engine(update_model)
evaluator = Engine(eval_fn)

metrics={"val loss": Loss(criterion)}

for name, metric in metrics.items():
    metric.attach(evaluator, name)


@trainer.on(Events.EPOCH_COMPLETED)
def run_validation():
    evaluator.run(data)

trainer.run(data, max_epochs=2)
print(evaluator.state.metrics)
&gt; {'val loss': 1.027790516614914}
</code></pre>
<p>Tell me if this works or does not work for you.</p>

xen0f0n: <p>It works! Thanks!<br>
Now, could you explain why?! <img src="https://discuss.pytorch.org/images/emoji/apple/stuck_out_tongue.png?v=9" title=":stuck_out_tongue:" class="emoji" alt=":stuck_out_tongue:"><br>
Why does the <code>eval_fn</code> return a dictionary? (kwargs for some function?)</p>
<p>According to the docs:<br>
<em>process_function (callable): A function receiving a handle to the engine and the current batch in each iteration, and returns data to be stored in the engine’s state.</em></p>
<p>How does <code>Loss</code> receive the <code>weight_map</code>?</p>
<p>And finally, for other metrics that only need outputs and target, do I have to run another evaluator again to log them?</p>

vfdev-5: <p>There is no magic there <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
<p>To be able to pass additional attribute to <code>Loss</code> we need to set the output in format <code>(prediction, target, kwargs)</code> as according to the <a href="https://pytorch.org/ignite/metrics.html#ignite.metrics.Loss" rel="noopener nofollow ugc">docs</a>:</p>
<blockquote>
<p>output_transform (callable): a callable that is used to transform the<br>
:class:<code>~ignite.engine.Engine</code>’s <code>process_function</code>’s output into the<br>
form expected by the metric.<br>
This can be useful if, for example, you have a multi-output model and<br>
you want to compute the metric with respect to one of the outputs.<br>
The output is expected to be a tuple <code>(prediction, target)</code> or<br>
(prediction, target, kwargs) where kwargs is a dictionary of extra<br>
keywords arguments. If extra keywords arguments are provided they are passed to <code>loss_fn</code>.</p>
</blockquote>
<p>You can also see how it is interpreted in the update function: <a href="https://pytorch.org/ignite/_modules/ignite/metrics/loss.html#Loss" class="inline-onebox" rel="noopener nofollow ugc">ignite.metrics.loss — PyTorch-Ignite v0.4.13 Documentation</a></p>
<p>Therefore, <code>eval_fn</code> as being engine’s processing function returns the output exactly in the good format for the Loss function.</p>
<p>For other metrics, you need to filter out the last element (dict) with <code>output_transform</code> and attach to this evaluator (no need another one):</p>
<pre data-code-wrap="python"><code class="lang-python">acc = Accuracy(output_transform=lambda out: out[0], out[1])
acc.attach(evaluator, "Accuracy")
</code></pre>