Nested event handlers

username: pytorchnewbie
cooked: <p>What if I want to run my evaluation engine (which itself has event handlers) with my evaluation dataloader every 20 epochs of the training engine? Right now this isn’t possible, so I made a wrapper function that allows for this below. However, I was wondering if there was an ignite way of doing this? I couldn’t find any.</p>
<pre><code class="lang-auto">def _lf_two(
    log_fn: Callable[[Engine, List[str]], None],
    inner_engine: Engine,
    loader: DataLoader,
    **kwargs
) -&gt; Callable[[Engine], Any]:
    """Returns a lambda calling custom log function with two engines (e.g. the training loop and validation loop)"""
    return lambda outer_engine: log_fn(
        inner_engine, loader, epoch_num=outer_engine.state.epoch, **kwargs
    )

def run_engine_and_log_output(
    engine: Engine, loader: DataLoader, fields: List[str], epoch_num=None,
) -&gt; None:

    engine.run(loader,max_epochs=1, epoch_length=10)
    ###Here i grab things from engine.state.output and do what i want###

trainer.add_event_handler(
        Events.EPOCH_COMPLETED(every=20),
        _lf_two(
            run_engine_and_log_output,
            evaluator,
            val_loader))
</code></pre>
<p>Also, I was wondering if I then wanted to run the evaluation outside of the training loop, and this time with different event handlers, would best practice be to create two different evaluation engines? One to pass into the training loop event handler into (__lf_two()), and one to run after my train.run ?</p>
<p>Thanks!</p>

username: vfdev-5
cooked: <p>Maybe I do not correctly understand the issue, but if you would like to run model evaluation every 20-th epochs and in the end of the training it is simply done like :</p>
<pre><code class="lang-python">
trainer = ...
evaluator = ...


@trainer.on(Events.EPOCH_COMPLETED(every=20) | Events.COMPLETED)
def validate_model():
    state = evaluator.run(val_loader)
    print(trainer.state.epoch, state.metrics)

</code></pre>
<p><a class="mention" href="/u/pytorchnewbie">@pytorchnewbie</a>, what do you think? Does it answer your question ?</p>

username: pytorchnewbie
cooked: <p>Sorry for the late reply!</p>
<p>Well sort of - I want to do :</p>
<p>handler = ModelCheckpoint(<br>
“models/”,<br>
“checkpoint”,<br>
score_function=score_function,<br>
n_saved=None<br>
)</p>
<p>evaluator.add_event_handler(Events.EPOCH_COMPLETED(every=50), handler, {‘textcnn’: model})</p>
<p>except I want it to be the EPOCH_COMPLETED(every=50) of the training epoch, not the evaluator epoch…</p>

username: vfdev-5
cooked: <blockquote>
<p>except I want it to be the EPOCH_COMPLETED(every=50) of the training epoch, not the evaluator epoch…</p>
</blockquote>
<p>It means that you would like to save a model (not the best one) every 50-th epoch completed. This is easy following the <a href="https://pytorch.org/ignite/handlers.html#ignite.handlers.ModelCheckpoint" rel="noopener nofollow ugc">doc of ModelCheckpoint</a></p>
<pre data-code-wrap="python"><code class="lang-python">handler = ModelCheckpoint(“models/”, “checkpoint”, n_saved=None)

trainer.add_event_handler(EPOCH_COMPLETED(every=50), handler)
</code></pre>
<p>so, here is a complete working example</p>
<pre data-code-wrap="python"><code class="lang-python">!rm -rf /tmp/models

import os
from ignite.engine import Engine, Events
from ignite.handlers import ModelCheckpoint
from torch import nn
trainer = Engine(lambda engine, batch: None)
handler = ModelCheckpoint('/tmp/models', 'myprefix', n_saved=None, create_dir=True)
model = nn.Linear(3, 3)
trainer.add_event_handler(Events.EPOCH_COMPLETED(every=50), handler, {'mymodel': model})
trainer.run([0], max_epochs=200)

!ls /tmp/models
</code></pre>
<p>HTH</p>
<p>PS. there was a typo in the docstring which i’ve just fixed</p>

username: pytorchnewbie
cooked: <p>oh oops sorry I forgot to mention that I want score_function to running the evaluation loop once, because I want the MSE from that. However, in that case I need to pass in the evaluator engine somehow.</p>

username: vfdev-5
cooked: <p><a class="mention" href="/u/pytorchnewbie">@pytorchnewbie</a> <code>score_function</code> option is used to save only the best N models.</p>
<p>To resume, there are two options how to save models:</p>
<ol>
<li>either based on any event. For example in like previous answer, model is always saved every 50-th epochs no matter its quality.</li>
<li>either based on a score. For example, evaluator engine provides a metric score, thus on the moment to save or not a model, <code>Checkpoint</code> verifies the current score, compares with registered score(s) and decides if we need or not to save current model.</li>
</ol>
<p>If you would like combine both, you need to provide your own <code>score_function</code> that would take into account event number and your score.</p>
<pre><code class="lang-python">trainer = ...
evaluator = ...

def score_function(_):
    # MAKE SURE THAT MSE SCORE IS AVAILABLE WHEN IT IS CALLED
    mse = evaluator.state.metrics["mse"]
    epoch = trainer.state.epoch
    # Create your own logic when to save the model
    # ... model with highest scores will be retained.
    return some_score

to_save = {'model': model}
handler = Checkpoint(..., score_function=score_function, ...)

trainer.add_event_handler(Events.COMPLETED, handler)
</code></pre>
<p>HTH</p>
<p>PS. If you want just to display eval score in the filename while saving a model every 50 epochs, unfortunately, this should be done by overriding the way you write the file…</p>

username: pytorchnewbie
cooked: <p>Thanks for your reply.</p>
<p>I want option <span class="hashtag">#2</span>.<br>
However, it doesn’t work. I have written:</p>
<pre><code class="lang-auto">def score_function(engine):
        return evaluator.state.metrics["mse"]

handler = ModelCheckpoint(
        "models/", "checkpoint", score_function=score_function, n_saved=5
    )

trainer.add_event_handler(
        Events.EPOCH_COMPLETED(every=10), handler, {"textcnn": model}
    )
</code></pre>
<p>The problem is that it saves the first 5 models every 10 epochs, so the model at epoch 10,20,30,40,50 . However, it doesn’t then replace them as the training continues (although the mse does drop)…</p>

username: vfdev-5
cooked: <p>Please, see the docs about the score function:</p>
<blockquote>
<p>Objects with highest scores will be retained.</p>
</blockquote>
<p>You need to return for example:</p>
<pre><code class="lang-auto">def score_function(engine):
    return -evaluator.state.metrics["mse"]
</code></pre>

username: pytorchnewbie
cooked: <p>Ah yes, thank you!! Really appreciate it</p>

username: pytorchnewbie
cooked: <p>Hi again. I apologize in advance but I’ve tried solving this for 2 hours now. I’m not sure what I did but suddenly the attaching doesn’t work anymore. It doesn’t recognize “mse” :</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/0/5/05d6397ade1e03c55a7bc44ddd881c465cf484fe.png" alt="finalerr" data-base62-sha1="PDmoOiTeEoDyvfXlBlHl7CONPM" width="580" height="92"></p>
<p>Which  is weird because it should since below is what I call in train():</p>
<pre><code class="lang-auto">model = get_network(cfg)

    to_save = {
        "model": model,
    }

    def score_function(engine):
        return -evaluator.state.metrics["mse"]

    handler = ModelCheckpoint(
        "models/", "checkpoint", score_function=score_function, n_saved=5
    )

    # Data Loaders
    train_loader, val_loader = get_dataloaders(cfg, num_workers=cfg.data_loader_workers)

    # Your training loop
    trainer = create_training_loop(model, cfg, "trainer", device=device)

    # Your evaluation loop
    evaluator = create_evaluation_loop(model, cfg, "evaluator", device=device)


    trainer.add_event_handler(Events.EPOCH_COMPLETED(every=10), handler, {"mlp": model})
</code></pre>
<p>And then here is a screenshot of  the actual code of create_evaluation_loop(), as well as the terminal where I use pdb to show that despite the for loop  where I attach running,  afterwards the engine still doesn’t have anything in :  engine.state.metrics …</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/5/05f7acb76868e29d696bae2e09178e54226969df.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/05f7acb76868e29d696bae2e09178e54226969df" title="pdbscreen"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/0/5/05f7acb76868e29d696bae2e09178e54226969df_2_690x408.jpeg" alt="pdbscreen" data-base62-sha1="QN1JdHUkuH5X0BTYccActUjIN9" width="690" height="408" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/0/5/05f7acb76868e29d696bae2e09178e54226969df_2_690x408.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/0/5/05f7acb76868e29d696bae2e09178e54226969df_2_1035x612.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/0/5/05f7acb76868e29d696bae2e09178e54226969df_2_1380x816.jpeg 2x" data-dominant-color="9F9B91"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">pdbscreen</span><span class="informations">2144×1268 776 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: vfdev-5
cooked: <p>You need to make sure that validation is run before saving the model.</p>
<p>Here is a synthetic example you can play with to understand how it works and fix your problem</p>
<pre><code class="lang-python">!rm -rf /tmp/ignite-mse-example/

import torch


from ignite.engine import Engine, Events
from ignite.handlers import Checkpoint, DiskSaver
from ignite.metrics import MeanSquaredError


num_epochs = 50
num_train_samples_per_epoch = 25
num_val_samples_per_epoch = 10

batch_size = 4
num_features = 5
val_targets = torch.rand(num_val_samples_per_epoch, batch_size, num_features)
val_preds = val_targets + torch.rand(num_val_samples_per_epoch, batch_size, num_features) * 0.01


trainer = Engine(lambda e, b: None)


def validation_step(e, b):
    i = e.state.iteration - 1    
    err = (num_epochs - trainer.state.epoch - 1) * torch.rand(batch_size, num_features)    
    y_preds = val_preds[i, ...] + err
    y = val_targets[i, ...]
    return y_preds, y
    

evaluator = Engine(validation_step)

mse_metric = MeanSquaredError()
mse_metric.attach(evaluator, "mse")


val_every = 5

@trainer.on(Events.EPOCH_COMPLETED(every=val_every))
def run_validation():
    print("{} : Run validation ...".format(trainer.state.epoch))
    val_data = range(num_val_samples_per_epoch)
    evaluator.run(val_data)
    print("Val MSE:", evaluator.state.metrics["mse"])


def score_function(_):
    # !!! MAKE SURE THAT MSE SCORE IS ALREADY COMPUTED !!!
    mse = evaluator.state.metrics["mse"]
    return -mse

to_save = {'model': torch.nn.Linear(10, 10)}
handler = Checkpoint(
    to_save, 
    DiskSaver("/tmp/ignite-mse-example"), 
    n_saved=3, 
    score_function=score_function, 
    score_name="val_mse",
    global_step_transform=lambda _1, _2: trainer.state.epoch
)

trainer.add_event_handler(Events.EPOCH_COMPLETED(every=val_every), handler)


train_data = range(num_train_samples_per_epoch)
trainer.run(train_data, max_epochs=num_epochs)

!ls /tmp/ignite-mse-example/
</code></pre>

username: pytorchnewbie
cooked: <p>Got it, makes great sense. Thanks alot!</p>