Model not saving

AfonsoSalgadoSousa: <p>Hi. In a standard train script, I have the following partial function:</p>
<pre><code class="lang-auto">best_model_handler = partial(Checkpoint,
      {"model": model},
      DiskSaver(dirname=cfg.work_dir.as_posix(), require_empty=False),
      filename_prefix="best",
      n_saved=2,
      global_step_transform=global_step_from_engine(trainer),
  )
</code></pre>
<p>If I want to evaluate, I do this:</p>
<pre><code class="lang-auto">best_model_handler = best_model_handler(
            score_name="val_bleu",
            score_function=Checkpoint.get_default_score_fn("bleu"),
        )
evaluator.add_event_handler(Events.COMPLETED, best_model_handler)
</code></pre>
<p>This works fine. On the other hand, when I try not to validate, I do the following:</p>
<pre><code class="lang-auto">best_model_handler = best_model_handler(
            score_name="train_loss",
            score_function=Checkpoint.get_default_score_fn("loss"),
        )
trainer.add_event_handler(Events.COMPLETED, best_model_handler)
</code></pre>
<p>But it throws the following error:</p>
<blockquote>
<p>File “/home/admin/anaconda3/envs/catbird/lib/python3.8/site-packages/ignite/handlers/checkpoint.py”, line 648, in wrapper<br>
return score_sign * engine.state.metrics[metric_name]<br>
KeyError: ‘loss’</p>
</blockquote>
<p>However, if I do not define <code>score_name</code> or <code>score_function</code>, the script runs fine, but the model is not saved. How can I fix this? Thanks in advance for any help you can provide.</p>

vfdev-5: <p><a class="mention" href="/u/afonsosalgadosousa">@AfonsoSalgadoSousa</a> thanks for reporting, I’ll check what happens in details and write back here.</p>
<p>Few remarks, <code>score_function=Checkpoint.get_default_score_fn("loss")</code> fetches metric from attached engine’s state and by default there is no “loss” key in trainer.state.metrics. See <a href="https://pytorch.org/ignite/v0.4.8/generated/ignite.handlers.checkpoint.Checkpoint.html#ignite.handlers.checkpoint.Checkpoint.get_default_score_fn" class="inline-onebox" rel="noopener nofollow ugc">Checkpoint — PyTorch-Ignite v0.4.8 Documentation</a></p>
<p>By the way, since v0.4.8 user can use str as save_handler without using DiskSaver.</p>
<blockquote>
<p>However, if I do not define <code>score_name</code> or <code>score_function</code> , the script runs fine, but the model is not saved. How can I fix this?</p>
</blockquote>
<p>This is very strange, I can try to reproduce the issue, but if you could provide a minimal code example, it would be very helpful. Thanks</p>

AfonsoSalgadoSousa: <p>Thanks for the reply. The following code runs successfully but the model is not saved to the newly-created directory.</p>
<pre><code class="lang-auto">from functools import partial

import ignite.distributed as idist
import torch
from ignite.engine import Engine, Events
from ignite.handlers import Checkpoint, DiskSaver, global_step_from_engine
from torch import nn, optim
from torch.nn import CrossEntropyLoss
from torch.utils.data import DataLoader, TensorDataset


def train_step(engine, batch):
    X, y = batch[0].to(idist.device()), batch[1].to(idist.device())
    model.train()
    loss_fct = CrossEntropyLoss()
    y_pred = model(X)
    loss = loss_fct(y_pred, y)
    print(loss)
    return loss.item()


x = torch.randn(8, 2)
y = torch.empty(8, dtype=torch.long).random_(4)
my_dataset = TensorDataset(x, y)
my_dataloader = DataLoader(my_dataset)
model = nn.Sequential(nn.Linear(2, 10), nn.ReLU(), nn.Linear(10, 4)).to(idist.device())
optimizer = optim.SGD(model.parameters(), lr=0.01, momentum=0.9)

trainer = Engine(train_step)

best_model_handler = partial(
    Checkpoint,
    {"model": model},
    DiskSaver(dirname="./testing", require_empty=False),
    filename_prefix="best",
    n_saved=2,
    global_step_transform=global_step_from_engine(trainer),
)
trainer.add_event_handler(Events.COMPLETED, best_model_handler)

trainer.run(my_dataloader, max_epochs=1)
</code></pre>

vfdev-5: <p><a class="mention" href="/u/afonsosalgadosousa">@AfonsoSalgadoSousa</a> I think there is a bug in this code snippet as <code>best_model_handler</code> is partial and not <code>Checkpoint</code>.<br>
If you update it as <code>trainer.add_event_handler(Events.COMPLETED, best_model_handler())</code> and rerun the code:</p>
<pre><code class="lang-auto">ls testing/

best_model_1.pt
</code></pre>

AfonsoSalgadoSousa: <p>I am sorry for the inconvenience. That was it—another bug.</p>

AfonsoSalgadoSousa: <p>BTW, <a class="mention" href="/u/vfdev-5">@vfdev-5</a>, what is the default behaviour when neither <code>score_name</code> nor <code>score_function</code> are set?</p>

vfdev-5: <p>Checkpoint should just save into the file the content of <code>to_save</code> on the called events.<br>
For example:</p>
<pre><code class="lang-auto">    handler = Checkpoint(
        to_save, '/tmp/models', n_saved=2
    )
    trainer.add_event_handler(Events.ITERATION_COMPLETED(every=1000), handler)
</code></pre>
<p>If trainer does 3000 iterations, <code>handler</code> will be called 3 times and will save 3 files and remove the first file (as n_saved=2).</p>