Repository: pytorch/ignite

Issue:
How to save model with same filename?
## 🚀 Feature

<!-- A clear and concise description of the feature proposal -->
<!-- Please outline the motivation for the proposal. Is your feature request related to a problem? e.g., I'm always frustrated when [...]. If this is related to another GitHub issue, please link here too -->
<!-- A clear and concise description of what you want to happen. -->
<!-- A clear and concise description of any alternative solutions or features you've considered, if any. -->
<!-- Add any other context or screenshots about the feature request here. -->
Hi @vfdev-5 ,

I found that `CheckpointHandler` can't use the fixed filename to save latest model:
https://github.com/pytorch/ignite/blob/v0.4.2/ignite/handlers/checkpoint.py#L149
But this is a very important requirement on our side...How can I achieve that?
For example, I want to always save the best model as `model.pt` after validations during training.

Thanks.



Discussion:
Hi @Nic-Ma 

Thanks for asking ! Yes, there is a limitation on the filename like that. Sometime ago I was thinking about supporting this feature. Maybe, it's time to add a support for that inside Checkpoint.
Let me see what we can do.
 
Hi @vfdev-5 ,

We need this feature urgent in NVIDIA project, so I developed a PR in MONAI by somehow `hack` your `Checkpoint` handler to manually delete the previously saved model:
https://github.com/Project-MONAI/MONAI/pull/1163
Could you please help review it?
If you can provide a simpler way than mine, that's would be awesome.
BTW, I already hacked several places to support some features, maybe we can think about releasing a special branch of ignite or some better idea?
Or can we add some my "reasonable" features to ignite 0.4.3 directly? When do you plan to release it?

Thanks.

@Nic-Ma actually, I forgot that there is simple workaround to do that using a custom disk saver:
```python
import torch
from ignite.engine import Engine, State
from ignite.handlers import Checkpoint
from ignite.handlers.checkpoint import BaseSaveHandler


class MyDiskSaver(BaseSaveHandler):
    
    def __call__(self, checkpoint, filename, metadata):
        print("-> save checkpoint:", filename, metadata)

    def remove(self, filename):
        print("remove:", filename)


model = torch.nn.Linear(1, 1)

save_handler = MyDiskSaver()

to_save = {"model": model}
checkpointer = Checkpoint(to_save, save_handler=save_handler, n_saved=1, score_function=lambda e: e.state.score)

trainer = Engine(lambda e, b: None)
trainer.state = State(epoch=1, iteration=1, score=0.77)

checkpointer(trainer)

trainer.state.epoch = 12
trainer.state.iteration = 1234
trainer.state.score = 0.78

checkpointer(trainer)
> -> save checkpoint: model_0.7700.pt {'basename': 'model', 'score_name': None, 'priority': 0.77}
> remove: model_0.7700.pt
> -> save checkpoint: model_0.7800.pt {'basename': 'model', 'score_name': None, 'priority': 0.78}
```
In `MyDiskSaver` you can save the file as you wish with any name you wish and keep `remove` implementation empty.
The only thing which may seem incoherent is 
```python
checkpointer.last_checkpoint
> 'model_0.7800.pt'
```
instead of the real filename...

What do you think ?
Hi @vfdev-5 ,

Thanks for your sugggestion, it's simpler than my previous idea, I updated my PR: https://github.com/Project-MONAI/MONAI/pull/1163
Could you please help review it again?

Thanks.

Hi @vfdev-5 ,

BTW, there is another problem with the `Checkpoint` handler: if we normally call `engine.run()` several times, every time it will run from the beginning because it has `_is_done()` to reset all the state, but `Checkpoint` handler doesn't reset the `self._saved` in it, right?

Thanks.

@Nic-Ma yes, you are right about that. Instance of `Checkpoint` keeps its `_saved` which can lead to unsaved checkpoints on further runs... Could you please open a feature request for that ? Thanks !
Hi @vfdev-5 , sure, submitted ticket: https://github.com/pytorch/ignite/issues/1422
Can we keep this issue open to track? Until you guys officially support fixed file name?

Thanks.

Resolving patch:
diff --git a/ignite/handlers/checkpoint.py b/ignite/handlers/checkpoint.py
--- a/ignite/handlers/checkpoint.py
+++ b/ignite/handlers/checkpoint.py
@@ -147,8 +147,8 @@ class Checkpoint(Serializable):
         ``30000-checkpoint-94.pt``
 
         **Warning:** Please, keep in mind that if filename collide with already used one to saved a checkpoint,
-        new checkpoint will not be stored. This means that filename like ``checkpoint.pt`` will be saved only once
-        and will not be overwritten by newer checkpoints.
+        new checkpoint will replace the older one. This means that filename like ``checkpoint.pt`` will be saved
+        every call and will always be overwritten by newer checkpoints.
 
     Note:
         To get the last stored filename, handler exposes attribute ``last_checkpoint``:
@@ -350,22 +350,26 @@ def __call__(self, engine: Engine) -> None:
             }
             filename = filename_pattern.format(**filename_dict)
 
-            if any(item.filename == filename for item in self._saved):
-                return
-
             metadata = {
                 "basename": "{}{}{}".format(self.filename_prefix, "_" * int(len(self.filename_prefix) > 0), name),
                 "score_name": self.score_name,
                 "priority": priority,
             }
 
-            if not self._check_lt_n_saved():
-                item = self._saved.pop(0)
+            try:
+                index = list(map(lambda it: it.filename == filename, self._saved)).index(True)
+                to_remove = True
+            except ValueError:
+                index = 0
+                to_remove = not self._check_lt_n_saved()
+
+            if to_remove:
+                item = self._saved.pop(index)
                 if isinstance(self.save_handler, BaseSaveHandler):
                     self.save_handler.remove(item.filename)
 
             self._saved.append(Checkpoint.Item(priority, filename))
-            self._saved.sort(key=lambda item: item[0])
+            self._saved.sort(key=lambda it: it[0])
 
             if self.include_self:
                 # Now that we've updated _saved, we can add our own state_dict.


Test patch:
diff --git a/tests/ignite/handlers/test_checkpoint.py b/tests/ignite/handlers/test_checkpoint.py
--- a/tests/ignite/handlers/test_checkpoint.py
+++ b/tests/ignite/handlers/test_checkpoint.py
@@ -1309,6 +1309,9 @@ def _test(
     )
     assert res == "best_model_12_acc=0.9999.pt"
 
+    pattern = "{name}.{ext}"
+    assert _test(to_save, filename_pattern=pattern) == "model.pt"
+
     pattern = "chk-{name}--{global_step}.{ext}"
     assert _test(to_save, to_save, filename_pattern=pattern) == "chk-model--203.pt"
     pattern = "chk-{filename_prefix}--{name}--{global_step}.{ext}"
@@ -1446,3 +1449,25 @@ def test_checkpoint_load_state_dict():
     sd = {"saved": [(0, "model_0.pt"), (10, "model_10.pt"), (20, "model_20.pt")]}
     checkpointer.load_state_dict(sd)
     assert checkpointer._saved == true_checkpointer._saved
+
+
+def test_checkpoint_fixed_filename():
+    model = DummyModel()
+    to_save = {"model": model}
+
+    def _test(n_saved):
+        save_handler = MagicMock(spec=BaseSaveHandler)
+        checkpointer = Checkpoint(to_save, save_handler=save_handler, n_saved=n_saved, filename_pattern="{name}.{ext}")
+
+        trainer = Engine(lambda e, b: None)
+
+        for i in range(10):
+            trainer.state = State(epoch=i, iteration=i)
+            checkpointer(trainer)
+            assert save_handler.call_count == i + 1
+            metadata = {"basename": "model", "score_name": None, "priority": i}
+            save_handler.assert_called_with(model.state_dict(), "model.pt", metadata)
+
+    _test(None)
+    _test(1)
+    _test(3)