Repository: pytorch/ignite

Issue:
Support options to compare in CheckpointHander
## 🚀 Feature

<!-- A clear and concise description of the feature proposal -->
<!-- Please outline the motivation for the proposal. Is your feature request related to a problem? e.g., I'm always frustrated when [...]. If this is related to another GitHub issue, please link here too -->
<!-- A clear and concise description of what you want to happen. -->
<!-- A clear and concise description of any alternative solutions or features you've considered, if any. -->
<!-- Add any other context or screenshots about the feature request here. -->
Hi @vfdev-5 ,

In current ignite CheckpointHander, it compares `priority`(may be metrics) > `previous priority` and save models:
https://github.com/pytorch/ignite/blob/master/ignite/handlers/checkpoint.py#L357
But we received several requests to customize this compare logic, is it possible to make options for it?

Thanks.



Discussion:
Hi @Nic-Ma , thanks for the idea! What kind of customizations it is about:
e.g. change the sign of comparision, e.g. `self._saved[0].priority > priority` or something more complex ?

Let us think about what can be done.

Maybe I think the easiest way is to support a `callable` function with a default `lambda`?
```py
compare_fn: Callable = lambda (new, old): new > old
```

Thanks.

@Nic-Ma What should be the arguments of this callable ? I mean, are `priority` values good enough ? Or maybe it could be useful to have an extra argument, for instance the checkpoint object ?
HI @sdesrozis ,

Thanks for your support and good idea.
Could you pass `engine` as extra arg to the callable? Just pass the `engine` from `Checkpoint.__call__(self, engine)` to this callable? Users can use `engine.state` as a data-sharing object.
For example, maybe I can store the average value of last 10 validation metrics into `engine.state.average_metric` in another handler, then compare current metric with it to save model:
```py
compare_fn: Callable = lambda (new_priority, old_priority, engine): new_priority > engine.state.average_metric
```

Thanks.

A bit tricky thing to keep in mind is that `self._saved[0].priority` is sorted such that best model is the first. If we enable another rule, this should be also taken into account.

Anyway, I think this feature could be nice to have in ignite. @Nic-Ma would you like to send a draft PR for that ?
@vfdev-5 @sdesrozis , thanks for your support here.
Sure, I can try to make a PR next week if you guys are busy with other development tasks.

Thanks.
 

Resolving patch:
diff --git a/ignite/handlers/checkpoint.py b/ignite/handlers/checkpoint.py
--- a/ignite/handlers/checkpoint.py
+++ b/ignite/handlers/checkpoint.py
@@ -98,6 +98,8 @@ class Checkpoint(Serializable):
             details.
         include_self (bool): Whether to include the `state_dict` of this object in the checkpoint. If `True`, then
             there must not be another object in ``to_save`` with key ``checkpointer``.
+        greater_or_equal (bool): if `True`, the latest equally scored model is stored. Otherwise, the first model.
+            Default, `False`.
 
     .. _DistributedDataParallel: https://pytorch.org/docs/stable/generated/
         torch.nn.parallel.DistributedDataParallel.html
@@ -261,6 +263,7 @@ def __init__(
         global_step_transform: Optional[Callable] = None,
         filename_pattern: Optional[str] = None,
         include_self: bool = False,
+        greater_or_equal: bool = False,
     ) -> None:
 
         if to_save is not None:  # for compatibility with ModelCheckpoint
@@ -301,6 +304,7 @@ def __init__(
         self.filename_pattern = filename_pattern
         self._saved = []  # type: List["Checkpoint.Item"]
         self.include_self = include_self
+        self.greater_or_equal = greater_or_equal
 
     def reset(self) -> None:
         """Method to reset saved checkpoint names.
@@ -339,6 +343,12 @@ def _check_lt_n_saved(self, or_equal: bool = False) -> bool:
             return True
         return len(self._saved) < self.n_saved + int(or_equal)
 
+    def _compare_fn(self, new: Union[int, float]) -> bool:
+        if self.greater_or_equal:
+            return new >= self._saved[0].priority
+        else:
+            return new > self._saved[0].priority
+
     def __call__(self, engine: Engine) -> None:
 
         global_step = None
@@ -354,7 +364,7 @@ def __call__(self, engine: Engine) -> None:
                 global_step = engine.state.get_event_attrib_value(Events.ITERATION_COMPLETED)
             priority = global_step
 
-        if self._check_lt_n_saved() or self._saved[0].priority < priority:
+        if self._check_lt_n_saved() or self._compare_fn(priority):
 
             priority_str = f"{priority}" if isinstance(priority, numbers.Integral) else f"{priority:.4f}"
 


Test patch:
diff --git a/tests/ignite/handlers/test_checkpoint.py b/tests/ignite/handlers/test_checkpoint.py
--- a/tests/ignite/handlers/test_checkpoint.py
+++ b/tests/ignite/handlers/test_checkpoint.py
@@ -1526,3 +1526,36 @@ def test_checkpoint_reset_with_engine(dirname):
     expected += [f"{_PREFIX}_{name}_{i}.pt" for i in [1 * 2, 2 * 2]]
     assert sorted(os.listdir(dirname)) == sorted(expected)
     assert "PREFIX_model_4.pt" in handler.last_checkpoint
+
+
+def test_greater_or_equal():
+    scores = iter([1, 2, 2, 2])
+
+    def score_function(_):
+        return next(scores)
+
+    class Saver:
+        def __init__(self):
+            self.counter = 0
+
+        def __call__(self, c, f, m):
+            if self.counter == 0:
+                assert f == "model_1.pt"
+            else:
+                assert f == "model_2.pt"
+            self.counter += 1
+
+    handler = Saver()
+
+    checkpointer = Checkpoint(
+        to_save={"model": DummyModel()},
+        save_handler=handler,
+        score_function=score_function,
+        n_saved=2,
+        greater_or_equal=True,
+    )
+    trainer = Engine(lambda e, b: None)
+
+    for _ in range(4):
+        checkpointer(trainer)
+    assert handler.counter == 4