Repository: pytorch/ignite

Issue:
"=" in the checkpoint name may be incompatible with OSs
## ❓ Questions/Help/Support
Issue raised by Eric from MONAI team:
File names with operators in them may be valid names in Linux but that's not the case in other OSs, plus shells like bash really don't like them.


Discussion:
@Nic-Ma Thank you for this question. 

I don't know any problematic OS with `=` operator in filenames but even with Linux, you have to be careful and sometimes use quotes.

The filename policy is here
https://github.com/pytorch/ignite/blob/cc78b3bbd764da9ae7dee0df21676b2b8a50fba8/ignite/handlers/checkpoint.py#L93-L105

Maybe we could add a format based on regex that user could set (default is existing policy) ? 
Hi @sdesrozis ,

Thanks for your quick response.
A regex or user-defined separator sounds good, the default still can be "=".
Thanks.

Shall I take this up @sdesrozis 
@Joel-hanson thank you to contribute on this issue 😊

The idea is to replace the operator `=` in filename of checkpoints. To do it, we can introduce a separator attribute (default is `=`) as suggested by @Nic-Ma. A more sophisticated option is to provide a string pattern like `{filename_prefix}_{name}_{score_name}={score}.{ext}` and process it by replacing labels. 

What do you think?
How about setting this pattern as a class member of Checkpoint:
```python
class Checkpoint:

     filename_pattern="{filename_prefix}_{name}_{score_name}={score}.{ext}"
     
     ...
```

If user would like to change it, there is a possibliity.
@sdesrozis replacing labels seems like a good option and the suggestion by @vfdev-5 is worth considering. I will note down what I have in mind to be more clear before moving forward.

I will set the `filename_pattern="{filename_prefix}_{name}_{score_name}={score}.{ext}"` as a class member for the Checkpoint. The filename will have the default pattern if there is no change in the class member. If a user changes his filename_pattern, then by using str.format_map I will replace each label with its values.

When we implement likewise whenever a user provides the pattern to be `{filename_prefix}_{name}_{score_name}`, without having the either `{score}` or `{ext}` shall I  append the default pattern to it like `{filename_prefix}_{name}_{score_name}{score}.{ext}` so that the file doesn't get replaced or not get saved without extension. Is there anything that I am missing out on? correct me if I am wrong.
@Joel-hanson What you have in mind seems good 😉 str + format with fixed keywords (`score`, `ext`, etc.) looks good. It’s similar to what is done in `logging` format. It should be a good inspiration.

And you are right, we have multiple patterns depending of the usage context. IMO if user give a pattern, we should use it. Otherwise, legacy policy remains used. Let’s see if it fits class member approach ! What do you think ? 
> When we implement likewise whenever a user provides the pattern to be {filename_prefix}_{name}_{score_name}, without having the either {score} or {ext} shall I append the default pattern to it like {filename_prefix}_{name}_{score_name}{score}.{ext} so that the file doesn't get replaced or not get saved without extension.

@Joel-hanson let's consider that if user changes the pattern he/she knows what he/she is doing. 
So, no internal additional appends. For example, if pattern is 
```
p = "{name}_{score}"
kw = {"name": "abc", "filename_prefix": "pref", "score_name":"acc", "score": 0.234, "ext": "pt"}
p.format(**kw)
> 'abc_0.234'
```

Thanks, @sdesrozis @vfdev-5 for the quick response. So I shall start implementing based on what @vfdev-5 has conveyed.

@Joel-hanson any updates on this ? Can we help you somehow to start implementation and send a PR ?
Sorry @vfdev-5 was busy with some pending works will start tomorrow.

Resolving patch:
diff --git a/ignite/handlers/checkpoint.py b/ignite/handlers/checkpoint.py
--- a/ignite/handlers/checkpoint.py
+++ b/ignite/handlers/checkpoint.py
@@ -93,6 +93,9 @@ class Checkpoint(Serializable):
             Default is None, global_step based on attached engine. If provided, uses function output as global_step.
             To setup global step from another engine, please use :meth:`~ignite.handlers.global_step_from_engine`.
         archived (bool, optional): Deprecated argument as models saved by ``torch.save`` are already compressed.
+        filename_pattern (str, optional): If ``filename_pattern`` is provided, this pattern will be used to render
+            checkpoint filenames. If the pattern is not defined, the default pattern would be used. See Note for
+            details.
         include_self (bool): Whether to include the `state_dict` of this object in the checkpoint. If `True`, then
             there must not be another object in ``to_save`` with key ``checkpointer``.
 
@@ -101,32 +104,52 @@ class Checkpoint(Serializable):
 
     Note:
         This class stores a single file as a dictionary of provided objects to save.
-        The filename has the following structure: `{filename_prefix}_{name}_{suffix}.{ext}` where
+        The filename is defined by ``filename_pattern`` and by default has the following
+        structure: ``{filename_prefix}_{name}_{suffix}.{ext}`` where
 
         - ``filename_prefix`` is the argument passed to the constructor,
         - `name` is the key in ``to_save`` if a single object is to store, otherwise `name` is "checkpoint".
-        - `suffix` is composed as following `{global_step}_{score_name}={score}`.
+        - `suffix` is composed as following ``{global_step}_{score_name}={score}``.
+
+    +----------------+------------+-----------------------+----------------------------------------------+
+    | score_function | score_name | global_step_transform |  suffix                                      |
+    +================+============+=======================+==============================================+
+    |      None      |   None     |        None           | ``{engine.state.iteration}``                 |
+    +----------------+------------+-----------------------+----------------------------------------------+
+    |       X        |   None     |        None           | ``{score}``                                  |
+    +----------------+------------+-----------------------+----------------------------------------------+
+    |       X        |   None     |         X             | ``{global_step}_{score}``                    |
+    +----------------+------------+-----------------------+----------------------------------------------+
+    |       X        |    X       |         X             | ``{global_step}_{score_name}={score}``       |
+    +----------------+------------+-----------------------+----------------------------------------------+
+    |      None      |   None     |         X             | ``{global_step}``                            |
+    +----------------+------------+-----------------------+----------------------------------------------+
+    |       X        |    X       |        None           | ``{score_name}={score}``                     |
+    +----------------+------------+-----------------------+----------------------------------------------+
+
+    Above `global_step` defined by the output of `global_step_transform` and `score` defined by the output
+    of `score_function`.
+
+    By default, none of ``score_function``, ``score_name``, ``global_step_transform`` is defined, then suffix is
+    setup by attached engine's current iteration. The filename will be
+    `{filename_prefix}_{name}_{engine.state.iteration}.{ext}`.
+
+    For example, ``score_name="neg_val_loss"`` and ``score_function`` that returns `-loss` (as objects with highest
+    scores will be retained), then saved filename will be ``{filename_prefix}_{name}_neg_val_loss=-0.1234.pt``.
 
-        Above `global_step` defined by the output of `global_step_transform` and `score` defined by the output
-        of `score_function`.
-
-        By default, none of ``score_function``, ``score_name``, ``global_step_transform`` is defined, then suffix is
-        setup by attached engine's current iteration. The filename will be
-        `{filename_prefix}_{name}_{engine.state.iteration}.{ext}`.
-
-        If only ``global_step_transform`` is defined, then suffix is setup using its return value.
-        The filename will be `{filename_prefix}_{name}_{global_step}.{ext}`.
-
-        If defined a ``score_function``, but without ``score_name``, then suffix is defined by provided score.
-        The filename will be `{filename_prefix}_{name}_{global_step}_{score}.pt`.
+    Note:
+        If ``filename_pattern`` is given, it will be used to render the filenames. ``filename_pattern`` is a string
+        that can contain ``{filename_prefix}``, ``{name}``, ``{score}``, ``{score_name}`` and ``{global_step}`` as
+        templates.
 
-        If defined ``score_function`` and ``score_name``, then the filename will
-        be `{filename_prefix}_{name}_{score_name}={score}.{ext}`. If ``global_step_transform`` is provided, then
-        the filename will be `{filename_prefix}_{name}_{global_step}_{score_name}={score}.{ext}`
+        For example, let ``filename_pattern="{global_step}-{name}-{score}.pt"`` then the saved filename will be
+        ``30000-checkpoint-94.pt``
 
-        For example, ``score_name="neg_val_loss"`` and ``score_function`` that returns `-loss` (as objects with highest
-        scores will be retained), then saved filename will be `{filename_prefix}_{name}_neg_val_loss=-0.1234.pt`.
+        **Warning:** Please, keep in mind that if filename collide with already used one to saved a checkpoint,
+        new checkpoint will not be stored. This means that filename like ``checkpoint.pt`` will be saved only once
+        and will not be overwritten by newer checkpoints.
 
+    Note:
         To get the last stored filename, handler exposes attribute ``last_checkpoint``:
 
         .. code-block:: python
@@ -143,8 +166,8 @@ class Checkpoint(Serializable):
 
     .. warning::
 
-        When running on TPUs, it should be run in all processes, otherwise application can get stuck on saving the
-        checkpoint.
+        When running on XLA devices, it should be run in all processes, otherwise application can get stuck on
+        saving the checkpoint.
 
         .. code-block:: python
 
@@ -219,6 +242,7 @@ def __init__(
         n_saved: Optional[int] = 1,
         global_step_transform: Callable = None,
         archived: bool = False,
+        filename_pattern: Optional[str] = None,
         include_self: bool = False,
     ):
 
@@ -254,14 +278,15 @@ def __init__(
             warnings.warn("Argument archived is deprecated and will be removed in 0.5.0")
 
         self.to_save = to_save
-        self._fname_prefix = filename_prefix + "_" if len(filename_prefix) > 0 else filename_prefix
+        self.filename_prefix = filename_prefix
         self.save_handler = save_handler
-        self._score_function = score_function
-        self._score_name = score_name
-        self._n_saved = n_saved
-        self._saved = []
-        self._ext = ".pt"
+        self.score_function = score_function
+        self.score_name = score_name
+        self.n_saved = n_saved
+        self.ext = "pt"
         self.global_step_transform = global_step_transform
+        self.filename_pattern = filename_pattern
+        self._saved = []
         self.include_self = include_self
 
     @property
@@ -271,27 +296,24 @@ def last_checkpoint(self) -> Optional[str]:
         return self._saved[-1].filename
 
     def _check_lt_n_saved(self, or_equal=False):
-        if self._n_saved is None:
+        if self.n_saved is None:
             return True
-        return len(self._saved) < self._n_saved + int(or_equal)
+        return len(self._saved) < self.n_saved + int(or_equal)
 
     def __call__(self, engine: Engine) -> None:
 
-        suffix = ""
         global_step = None
         if self.global_step_transform is not None:
             global_step = self.global_step_transform(engine, engine.last_event_name)
-            suffix = "{}".format(global_step)
 
-        if self._score_function is not None:
-            priority = self._score_function(engine)
+        if self.score_function is not None:
+            priority = self.score_function(engine)
             if not isinstance(priority, numbers.Number):
                 raise ValueError("Output of score_function should be a number")
         else:
-            if global_step is not None:
-                priority = global_step
-            else:
-                priority = engine.state.get_event_attrib_value(Events.ITERATION_COMPLETED)
+            if global_step is None:
+                global_step = engine.state.get_event_attrib_value(Events.ITERATION_COMPLETED)
+            priority = global_step
 
         if self._check_lt_n_saved() or self._saved[0].priority < priority:
 
@@ -299,17 +321,6 @@ def __call__(self, engine: Engine) -> None:
                 "{}".format(priority) if isinstance(priority, numbers.Integral) else "{:.4f}".format(priority)
             )
 
-            if self._score_name is not None:
-                if len(suffix) > 0:
-                    suffix += "_"
-                suffix = "{}{}={}".format(suffix, self._score_name, priority_str)
-            elif self._score_function is not None:
-                if len(suffix) > 0:
-                    suffix += "_"
-                suffix = "{}{}".format(suffix, priority_str)
-            elif len(suffix) == 0:
-                suffix = "{}".format(priority_str)
-
             checkpoint = self._setup_checkpoint()
 
             name = "checkpoint"
@@ -317,14 +328,33 @@ def __call__(self, engine: Engine) -> None:
                 for k in checkpoint:
                     name = k
                 checkpoint = checkpoint[name]
-            filename = "{}{}_{}{}".format(self._fname_prefix, name, suffix, self._ext)
+
+            if self.filename_pattern is None:
+                filename_pattern = self.setup_filename_pattern(
+                    with_prefix=len(self.filename_prefix) > 0,
+                    with_score=self.score_function is not None,
+                    with_score_name=self.score_name is not None,
+                    with_global_step=global_step is not None,
+                )
+            else:
+                filename_pattern = self.filename_pattern
+
+            filename_dict = {
+                "filename_prefix": self.filename_prefix,
+                "ext": self.ext,
+                "name": name,
+                "score_name": self.score_name,
+                "score": priority_str if (self.score_function is not None) else None,
+                "global_step": global_step,
+            }
+            filename = filename_pattern.format(**filename_dict)
 
             if any(item.filename == filename for item in self._saved):
                 return
 
             metadata = {
-                "basename": "{}{}".format(self._fname_prefix, name),
-                "score_name": self._score_name,
+                "basename": "{}{}{}".format(self.filename_prefix, "_" * int(len(self.filename_prefix) > 0), name),
+                "score_name": self.score_name,
                 "priority": priority,
             }
 
@@ -353,6 +383,56 @@ def _setup_checkpoint(self) -> dict:
             checkpoint[k] = obj.state_dict()
         return checkpoint
 
+    @staticmethod
+    def setup_filename_pattern(
+        with_prefix: bool = True, with_score: bool = True, with_score_name: bool = True, with_global_step: bool = True,
+    ) -> str:
+        """Helper method to get the default filename pattern for a checkpoint.
+
+        Args:
+            with_prefix (bool): If True, the ``filename_prefix`` is added to the filename pattern:
+                ``{filename_prefix}_{name}...``. Default, True.
+            with_score (bool): If True, ``score`` is added to the filename pattern: ``..._{score}.{ext}``.
+                Default, True. At least one of ``with_score`` and ``with_global_step`` should be True.
+            with_score_name (bool): If True, ``score_name`` is added to the filename pattern:
+                ``..._{score_name}={score}.{ext}``. If activated, argument ``with_score`` should be
+                also True, otherwise an error is raised. Default, True.
+            with_global_step (bool): If True, ``{global_step}`` is added to the
+                filename pattern: ``...{name}_{global_step}...``.
+                At least one of ``with_score`` and ``with_global_step`` should be True.
+
+        Example:
+
+            .. code-block:: python
+
+                from ignite.handlers import Checkpoint
+
+                filename_pattern = Checkpoint.setup_filename_pattern()
+
+                print(filename_pattern)
+                > "{filename_prefix}_{name}_{global_step}_{score_name}={score}.{ext}"
+        """
+        filename_pattern = "{name}"
+
+        if not (with_global_step or with_score):
+            raise ValueError("At least one of with_score and with_global_step should be True.")
+
+        if with_global_step:
+            filename_pattern += "_{global_step}"
+
+        if with_score_name and with_score:
+            filename_pattern += "_{score_name}={score}"
+        elif with_score:
+            filename_pattern += "_{score}"
+        elif with_score_name:
+            raise ValueError("If with_score_name is True, with_score should be also True")
+
+        if with_prefix:
+            filename_pattern = "{filename_prefix}_" + filename_pattern
+
+        filename_pattern += ".{ext}"
+        return filename_pattern
+
     @staticmethod
     def _check_objects(objs: Mapping, attr: str) -> None:
         for k, obj in objs.items():


Test patch:
diff --git a/tests/ignite/handlers/test_checkpoint.py b/tests/ignite/handlers/test_checkpoint.py
--- a/tests/ignite/handlers/test_checkpoint.py
+++ b/tests/ignite/handlers/test_checkpoint.py
@@ -1211,6 +1211,167 @@ def test_distrib_single_device_xla_nprocs(xmp_executor, dirname):
     xmp_executor(_test_tpu_saves_to_cpu_nprocs, args=(dirname,), nprocs=n)
 
 
+def test_checkpoint_filename_pattern():
+    def _test(
+        to_save,
+        filename_prefix="",
+        score_function=None,
+        score_name=None,
+        global_step_transform=None,
+        filename_pattern=None,
+    ):
+        save_handler = MagicMock(spec=BaseSaveHandler)
+
+        checkpointer = Checkpoint(
+            to_save,
+            save_handler=save_handler,
+            filename_prefix=filename_prefix,
+            score_function=score_function,
+            score_name=score_name,
+            global_step_transform=global_step_transform,
+            filename_pattern=filename_pattern,
+        )
+
+        trainer = Engine(lambda e, b: None)
+        trainer.state = State(epoch=12, iteration=203, score=0.9999)
+
+        checkpointer(trainer)
+        return checkpointer.last_checkpoint
+
+    model = DummyModel()
+    to_save = {"model": model}
+
+    assert _test(to_save) == "model_203.pt"
+    assert _test(to_save, "best") == "best_model_203.pt"
+    assert _test(to_save, score_function=lambda e: e.state.score) == "model_0.9999.pt"
+
+    res = _test(to_save, score_function=lambda e: e.state.score, global_step_transform=lambda e, _: e.state.epoch)
+    assert res == "model_12_0.9999.pt"
+
+    assert _test(to_save, score_function=lambda e: e.state.score, score_name="acc") == "model_acc=0.9999.pt"
+
+    res = _test(
+        to_save,
+        score_function=lambda e: e.state.score,
+        score_name="acc",
+        global_step_transform=lambda e, _: e.state.epoch,
+    )
+    assert res == "model_12_acc=0.9999.pt"
+
+    assert _test(to_save, "best", score_function=lambda e: e.state.score) == "best_model_0.9999.pt"
+
+    res = _test(
+        to_save, "best", score_function=lambda e: e.state.score, global_step_transform=lambda e, _: e.state.epoch
+    )
+    assert res == "best_model_12_0.9999.pt"
+
+    res = _test(to_save, "best", score_function=lambda e: e.state.score, score_name="acc")
+    assert res == "best_model_acc=0.9999.pt"
+
+    res = _test(
+        to_save,
+        "best",
+        score_function=lambda e: e.state.score,
+        score_name="acc",
+        global_step_transform=lambda e, _: e.state.epoch,
+    )
+    assert res == "best_model_12_acc=0.9999.pt"
+
+    pattern = "chk-{name}--{global_step}.{ext}"
+    assert _test(to_save, to_save, filename_pattern=pattern) == "chk-model--203.pt"
+    pattern = "chk-{filename_prefix}--{name}--{global_step}.{ext}"
+    assert _test(to_save, "best", filename_pattern=pattern) == "chk-best--model--203.pt"
+    pattern = "chk-{name}--{score}.{ext}"
+    assert _test(to_save, score_function=lambda e: e.state.score, filename_pattern=pattern) == "chk-model--0.9999.pt"
+    pattern = "{global_step}-{name}-{score}.chk.{ext}"
+    res = _test(
+        to_save,
+        score_function=lambda e: e.state.score,
+        global_step_transform=lambda e, _: e.state.epoch,
+        filename_pattern=pattern,
+    )
+    assert res == "12-model-0.9999.chk.pt"
+
+    pattern = "chk-{name}--{score_name}--{score}.{ext}"
+    res = _test(to_save, score_function=lambda e: e.state.score, score_name="acc", filename_pattern=pattern)
+    assert res == "chk-model--acc--0.9999.pt"
+
+    pattern = "chk-{name}-{global_step}-{score_name}-{score}.{ext}"
+    res = _test(
+        to_save,
+        score_function=lambda e: e.state.score,
+        score_name="acc",
+        global_step_transform=lambda e, _: e.state.epoch,
+        filename_pattern=pattern,
+    )
+    assert res == "chk-model-12-acc-0.9999.pt"
+
+    pattern = "{filename_prefix}-{name}-{score}.chk"
+    res = _test(to_save, "best", score_function=lambda e: e.state.score, filename_pattern=pattern)
+    assert res == "best-model-0.9999.chk"
+
+    pattern = "resnet-{filename_prefix}-{name}-{global_step}-{score}.chk"
+    res = _test(
+        to_save,
+        "best",
+        score_function=lambda e: e.state.score,
+        global_step_transform=lambda e, _: e.state.epoch,
+        filename_pattern=pattern,
+    )
+    assert res == "resnet-best-model-12-0.9999.chk"
+
+    pattern = "{filename_prefix}-{name}-{score_name}-{score}.chk"
+    res = _test(to_save, "best", score_function=lambda e: e.state.score, score_name="acc", filename_pattern=pattern)
+    assert res == "best-model-acc-0.9999.chk"
+
+    pattern = "{global_step}-{filename_prefix}-{name}-{score_name}-{score}"
+    res = _test(
+        to_save,
+        "best",
+        score_function=lambda e: e.state.score,
+        score_name="acc",
+        global_step_transform=lambda e, _: e.state.epoch,
+        filename_pattern=pattern,
+    )
+    assert res == "12-best-model-acc-0.9999"
+
+    pattern = "SAVE:{name}-{score_name}-{score}.pth"
+    res = _test(
+        to_save,
+        "best",
+        score_function=lambda e: e.state.score,
+        score_name="acc",
+        global_step_transform=lambda e, _: e.state.epoch,
+        filename_pattern=pattern,
+    )
+
+    assert res == "SAVE:model-acc-0.9999.pth"
+
+    pattern = "{global_step}-chk-{filename_prefix}-{name}-{score_name}-{score}.{ext}"
+    assert _test(to_save, filename_pattern=pattern) == "203-chk--model-None-None.pt"
+
+    with pytest.raises(KeyError, match=r"random_key"):
+        pattern = "SAVE:{random_key}.{ext}"
+        _test(to_save, filename_pattern=pattern)
+
+
+def test_setup_filename_pattern():
+    # default filename pattern
+    assert Checkpoint.setup_filename_pattern() == "{filename_prefix}_{name}_{global_step}_{score_name}={score}.{ext}"
+
+    assert Checkpoint.setup_filename_pattern(False) == "{name}_{global_step}_{score_name}={score}.{ext}"
+    assert Checkpoint.setup_filename_pattern(False, False, False) == "{name}_{global_step}.{ext}"
+    assert Checkpoint.setup_filename_pattern(False, True, False) == "{name}_{global_step}_{score}.{ext}"
+    assert Checkpoint.setup_filename_pattern(False, True, False, False) == "{name}_{score}.{ext}"
+    assert Checkpoint.setup_filename_pattern(False, True, True, False) == "{name}_{score_name}={score}.{ext}"
+
+    with pytest.raises(ValueError, match=r"At least one of with_score and with_global_step should be True."):
+        Checkpoint.setup_filename_pattern(False, False, False, False)
+
+    with pytest.raises(ValueError, match=r"If with_score_name is True, with_score should be also True"):
+        Checkpoint.setup_filename_pattern(True, False, True, True)
+
+
 def _setup_checkpoint():
     save_handler = MagicMock(spec=BaseSaveHandler)
     model = DummyModel()