Repository: Project-MONAI/MONAI

Issue:
CheckpointSaver: incorrect labelling of final model
**Describe the bug**
When using a `CheckpointSaver` among the validation handlers, the argument `save_final=True` results in every epoch being stored under `net_final_iteration=1.pt` when `final_filename=None`. I interpret the docs (`default to checkpoint_final_iteration=N.pt`) that the file will be renamed after each completed epoch.

**To Reproduce**
add `save_final=True` to the `CheckpointHandler` in the Lung Segmentation Tutorial

**Expected behavior**
`net_final_iteration=1.pt` ->  `net_final_iteration=2.pt` -> ... ->  `net_final_iteration=MAX_EPOCHS.pt`

**Screenshots**
The logging function also prints incorrect iteration numbers:

```
INFO:ignite.engine.engine.SupervisedEvaluator:Train completed, saved final checkpoint: net_final_iteration=1.pt
...
INFO:ignite.engine.engine.SupervisedEvaluator:Train completed, saved final checkpoint: net_final_iteration=1.pt
...
INFO:ignite.engine.engine.SupervisedEvaluator:Train completed, saved final checkpoint: net_final_iteration=1.pt
```

**Environment**

Ensuring you use the relevant python executable, please paste the output of:

```
MONAI version: 0.3.0+95.g535561e
Python version: 3.8.6 | packaged by conda-forge | (default, Oct  7 2020, 18:42:56)  [Clang 10.0.1 ]
OS version: Darwin (19.6.0)
Numpy version: 1.19.2
Pytorch version: 1.7.0
MONAI flags: HAS_EXT = False, USE_COMPILED = False

Optional dependencies:
Pytorch Ignite version: 0.4.2
Nibabel version: 3.2.0
scikit-image version: NOT INSTALLED or UNKNOWN VERSION.
Pillow version: 8.0.1
Tensorboard version: 2.4.0
gdown version: NOT INSTALLED or UNKNOWN VERSION.
TorchVision version: 0.8.1
ITK version: NOT INSTALLED or UNKNOWN VERSION.
tqdm version: 4.53.0
lmdb version: NOT INSTALLED or UNKNOWN VERSION.

For details about installing the optional dependencies, please visit:
    https://docs.monai.io/en/latest/installation.html#installing-the-recommended-dependencies
```

**Additional context**
Using `SupervisedEvaluator` and `SupervisedTrainer`.


Discussion:

Resolving patch:
diff --git a/monai/handlers/checkpoint_saver.py b/monai/handlers/checkpoint_saver.py
--- a/monai/handlers/checkpoint_saver.py
+++ b/monai/handlers/checkpoint_saver.py
@@ -188,6 +188,13 @@ def attach(self, engine: Engine) -> None:
             else:
                 engine.add_event_handler(Events.ITERATION_COMPLETED(every=self.save_interval), self.interval_completed)
 
+    def _delete_previous_final_ckpt(self):
+        saved = self._final_checkpoint._saved
+        if len(saved) > 0:
+            item = saved.pop(0)
+            self._final_checkpoint.save_handler.remove(item.filename)
+            self.logger.info(f"Deleted previous saved final checkpoint: {item.filename}")
+
     def completed(self, engine: Engine) -> None:
         """Callback for train or validation/evaluation completed Event.
         Save final checkpoint if configure save_final is True.
@@ -196,6 +203,8 @@ def completed(self, engine: Engine) -> None:
             engine: Ignite Engine, it can be a trainer, validator or evaluator.
         """
         assert callable(self._final_checkpoint), "Error: _final_checkpoint function not specified."
+        # delete previous saved final checkpoint if existing
+        self._delete_previous_final_ckpt()
         self._final_checkpoint(engine)
         assert self.logger is not None
         assert hasattr(self.logger, "info"), "Error, provided logger has not info attribute."
@@ -211,6 +220,8 @@ def exception_raised(self, engine: Engine, e: Exception) -> None:
             e: the exception caught in Ignite during engine.run().
         """
         assert callable(self._final_checkpoint), "Error: _final_checkpoint function not specified."
+        # delete previous saved final checkpoint if existing
+        self._delete_previous_final_ckpt()
         self._final_checkpoint(engine)
         assert self.logger is not None
         assert hasattr(self.logger, "info"), "Error, provided logger has not info attribute."


Test patch:
diff --git a/tests/test_handler_checkpoint_saver.py b/tests/test_handler_checkpoint_saver.py
--- a/tests/test_handler_checkpoint_saver.py
+++ b/tests/test_handler_checkpoint_saver.py
@@ -117,6 +117,7 @@ def _train_func(engine, batch):
                 n_saved,
             )
             handler.attach(engine)
+            engine.run(data, max_epochs=2)
             engine.run(data, max_epochs=5)
             for filename in filenames:
                 self.assertTrue(os.path.exists(os.path.join(tempdir, filename)))