Repository: Lightning-AI/pytorch-lightning

Issue:
ModelCheckpoint monitor should be optional
## 🚀 Feature

We should be able to set `monitor = None`, meaning that it would always save

### Motivation

Current behaviour forces you to have a monitor, so to always save you would have to monitor something like the current epoch.


Discussion:
how would you know when to save without monitoring something?

I personally wouldn't just want a checkpoint saved on every epoch
Maybe just by itself is a bit useless, but combined with `period` one can have a "every_k_epochs" `ModelCheckpoint`
well, generally checkpoints correspond to some sort of improvement in performance...

there is a (rare) case (image synthesis, speech synthesis), etc... where longer training does make more sense, in which case saving the last checkpoint makes sense.

in this case though, the answer is just that whenever save last is enabled, it always saves without having to monitor something?

There is something to be said about the default behavior of saving checkpoint on every epoch by default until the user monitors a specific metric. that might not be a bad option (thoughts @ananthsub ?)

Couldn't you set `save_top_k = -1`? Then we could ignore `monitor`
Not really, because you still would have to give a placeholder monitor.

Currently, if monitor is None, this line fails:

https://github.com/PyTorchLightning/pytorch-lightning/blob/c61e1e697d2e4ed5cc97c187e6704a44f2f77aff/pytorch_lightning/callbacks/model_checkpoint.py#L177

And if you give it a placeholder monitor `foo` (even with `save_top_k = -1`) it has to be present in the logged metrics. see:

https://github.com/PyTorchLightning/pytorch-lightning/blob/c61e1e697d2e4ed5cc97c187e6704a44f2f77aff/pytorch_lightning/callbacks/model_checkpoint.py#L340-L345

https://github.com/PyTorchLightning/pytorch-lightning/blob/c61e1e697d2e4ed5cc97c187e6704a44f2f77aff/pytorch_lightning/callbacks/model_checkpoint.py#L414-L415
> well, generally checkpoints correspond to some sort of improvement in performance...

They do correspond to that right now, but we should see them just as snapshot takers.

When we support having multiple checkpoint callbacks, if you are training for a long time you might want to take snapshots of your model not only on performance improvements but also every N epochs (regardless of performance). For example, to have an history of how the model improved.

This issue is just one step forward to support that.
@carmocca with your None approach, what will the value of best_model_path (forgot exact name) in checkpoint be? Can these checkpoints be properly loaded and resumed? I'm just trying to look for edge cases here. 

@ananthsub @carmocca on the setting top_k=-1, yes it would save all checkpoints, but it is an important detail that the name "topk" would imply that we have a sorting based on the monitor. So even with top_k=-1 the monitor would be necessary. So @carmocca 's proposal with None does make sense to me at least for now :)
>  what will the value of best_model_path (forgot exact name) in checkpoint be?

https://github.com/PyTorchLightning/pytorch-lightning/blob/f05244e036c0cf5c81623d4689d5a12815cab7cc/tests/callbacks/test_model_checkpoint.py#L219-L223

Resolving patch:
diff --git a/pytorch_lightning/callbacks/model_checkpoint.py b/pytorch_lightning/callbacks/model_checkpoint.py
--- a/pytorch_lightning/callbacks/model_checkpoint.py
+++ b/pytorch_lightning/callbacks/model_checkpoint.py
@@ -36,7 +36,7 @@
 
 class ModelCheckpoint(Callback):
     r"""
-    Save the model after every epoch if it improves.
+    Save the model after every epoch by monitoring a quantity.
 
     After training finishes, use :attr:`best_model_path` to retrieve the path to the
     best checkpoint file and :attr:`best_model_score` to retrieve its score.
@@ -63,7 +63,7 @@ class ModelCheckpoint(Callback):
             :paramref:`~pytorch_lightning.trainer.trainer.Trainer.weights_save_path` arguments,
             and if the Trainer uses a logger, the path will also contain logger name and version.
 
-        monitor: quantity to monitor.
+        monitor: quantity to monitor. If None, a checkpoint will be saved every epoch.
         verbose: verbosity mode. Default: ``False``.
         save_last: always saves the model at the end of the epoch. Default: ``False``.
         save_top_k: if ``save_top_k == k``,
@@ -120,7 +120,7 @@ class ModelCheckpoint(Callback):
     def __init__(
         self,
         filepath: Optional[str] = None,
-        monitor: str = "checkpoint_on",
+        monitor: Optional[str] = "checkpoint_on",
         verbose: bool = False,
         save_last: bool = False,
         save_top_k: int = 1,
@@ -174,7 +174,7 @@ def __init__(
             "min": (torch_inf, "min"),
             "max": (-torch_inf, "max"),
             "auto": (-torch_inf, "max")
-            if "acc" in self.monitor or self.monitor.startswith("fmeasure")
+            if monitor is not None and ("acc" in monitor or monitor.startswith("fmeasure"))
             else (torch_inf, "min"),
         }
 
@@ -337,11 +337,11 @@ def on_validation_end(self, trainer, pl_module):
         epoch = trainer.current_epoch
 
         # validate metric
-        if not self._is_valid_monitor_key(metrics):
-            keys = list(metrics.keys())
-            m = f"""
-                ModelCheckpoint(monitor='{self.monitor}') not found in the returned metrics ({keys}),
-                "did you call result.log(f'{self.monitor}', tensor)?"""
+        if not (self.monitor is None or self._is_valid_monitor_key(metrics)):
+            m = (
+                f"ModelCheckpoint(monitor='{self.monitor}') not found in the returned metrics:"
+                f" {list(metrics.keys())}. HINT: Did you call result.log('{self.monitor}', tensor)?"
+            )
             raise MisconfigurationException(m)
 
         if (


Test patch:
diff --git a/tests/callbacks/test_model_checkpoint.py b/tests/callbacks/test_model_checkpoint.py
--- a/tests/callbacks/test_model_checkpoint.py
+++ b/tests/callbacks/test_model_checkpoint.py
@@ -204,6 +204,30 @@ def test_model_checkpoint_save_last_checkpoint_contents(tmpdir):
         assert w0.eq(w1).all()
 
 
+def test_model_checkpoint_none_monitor(tmpdir):
+    model = EvalModelTemplate()
+    epochs = 2
+    checkpoint_callback = ModelCheckpoint(filepath=tmpdir, monitor=None, save_top_k=-1)
+    trainer = Trainer(
+        default_root_dir=tmpdir,
+        early_stop_callback=False,
+        checkpoint_callback=checkpoint_callback,
+        max_epochs=epochs,
+    )
+    trainer.fit(model)
+
+    # these should not be set if monitor is None
+    assert checkpoint_callback.best_model_path == ''
+    assert checkpoint_callback.best_model_score == 0
+    assert checkpoint_callback.best_k_models == {}
+    assert checkpoint_callback.kth_best_model_path == ''
+
+    # check that the correct ckpts were created
+    expected = ['lightning_logs']
+    expected.extend(f'epoch={e}.ckpt' for e in range(epochs))
+    assert set(os.listdir(tmpdir)) == set(expected)
+
+
 def test_ckpt_metric_names(tmpdir):
     model = EvalModelTemplate()