Repository: Lightning-AI/pytorch-lightning

Issue:
Best model path is None and best model score is always 0 when using save_top_k=-1
I am currently using lightning `0.8.4` and configuring model checkpoint and doing the training as described in the docs. However, the checkpoint `best_model_path` is always None and the `best_model_sccore` is 0.

Here is my usage:

```
checkpoint_callback = ModelCheckpoint(filepath=hparams['chkpt_dir'],
                                          save_top_k=-1)
trainer = Trainer(gpus=1,
                          distributed_backend='dp',
                          logger=logger,
                          early_stop_callback=early_stop_callback,
                          checkpoint_callback=checkpoint_callback,
                          min_epochs=1, max_epochs=50)
model = MyNet(params)
trainer.fit(model)
print('Best path: ', checkpoint_callback.best_model_path)  # None
print('Best score: ', checkpoint_callback.best_model_score)  # 0
```

I can confirm the model does indeed train. Here is the console output:

```
Validation sanity check: 100%|████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:02<00:00,  2.00s/it]Mean Val loss:  0.9996303915977478
Epoch 1: 100%|█████████████████████████████████████████████████████████████████████████████████████▉| 692/693 [02:06<00:00,  5.47it/s, loss=0.222, v_num=20]Mean Val loss:  0.21550895273685455█████████████████████████████████████████████████████████████████████████████▍             | 7/8 [00:03<00:00,  2.55it/s]
Epoch 2: 100%|███████████████████████████████████████████████████████| 693/693 [01:06<00:00, 10.39it/s, loss=0.176, v_num=20, validation_loss=0.216, step=0]Mean Val loss:  0.18335537612438202█████████████████████████████████████████████████████████████████████████████▍             | 7/8 [00:03<00:00,  2.50it/s]
Epoch 3: 100%|███████████████████████████████████████████████████████| 693/693 [01:06<00:00, 10.47it/s, loss=0.156, v_num=20, validation_loss=0.183, step=1]Mean Val loss:  0.1659010648727417██████████████████████████████████████████████████████████████████████████████▍             | 7/8 [00:03<00:00,  2.67it/s]
Epoch 4: 100%|███████████████████████████████████████████████████████| 693/693 [01:06<00:00, 10.46it/s, loss=0.119, v_num=20, validation_loss=0.166, step=2]Mean Val loss:  0.13563179969787598█████████████████████████████████████████████████████████████████████████████▍             | 7/8 [00:03<00:00,  2.66it/s]
Epoch 5: 100%|███████████████████████████████████████████████████████| 693/693 [01:06<00:00, 10.40it/s, loss=0.120, v_num=20, validation_loss=0.136, step=3]Mean Val loss:  0.13059481978416443█████████████████████████████████████████████████████████████████████████████▍             | 7/8 [00:03<00:00,  2.60it/s]
Epoch 6: 100%|███████████████████████████████████████████████████████| 693/693 [01:05<00:00, 10.51it/s, loss=0.094, v_num=20, validation_loss=0.131, step=4]Mean Val loss:  0.11244599521160126█████████████████████████████████████████████████████████████████████████████▍             | 7/8 [00:03<00:00,  2.59it/s]
Epoch 7: 100%|███████████████████████████████████████████████████████| 693/693 [01:06<00:00, 10.43it/s, loss=0.089, v_num=20, validation_loss=0.112, step=5]Mean Val loss:  0.10584709048271179█████████████████████████████████████████████████████████████████████████████▍             | 7/8 [00:03<00:00,  2.31it/s]
Epoch 8: 100%|███████████████████████████████████████████████████████| 693/693 [01:06<00:00, 10.43it/s, loss=0.081, v_num=20, validation_loss=0.106, step=6]Mean Val loss:  0.09358386695384979█████████████████████████████████████████████████████████████████████████████▍             | 7/8 [00:03<00:00,  2.61it/s]
Epoch 9: 100%|██████████████████████████████████████████████████████| 693/693 [01:05<00:00, 10.55it/s, loss=0.072, v_num=20, validation_loss=0.0936, step=7]Mean Val loss:  0.09778335690498352█████████████████████████████████████████████████████████████████████████████▍             | 7/8 [00:02<00:00,  2.73it/s]
Epoch 10: 100%|█████████████████████████████████████████████████████| 693/693 [01:06<00:00, 10.46it/s, loss=0.077, v_num=20, validation_loss=0.0978, step=8]Mean Val loss:  0.11586500704288483█████████████████████████████████████████████████████████████████████████████▍             | 7/8 [00:03<00:00,  2.61it/s]
Epoch 11: 100%|██████████████████████████████████████████████████████| 693/693 [01:05<00:00, 10.52it/s, loss=0.072, v_num=20, validation_loss=0.116, step=9]Mean Val loss:  0.08725202083587646█████████████████████████████████████████████████████████████████████████████▍             | 7/8 [00:02<00:00,  2.76it/s]
Epoch 12: 100%|████████████████████████████████████████████████████| 693/693 [01:06<00:00, 10.49it/s, loss=0.072, v_num=20, validation_loss=0.0873, step=10]Mean Val loss:  0.1264028400182724██████████████████████████████████████████████████████████████████████████████▍             | 7/8 [00:03<00:00,  2.68it/s]
Epoch 13: 100%|█████████████████████████████████████████████████████| 693/693 [01:06<00:00, 10.48it/s, loss=0.067, v_num=20, validation_loss=0.126, step=11]Mean Val loss:  0.10011263191699982█████████████████████████████████████████████████████████████████████████████▍             | 7/8 [00:03<00:00,  2.70it/s]
Epoch 14: 100%|███████████████████████████████████████████████████████| 693/693 [01:05<00:00, 10.60it/s, loss=0.073, v_num=20, validation_loss=0.1, step=12]Mean Val loss:  0.08872538805007935█████████████████████████████████████████████████████████████████████████████▍             | 7/8 [00:02<00:00,  2.81it/s]
Epoch 14: 100%|████████████████████████████████████████████████████| 693/693 [01:05<00:00, 10.59it/s, loss=0.073, v_num=20, validation_loss=0.0887, step=13Epoch 00014: early stopping triggered.
Epoch 14: 100%|████████████████████████████████████████████████████| 693/693 [01:05<00:00, 10.58it/s, loss=0.073, v_num=20, validation_loss=0.0887, step=13]
Best path:
Best score:  0
```



Discussion:
Tried on 0.8.4. Working for me. Can you share a colab notebook with code??

![Screenshot from 2020-07-12 16-43-04](https://user-images.githubusercontent.com/30778939/87244957-ee2b3380-c45e-11ea-97d6-36b4afff934d.png)

ok, so this only happens when ` save_top_k=-1`. If I do ` save_top_k=1`, this seems to work. Is that expected? Surely the utility of this method only comes into play when models besides the best one are also saved?

@williamFalcon 
Yeah! not working when `save_top_k=-1`. I think it should work regardless of `save_top_k` value. The problem is here:
https://github.com/PyTorchLightning/pytorch-lightning/blob/1d565e175d98103c2ebd6164e681f76143501da9/pytorch_lightning/callbacks/model_checkpoint.py#L293
This is called only when the value is `save_top_k=-1`:
https://github.com/PyTorchLightning/pytorch-lightning/blob/1d565e175d98103c2ebd6164e681f76143501da9/pytorch_lightning/callbacks/model_checkpoint.py#L309
And this is updated in `self._do_check_save`:
https://github.com/PyTorchLightning/pytorch-lightning/blob/1d565e175d98103c2ebd6164e681f76143501da9/pytorch_lightning/callbacks/model_checkpoint.py#L337-L339
@williamFalcon Should I file a more detailed bug report about this?
*bump*: Is there any interest in this?
@pamparana34 sorry for the delay, mind test actual master, and if it is still there share a complete example to reproduce...
Ideally, an example that can be also used as a test to prevent such behavior in the future...
@Borda this is still an issue.
@pamparana34 could you send us a failing test case with your example?

Resolving patch:
diff --git a/pytorch_lightning/callbacks/model_checkpoint.py b/pytorch_lightning/callbacks/model_checkpoint.py
--- a/pytorch_lightning/callbacks/model_checkpoint.py
+++ b/pytorch_lightning/callbacks/model_checkpoint.py
@@ -212,10 +212,7 @@ def save_checkpoint(self, trainer, pl_module):
 
         # Mode 2: save all checkpoints OR only the top k
         if self.save_top_k:
-            if self.save_top_k == -1:
-                self._save_all_checkpoints(trainer, pl_module, epoch, filepath)
-            else:
-                self._save_top_k_checkpoints(monitor_candidates, trainer, pl_module, epoch, filepath)
+            self._save_top_k_checkpoints(monitor_candidates, trainer, pl_module, epoch, filepath)
 
     def __validate_init_configuration(self):
         if self.save_top_k is not None and self.save_top_k < -1:
@@ -297,6 +294,9 @@ def _save_model(self, filepath: str, trainer, pl_module):
             raise ValueError(".save_function() not set")
 
     def check_monitor_top_k(self, current) -> bool:
+        if self.save_top_k == -1:
+            return True
+
         less_than_k_models = len(self.best_k_models) < self.save_top_k
         if less_than_k_models:
             return True
@@ -477,8 +477,7 @@ def _save_top_k_checkpoints(self, metrics, trainer, pl_module, epoch, filepath):
         current = metrics.get(self.monitor)
 
         if not isinstance(current, torch.Tensor) and current is not None:
-            if current is not None:
-                current = torch.tensor(current).to(pl_module.device)
+            current = torch.tensor(current, device=pl_module.device)
 
         if current is None:
             m = f"Can save best model only with {self.monitor} available, skipping."
@@ -489,23 +488,16 @@ def _save_top_k_checkpoints(self, metrics, trainer, pl_module, epoch, filepath):
                 )
             rank_zero_warn(m, RuntimeWarning)
         elif self.check_monitor_top_k(current):
-            self._do_check_save(filepath, current, epoch, trainer, pl_module)
+            self._update_best_and_save(filepath, current, epoch, trainer, pl_module)
         elif self.verbose:
             log.info(
                 f"Epoch {epoch:d}: {self.monitor} was not in top {self.save_top_k}"
             )
 
-    def _save_all_checkpoints(self, trainer, pl_module, epoch, filepath):
-        if self.verbose:
-            log.info(f"Epoch {epoch:d}: saving model to {filepath}")
-
-        assert (trainer.global_rank == 0), "tried to make a checkpoint from non global_rank=0"
-        self._save_model(filepath, trainer, pl_module)
-
     def _is_valid_monitor_key(self, metrics):
         return self.monitor in metrics or len(metrics) == 0
 
-    def _do_check_save(
+    def _update_best_and_save(
         self,
         filepath: str,
         current: torch.Tensor,
@@ -513,16 +505,17 @@ def _do_check_save(
         trainer,
         pl_module,
     ):
-        # remove kth
+
+        k = epoch + 1 if self.save_top_k == -1 else self.save_top_k
 
         del_list = []
-        if len(self.best_k_models) == self.save_top_k and self.save_top_k > 0:
+        if len(self.best_k_models) == k and k > 0:
             delpath = self.kth_best_model_path
             self.best_k_models.pop(self.kth_best_model_path)
             del_list.append(delpath)
 
         self.best_k_models[filepath] = current
-        if len(self.best_k_models) == self.save_top_k:
+        if len(self.best_k_models) == k:
             # monitor dict has reached k elements
             _op = max if self.mode == "min" else min
             self.kth_best_model_path = _op(
@@ -538,7 +531,7 @@ def _do_check_save(
             log.info(
                 f"Epoch {epoch:d}: {self.monitor} reached"
                 f" {current:0.5f} (best {self.best_model_score:0.5f}),"
-                f" saving model to {filepath} as top {self.save_top_k}"
+                f" saving model to {filepath} as top {k}"
             )
         self._save_model(filepath, trainer, pl_module)
 


Test patch:
diff --git a/tests/callbacks/test_model_checkpoint.py b/tests/callbacks/test_model_checkpoint.py
--- a/tests/callbacks/test_model_checkpoint.py
+++ b/tests/callbacks/test_model_checkpoint.py
@@ -320,6 +320,26 @@ def test_model_checkpoint_topk_zero(tmpdir):
     assert len(os.listdir(tmpdir)) == 0
 
 
+def test_model_checkpoint_topk_all(tmpdir):
+    """ Test that save_top_k=-1 tracks the best models when monitor key is provided. """
+    seed_everything(1000)
+    epochs = 2
+    model = EvalModelTemplate()
+    checkpoint_callback = ModelCheckpoint(filepath=tmpdir, monitor="val_loss", save_top_k=-1)
+    trainer = Trainer(
+        default_root_dir=tmpdir,
+        early_stop_callback=False,
+        checkpoint_callback=checkpoint_callback,
+        max_epochs=epochs,
+        logger=False,
+    )
+    trainer.fit(model)
+    assert checkpoint_callback.best_model_path == tmpdir / "epoch=1.ckpt"
+    assert checkpoint_callback.best_model_score > 0
+    assert set(checkpoint_callback.best_k_models.keys()) == set(str(tmpdir / f"epoch={i}.ckpt") for i in range(epochs))
+    assert checkpoint_callback.kth_best_model_path == tmpdir / "epoch=0.ckpt"
+
+
 def test_ckpt_metric_names(tmpdir):
     model = EvalModelTemplate()
 
diff --git a/tests/trainer/test_trainer.py b/tests/trainer/test_trainer.py
--- a/tests/trainer/test_trainer.py
+++ b/tests/trainer/test_trainer.py
@@ -669,7 +669,7 @@ def test_test_checkpoint_path(tmpdir, ckpt_path, save_top_k):
     trainer.fit(model)
     if ckpt_path == 'best':
         # ckpt_path is 'best', meaning we load the best weights
-        if save_top_k <= 0:
+        if save_top_k == 0:
             with pytest.raises(MisconfigurationException, match='.*is not configured to save the best.*'):
                 trainer.test(ckpt_path=ckpt_path)
         else: