Repository: Lightning-AI/pytorch-lightning
Pull request: Add ModelCheckpoint.to_yaml method
## What does this PR do

Add simple `to_json()` method to `ModelCheckpoint`. 
In the current code, it requires three lines, and it could be automated by lightning.
Fixes #3047 (disclaimer: my issue with no answer yet)

# Before submitting

- [x] Did you read the [contributor guideline](https://github.com/PyTorchLightning/pytorch-lightning/blob/master/.github/CONTRIBUTING.md), Pull Request section?
- [x] Did you make sure your PR does only one thing, instead of bundling different changes together? Otherwise, we ask you to create a separate PR for every change.
- [x] Did you make sure to update the documentation with your changes?
- [x] Did you write any new necessary tests? 

## PR review    
Anyone in the community is free to review the PR once the tests have passed.     
If we didn't discuss your PR in Github issues there's a high chance it will not be merged.

## Did you have fun?
Make sure you had fun coding 🙃


Problem statement:
ModelCheckpoint instance to JSON
## 🚀 Feature

Have a `to_json(path)` function that writes the `best_k_models` to a json file in `path`.

### Motivation

Why having a JSON is good? 
Then you don't need to load the checkpoints to know the metrics and things like a posteriori stochastic weight averaging gets  easier. 

It used to be easier in 0.6, but loss values are now tensors, so it's not directly serializable.
This is currently needed: 
```python
best_k = {k: v.item() for k, v in checkpoint.best_k_models.items()}
with open(json_path, "w") as f:
    json.dump(best_k, f, indent=0)
```

Instead, we could have 
```python
checkpoint.to_json(json_path)
```

I'm ready to make a PR for it. Just tell me if this would be useful for you.

Hints from issue discussion:
should this file be saved with every checkpoint, or one file that gets updated/overwritten? 
I just thought about a single function that can be called by the user at the end of training. 
But actually, updating/overwritting at every checkpoint also would be cool, at least the user never has to call it. 
Link to the PR #3048

Commit message: Update pytorch_lightning/callbacks/model_checkpoint.py

Patch to review:
diff --git a/pytorch_lightning/callbacks/model_checkpoint.py b/pytorch_lightning/callbacks/model_checkpoint.py
index 1670cf0192eff..dbab9803c8276 100644
--- a/pytorch_lightning/callbacks/model_checkpoint.py
+++ b/pytorch_lightning/callbacks/model_checkpoint.py
@@ -22,8 +22,10 @@
 
 import os
 import re
+import yaml
 from copy import deepcopy
-from typing import Any, Dict, Optional
+from typing import Any, Dict, Optional, Union
+from pathlib import Path
 
 import numpy as np
 import torch
@@ -531,3 +533,13 @@ def _do_check_save(
             if cur_path != filepath:
                 self._del_model(cur_path)
 
+    def to_yaml(self, filepath: Optional[Union[str, Path]] = None):
+        """
+        Saves the `best_k_models` dict containing the checkpoint
+        paths with the corresponding scores to a YAML file.
+        """
+        best_k = {k: v.item() for k, v in self.best_k_models.items()}
+        if filepath is None:
+            filepath = os.path.join(self.dirpath, "best_k_models.yaml")
+        with open(filepath, "w") as fp:
+            yaml.dump(best_k, fp)
diff --git a/tests/callbacks/test_model_checkpoint.py b/tests/callbacks/test_model_checkpoint.py
index 2db905f649938..78240ade4cd57 100644
--- a/tests/callbacks/test_model_checkpoint.py
+++ b/tests/callbacks/test_model_checkpoint.py
@@ -1,4 +1,5 @@
 import os
+import yaml
 import pickle
 import platform
 import re
@@ -36,6 +37,23 @@ def test_model_checkpoint_with_non_string_input(tmpdir, save_top_k):
     )
 
 
+@pytest.mark.parametrize('save_top_k', [-1, 0, 1, 2])
+def test_model_checkpoint_to_yaml(tmpdir, save_top_k):
+    """ Test that None in checkpoint callback is valid and that chkp_path is set correctly """
+    tutils.reset_seed()
+    model = EvalModelTemplate()
+
+    checkpoint = ModelCheckpoint(filepath=None, save_top_k=save_top_k)
+
+    trainer = Trainer(default_root_dir=tmpdir, checkpoint_callback=checkpoint, overfit_batches=0.20, max_epochs=2)
+    trainer.fit(model)
+
+    checkpoint.to_yaml('./best_k_models.yaml')
+    d = yaml.full_load(open('./best_k_models.yaml', 'r'))
+    best_k = {k: v.item() for k, v in checkpoint.best_k_models.items()}
+    assert d == best_k
+
+
 @pytest.mark.parametrize(
     "logger_version,expected",
     [(None, "version_0"), (1, "version_1"), ("awesome", "awesome")],


Code review comments:
diff_hunk: @@ -36,6 +37,23 @@ def test_model_checkpoint_with_non_string_input(tmpdir, save_top_k):
     )
 
 
+@pytest.mark.parametrize('save_top_k', [-1, 0, 1, 2])
+def test_model_checkpoint_to_yaml(tmpdir, save_top_k):
+    """ Test that None in checkpoint callback is valid and that chkp_path is set correctly """
+    tutils.reset_seed()
+    model = EvalModelTemplate()
+
+    checkpoint = ModelCheckpoint(filepath=None, save_top_k=save_top_k)
+
+    trainer = Trainer(default_root_dir=tmpdir, checkpoint_callback=checkpoint, overfit_batches=0.20, max_epochs=2)
+    trainer.fit(model)
+
+    checkpoint.to_yaml('./best_k_models.yaml')
+    d = yaml.full_load(open('./best_k_models.yaml', 'r'))
path: tests/callbacks/test_model_checkpoint.py
text: @user1:
```suggestion
    path_yaml = os.path.join(tmpdir, 'best_k_models.yaml')
    checkpoint.to_yaml(path_yaml)
    d = yaml.full_load(open(path_yaml, 'r'))
```