Repository: Lightning-AI/pytorch-lightning

Issue:
Add support to log hparams and metrics to tensorboard?
How can I log metrics (_e.g._ validation loss of best epoch) together with the set of hyperparameters?

I have looked through the docs and through the code.
It seems like an obvious thing, so maybe I'm just not getting it.

Currently, the only way that I found was to extend the logger class:
```python
class MyTensorBoardLogger(TensorBoardLogger):

    def __init__(self, *args, **kwargs):
        super(MyTensorBoardLogger, self).__init__(*args, **kwargs)

    def log_hyperparams(self, *args, **kwargs):
        pass

    @rank_zero_only
    def log_hyperparams_metrics(self, params: dict, metrics: dict) -> None:
        params = self._convert_params(params)
        exp, ssi, sei = hparams(params, metrics)
        writer = self.experiment._get_file_writer()
        writer.add_summary(exp)
        writer.add_summary(ssi)
        writer.add_summary(sei)
        # some alternative should be added
        self.tags.update(params)
```
And then I'm writing the hparams with metrics in a callback:
```
def on_train_end(self, trainer, module):
        module.logger.log_hyperparams_metrics(module.hparams, {'val_loss': self.best_val_loss})
```
But that doesn't seem right. 
Is there a better way to write some metric together with the hparams as well?

#### Environment

 - OS: Ubuntu18.04
 - conda4.8.3
 - pytorch-lightning==0.7.1
- torch==1.4.0



Discussion:
Hi! thanks for your contribution!, great first issue!
it seems to be duplicated, pls continue in #1225
Actually #1225 is not related. In that issue it's about providing a Namespace as hparams. Here, its about logging a metric such as validation accuracy together with a set of hparams.
at which point would you like to log that? a) at each training step b) at the end of training c) something else?
Did you try this: In training_step for example:
```
self.hparams.my_custom_metric = 3.14
self.logger.log_hyperparams(hparams)
```
Yes I tried it. It seems that updating the hyperparams (writing a seond time) doesn't work. That's why I overwrite the original `log_hyperparams` to do nothing, and then only call my own implementation `log_hyperparams_metrics` at the very end of the training. (`log_hyperparams` is called automatically by the pytorch lightning framework at some point during the start of the trianing).
I want to achieve b).
_E.g._ I run 10 random hparams sampling rounds, then I want to know which hparam set gave the best validation loss.
I am maybe something missing on your use-case... you are running some hyperparameter search with just one logger for all results? Not sure if  storing all into single logger run is a good idea :rabbit: 
Usually I am training and validating a model with different sets of hyperparameters in a loop. After each round the final output is often something like "val-loss". This would be the validation loss of the best epoch achieved in this particular round.
Eventually, I have a number of "best validation losses" and each of these represents a certain set of hyper parameters.

After the last training round I am looking at the various sets of hyperparameters and compare them to their associated best validation loss. Tensorboard already provides tools for that: [Visualize the results in TensorBoard's HParams plugin](https://www.tensorflow.org/tensorboard/hyperparameter_tuning_with_hparams#4_visualize_the_results_in_tensorboards_hparams_plugin).

In your `TensorBoardLogger` you are already using the `hparams` function to summarize the hyperparameters. So, you are almost there. This function can also take `metrics` as a second argument. However, in your current implementation you always pass a `{}`. That's why I had to overwrite your original implementation. Furthermore you are writing this summary once in the beginning of the round. But the metrics are only known at the very end.
I wonder, how do you compare different hyperparameter sets? Maybe there is a functionality that I didn't find...
I am also seeing this same issue. No matter what I write to with ```log_hyperparams ```, no data is outptu to Tensorboard. I only see a line in tensorboard for my log with no data for each log. The input I am using is a dict with values filled. I tried both before and after trainer.fit() and no results.
So it appears like ```Trainer.fit()``` is calling ```run_pretrain_routine``` which checks if the trainer has the ```hparams``` attribute. Since this attribute is defined in the ```__init__``` function of ```Trainer```, even though it is set to ```None``` by default, Lightning will still write out the empty hyperparameters to the logger. In the case of Tensorfboard, this causes all subsequent writes to the hyper-parameters to be ignored. You can sole this in one of two ways:

1. Call ```delattr(model, "hparams")``` on your ```LightningModule``` before ```trainer.fit()``` to ensure the hyperparameters are not automatically written out.
2. Use @mRcSchwering code above. This will not write initially since he is passing the original ```log_hyperparams```. 

> ```python
> class MyTensorBoardLogger(TensorBoardLogger):
> 
>     def __init__(self, *args, **kwargs):
>         super(MyTensorBoardLogger, self).__init__(*args, **kwargs)
> 
>     def log_hyperparams(self, *args, **kwargs):
>         pass
> 
>     @rank_zero_only
>     def log_hyperparams_metrics(self, params: dict, metrics: dict) -> None:
>         from torch.utils.tensorboard.summary import hparams
>         params = self._convert_params(params)
>         exp, ssi, sei = hparams(params, metrics)
>         writer = self.experiment._get_file_writer()
>         writer.add_summary(exp)
>         writer.add_summary(ssi)
>         writer.add_summary(sei)
>         # some alternative should be added
>         self.tags.update(params)
> ```

Tip: I was only able to get metric values to display when they matched the tag name of a scalar I had logged previously in the log.

umm. so weird. my tb logs hparams automatically when they are attached tk the module (self.hparams=hparams).

and metrics are logged normally through the log key. 

did you try that? what exactly are you trying to do?
What log key are you using? I was never able to get metrics to show up in the hyperparameters window without using the above method. I can log the metrics to the scalars tab but not the hparams tab. I am going for something like this in my hparams tab: 
![image](https://user-images.githubusercontent.com/23267120/79457968-1b328c00-7fbf-11ea-8894-350e047c51d7.png)
I was able to get this using the above code. Setting the ```hparams``` attribute gave me the first column with the hyperparameters but I could not figure out how to add the accuracy and loss columns without the modified logger. Thanks!
Maybe a full simple example in Colab could be easier for this discussion?
@SpontaneousDuck  I tried your fix with `pytorch-lightning         0.7.4rc7 ` but it did not work for me.
I got the hparams in tb but the metrics are not enabled.
> and metrics are logged normally through the log key.

Can you please give a simple example? 
@mRcSchwering 
1. if the cluster or (you) kill your job, how would you log the parameters?
2. What if I need to know the params AS the job is running so that i can see the effect each has on the loss curve?

These are the reasons we do this in the beginning of training... but agree this is not optimal. So, this seems to be a fundamental design problem with tensorboard unless I'm missing something obvious?

FYI:
@awaelchli , @justusschock 
@williamFalcon indeed both cases wont work with my solution.
Currently, I just wrote a function that writes the set of hparams with the best loss metric into a csv and updates it after every round. 

It's not really nice because I have all the epoch-wise metrics nicely organized in the tensorboard, but the hparams overview is in a separate .csv without all the tensorboard features.
Just a question: If you log them multiple times, would they be overwritten on tensorboard? Because then we could probably also log them each epoch.
From some comments I get the feeling we might be talking about 2 things here.
There are 2 levels of metrics here. Let me clarify with an example:

Say I have a model which has the hyperparameter learning rate, it's a classifier, I want to train it for 100 epochs, I want to try out 2 different learning rates (say 1e-4 and 1e-5). 

On the one hand, I want to describe the training progress. I will do that with a training/validation loss and accuracy, which I write down for every epoch. Let me call these metrics **epoch-wise metrics**.

On the other hand, I want to have an overview of which learning rate worked better.
So maybe 1e-4 had the lowest validation loss at epoch 20 with 0.6, and 1e-5 at epoch 80 with 0.5.
This summary might get updates every epoch, but it only summarizes the best achieved epoch of every training run. In this summary I can see what influence the learning rate has on the training. Let me call this **run metrics**.

What I was talking about the whole time is the second case, the **run metrics**.
The first case (epoch-wise metrics) work fine for me.
What I was trying to get is the second case.
In Tensorboard there is a extra tab for this ([4_visualize_the_results_in_tensorboards_hparams_plugin](https://www.tensorflow.org/tensorboard/hyperparameter_tuning_with_hparams#4_visualize_the_results_in_tensorboards_hparams_plugin)).
Okay, so just to be sure: for your run metrics, you would like to log metrics *and* hparams in each step? 

I think, that's a bit overkill to add it as default behaviour. If it's just about the learning rate maybe #1498 could help you there. Otherwise I'm afraid I can't think of a generic implementation here, that won't add much overhead for most users :/
Yes one solution would be to log every epoch. And no, I think it should not be automatically because the kind of metrics you are interested in depends heavily on the whole experiment.

I just described an easy example above with 1 hyperparameter (learning rate).  In reality you have many hyperparameters which each can have different reasonable values. Eventually, you are looking for the best set of values. For that it is important to have an overview, where you can see how each set of values influence the training (_e.g._ the best achieved training loss in a training run).
As I described [here](https://github.com/PyTorchLightning/pytorch-lightning/issues/1228#issuecomment-611563428) the feature is _almost_ already in pytorch lightning.
Personally, I think it would be cool if `log_hyperparams` would already take the `metrics` argument. Then, I can inject that using a callback class which updates the _best achieved loss_ every epoch.
Maybe we could add an optional argument for that

Although I'm afraid, that it will be hard to change the time of logging HParams per default (since I think @williamFalcon has a point there). 

So I think we can definitely update the logger, but you would still have to deal with your time of logging manually. Maybe you and we can also think of a better solution to automate the time of logging...

Resolving patch:
diff --git a/pytorch_lightning/loggers/tensorboard.py b/pytorch_lightning/loggers/tensorboard.py
--- a/pytorch_lightning/loggers/tensorboard.py
+++ b/pytorch_lightning/loggers/tensorboard.py
@@ -101,7 +101,8 @@ def experiment(self) -> SummaryWriter:
         return self._experiment
 
     @rank_zero_only
-    def log_hyperparams(self, params: Union[Dict[str, Any], Namespace]) -> None:
+    def log_hyperparams(self, params: Union[Dict[str, Any], Namespace],
+                        metrics: Optional[Dict[str, Any]] = None) -> None:
         params = self._convert_params(params)
         params = self._flatten_dict(params)
         sanitized_params = self._sanitize_params(params)
@@ -114,7 +115,9 @@ def log_hyperparams(self, params: Union[Dict[str, Any], Namespace]) -> None:
             )
         else:
             from torch.utils.tensorboard.summary import hparams
-            exp, ssi, sei = hparams(sanitized_params, {})
+            if metrics is None:
+                metrics = {}
+            exp, ssi, sei = hparams(sanitized_params, metrics)
             writer = self.experiment._get_file_writer()
             writer.add_summary(exp)
             writer.add_summary(ssi)


Test patch:
diff --git a/tests/loggers/test_tensorboard.py b/tests/loggers/test_tensorboard.py
--- a/tests/loggers/test_tensorboard.py
+++ b/tests/loggers/test_tensorboard.py
@@ -77,3 +77,19 @@ def test_tensorboard_log_hyperparams(tmpdir):
         "layer": torch.nn.BatchNorm1d
     }
     logger.log_hyperparams(hparams)
+
+
+def test_tensorboard_log_hparams_and_metrics(tmpdir):
+    logger = TensorBoardLogger(tmpdir)
+    hparams = {
+        "float": 0.3,
+        "int": 1,
+        "string": "abc",
+        "bool": True,
+        "dict": {'a': {'b': 'c'}},
+        "list": [1, 2, 3],
+        "namespace": Namespace(foo=Namespace(bar='buzz')),
+        "layer": torch.nn.BatchNorm1d
+    }
+    metrics = {'abc': torch.tensor([0.54])}
+    logger.log_hyperparams(hparams, metrics)