Repository: Lightning-AI/pytorch-lightning

Issue:
Metric aggragation is broken for LoggerCollection
<!-- 
### Common bugs:
1. Tensorboard not showing in Jupyter-notebook see [issue 79](https://github.com/PyTorchLightning/pytorch-lightning/issues/79).    
2. PyTorch 1.1.0 vs 1.2.0 support [see FAQ](https://github.com/PyTorchLightning/pytorch-lightning#faq)    
-->

## 🐛 Bug

<!-- A clear and concise description of what the bug is. -->
After changes in #1278 it is now not possible to log testing metrics after traning while using several loggers.
### To Reproduce
Say we want to run a [MINST example](https://github.com/PyTorchLightning/pytorch-lightning/blob/master/pl_examples/basic_examples/cpu_template.py) and also want to add a change - log testing metrics after training. For that we define a Callback

```
class TestCallback(Callback):
    def on_train_end(self, trainer, pl_module):
        # note that it would crash if you don't pass the `pl_module`
        trainer.test(pl_module)
```
and pass it to trainer `callbacks` argument. 

We would also like to use several loggers to track all metrics, say `MLFlowLogger` and `TensorBoardLogger`. For this we create instances of these loggers and pass them into `Trainer` in a list.

### Expected behavior
Testing metrics should be logged - but they don't as there's no final aggregation when our logger is a `LoggerCollection`

### Additional context
In my opinion, the logic in `agg_and_log_metrics`  and `_finalize_agg_metrics` is hard to follow, so I'd be happy if user could choose plain old `log_metrics` which worked nicely.



Discussion:
are you using the last master?
@Borda thanks! It now works for a single logger. But it seems that problem persists when several loggers are used, so I updated the issue. This happens because metrics to aggregate are not propagated to child loggeres
I'm also having this issue with multiple loggers on master. Seems that the test results are printed at the end, but the loggers don't log the test metrics.
@Borda any update on this issue? I'm still encountering this on the latest release, and it seems related to #1859 
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


Resolving patch:
diff --git a/pytorch_lightning/loggers/base.py b/pytorch_lightning/loggers/base.py
--- a/pytorch_lightning/loggers/base.py
+++ b/pytorch_lightning/loggers/base.py
@@ -272,24 +272,46 @@ def __init__(self, logger_iterable: Iterable[LightningLoggerBase]):
     def __getitem__(self, index: int) -> LightningLoggerBase:
         return [logger for logger in self._logger_iterable][index]
 
+    def update_agg_funcs(
+            self,
+            agg_key_funcs: Optional[Mapping[str, Callable[[Sequence[float]], float]]] = None,
+            agg_default_func: Callable[[Sequence[float]], float] = np.mean
+    ):
+        for logger in self._logger_iterable:
+            logger.update_agg_funcs(agg_key_funcs, agg_default_func)
+
     @property
     def experiment(self) -> List[Any]:
         return [logger.experiment for logger in self._logger_iterable]
 
+    def agg_and_log_metrics(self, metrics: Dict[str, float], step: Optional[int] = None):
+        for logger in self._logger_iterable:
+            logger.agg_and_log_metrics(metrics, step)
+
     def log_metrics(self, metrics: Dict[str, float], step: Optional[int] = None) -> None:
-        [logger.log_metrics(metrics, step) for logger in self._logger_iterable]
+        for logger in self._logger_iterable:
+            logger.log_metrics(metrics, step)
 
     def log_hyperparams(self, params: Union[Dict[str, Any], Namespace]) -> None:
-        [logger.log_hyperparams(params) for logger in self._logger_iterable]
+        for logger in self._logger_iterable:
+            logger.log_hyperparams(params)
 
     def save(self) -> None:
-        [logger.save() for logger in self._logger_iterable]
+        for logger in self._logger_iterable:
+            logger.save()
 
     def finalize(self, status: str) -> None:
-        [logger.finalize(status) for logger in self._logger_iterable]
+        for logger in self._logger_iterable:
+            logger.finalize(status)
 
     def close(self) -> None:
-        [logger.close() for logger in self._logger_iterable]
+        for logger in self._logger_iterable:
+            logger.close()
+
+    @property
+    def save_dir(self) -> Optional[str]:
+        # Checkpoints should be saved to default / chosen location when using multiple loggers
+        return None
 
     @property
     def name(self) -> str:


Test patch:
diff --git a/tests/loggers/test_base.py b/tests/loggers/test_base.py
--- a/tests/loggers/test_base.py
+++ b/tests/loggers/test_base.py
@@ -22,6 +22,16 @@ def test_logger_collection():
     assert logger.experiment[0] == mock1.experiment
     assert logger.experiment[1] == mock2.experiment
 
+    assert logger.save_dir is None
+
+    logger.update_agg_funcs({'test': np.mean}, np.sum)
+    mock1.update_agg_funcs.assert_called_once_with({'test': np.mean}, np.sum)
+    mock2.update_agg_funcs.assert_called_once_with({'test': np.mean}, np.sum)
+
+    logger.agg_and_log_metrics({'test': 2.0}, 4)
+    mock1.agg_and_log_metrics.assert_called_once_with({'test': 2.0}, 4)
+    mock2.agg_and_log_metrics.assert_called_once_with({'test': 2.0}, 4)
+
     logger.close()
     mock1.close.assert_called_once()
     mock2.close.assert_called_once()