Repository: Lightning-AI/pytorch-lightning

Issue:
Neptune logger with a validation epoch end conflict due to the 'epoch' key added on the fly.
Hi everybody! First thanks for this lib, it is very handy!

## 🐛 Bug

When using pytorch lightning in conjunction with the neptune logger, one can see this kind of error popping every time an epoch ends:
`
neptune.api_exceptions.ChannelsValuesSendBatchError: Received batch errors sending channels' values to experiment SOC-114. Cause: Error(code=400, message='X-coordinates must be strictly increasing for channel: e4e2635d-b707-46fa-9a1b-996dd009790f. Invalid point: InputChannelValue(timestamp=2020-08-13T11:55:38.422Z, x=5.0, numericValue=2.0, textValue=null, image', type=None) (metricId: 'e4e2635d-b707-46fa-9a1b-996dd009790f', x: 5.0) Skipping 1 values.
`

the import part in this error is the following line: `X-coordinates must be strictly increasing`

This is because, in `trainer/logging.py`, the `epoch` key is added on the fly on line 69: 
```python
scalar_metrics['epoch'] = self.current_epoch
```

But why does Neptune complains?

If you log all the timesteps (using `row_log_interval = 1`), at the end of an epoch, 2 calls are emitted to the logger: One to log the training logs and one for the validation logs. 
Both of those have the same `step` value which is the current training `step` value. Since the key `epoch` is duplicated in both those calls, Neptune receives the key `epoch` twice with the same `step` value leading to the exception.

### To Reproduce

launch training with:
- Neptune logger
- training logs
- validation logs
- row_log_interval=1

### Expected behaviour

Don't add the `epoch` key on the fly which force the logger to log it.




Discussion:
Hi! thanks for your contribution!, great first issue!
thanks!

yeah, unfortunately, this might be a problem on their end since we have to track the epoch.

In the meantime, can you post a colab that replicates this issue?

thanks!
Hi,

Thanks for answering.

I've been looking at the code and I'm not sure why you need that `epoch` at that moment.
When I look at the following code, I see that the `epoch` key is added on only one part of the "if statement". So the code used after this part can't rely on this key to exist. What am I missing?

```
if "step" in scalar_metrics and step is None:
    step = scalar_metrics.pop("step")
else:
    # added metrics by Lightning for convenience
    scalar_metrics['epoch'] = self.current_epoch
    step = step if step is not None else self.global_step
```


Resolving patch:
diff --git a/pytorch_lightning/trainer/evaluation_loop.py b/pytorch_lightning/trainer/evaluation_loop.py
--- a/pytorch_lightning/trainer/evaluation_loop.py
+++ b/pytorch_lightning/trainer/evaluation_loop.py
@@ -217,7 +217,7 @@ def add_progress_bar_metrics(self, *args):
         """Warning: this is just empty shell for code implemented in other class."""
 
     @abstractmethod
-    def log_metrics(self, *args):
+    def log_metrics(self, *args, **kwargs):
         """Warning: this is just empty shell for code implemented in other class."""
 
     @abstractmethod
@@ -379,7 +379,7 @@ def _evaluate(
 
                     dl_outputs.append(output)
 
-                self.__eval_add_step_metrics(output)
+                self.__eval_add_step_metrics(output, batch_idx)
 
                 # track debug metrics
                 self.dev_debugger.track_eval_loss_history(test_mode, batch_idx, dataloader_idx, output)
@@ -505,14 +505,19 @@ def __gather_epoch_end_eval_results(self, outputs):
             eval_results = eval_results[0]
         return eval_results
 
-    def __eval_add_step_metrics(self, output):
+    def __eval_add_step_metrics(self, output, batch_idx):
         # track step level metrics
         if isinstance(output, EvalResult) and not self.running_sanity_check:
             step_log_metrics = output.batch_log_metrics
             step_pbar_metrics = output.batch_pbar_metrics
 
             if len(step_log_metrics) > 0:
-                self.log_metrics(step_log_metrics, {})
+                # make the metrics appear as a different line in the same graph
+                metrics_by_epoch = {}
+                for k, v in step_log_metrics.items():
+                    metrics_by_epoch[f'{k}/epoch_{self.current_epoch}'] = v
+
+                self.log_metrics(metrics_by_epoch, {}, step=batch_idx)
 
             if len(step_pbar_metrics) > 0:
                 self.add_progress_bar_metrics(step_pbar_metrics)
diff --git a/pytorch_lightning/trainer/logging.py b/pytorch_lightning/trainer/logging.py
--- a/pytorch_lightning/trainer/logging.py
+++ b/pytorch_lightning/trainer/logging.py
@@ -64,10 +64,12 @@ def log_metrics(self, metrics, grad_norm_dic, step=None):
 
         if "step" in scalar_metrics and step is None:
             step = scalar_metrics.pop("step")
-        else:
+
+        elif step is None:
             # added metrics by Lightning for convenience
             scalar_metrics['epoch'] = self.current_epoch
             step = step if step is not None else self.global_step
+
         # log actual metrics
         if self.is_global_zero and self.logger is not None:
             self.logger.agg_and_log_metrics(scalar_metrics, step=step)


Test patch:
diff --git a/tests/trainer/test_validation_steps_result_return.py b/tests/trainer/test_validation_steps_result_return.py
--- a/tests/trainer/test_validation_steps_result_return.py
+++ b/tests/trainer/test_validation_steps_result_return.py
@@ -214,12 +214,15 @@ def test_val_step_only_step_metrics(tmpdir):
 
     # make sure we logged the correct epoch metrics
     total_empty_epoch_metrics = 0
+    epoch = 0
     for metric in trainer.dev_debugger.logged_metrics:
+        if 'epoch' in metric:
+            epoch += 1
         if len(metric) > 2:
             assert 'no_val_no_pbar' not in metric
             assert 'val_step_pbar_acc' not in metric
-            assert metric['val_step_log_acc']
-            assert metric['val_step_log_pbar_acc']
+            assert metric[f'val_step_log_acc/epoch_{epoch}']
+            assert metric[f'val_step_log_pbar_acc/epoch_{epoch}']
         else:
             total_empty_epoch_metrics += 1
 
@@ -228,6 +231,8 @@ def test_val_step_only_step_metrics(tmpdir):
     # make sure we logged the correct epoch pbar metrics
     total_empty_epoch_metrics = 0
     for metric in trainer.dev_debugger.pbar_added_metrics:
+        if 'epoch' in metric:
+            epoch += 1
         if len(metric) > 2:
             assert 'no_val_no_pbar' not in metric
             assert 'val_step_log_acc' not in metric
@@ -288,11 +293,12 @@ def test_val_step_epoch_step_metrics(tmpdir):
     for metric_idx in range(0, len(trainer.dev_debugger.logged_metrics), batches + 1):
         batch_metrics = trainer.dev_debugger.logged_metrics[metric_idx: metric_idx + batches]
         epoch_metric = trainer.dev_debugger.logged_metrics[metric_idx + batches]
+        epoch = epoch_metric['epoch']
 
         # make sure the metric was split
         for batch_metric in batch_metrics:
-            assert 'step_val_step_log_acc' in batch_metric
-            assert 'step_val_step_log_pbar_acc' in batch_metric
+            assert f'step_val_step_log_acc/epoch_{epoch}' in batch_metric
+            assert f'step_val_step_log_pbar_acc/epoch_{epoch}' in batch_metric
 
         # make sure the epoch split was correct
         assert 'epoch_val_step_log_acc' in epoch_metric
@@ -421,11 +427,11 @@ def test_val_step_full_loop_result_dp(tmpdir):
     assert 'train_step_metric' in seen_keys
     assert 'train_step_end_metric' in seen_keys
     assert 'epoch_train_epoch_end_metric' in seen_keys
-    assert 'step_validation_step_metric' in seen_keys
+    assert 'step_validation_step_metric/epoch_0' in seen_keys
     assert 'epoch_validation_step_metric' in seen_keys
     assert 'validation_step_end_metric' in seen_keys
     assert 'validation_epoch_end_metric' in seen_keys
-    assert 'step_test_step_metric' in seen_keys
+    assert 'step_test_step_metric/epoch_2' in seen_keys
     assert 'epoch_test_step_metric' in seen_keys
     assert 'test_step_end_metric' in seen_keys
     assert 'test_epoch_end_metric' in seen_keys