Repository: Lightning-AI/pytorch-lightning

Issue:
Early stopping conditioned on metric `val_loss` isn't recognised when setting the val_check_interval
**Describe the bug**
Training stops when setting `val_check_interval`<1.0 in the Trainer class as it doesn't recognise `val_loss`. I get the following warning at the end of the 3rd epoch:
```
Early stopping conditioned on metric `val_loss` which is not available. Available metrics are: loss,train_loss
```

**To Reproduce**
Steps to reproduce the behavior:
1. Run the CoolModel example but change the `trainer` line to
trainer = Trainer(val_check_interval=0.5,default_save_path="test")    
2. Training will stop at the end of the third epoch and the above warning will show.

**Expected behavior**
Training shouldn't stop and `val_loss` should be recognised.

**Desktop (please complete the following information):**
 - VM: Google Colab
 - Version 0.5.3.2

**Additional context**
This doesn't happen with 0.5.2.1 although it looks like something has changed with model saving mechanism since it only seems to save the best model in 0.5.3.2.

EDIT: Also seems to happen when setting `train_percent_check`<1.0


Discussion:
can you post your test_end step?
I didn't use a test set since it is optional. The default MNIST example in the README will reproduce the behaviour when changing the trainer line to:
```
trainer = Trainer(val_check_interval=0.5,default_save_path="log_dir")
# or
trainer = Trainer(train_percent_check=0.5,default_save_path="log_dir")
````
sorry, meant validation_end
```python
    def validation_end(self, outputs):
        # OPTIONAL
        avg_loss = torch.stack([x['val_loss'] for x in outputs]).mean()
        tensorboard_logs = {'val_loss': avg_loss}
        return {'avg_val_loss': avg_loss, 'log': tensorboard_logs}
````
I tried changing 'avg_val_loss' -> 'val_loss' but the same issue occurs.
it should be val_loss
I tried it with `val_loss` too. 
```
    def validation_end(self, outputs):
        # OPTIONAL
        avg_loss = torch.stack([x['val_loss'] for x in outputs]).mean()
        tensorboard_logs = {'val_loss': avg_loss}
        return {'val_loss': avg_loss, 'log': tensorboard_logs}
```
The issue still occurs.

The issue only doesn't happen when using the default `val_check_interval` and `train_percent_check` in the Trainer.
ok got it. can you share the stacktrace? 
There is no error just a warning at the end of epoch 3 and then training stops.
```
Epoch 3: : 1894batch [00:04, 403.95batch/s, batch_nb=18, loss=1.014, v_nb=0] /usr/local/lib/python3.6/dist-packages/pytorch_lightning/callbacks/pt_callbacks.py:128: RuntimeWarning: Early stopping conditioned on metric `val_loss` which is not available. Available metrics are: loss,train_loss
  RuntimeWarning)
```
It looks like the problem is that there is only one `self.callback_metrics` which is sometimes overwritten by `self.run_training_batch` and sometimes by `self.run_evaluation`. At the same time, early stopping callback uses `self.callback_metrics` at the end of the training epoch. And the problem is that there can be no validation run at the last training batch. In that case `self.callback_metrics` will contain only the metrics from the last training batch.

If it is true, we can just force validation computation at the end of the training epoch.
@kuynzereb we shouldn't force computation. just partition self.callback_metrics to have 

```python
self.callback_metrics['train'] = {}
self.callback_metrics['val'] = {}
self.callback_metrics['test'] = {}
```

anyone interested in the PR?

Resolving patch:
diff --git a/pytorch_lightning/trainer/evaluation_loop_mixin.py b/pytorch_lightning/trainer/evaluation_loop_mixin.py
--- a/pytorch_lightning/trainer/evaluation_loop_mixin.py
+++ b/pytorch_lightning/trainer/evaluation_loop_mixin.py
@@ -138,7 +138,7 @@ def run_evaluation(self, test=False):
             self.log_metrics(log_metrics, {})
 
             # track metrics for callbacks
-            self.callback_metrics = callback_metrics
+            self.callback_metrics.update(callback_metrics)
 
             # hook
             model.on_post_performance_check()
diff --git a/pytorch_lightning/trainer/train_loop_mixin.py b/pytorch_lightning/trainer/train_loop_mixin.py
--- a/pytorch_lightning/trainer/train_loop_mixin.py
+++ b/pytorch_lightning/trainer/train_loop_mixin.py
@@ -258,7 +258,7 @@ def optimizer_closure():
         all_log_metrics = {k: v for d in all_log_metrics for k, v in d.items()}
 
         # track all metrics for callbacks
-        self.callback_metrics = {k: v for d in all_callback_metrics for k, v in d.items()}
+        self.callback_metrics.update({k: v for d in all_callback_metrics for k, v in d.items()})
 
         return 0, grad_norm_dic, all_log_metrics
 


Test patch: