Repository: Lightning-AI/pytorch-lightning

Issue:
Early Stopping + result dictionary + no validation not working.
## 🐛 Bug

The case where the user does not use validation __and__ returns a dictionary (instead of a TrainResult) during training does not work in combination with early stopping.
The test case which should check this is here:

https://github.com/PyTorchLightning/pytorch-lightning/blob/bd35c869ee0b856fc30229ef36957b058fe97461/tests/callbacks/test_early_stopping.py#L136-L159

The check in the last line is wrong. It should actually compare:
```python3
    assert trainer.current_epoch < trainer.max_epochs - 1
```

### To Reproduce

Steps to reproduce the behavior:

1. Fix the test case
2. Run tests.


#### Code sample

I guess using the test case is simpler and easier.

### Expected behavior

That is an interesting question indeed. Possibilities are:

1. Test case should pass with correct comparison
2. The [docs](https://github.com/PyTorchLightning/pytorch-lightning/blob/bd35c869ee0b856fc30229ef36957b058fe97461/docs/source/results.rst#checkpointearly_stop) and  @williamFalcon in https://github.com/PyTorchLightning/pytorch-lightning/pull/3193#discussion_r477245884 suggest that only 'loss' should work.

So before fixing this issue, it should be settled what the expected behavior is.
If you tell me, I'm happy to help.
I could also include it in the pull request where I already tried to bring the docs in line with the test cases.

### Environment

* CUDA:
	- GPU:
		- GeForce GTX 1080 Ti
	- available:         True
	- version:           10.1
* Packages:
	- numpy:             1.19.1
	- pyTorch_debug:     False
	- pyTorch_version:   1.6.0
	- pytorch-lightning: 0.9.1dev
	- tensorboard:       2.2.0
	- tqdm:              4.48.2
* System:
	- OS:                Linux
	- architecture:
		- 64bit
		- ELF
	- processor:         x86_64
	- python:            3.8.3
	- version:           #113~16.04.1-Ubuntu SMP Fri Jul 10 04:37:08 UTC 2020

### Additional context



Discussion:
@Lucas-Steinmann mind sending a PR for this?
I'd love to, but what is the intended behavior? 
Possibility 1, 2, or doesn't it matter? 
Maybe we should wait on @williamFalcon answer, since he already expressed his opinion.
There is a third possibility, which is suggested by the code which would have to be changed:
https://github.com/PyTorchLightning/pytorch-lightning/blob/4d98419bb84e06c06aaee0a86530e322e24f6d11/pytorch_lightning/callbacks/early_stopping.py#L169

This comment says, that the current behavior is correct. (assuming "structured result" means Result object)
So either code, docs or test is correct and the other two have to be adjusted. :)
if you're not using the trainresult then the key 'loss' needs to be present

All of these are equivalent

```python

return loss

return {'loss': loss}

return TrainResult(minimize=loss)

return TrainResult(loss)
```

But the recommended way is:

```
return TrainResult(loss)
```

We're moving away from plain dictionaries (TrainResult is just a dict also, but adds type checking)
But ``return {'loss': loss}`` doesn't work with early stopping.

Also the test, which suggests that any key should work still contradicts you.

Furthermore the test is still wrong because the last line does not actually check whether early stopping was triggered (**off-by-one error**)
@Lucas-Steinmann reopening this then!
Just to be clear. The next step is for you (the project owners) to decide what the correct behaviour is:

1. The docs (only dicts with 'loss' key should work with early stopping): Then we have to fix the code and the tests.
2. The code (early stopping should only work with TrainResult): Then we have to remove the test and fix the docs
3. The test (early stopping should work with dicts and any key): Then we have to fix the code and the docs.

I would love to help but I think it is not for me to decide what is correct.
If you pick one, I'll implement it.
The correct behavior is neither of the above...

1. if using dicts, it should work with whatever you put in the monitor argument of the EarlyStopping or ModelCheckpoint.

```python
EarlyStopping(monitor='jiraffe')

# must have this key
return {'jiraffe': x}
```

2. If using results it should work with whatever is in the early_stop_on argument:

```python
acc = ...
loss = ...
f1 = ...

# pick whatever you want to early stop on
key_i_care_about = f1 or loss or acc
result = pl.TrainResult(early_stop_on=key_i_care_about)
```

If you ALSO pass the key in the EvalResult then the EvalResult takes precedence...

```python
def training_step(...):
   # early stop on has NO effect because you ALSO have one in validation_step
    result = TrainResult(early_stop_on=acc)

def validation_step(...)
    my_val_loss = MSE(y, y_hat)
    result = EvalResult(early_stop_on=my_val_loss)
```


Finally, I'm pretty sure this is how it works today given that I personally implemented these and wrote a ton of tests...
So, my question is:

1. does it not work like I described? then we need to fix the bugs
2. if it does work, then we need to clean up the docs to explain this better.
Ok. 1. currently does not work (atleast a few days ago, when I wrote the issue, see my minimal example here: https://github.com/PyTorchLightning/pytorch-lightning/pull/3193#discussion_r478517184)

There is a test, which exactly checks 1.  but has an off-by-one error and would actually fail if the error is fixed (see my first comment on this issue).

If I understand the docs correctly, they currently say it should only work with 'loss' key. Maybe it was meant to say the default behaviour is that it only works with 'loss' key: https://github.com/PyTorchLightning/pytorch-lightning/blob/3910ad033074367f6abfe0001562db725a75cb73/docs/source/results.rst#L109


I think the bug in the code is located here, where it only checks for the ``early_stop_on`` key:
https://github.com/PyTorchLightning/pytorch-lightning/blob/3910ad033074367f6abfe0001562db725a75cb73/pytorch_lightning/callbacks/early_stopping.py#L170-L173

Resolving patch:
diff --git a/pytorch_lightning/callbacks/early_stopping.py b/pytorch_lightning/callbacks/early_stopping.py
--- a/pytorch_lightning/callbacks/early_stopping.py
+++ b/pytorch_lightning/callbacks/early_stopping.py
@@ -85,6 +85,9 @@ def __init__(self, monitor: str = 'val_loss', min_delta: float = 0.0, patience:
         self.stopped_epoch = 0
         self.mode = mode
         self.warned_result_obj = False
+        # Indicates, if eval results are used as basis for early stopping
+        # It is set to False initially and overwritten, if eval results have been validated
+        self.based_on_eval_results = False
 
         if mode not in self.mode_dict:
             if self.verbose > 0:
@@ -157,19 +160,25 @@ def on_validation_epoch_end(self, trainer, pl_module):
         if val_es_key in trainer.logger_connector.callback_metrics:
             self.strict = False
 
-        self._validate_condition_metric(trainer.logger_connector.callback_metrics)
+        if self._validate_condition_metric(trainer.logger_connector.callback_metrics):
+            # turn off early stopping in on_train_epoch_end
+            self.based_on_eval_results = True
 
     def on_train_epoch_end(self, trainer, pl_module):
         # disable early stopping in train loop when there's a val loop
-        if self.monitor == 'val_early_stop_on':
+        if self.based_on_eval_results:
             return
 
-        # early stopping can also work in the train loop when there is no val loop and when using structured results
+        # early stopping can also work in the train loop when there is no val loop
         should_check_early_stop = False
+        # early_stop_on takes precedence over monitor key
         train_es_key = 'early_stop_on'
         if trainer.logger_connector.callback_metrics.get(train_es_key, None) is not None:
             self.monitor = train_es_key
             should_check_early_stop = True
+        # fallback to monitor key in result dict
+        if trainer.logger_connector.callback_metrics.get(self.monitor, None) is not None:
+            should_check_early_stop = True
 
         if should_check_early_stop:
             self._run_early_stopping_check(trainer, pl_module)
@@ -187,6 +196,10 @@ def __warn_deprecated_monitor_key(self):
             rank_zero_warn(m)
 
     def _run_early_stopping_check(self, trainer, pl_module):
+        """
+        Checks whether the early stopping condition is met
+        and if so tells the trainer to stop the training.
+        """
         logs = trainer.logger_connector.callback_metrics
 
         if not self._validate_condition_metric(logs):


Test patch:
diff --git a/tests/callbacks/test_early_stopping.py b/tests/callbacks/test_early_stopping.py
--- a/tests/callbacks/test_early_stopping.py
+++ b/tests/callbacks/test_early_stopping.py
@@ -137,17 +137,17 @@ def training_step(self, *args, **kwargs):
     model.validation_step = None
     model.val_dataloader = None
 
-    stopping = EarlyStopping(monitor='my_train_metric', min_delta=0.1)
+    stopping = EarlyStopping(monitor='my_train_metric', min_delta=0.1, patience=0)
     trainer = Trainer(
         default_root_dir=tmpdir,
         early_stop_callback=stopping,
         overfit_batches=0.20,
-        max_epochs=2,
+        max_epochs=10,
     )
     result = trainer.fit(model)
 
     assert result == 1, 'training failed to complete'
-    assert trainer.current_epoch < trainer.max_epochs
+    assert trainer.current_epoch < trainer.max_epochs - 1
 
 
 def test_early_stopping_functionality(tmpdir):
@@ -168,3 +168,24 @@ def validation_epoch_end(self, outputs):
     )
     trainer.fit(model)
     assert trainer.current_epoch == 5, 'early_stopping failed'
+
+
+def test_early_stopping_functionality_arbitrary_key(tmpdir):
+    """Tests whether early stopping works with a custom key and dictionary results on val step."""
+
+    class CurrentModel(EvalModelTemplate):
+        def validation_epoch_end(self, outputs):
+            losses = [8, 4, 2, 3, 4, 5, 8, 10]
+            val_loss = losses[self.current_epoch]
+            return {'jiraffe': torch.tensor(val_loss)}
+
+    model = CurrentModel()
+
+    trainer = Trainer(
+        default_root_dir=tmpdir,
+        early_stop_callback=EarlyStopping(monitor='jiraffe'),
+        overfit_batches=0.20,
+        max_epochs=20,
+    )
+    trainer.fit(model)
+    assert trainer.current_epoch >= 5, 'early_stopping failed'