Training is cancelled when check_val_every_n_epoch > 1 and early stopping is used.
🐛 Bug
Using early stopping callback on metric emitted during validation with check_val_every_n_epoch > 1 flag of trainer leads to the training being aborted.
Reproduced here.
Expected behavior
Training should continue even when check_val_every_n_epoch > 1 and early stopping is used.
One has to think how the patience parameter  of the early stopping is handled in this case.
I see two possible directions


Interpreting patience as the number of validation runs which did satisfy the early stopping criterion.
This would mean e.g. choosing check_val_every_n_epoch = 2 and a patience = 10 would train for an additional 2 * 10 = 20 epochs which might not be what one would expect.


Interpreting patience as the number of training epochs since the last validation improvement which have to be at least completed until the training is stopped, i.e., if the number of training epochs since the last improvement as measured by validation is larger than patience then stop.


Environment
* CUDA:
	- GPU:
		- Tesla T4
	- available:         True
	- version:           10.1
* Packages:
	- numpy:             1.19.5
	- pyTorch_debug:     True
	- pyTorch_version:   1.7.0+cu101
	- pytorch-lightning: 1.2.0
	- tqdm:              4.41.1
* System:
	- OS:                Linux
	- architecture:
		- 64bit
		- 
	- processor:         x86_64
	- python:            3.6.9
	- version:           #1 SMP Thu Jul 23 08:00:38 PDT 2020