Upgrade PyTorch Lightning to 1.0.2
Author: SeanNaren
Pull request: yes

Updates Pytorch Lightning to 1.0.2, and moves early stopping callback into the general callbacks due to deprecation.

Comments:
@sshleifer @julien-c 
mind helping land this?

this upgrades HF to use lightning 1.0+ 

the 1.0 version has the stable API along with major fixes (many of the open issues with lightning and HF are fixed here).  

Finally, a few notes about the changes with the earlier versions (earliest versions) of lightning:

1. models are decoupled from data (but still supported for research purposes). 
2. lightningModules for the HF case are likely better used to pass in models to be trained in a specific way or even easier defined as a lightningmodule to modifying and subclassing can be easier. 
3. all LMs now offer first class support for exporting to onnx and torchscript.

thanks for the help!

Thanks for the contribution!
I updated (pushed to) your branch to include some new multi-gpu tests that are on master from @stas00.

For hf tests marked `@slow`, circleci won't run them, so you should run:

```
RUN_SLOW=1 pytest/examples/seq2seq/finetune.py
RUN_SLOW=1 pytest/examples/seq2seq/test_bash_script.py  # I added --do_predict to this to document another issue. This is failing on master, so feel free to remove it if you can't get it working
```
On a machine with >1 GPUs, run:

```bash
RUN_SLOW=1 pytest/examples/seq2seq/test_seq2seq_examples_multi_gpu.py
```


We have had some issues getting `trainer.test` working in DDP (@nateraw thinks our usage of `hparams` is the culprit). I can't replicate the issue with the PL `BoringModel`, but you may run into it now. Would love to know if you see anything, but if it's unfixable, we can remove `--do_predict` from test_bash_script.py.




Thanks @sshleifer will investigate :)

I've made a few changes in the code and the tests are passing!

Regarding the barrier I don't think it should stay like this, but its necessary due to the test logic after picking the best model. Current functionality means testing happens across all GPUs, in sync.

The save function in PL probably should add a barrier to ensure all processes are in sync when saving to prevent further issues, and I can open a followup issue to handle this (cc @williamFalcon)

@sshleifer could you verify things work from your end as well if poss?

@sshleifer @julien-c any chance we can land this? everything is passing and things are fixed! 

:)

@SeanNaren is
> Running into other failing tests, this might be the pytorch 1.7 upgrade...

resolved?

cc @stas00, @patil-suraj for awareness.

@sshleifer yeah just needed to merge master

Thanks @SeanNaren for your work on this. It is very much appreciated!

I propose an easy solution for such changes - in addition to updating the requirement file, we could add a run-time version check at the top of `lightning_base.py`, which gets adjusted when a breaking change is done like in this PR.

Incidentally this upgrade solves the problem I have been battling since last night.

a test using PL w/ 1 gpu, followed by a test using 2 gpus was failing with (same pytest process):

```
pytorch_lightning.utilities.exceptions.MisconfigurationException:
                You requested GPUs: [0, 1]
                But your machine only has: [0]
```
after this update, it's gone! Clearly it was some bug in PL that kept some stale state.


Thank you guys for this awesome repo :)

@stas00 that's a neat solution, we're hoping to assist out on doing some small refactors to the lightning code to make it simpler and easier whilst being less intrusive. More to follow in time and we'll keep you guys in the loop

How hard would it be to get the validation metrics sync'd/averaged between processes?

I posted an issue a while back on your repo where checkpoint saving was not making decisions based on averaged metrics, and I know you guys have made some progress there.

I'm specifically interested in `examples/seq2seq/finetune.py` which uses [this function](https://github.com/huggingface/transformers/blob/master/examples/seq2seq/callbacks.py#L87) to build a `ModelCheckpoint` callback.

Even if it were just a simple average across processes of the relevant metric this would be very valuable to us.

> How hard would it be to get the validation metrics sync'd/averaged between processes?

The new Metrics API in Lightning 1.0 supports distributed syncing out of the box (doesn't even require lightning to work). Any metrics you need can definitely be [implemented](https://pytorch-lightning.readthedocs.io/en/latest/metrics.html#implementing-a-metric) fairly easily.


Are there any documented examples where `dist_reduce_fx="mean"`?

@sshleifer Right now I can't find any of our current metrics using ``dist_reduce_fx="mean"``. Is there a specific issue you are running into?

Since I already have code to compute all the metrics I want, I am trying to make a class that just gathers the computed metrics from all processes and averages whatever it is sent. (Maybe in future I will try to overweight ranks that processed more data.)
My issue is that the type of value sent to update has to be coerced to a `torch.tensor` on the correct device, which I have not figured out yet. My attempt at
https://github.com/huggingface/transformers/pull/8269

raises `RuntimeError: Tensors must be CUDA and dense` during the `allgather` step.

(pl 1.0.4, torch 1.6)

Will take a look @sshleifer :)