[examples tests] various fixes
Author: stas00
Pull request: yes

This PR is fixing slow examples tests that currently fail on scheduled CI

2 more tests will be fixed by https://github.com/huggingface/transformers/pull/10551

This PR:

Sharded DDP issues:
* fixes fully sharded ddp enum - and the corresponding tests
* 2 sharded ddp tests currently hang with master fairscale - add skip until this is sorted out - didn't want to step on @sgugger's toes - so for now just skipping


Tests:
* changes a large group of tests to check loss is not nan
* fix `test_run_seq2seq_slow` test 
   - missed by my PR https://github.com/huggingface/transformers/pull/10428
  - make it more resilient - was failing quality-wise on multi-gpu
* then we have an issue with apex - once run in a worker directly it breaks other tests running directly in the same pytest worker:

```
        # XXX: apex breaks the trainer if it's run twice e.g. run_seq2seq.main() from the same
        # program and it breaks other tests that run from the same pytest worker, therefore until this is
        # sorted out it must be run only in an external program, that is distributed=True in this
        # test and only under one or more gpus - if we want cpu will need to make a special test
        #
        # specifically to the problem traced it to self.optimizer.step() - if it's run 2nd time via
        # 2nd main() call it botches the future eval.
```
I'm not quite sure what happens but I think no-one will run into this in a normal situation, here we basically end up running:
```
main()
main()
```
in the same process. We have some internal state that doesn't get reset. 

So as I wrote above I used a workaround of running the apex integration test in a separate process so that it doesn't affect the rest of the test suite.

Of course if you have ideas on what the problem is and how to fix it I'm all ears. It's very simple to reproduce it:
```
    def test_run_seq2seq_apex(self):
        self.run_seq2seq_quick(distributed=True, extra_args_str="--fp16 --fp16_backend=apex")
        # test 2nd time - was getting eval_loss': nan'
        # to reproduce the problem set distributed=False
        self.run_seq2seq_quick(distributed=True, extra_args_str="--fp16 --fp16_backend=apex")
```

I can move it out into a separate PR if it's too much for one.

@LysandreJik, @sgugger 

Comments:
It's still hanging waiting for something. If I'm not mistaken it's hanging in saving the model. but there is another thread that might be relevant:
```
Thread 0x00007f6dd984b740 (most recent call first):
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/torch/_utils.py", line 45 in _type
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/torch/cuda/__init__.py", line 496 in type
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/torch/storage.py", line 72 in cpu
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/torch/serialization.py", line 488 in _save
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/torch/serialization.py", line 372 in save
  File "/mnt/nvme1/code/huggingface/transformers-test-fix/src/transformers/modeling_utils.py", line 835 in save_pretrained
  File "/mnt/nvme1/code/huggingface/transformers-test-fix/src/transformers/trainer.py", line 1535 in _save
  File "/mnt/nvme1/code/huggingface/transformers-test-fix/src/transformers/trainer.py", line 1496 in save_model
  File "/mnt/nvme1/code/huggingface/transformers-test-fix/src/transformers/trainer.py", line 1204 in _save_checkpoint
  File "/mnt/nvme1/code/huggingface/transformers-test-fix/src/transformers/trainer.py", line 1179 in _maybe_log_save_evaluate
  File "/mnt/nvme1/code/huggingface/transformers-test-fix/src/transformers/trainer.py", line 1088 in train
  File "/mnt/nvme1/code/huggingface/transformers-test-fix/examples/seq2seq/run_seq2seq.py", line 590 in main
  File "/mnt/nvme1/code/huggingface/transformers-test-fix/examples/seq2seq/run_seq2seq.py", line 654 in <module>

Thread 0x00007f59389d2740 (most recent call first):
  File "/mnt/nvme1/code/huggingface/transformers-test-fix/src/transformers/trainer.py", line 1552 in store_flos
  File "/mnt/nvme1/code/huggingface/transformers-test-fix/src/transformers/trainer.py", line 1132 in train
  File "/mnt/nvme1/code/huggingface/transformers-test-fix/examples/seq2seq/run_seq2seq.py", line 590 in main
  File "/mnt/nvme1/code/huggingface/transformers-test-fix/examples/seq2seq/run_seq2seq.py", line 654 in <module>
```
Right after:
```
Saving model checkpoint to /tmp/tmp4rujlsox/checkpoint-1 'epoch': 1.0}
stderr: Configuration saved in /tmp/tmp4rujlsox/checkpoint-1/config.json
```

You can add:

```
--- a/examples/seq2seq/run_seq2seq.py
+++ b/examples/seq2seq/run_seq2seq.py
@@ -649,4 +649,6 @@ def _mp_fn(index):


 if __name__ == "__main__":
+    import faulthandler
+    faulthandler.dump_traceback_later(20, repeat=True)
     main()
```

To get the hanging bt.

And it's best debug outside of pytest, otherwise some bt gets messed up it seems.

Oh? It was running to completion on my side after the removal. So let's merge as is and I will look into the failure when I have time.