[s2s trainer] tests to use distributed on multi-gpu machine
Author: stas00
Pull request: yes

This PR:

* [x] abstracts the async forking io into local `utils.py`
* [x] deploys distributed training with special async io forking for `examples/seq2seq/test_finetune_trainer.py`

So now this works (2 gpus):
```
CUDA_VISIBLE_DEVICES="0,1" RUN_SLOW=1 pytest -sv examples/seq2seq/test_finetune_trainer.py
```
and this still works (1 gpu)
```
CUDA_VISIBLE_DEVICES="0" RUN_SLOW=1 pytest -sv examples/seq2seq/test_finetune_trainer.py
```

Fixes: #7833
Fixes: #7982

@sshleifer  

Comments:
@sshleifer, the slow test isn't working for me prior to this PR with 0 or 1 gpu:

```
CUDA_VISIBLE_DEVICES="" RUN_SLOW=1 pytest -sv examples/seq2seq/test_finetune_trainer.py::TestFinetuneTrainer::test_finetune_trainer_slow   
[...]

self = <seq2seq.test_finetune_trainer.TestFinetuneTrainer testMethod=test_finetune_trainer_slow>

    @slow
    def test_finetune_trainer_slow(self):
        # There is a missing call to __init__process_group somewhere
>       output_dir = self.run_trainer(eval_steps=2, max_len="128", model_name=MARIAN_MODEL, num_train_epochs=3)

examples/seq2seq/test_finetune_trainer.py:35: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
examples/seq2seq/test_finetune_trainer.py:113: in run_trainer
    main()
examples/seq2seq/finetune_trainer.py:199: in main
    model = AutoModelForSeq2SeqLM.from_pretrained(
src/transformers/modeling_auto.py:1118: in from_pretrained
    return MODEL_FOR_SEQ_TO_SEQ_CAUSAL_LM_MAPPING[type(config)].from_pretrained(
src/transformers/modeling_utils.py:947: in from_pretrained
    model = cls(config, *model_args, **model_kwargs)
src/transformers/modeling_bart.py:964: in __init__
    base_model = BartModel(config)
src/transformers/modeling_bart.py:843: in __init__
    self.encoder = BartEncoder(config, self.shared)
src/transformers/modeling_bart.py:315: in __init__
    self.embed_positions = SinusoidalPositionalEmbedding(
src/transformers/modeling_bart.py:1331: in __init__
    self.weight = self._init_weight(self.weight)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

out = Parameter containing:
tensor([[ 2.2244e+00, -1.2380e+00, -3.5307e-01,  ..., -1.0924e+00,
         -1.3130e+00,  1.7737...    [-3.8906e-01,  9.2203e-01,  1.7887e-01,  ..., -1.7493e-01,
         -1.6993e+00,  2.0896e-01]], requires_grad=True)

    @staticmethod
    def _init_weight(out: nn.Parameter):
        """Identical to the XLM create_sinusoidal_embeddings except features are not interleaved.
        The cos features are in the 2nd half of the vector. [dim // 2:]
        """
        n_pos, dim = out.shape
        position_enc = np.array(
            [[pos / np.power(10000, 2 * (j // 2) / dim) for j in range(dim)] for pos in range(n_pos)]
        )
>       out[:, 0 : dim // 2] = torch.FloatTensor(np.sin(position_enc[:, 0::2]))  # This line breaks for odd n_pos
E       RuntimeError: a view of a leaf Variable that requires grad is being used in an in-place operation
```
**edit**: After messing around with my conda env due to ever-breaking tf, this went away, but a new thing came instead: https://github.com/huggingface/transformers/issues/7982


interesting, I can't reproduce that. What's your `transformers-cli env`? Does it fail after the change?

No, it fails on master.

```
- `transformers` version: 3.4.0
- Platform: Linux-4.15.0-118-generic-x86_64-with-glibc2.10
- Python version: 3.8.5
- PyTorch version (GPU?): 1.8.0.dev20201020 (True)
- Tensorflow version (GPU?): 2.3.1 (True)
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>
```

I'd open an issue "sinusoidal positional embedding broken on torch 1.8".
Reasoing:  [these](https://github.com/huggingface/transformers/blob/master/tests/test_modeling_bart.py#L606) fast tests pass CI and I can't replicate on on torch 1.5.

Is this ready to merge otherwise?

Need to sort out this first: https://github.com/huggingface/transformers/issues/7982

It's mostly ready otherwise, but the slow test will fail as it has nothing to do with this PR. 

**edit** resolved in this PR.

Out of curiosity, how did you resolve the bleu issue?

Great work, btw! This is awesome. Now we can tell people to run these tests before they break things :) 
Apparently there is multi-gpu ci running src/ tests at some frequency FYI, I think through gh actions.

I replied in the other issue: I used more iterations - 6 was enough for 1 gpu, 10 for 2, so I went with 10.

I think if someone tries it on more than 2 gpus it might need even more iterations - could probably codify this with a factor of n_gpus.


Documenting it now https://github.com/huggingface/transformers/pull/7993
If you think anything needs to be added please let me know.
It will get better over time.

> Apparently there is multi-gpu ci running src/ tests at some frequency FYI, I think through gh actions.

Once a day yes:
https://github.com/huggingface/transformers/blob/master/.github/workflows/self-scheduled.yml#L74

Later I want to add the non-interactive IO pipe options - in case it hangs for someone - by default it could be non-interactive - always works, and only make it interactive for debug purposes.