[s2s finetune_trainer] a mess around distributed 
Author: stas00
Pull request: no

Currently `examples/seq2seq/finetune_trainer.py` bails with multigpu and w/o an explicit ddp mode invoked w/ `-m torch.distributed.launch` - it tries to get the world size thinking it's under ddp, when it's actually under dp.

```
Traceback (most recent call last):
  File "finetune_trainer.py", line 310, in <module>
    main()
  File "finetune_trainer.py", line 254, in main
    trainer.train(
  File "/mnt/nvme1/code/huggingface/transformers-master/src/transformers/trainer.py", line 595, in train
    train_dataloader = self.get_train_dataloader()
  File "/mnt/nvme1/code/huggingface/transformers-master/src/transformers/trainer.py", line 390, in get_train_dataloader
    train_sampler = self._get_train_sampler()
  File "/mnt/nvme1/code/huggingface/transformers-deepspeed/examples/seq2seq/seq2seq_trainer.py", line 124, in _get_train_sampler
    self.train_dataset.make_sortish_sampler(
  File "/mnt/nvme1/code/huggingface/transformers-deepspeed/examples/seq2seq/utils.py", line 156, in make_sortish_sampler
    return DistributedSortishSampler(self, batch_size, shuffle=shuffle, **kwargs)
  File "/mnt/nvme1/code/huggingface/transformers-deepspeed/examples/seq2seq/utils.py", line 368, in __init__
    num_replicas = dist.get_world_size()
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py", line 671, in get_world_size
    return _get_group_size(group)
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py", line 233, in _get_group_size
    default_pg = _check_default_pg()
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py", line 225, in _check_default_pg
    raise RuntimeError("Default process group is not initialized")
RuntimeError: Default process group is not initialized
```

The problem is that the HF trainer doesn't have a very clear way about the different dist modes. There is a bunch of different checks at different places and no simple single flag to tell the downstream code which mode it is in, leading to such bugs.

I sent a fix PR, with:
```
distributed=(self.args.n_gpu > 1 and self.args.local_rank != -1),
```
but it just shows how fragile the downstream code is because there is no loud and clear flag :(

I propose to set a new attribute`self.distributed_mode={None|dp|ddp}` perhaps in `_setup_devices` in `training_args.py`? 

@patrickvonplaten, @sgugger, @LysandreJik 


Comments:
Hi, thanks @stas00 , I would be grateful to integrate this fix, I am currently dealing with this issue and using this script. thanks.

Once https://github.com/huggingface/transformers/pull/8823 is merged (hopefully Monday), it will be in master, but feel free to use that branch until then.


Awesome, I am really thankful, it helps me a lot.

On Sun, Nov 29, 2020 at 1:48 AM Stas Bekman <notifications@github.com>
wrote:

> Once #8823 <https://github.com/huggingface/transformers/pull/8823> is
> merged (hopefully Monday), it will be in master, but feel free to use that
> branch until then.
>
> —
> You are receiving this because you commented.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/transformers/issues/8822#issuecomment-735311547>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/ABP4ZCDF7U67LXQTURXHFRTSSGK6TANCNFSM4UFIIMLA>
> .
>


@rabeehk, here is a quick update - as @sgugger pointed out my fix wasn't correct if you wanted the sortish_sampler and we are trying to figure out how to fix it correctly. If all you want is to use sortish_sample with dpp only then the correct fix is most likely this:
```
 self.train_dataset.make_sortish_sampler(
                    self.args.per_device_train_batch_size, distributed=self.args.local_rank != -1)
```
please watch the development in https://github.com/huggingface/transformers/pull/8823

OK, the right fix has been merged into master https://github.com/huggingface/transformers/pull/8823 so just update the master and you should have it working, @rabeehk