Failing Multi-GPU trainer test
Author: LysandreJik
Pull request: no

This test is currently [failing in a multi-GPU setup](https://github.com/huggingface/transformers/runs/1902689616?check_suite_focus=true):

```
FAILED tests/test_trainer_distributed.py::TestTrainerDistributed::test_trainer
```

The error is the following:

```
RuntimeError: NCCL error in: /pytorch/torch/lib/c10d/ProcessGroupNCCL.cpp:784, unhandled system error, NCCL version 2.7.8
```

cc @sgugger @stas00 

Comments:
I have tried on my machine and the test passes, so the bug is linked to the setup of the machine executing the multi-GPU tests. I have never seen that error before but I would guess there is something wrong with nccl/cuda?

Retrieved the backtrace:

The error is: `RuntimeError: Address already in use`

Which means that more than one of these was running at the same time or one from a previous run is zombie and is holding this port. Probably need to try to catch that case and try a different port. I will have a look

Full trace:
```
tests/test_trainer_distributed.py:72: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

cmd = ['/home/github_actions/actions-runner/_work/transformers/transformers/.env/bin/python', '-m', 'torch.distributed.launc.../github_actions/actions-runner/_work/transformers/transformers/tests/test_trainer_distributed.py', '--output_dir', ...]
env = {'CI': 'true', 'GITHUB_ACTION': 'run7', 'GITHUB_ACTIONS': 'true', 'GITHUB_ACTION_REF': '', ...}
stdin = None, timeout = 180, quiet = False, echo = True

    def execute_subprocess_async(cmd, env=None, stdin=None, timeout=180, quiet=False, echo=True) -> _RunOutput:
    
        loop = asyncio.get_event_loop()
        result = loop.run_until_complete(
            _stream_subprocess(cmd, env=env, stdin=stdin, timeout=timeout, quiet=quiet, echo=echo)
        )
    
        cmd_str = " ".join(cmd)
        if result.returncode > 0:
            stderr = "\n".join(result.stderr)
            raise RuntimeError(
>               f"'{cmd_str}' failed with returncode {result.returncode}\n\n"
                f"The combined stderr from workers follows:\n{stderr}"
            )
E           RuntimeError: '/home/github_actions/actions-runner/_work/transformers/transformers/.env/bin/python -m torch.distributed.launch --nproc_per_node=2 /home/github_actions/actions-runner/_work/transformers/transformers/tests/test_trainer_distributed.py --output_dir /tmp/tmpsdmi_ca2' failed with returncode 1
E           
E           The combined stderr from workers follows:
E           Traceback (most recent call last):
E             File "/home/github_actions/actions-runner/_work/transformers/transformers/tests/test_trainer_distributed.py", line 82, in <module>
E               training_args = parser.parse_args_into_dataclasses()[0]
E             File "/home/github_actions/actions-runner/_work/transformers/transformers/src/transformers/hf_argparser.py", line 180, in parse_args_into_dataclasses
E               obj = dtype(**inputs)
E             File "<string>", line 59, in __init__
E             File "/home/github_actions/actions-runner/_work/transformers/transformers/src/transformers/training_args.py", line 479, in __post_init__
E               if is_torch_available() and self.device.type != "cuda" and self.fp16:
E             File "/home/github_actions/actions-runner/_work/transformers/transformers/src/transformers/file_utils.py", line 1346, in wrapper
E               return func(*args, **kwargs)
E             File "/home/github_actions/actions-runner/_work/transformers/transformers/src/transformers/training_args.py", line 601, in device
E               return self._setup_devices
E             File "/home/github_actions/actions-runner/_work/transformers/transformers/src/transformers/file_utils.py", line 1336, in __get__
E               cached = self.fget(obj)
E             File "/home/github_actions/actions-runner/_work/transformers/transformers/src/transformers/file_utils.py", line 1346, in wrapper
E               return func(*args, **kwargs)
E             File "/home/github_actions/actions-runner/_work/transformers/transformers/src/transformers/training_args.py", line 586, in _setup_devices
E               torch.distributed.init_process_group(backend="nccl")
E             File "/home/github_actions/actions-runner/_work/transformers/transformers/.env/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py", line 436, in init_process_group
E               store, rank, world_size = next(rendezvous_iterator)
E             File "/home/github_actions/actions-runner/_work/transformers/transformers/.env/lib/python3.7/site-packages/torch/distributed/rendezvous.py", line 179, in _env_rendezvous_handler
E               store = TCPStore(master_addr, master_port, world_size, start_daemon, timeout)
E           RuntimeError: Address already in use
E           Traceback (most recent call last):
E             File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
E               "__main__", mod_spec)
E             File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
E               exec(code, run_globals)
E             File "/home/github_actions/actions-runner/_work/transformers/transformers/.env/lib/python3.7/site-packages/torch/distributed/launch.py", line 260, in <module>
E               main()
E             File "/home/github_actions/actions-runner/_work/transformers/transformers/.env/lib/python3.7/site-packages/torch/distributed/launch.py", line 256, in main
E               cmd=cmd)
E           subprocess.CalledProcessError: Command '['/home/github_actions/actions-runner/_work/transformers/transformers/.env/bin/python', '-u', '/home/github_actions/actions-runner/_work/transformers/transformers/tests/test_trainer_distributed.py', '--local_rank=1', '--output_dir', '/tmp/tmpsdmi_ca2']' returned non-zero exit status 1.
E           Traceback (most recent call last):
E             File "/home/github_actions/actions-runner/_work/transformers/transformers/tests/test_trainer_distributed.py", line 82, in <module>
E               training_args = parser.parse_args_into_dataclasses()[0]
E             File "/home/github_actions/actions-runner/_work/transformers/transformers/src/transformers/hf_argparser.py", line 180, in parse_args_into_dataclasses
E               obj = dtype(**inputs)
E             File "<string>", line 59, in __init__
E             File "/home/github_actions/actions-runner/_work/transformers/transformers/src/transformers/training_args.py", line 479, in __post_init__
E               if is_torch_available() and self.device.type != "cuda" and self.fp16:
E             File "/home/github_actions/actions-runner/_work/transformers/transformers/src/transformers/file_utils.py", line 1346, in wrapper
E               return func(*args, **kwargs)
E             File "/home/github_actions/actions-runner/_work/transformers/transformers/src/transformers/training_args.py", line 601, in device
E               return self._setup_devices
E             File "/home/github_actions/actions-runner/_work/transformers/transformers/src/transformers/file_utils.py", line 1336, in __get__
E               cached = self.fget(obj)
E             File "/home/github_actions/actions-runner/_work/transformers/transformers/src/transformers/file_utils.py", line 1346, in wrapper
E               return func(*args, **kwargs)
E             File "/home/github_actions/actions-runner/_work/transformers/transformers/src/transformers/training_args.py", line 586, in _setup_devices
E               torch.distributed.init_process_group(backend="nccl")
E             File "/home/github_actions/actions-runner/_work/transformers/transformers/.env/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py", line 455, in init_process_group
E               barrier()
E             File "/home/github_actions/actions-runner/_work/transformers/transformers/.env/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py", line 1960, in barrier
E               work = _default_pg.barrier()
E           RuntimeError: NCCL error in: /pytorch/torch/lib/c10d/ProcessGroupNCCL.cpp:784, unhandled system error, NCCL version 2.7.8

src/transformers/testing_utils.py:1062: RuntimeError

```

We can close this one, right, @LysandreJik?

It was happening because some zombies from previous jobs, which will now kill before the job starts. i.e. this test wasn't at fault.

Yes, we can close! Thanks.