torch.distributed.barrier() have NCCL error
Author: Limtle
Pull request: no

## Details
<!-- Description of your issue -->
**singularity :** ```singularity build pytorch20.02.simg docker://nvcr.io/nvidia/pytorch:20.02-py3```

I use Slurm and singularity to run run_glue.py but have NCCL error on torch.distributed.barrier()
### test.sh
```
#!/bin/bash
#SBATCH --nodes=1    
#SBATCH --ntasks=2   
#SBATCH --cpus-per-task=4
#SBATCH --gres=gpu:2        
#SBATCH --mem=161920
module purge
module load compiler/gnu/7.3.0 openmpi3 singularity

singularity exec pytorch20.02.simg python -m torch.distributed.launch --nproc_per_node 2 run_glue.py --model_type bert --model_name_or_path bert-base-uncased --task_name MRPC --do_train --do_eval --do_lower_case --data_dir ./glue_data/MRPC --max_seq_length 128 --per_gpu_eval_batch_size=8 --per_gpu_train_batch_size=8 --learning_rate 2e-5 --num_train_epochs 3.0 --output_dir /tmp/MRPC/
```
### Output
```
Traceback (most recent call last):
Traceback (most recent call last):
  File "run_glue.py", line 701, in <module>
  File "run_glue.py", line 701, in <module>
    main()
  File "run_glue.py", line 618, in main
    main()
  File "run_glue.py", line 641, in main
    torch.distributed.barrier()  # Make sure only the first process in distributed training will download model & vocab
  File "/opt/conda/lib/python3.6/site-packages/torch/distributed/distributed_c10d.py", line 1489, in barrier
    torch.distributed.barrier()  # Make sure only the first process in distributed training will download model & vocab
  File "/opt/conda/lib/python3.6/site-packages/torch/distributed/distributed_c10d.py", line 1489, in barrier
    work = _default_pg.barrier()
RuntimeError: NCCL error in: ../torch/lib/c10d/ProcessGroupNCCL.cpp:450, unhandled system error, NCCL version 2.5.6
    work = _default_pg.barrier()
RuntimeError: NCCL error in: ../torch/lib/c10d/ProcessGroupNCCL.cpp:450, unhandled system error, NCCL version 2.5.6
[E ProcessGroupNCCL.cpp:284] NCCL watchdog thread terminated
[E ProcessGroupNCCL.cpp:284] NCCL watchdog thread terminated
Traceback (most recent call last):
  File "/opt/conda/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/opt/conda/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/opt/conda/lib/python3.6/site-packages/torch/distributed/launch.py", line 263, in <module>
    main()
  File "/opt/conda/lib/python3.6/site-packages/torch/distributed/launch.py", line 259, in main
    cmd=cmd)
subprocess.CalledProcessError: Command '['/opt/conda/bin/python', '-u', 'run_glue.py', '--local_rank=1', '--model_type', 'bert', '--model_name_or_path', 'bert-base-uncased', '--task_name', 'MRPC', '--do_train', '--do_eval', '--do_lower_case', '--data_dir', '.glue_data/MRPC', '--max_seq_length', '128', '--per_gpu_eval_batch_size=8', '--per_gpu_train_batch_size=8', '--learning_rate', '2e-5', '--num_train_epochs', '3.0', '--output_dir', '/tmp/MRPC/']' returned non-zero exit status 1.
*****************************************
Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. 
*****************************************
```
Sloved
```
#!/bin/bash
#SBATCH --nodes=1    
#SBATCH --ntasks=2   
#SBATCH --cpus-per-task=4
#SBATCH --gres=gpu:2        
#SBATCH --mem=161920
module purge
module load compiler/gnu/7.3.0 openmpi3 singularity

singularity exec --nv pytorch20.02.simg python -m torch.distributed.launch --nproc_per_node 2 run_glue.py --model_type bert --model_name_or_path bert-base-uncased --task_name MRPC --do_train --do_eval --do_lower_case --data_dir ./glue_data/MRPC --max_seq_length 128 --per_gpu_eval_batch_size=8 --per_gpu_train_batch_size=8 --learning_rate 2e-5 --num_train_epochs 3.0 --output_dir /tmp/MRPC/
```

Comments: