Cannot train RoBERTa from scratch with multiple nodes and multiple GPUs
Author: chiyuzhang94
Pull request: no

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...):

Language I am using the model on (English, Chinese ...):

The problem arises when using:
- [x] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
- [x] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. Use the `transformers/examples/language-modeling/run_language_modeling.py` to train RoBERT model from scratch. 
2. I am using the Slurm tool to submit a job that requires multiple nodes and multiple GPUs.  In this case, it requests 2 nodes and 2 GPUs for each. The shell scripts: 
```
#!/bin/bash
#SBATCH --time=3:00:00
#SBATCH --ntasks=2
#SBATCH --nodes=2
#SBATCH --cpus-per-task=2
#SBATCH --gres=gpu:2
#SBATCH --mem=64G
#SBATCH --job-name=pre-train
#SBATCH --output=pre-train.out
#SBATCH --account=XXXXXX

module load gcc
module load cuda cudnn
module load openmpi nccl

source ~/roberta/bin/activate
export NCCL_DEBUG=INFO
export NPROC_PER_NODE=2
export HDF5_USE_FILE_LOCKING='FALSE'
export PARENT=`/bin/hostname -s`
export MPORT=13000
export CHILDREN=`scontrol show hostnames $SLURM_JOB_NODELIST | grep -v $PARENT`
export HOSTLIST="$PARENT $CHILDREN"
echo $HOSTLIST
export WORLD_SIZE=$SLURM_NTASKS

srun distributed_runner.sh
```
3. `distributed_runner.sh` script is: 
```
#!/bin/bash
/bin/hostname -s
source ~/roberta/bin/activate
python3 -m torch.distributed.launch \
		--nproc_per_node=$NPROC_PER_NODE \
		--nnodes=$SLURM_JOB_NUM_NODES \
		--node_rank=$SLURM_PROCID \
		--master_addr="$PARENT" --master_port="$MPORT" \
		run_language_modeling.py \
		--gradient_accumulation_steps=16 \
		--train_data_file="./data/sample.txt" \
		--output_dir="./sample_model/" \
		--model_type=roberta \
		--mlm \
		--local_rank=$SLURM_LOCALID \
		--config_name="./sample_config" \
		--tokenizer_name="./sample_config" \
		--do_train \
		--line_by_line \
		--learning_rate=1e-4 \
		--num_train_epochs=40 \
		--save_total_limit=5 \
		--save_steps=20 \
		--per_gpu_train_batch_size=16 \
		--seed=42
```
4. `sample.txt` is a dummy training set with 20K lines where each line is a text string. Data is already pre-processed following the official instruction. `sample_config/` includes all the pre-processed vocabulary and config files: config.json, merges.txt, tokenizer_config.json, and vocab.json.  

5. The job can be successfully launched on two nodes, with no error. But **each node only uses one GPU** based on the GPU usage with `nvidia-smi` command. 

**Node 1**
```
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 440.33.01    Driver Version: 440.33.01    CUDA Version: 10.2     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla V100-SXM2...  On   | 00000000:3B:00.0 Off |                    0 |
| N/A   40C    P0    70W / 300W |  21175MiB / 32510MiB |    100%      Default |
+-------------------------------+----------------------+----------------------+
|   1  Tesla V100-SXM2...  On   | 00000000:86:00.0 Off |                    0 |
| N/A   35C    P0    42W / 300W |     11MiB / 32510MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0    190822      C   /home/py3.6/bin/python3                    13003MiB |
|    0    190823      C   /home/py3.6/bin/python3                     8161MiB |
+-----------------------------------------------------------------------------+
```

**Node 2**
```
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 440.33.01    Driver Version: 440.33.01    CUDA Version: 10.2     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla V100-SXM2...  On   | 00000000:18:00.0 Off |                    0 |
| N/A   43C    P0    77W / 300W |  17127MiB / 32510MiB |     97%      Default |
+-------------------------------+----------------------+----------------------+
|   1  Tesla V100-SXM2...  On   | 00000000:3B:00.0 Off |                    0 |
| N/A   33C    P0    40W / 300W |     11MiB / 32510MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0    209232      C   /home/py3.6/bin/python3                     6899MiB |
|    0    209233      C   /home/py3.6/bin/python3                    10217MiB |
+-----------------------------------------------------------------------------+
```

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->
Expect each node can use two GPUs. Namely, the Processes should show the usage of GPU 0 and 1 for both nodes. 
## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version: 3.0.1
- Platform: CentOS Linux 7 (Core)
- Python version: Python 3.6.3
- PyTorch version (GPU?): torch==1.5.1
- Tensorflow version (GPU?): tensorflow-gpu==2.1.0
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: Yes


Comments:
I figured out the issue. I should not set `--local_rank=$SLURM_LOCALID` in the argument. `torch.distributed.launch` will automatically pass the right --local_rank value to run_language_modeling.py.