BERT Trainer.train() CUDA out of memory error
Author: choidongyeon
Pull request: no
Labels: name: wontfix

## Environment info

- `transformers` version: 3.0.2
- Platform: Linux-5.3.0
- Python version: 3.6.10
- PyTorch version (GPU?): 1.6.0 (True)
- Tensorflow version (GPU?): 2.3.0 (True)
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: Distributed

### Who can help
@LysandreJik, @sgugger, @patrickvonplaten

## Information
I am using 8xV100s (32GB). The script (`run_training.py`) works when running on a single machine but I am running into the ```CUDA out of memory``` when trying to run distributed training. The behavior is consistent whether or not `fp16` is `True`. I am using the publicly available wikitext data. 

Model I am using (Bert, XLNet ...): Bert

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. My code is `run_training.py`:
```
bert_config = BertConfig(hidden_size=768, num_attention_heads=12)
tokenizer = BertTokenizer.from_pretrained("bert-base-uncased")
model = BertForPreTraining(config=bert_config)
train_dataset = TextDatasetForNextSentencePrediction(tokenizer=tokenizer, file_path=args.train_data_file, block_size=args.max_length)
data_collator = DataCollatorForNextSentencePrediction(tokenizer=tokenizer, mlm_probability=args.mlm_probability)
trainer = Trainer(
        model=model,
        args=training_args,
        data_collator=data_collator,
        train_dataset=train_dataset
    )
global_steps, training_loss = trainer.train()
```

2. Run the script in distributed mode:
```
python -m torch.distributed.launch \
  --nnodes={num_nodes} \
  --node_rank={rank} \
  --nproc_per_node=8  \
  --run_training.py \
  --max_steps 50 " \
  --train_data_file {train_data_file} " \
  --eval_data_file {test_data_file} " \
  --logging_dir {output_dir} \
  --fp16 False " \
  --gradient_accumulation_steps 1 " \
  --per_gpu_train_batch_size 1 \
  --per_gpu_eval_batch_size 1
```

## Expected behavior

Given that the script works fine (i.e., not run into the out of memory issue) on a single machine, I would expect multi-node to be the same. Any insight into what might be going on is appreciated!

Comments:
Haven’t we fixed a memory leak on master recently?

Yes we did, though this might be a different problem.

To make sure, can you please check if you have the bug with an install from source?

@sgugger yeah, I'm installing from source (not sure why the script yielded 3.0.2)

I encountered the problem, too, especially using NSP objective, the memory usage much higher than MLM.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


I encounter this problem too when I try to train on multi-nodes and multi GPUs. 
And my transformers version is 4.15.0, so does this issue have the answer?
@choidongyeon @julien-c @sgugger