Issue 1:
I tried to run an 8 GPU training job, and it OOM'd, so I investigated whether it could run on 1 GPU. It could!

So here are two commands, the first one says that it is using 14814MiB on GPU 0.

The second says it is using

15594MiB on each.

This doesn't happen in PL, which leads me to believe that distributed_scalars is to blame, but I am not sure. Has anyone run into this?

cc @sgugger @patil-suraj

Delta between two commands

- CUDA_VISIBLE_DEVICES=0 python
+ CUDA_VISIBLE_DEVICES=0,1 python -m torch.distributed.launch --nproc_per_node=2

Command 1

CUDA_VISIBLE_DEVICES=0 python finetune_trainer.py \
--model_name_or_path student_pegasus_cnn_12_2 \
--data_dir cnn_dm \
--output_dir dpx_cnn_12_2_pl_comb_noDO --overwrite_output_dir --freeze_embeds \
--learning_rate=3e-5 \
--warmup_steps 500 --sortish_sampler \
--gradient_accumulation_steps=4 \
--per_device_train_batch_size=4 --per_device_eval_batch_size=8 --eval_beams 2 \
--num_train_epochs=5 \
--save_steps 3000 --eval_steps 3000 \
--logging_first_step \
--max_target_length 56 --val_max_target_length 142 --test_max_target_length 142 \
--do_train --do_eval --do_predict --evaluate_during_training \
--predict_with_generate --load_best_model_at_end

Command 2

CUDA_VISIBLE_DEVICES=0,1 python -m torch.distributed.launch --nproc_per_node=2 fine
tune_trainer.py \
--model_name_or_path student_pegasus_cnn_12_2 \
--data_dir cnn_dm \
--output_dir dpx_cnn_12_2_pl_comb_noDO --overwrite_output_dir --freeze_embeds \
--learning_rate=3e-5 \
--warmup_steps 500 --sortish_sampler \
--gradient_accumulation_steps=4 \
--per_device_train_batch_size=4 --per_device_eval_batch_size=8 --eval_beams 2 \
--num_train_epochs=5 \
--save_steps 3000 --eval_steps 3000 \
--logging_first_step \
--max_target_length 56 --val_max_target_length 142 --test_max_target_length 142 \
--do_train --do_eval --do_predict --evaluate_during_training \
--predict_with_generate --load_best_model_at_end

Issue 2:
Environment info

transformers version: 3.0.2

Platform: Linux-5.3.0

Python version: 3.6.10

PyTorch version (GPU?): 1.6.0 (True)

Tensorflow version (GPU?): 2.3.0 (True)

Using GPU in script?: Yes

Using distributed or parallel set-up in script?: Distributed

Who can help

@LysandreJik, @sgugger, @patrickvonplaten

Information

I am using 8xV100s (32GB). The script (run_training.py) works when running on a single machine but I am running into the CUDA out of memory when trying to run distributed training. The behavior is consistent whether or not fp16 is True. I am using the publicly available wikitext data.

Model I am using (Bert, XLNet ...): Bert

The problem arises when using:

the official example scripts: (give details below)

my own modified scripts: (give details below)

The tasks I am working on is:

an official GLUE/SQUaD task: (give the name)

my own task or dataset: (give details below)

To reproduce

Steps to reproduce the behavior:

My code is run_training.py:

bert_config = BertConfig(hidden_size=768, num_attention_heads=12)
tokenizer = BertTokenizer.from_pretrained("bert-base-uncased")
model = BertForPreTraining(config=bert_config)
train_dataset = TextDatasetForNextSentencePrediction(tokenizer=tokenizer, file_path=args.train_data_file, block_size=args.max_length)
data_collator = DataCollatorForNextSentencePrediction(tokenizer=tokenizer, mlm_probability=args.mlm_probability)
trainer = Trainer(
model=model,
args=training_args,
data_collator=data_collator,
train_dataset=train_dataset
)
global_steps, training_loss = trainer.train()

Run the script in distributed mode:

python -m torch.distributed.launch \
--nnodes={num_nodes} \
--node_rank={rank} \
--nproc_per_node=8 \
--run_training.py \
--max_steps 50 " \
--train_data_file {train_data_file} " \
--eval_data_file {test_data_file} " \
--logging_dir {output_dir} \
--fp16 False " \
--gradient_accumulation_steps 1 " \
--per_gpu_train_batch_size 1 \
--per_gpu_eval_batch_size 1

Expected behavior

Given that the script works fine (i.e., not run into the out of memory issue) on a single machine, I would expect multi-node to be the same. Any insight into what might be going on is appreciated!