Issue 1:
Environment info

transformers version: 3.1.0

Platform: Linux-3.10.0-1127.el7.x86_64-x86_64-with-debian-buster-sid

Python version: 3.7.0

PyTorch version (GPU?): 1.5.1 (True)

Tensorflow version (GPU?): 2.2.0 (False)

Using GPU in script?: yes

Using distributed or parallel set-up in script?: distributed

Who can help

@LysandreJik, @sgugger, @patrickvonplaten

Information

Model I am using (Bert, GPT2):

The problem arises when using:

[ X] the official example scripts: (give details below)

my own modified scripts: (give details below)

The tasks I am working on is:

an official GLUE/SQUaD task: (give the name)

[X ] my own task or dataset: (give details below)

To reproduce

When I pretrain or fine tune a model (in my case BERT and GPT2) using torch.distributed.launch, the CPU memory usage will grow up to the memory limit (>500GB) until the first process is killed due to this issue. If I train bert-base, it takes around 30 epochs until the first process is killed, but when I train gpt-large, it just need 3 epochs until it is killed. Following is the command line I run to train/fine tune the bert-base (similar with gpt2). The script run_language_modeling.py is a copy of transformers/examples/language-modeling/run_language_modeling.py (vers. 3.1.0)

python -m torch.distributed.launch --nproc_per_node=8

../run_language_modeling.py

--output_dir $model_target

--model_name_or_path $model_source

--config_name $model_source

--tokenizer_name $model_source

--train_data_file $target_train

--eval_data_file $target_test

--save_total_limit 5

--block_size 128

--overwrite_output_dir

--fp16

--num_train_epochs 50

--do_train --do_eval

--per_device_train_batch_size 32

--per_device_eval_batch_size 4

--mlm

Expected behavior

I would expect that the distributed training run until it is done without any memory issue.

Thanks for checking it.

Issue 2:
Environment info

transformers version: 3.0.2

Platform: Linux-5.3.0

Python version: 3.6.10

PyTorch version (GPU?): 1.6.0 (True)

Tensorflow version (GPU?): 2.3.0 (True)

Using GPU in script?: Yes

Using distributed or parallel set-up in script?: Distributed

Who can help

@LysandreJik, @sgugger, @patrickvonplaten

Information

I am using 8xV100s (32GB). The script (run_training.py) works when running on a single machine but I am running into the CUDA out of memory when trying to run distributed training. The behavior is consistent whether or not fp16 is True. I am using the publicly available wikitext data.

Model I am using (Bert, XLNet ...): Bert

The problem arises when using:

the official example scripts: (give details below)

my own modified scripts: (give details below)

The tasks I am working on is:

an official GLUE/SQUaD task: (give the name)

my own task or dataset: (give details below)

To reproduce

Steps to reproduce the behavior:

My code is run_training.py:

bert_config = BertConfig(hidden_size=768, num_attention_heads=12)
tokenizer = BertTokenizer.from_pretrained("bert-base-uncased")
model = BertForPreTraining(config=bert_config)
train_dataset = TextDatasetForNextSentencePrediction(tokenizer=tokenizer, file_path=args.train_data_file, block_size=args.max_length)
data_collator = DataCollatorForNextSentencePrediction(tokenizer=tokenizer, mlm_probability=args.mlm_probability)
trainer = Trainer(
model=model,
args=training_args,
data_collator=data_collator,
train_dataset=train_dataset
)
global_steps, training_loss = trainer.train()

Run the script in distributed mode:

python -m torch.distributed.launch \
--nnodes={num_nodes} \
--node_rank={rank} \
--nproc_per_node=8 \
--run_training.py \
--max_steps 50 " \
--train_data_file {train_data_file} " \
--eval_data_file {test_data_file} " \
--logging_dir {output_dir} \
--fp16 False " \
--gradient_accumulation_steps 1 " \
--per_gpu_train_batch_size 1 \
--per_gpu_eval_batch_size 1

Expected behavior

Given that the script works fine (i.e., not run into the out of memory issue) on a single machine, I would expect multi-node to be the same. Any insight into what might be going on is appreciated!