Issue 1:
Environment info

transformers version: 4.1.1

Platform: AWS Sagemaker

Python version: 3.6

PyTorch version (GPU?): 1.6

Tensorflow version (GPU?):

Using GPU in script?: YES

Using distributed or parallel set-up in script?: YES

Who can help

@LysandreJik @sgugger @alexorona

Information

Model I am using (Bert, XLNet ...): GPT2

The problem arises when using:

the official example scripts: (give details below)

my own modified scripts: (give details below)

The tasks I am working on is:

an official GLUE/SQUaD task: (give the name)

my own task or dataset: (give details below)

I am fine-tuning GPT2 using my own dataset, using the examples/language-modelling/run_clm.py script, and I want to use the new model_parallel feature in v4.1.1. I am using a multi-gpu instance (AWS p3.8xlarge - with 4 gpus).

But I get this error:

AssertionError: DistributedDataParallel device_ids and output_device arguments only work with single-device CUDA modules, but got device_ids [2], output_device 2, and module parameters {device(type='cpu')}.

To reproduce

Steps to reproduce the behavior:

Run run_clm.py with the following params:

python -m torch.distributed.launch \
--nproc_per_node 4 run_clm.py \
--do_train \
--do_eval \
--fp16 \
--logging_first_step \
--model_parallel \
--evaluation_strategy epoch \
--logging_steps 50 \
--model_name_or_path gpt2 \
--model_type gpt2 \
--num_train_epochs 1 \
--output_dir /opt/ml/model/ \
--per_device_eval_batch_size 2 \
--per_device_train_batch_size 2 \
--save_steps 50 \
--save_total_limit 1 \
--train_file /opt/ml/input/data/data/train.txt \
--validation_file /opt/ml/input/data/data/val.txt

I get this error when training starts:

Traceback (most recent call last):
File "run_clm.py", line 374, in <module>
main()
File "run_clm.py", line 344, in main
trainer.train(model_path=model_path)
File "/opt/conda/lib/python3.6/site-packages/transformers/trainer.py", line 681, in train
else True
File "/opt/conda/lib/python3.6/site-packages/torch/nn/parallel/distributed.py", line 282, in __init__
#015 95%|â��â��â��â��â��â��â��â��â��â��| 62/65 [00:04<00:00, 12.58ba/s] ).format(device_ids, output_device, {p.device for p in module.parameters()})
AssertionError: DistributedDataParallel device_ids and output_device arguments only work with single-device CUDA modules, but got device_ids [1], output_device 1, and module parameters {device(type='cpu')}.
#015 98%|â��â��â��â��â��â��â��â��â��â��| 64/65 [00:05<00:00, 12.26ba/s]#015100%|â��â��â��â��â��â��â��â��â��â��| 65/65 [00:05<00:00, 12.61ba/s]
Traceback (most recent call last):
File "run_clm.py", line 374, in <module>
main()
File "run_clm.py", line 344, in main
trainer.train(model_path=model_path)
File "/opt/conda/lib/python3.6/site-packages/transformers/trainer.py", line 681, in train
else True
File "/opt/conda/lib/python3.6/site-packages/torch/nn/parallel/distributed.py", line 282, in __init__
).format(device_ids, output_device, {p.device for p in module.parameters()})
AssertionError: DistributedDataParallel device_ids and output_device arguments only work with single-device CUDA modules, but got device_ids [2], output_device 2, and module parameters {device(type='cpu')}.
#015 95%|â��â��â��â��â��â��â��â��â��â��| 62/65 [00:05<00:00, 11.84ba/s]#015 98%|â��â��â��â��â��â��â��â��â��â��| 64/65 [00:05<00:00, 12.18ba/s]#015100%|â��â��â��â��â��â��â��â��â��â��| 65/65 [00:05<00:00, 12.60ba/s]
Traceback (most recent call last):
File "run_clm.py", line 374, in <module>
main()
File "run_clm.py", line 344, in main
trainer.train(model_path=model_path)
File "/opt/conda/lib/python3.6/site-packages/transformers/trainer.py", line 681, in train
else True
File "/opt/conda/lib/python3.6/site-packages/torch/nn/parallel/distributed.py", line 282, in __init__
).format(device_ids, output_device, {p.device for p in module.parameters()})
AssertionError: DistributedDataParallel device_ids and output_device arguments only work with single-device CUDA modules, but got device_ids [3], output_device 3, and module parameters {device(type='cpu')}.
#015 98%|â��â��â��â��â��â��â��â��â��â��| 64/65 [00:05<00:00, 11.50ba/s]#015100%|â��â��â��â��â��â��â��â��â��â��| 65/65 [00:05<00:00, 12.48ba/s]
[INFO|trainer.py:388] 2020-12-22 00:54:34,892 >> The following columns in the training set don't have a corresponding argument in `GPT2LMHeadModel.forward` and have been ignored: .
[INFO|trainer.py:388] 2020-12-22 00:54:34,892 >> The following columns in the evaluation set don't have a corresponding argument in `GPT2LMHeadModel.forward` and have been ignored: .
Traceback (most recent call last):
File "run_clm.py", line 374, in <module>
main()
File "run_clm.py", line 344, in main
trainer.train(model_path=model_path)
File "/opt/conda/lib/python3.6/site-packages/transformers/trainer.py", line 681, in train
else True
File "/opt/conda/lib/python3.6/site-packages/torch/nn/parallel/distributed.py", line 282, in __init__
).format(device_ids, output_device, {p.device for p in module.parameters()})
AssertionError: DistributedDataParallel device_ids and output_device arguments only work with single-device CUDA modules, but got device_ids [0], output_device 0, and module parameters {device(type='cpu')}.
Traceback (most recent call last):
File "/opt/conda/lib/python3.6/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/opt/conda/lib/python3.6/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/opt/conda/lib/python3.6/site-packages/torch/distributed/launch.py", line 261, in <module>
main()
File "/opt/conda/lib/python3.6/site-packages/torch/distributed/launch.py", line 257, in main
cmd=cmd)
subprocess.CalledProcessError: Command '['/opt/conda/bin/python', '-u', 'run_clm.py', '--local_rank=3', '--do_train', '--do_eval', '--fp16', '--logging_first_step', '--model_parallel', '--evaluation_strategy', 'epoch', '--logging_steps', '50', '--model_name_or_path', 'gpt2', '--model_type', 'gpt2', '--num_train_epochs', '1', '--output_dir', '/opt/ml/model/', '--per_device_eval_batch_size', '2', '--per_device_train_batch_size', '2', '--save_steps', '50', '--save_total_limit', '1', '--train_file', '/opt/ml/input/data/data/train.txt', '--validation_file', '/opt/ml/input/data/data/val.txt']' returned non-zero exit status 1.

Expected behavior

I expect the fine-tuning script to run successfully.

If I remove the --model_parallel in the args, then it does run successfully in distributed mode. But I want to use this new feature to reduce memory usage, and increase batch_size

Issue 2:
❓ Questions & Help

I saw that "dynamic masking" was mentioned on the README file for language modeling:

"In accordance to the RoBERTa paper, we use dynamic masking rather than static masking. The model may, therefore, converge slightly slower (over-fitting takes more epochs)."

I couldn't find which class this method is implemented in and how to enable this feature during pre-training using the Trainer class. Could someone please help me?

Thank you very much in advance.