text: 
  
  TITLE
  AssertionError with model_parallel in run_clm.py
  
  COMMENTS
  9
  
  REACTIONS
  +1: 0
  -1: 0
  laugh: 0
  hooray: 0
  heart: 0
  rocket: 0
  eyes: 0
  
  BODY
  ## Environment info
  <!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
       Don't forget to fill out the missing fields in that output! -->
       
  - `transformers` version: 4.1.1
  - Platform: AWS Sagemaker
  - Python version:  3.6
  - PyTorch version (GPU?): 1.6
  - Tensorflow version (GPU?):
  - Using GPU in script?: YES
  - Using distributed or parallel set-up in script?: YES
  
  ### Who can help
  <!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
   If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
   Please tag fewer than 3 people.
   
   albert, bert, GPT2, XLM: @LysandreJik 
   tokenizers: @mfuntowicz
   Trainer: @sgugger
   Speed and Memory Benchmarks: @patrickvonplaten
   Model Cards: @julien-c
   TextGeneration: @TevenLeScao 
   examples/distillation: @VictorSanh
   nlp datasets: [different repo](https://github.com/huggingface/nlp)
   rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
   Text Generation: @patrickvonplaten @TevenLeScao
   Blenderbot: @patrickvonplaten
   Bart: @patrickvonplaten
   Marian: @patrickvonplaten
   Pegasus: @patrickvonplaten
   mBART: @patrickvonplaten
   T5: @patrickvonplaten
   Longformer/Reformer: @patrickvonplaten
   TransfoXL/XLNet: @TevenLeScao
   RAG: @patrickvonplaten, @lhoestq
   FSMT: @stas00
   examples/seq2seq: @patil-suraj
   examples/bert-loses-patience: @JetRunner
   tensorflow: @jplu
   examples/token-classification: @stefan-it
   documentation: @sgugger
   -->
  @LysandreJik  @sgugger @alexorona 
  
  
  ## Information
  
  Model I am using (Bert, XLNet ...): GPT2
  
  The problem arises when using:
  * [X] the official example scripts: (give details below)
  * [ ] my own modified scripts: (give details below)
  
  The tasks I am working on is:
  * [ ] an official GLUE/SQUaD task: (give the name)
  * [X] my own task or dataset: (give details below)
  
  I am fine-tuning GPT2 using my own dataset, using the examples/language-modelling/run_clm.py script, and I want to use the new model_parallel feature in v4.1.1. I am using a multi-gpu instance (AWS p3.8xlarge - with 4 gpus). 
  
  But I get this error:
  AssertionError: DistributedDataParallel device_ids and output_device arguments only work with single-device CUDA modules, but got device_ids [2], output_device 2, and module parameters {device(type='cpu')}.
  
  ## To reproduce
  
  Steps to reproduce the behavior:
  
  1. Run run_clm.py with the following params:
  ```
  python -m torch.distributed.launch \
      --nproc_per_node 4 run_clm.py \
      --do_train \
      --do_eval \
      --fp16 \
      --logging_first_step \
      --model_parallel \
      --evaluation_strategy epoch \
      --logging_steps 50 \
      --model_name_or_path gpt2 \
      --model_type gpt2 \
      --num_train_epochs 1 \
      --output_dir /opt/ml/model/ \
      --per_device_eval_batch_size 2 \
      --per_device_train_batch_size 2 \
      --save_steps 50 \
      --save_total_limit 1 \
      --train_file /opt/ml/input/data/data/train.txt \
      --validation_file /opt/ml/input/data/data/val.txt
  ```
  
  2. I get this error when training starts:
  ```
  Traceback (most recent call last):
    File "run_clm.py", line 374, in <module>
      main()
    File "run_clm.py", line 344, in main
      trainer.train(model_path=model_path)
    File "/opt/conda/lib/python3.6/site-packages/transformers/trainer.py", line 681, in train
      else True
    File "/opt/conda/lib/python3.6/site-packages/torch/nn/parallel/distributed.py", line 282, in __init__
  #015 95%|ââââââââââ| 62/65 [00:04<00:00, 12.58ba/s]    ).format(device_ids, output_device, {p.device for p in module.parameters()})
  AssertionError: DistributedDataParallel device_ids and output_device arguments only work with single-device CUDA modules, but got device_ids [1], output_device 1, and module parameters {device(type='cpu')}.
  #015 98%|ââââââââââ| 64/65 [00:05<00:00, 12.26ba/s]#015100%|ââââââââââ| 65/65 [00:05<00:00, 12.61ba/s]
  Traceback (most recent call last):
    File "run_clm.py", line 374, in <module>
      main()
    File "run_clm.py", line 344, in main
      trainer.train(model_path=model_path)
    File "/opt/conda/lib/python3.6/site-packages/transformers/trainer.py", line 681, in train
      else True
    File "/opt/conda/lib/python3.6/site-packages/torch/nn/parallel/distributed.py", line 282, in __init__
      ).format(device_ids, output_device, {p.device for p in module.parameters()})
  AssertionError: DistributedDataParallel device_ids and output_device arguments only work with single-device CUDA modules, but got device_ids [2], output_device 2, and module parameters {device(type='cpu')}.
  #015 95%|ââââââââââ| 62/65 [00:05<00:00, 11.84ba/s]#015 98%|ââââââââââ| 64/65 [00:05<00:00, 12.18ba/s]#015100%|ââââââââââ| 65/65 [00:05<00:00, 12.60ba/s]
  Traceback (most recent call last):
    File "run_clm.py", line 374, in <module>
      main()
    File "run_clm.py", line 344, in main
      trainer.train(model_path=model_path)
    File "/opt/conda/lib/python3.6/site-packages/transformers/trainer.py", line 681, in train
      else True
    File "/opt/conda/lib/python3.6/site-packages/torch/nn/parallel/distributed.py", line 282, in __init__
      ).format(device_ids, output_device, {p.device for p in module.parameters()})
  AssertionError: DistributedDataParallel device_ids and output_device arguments only work with single-device CUDA modules, but got device_ids [3], output_device 3, and module parameters {device(type='cpu')}.
  #015 98%|ââââââââââ| 64/65 [00:05<00:00, 11.50ba/s]#015100%|ââââââââââ| 65/65 [00:05<00:00, 12.48ba/s]
  [INFO|trainer.py:388] 2020-12-22 00:54:34,892 >> The following columns in the training set don't have a corresponding argument in `GPT2LMHeadModel.forward` and have been ignored: .
  [INFO|trainer.py:388] 2020-12-22 00:54:34,892 >> The following columns in the evaluation set don't have a corresponding argument in `GPT2LMHeadModel.forward` and have been ignored: .
  Traceback (most recent call last):
    File "run_clm.py", line 374, in <module>
      main()
    File "run_clm.py", line 344, in main
      trainer.train(model_path=model_path)
    File "/opt/conda/lib/python3.6/site-packages/transformers/trainer.py", line 681, in train
      else True
    File "/opt/conda/lib/python3.6/site-packages/torch/nn/parallel/distributed.py", line 282, in __init__
      ).format(device_ids, output_device, {p.device for p in module.parameters()})
  AssertionError: DistributedDataParallel device_ids and output_device arguments only work with single-device CUDA modules, but got device_ids [0], output_device 0, and module parameters {device(type='cpu')}.
  Traceback (most recent call last):
    File "/opt/conda/lib/python3.6/runpy.py", line 193, in _run_module_as_main
      "__main__", mod_spec)
    File "/opt/conda/lib/python3.6/runpy.py", line 85, in _run_code
      exec(code, run_globals)
    File "/opt/conda/lib/python3.6/site-packages/torch/distributed/launch.py", line 261, in <module>
      main()
    File "/opt/conda/lib/python3.6/site-packages/torch/distributed/launch.py", line 257, in main
      cmd=cmd)
  subprocess.CalledProcessError: Command '['/opt/conda/bin/python', '-u', 'run_clm.py', '--local_rank=3', '--do_train', '--do_eval', '--fp16', '--logging_first_step', '--model_parallel', '--evaluation_strategy', 'epoch', '--logging_steps', '50', '--model_name_or_path', 'gpt2', '--model_type', 'gpt2', '--num_train_epochs', '1', '--output_dir', '/opt/ml/model/', '--per_device_eval_batch_size', '2', '--per_device_train_batch_size', '2', '--save_steps', '50', '--save_total_limit', '1', '--train_file', '/opt/ml/input/data/data/train.txt', '--validation_file', '/opt/ml/input/data/data/val.txt']' returned non-zero exit status 1.
  ```
  
  
  <!-- If you have code snippets, error messages, stack traces please provide them here as well.
       Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
       Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->
  
  ## Expected behavior
  
  <!-- A clear and concise description of what you would expect to happen. -->
  I expect the fine-tuning script to run successfully. 
  
  If I remove the --model_parallel in the args, then it does run successfully in distributed mode. But I want to use this new feature to reduce memory usage, and increase batch_size
  