run_glue.py RuntimeError: module must have its parameters and buffers on device cuda:0 (device_ids[0]) but found one of them on device: cuda:3
Author: insublee
Pull request: no
Labels: name: wontfix

## 🐛 Bug

<!-- Important information -->

Model I am using (Bert, XLNet....): Bert

Language I am using the model on (English, Chinese....): English

The problem arise when using:
* [ ] the official example scripts: (give details)  : transformers/examples/run_glue.py
* [ ] my own modified scripts: (give details)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name) :  MRPC
* [ ] my own task or dataset: (give details)

## To Reproduce

Steps to reproduce the behavior:

1.
I've tested using
python -m pytest -sv ./transformers/tests/
python -m pytest -sv ./examples/
and it works fine without couple of tesks.

2.
after test, i downloaded glue datafile via
https://gist.github.com/W4ngatang/60c2bdb54d156a41194446737ce03e2e
and tried run_glue.py

pip install -r ./examples/requirements.txt
export GLUE_DIR=/path/to/glue
export TASK_NAME=MRPC


3.
python ./examples/run_glue.py \
    --model_type bert \
    --model_name_or_path bert-base-uncased \
    --task_name $TASK_NAME \
    --do_train \
    --do_eval \
    --do_lower_case \
    --data_dir $GLUE_DIR/$TASK_NAME \
    --max_seq_length 128 \
    --per_gpu_eval_batch_size=8   \
    --per_gpu_train_batch_size=8   \
    --learning_rate 2e-5 \
    --num_train_epochs 3.0 \
    --output_dir /tmp/$TASK_NAME/

and i got this error.

`11/11/2019 21:10:50 - INFO - __main__ -     Total optimization steps = 345
Epoch:   0%|                                                                                    | 0/3 [00:00<?, ?it/sTraceback (most recent call last):                                                             | 0/115 [00:00<?, ?it/s]
  File "./examples/run_glue.py", line 552, in <module>
    main()
  File "./examples/run_glue.py", line 503, in main
    global_step, tr_loss = train(args, train_dataset, model, tokenizer)
  File "./examples/run_glue.py", line 146, in train
    outputs = model(**inputs)
  File "/home/insublee/anaconda3/envs/py_torch4/lib/python3.7/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/insublee/anaconda3/envs/py_torch4/lib/python3.7/site-packages/torch/nn/parallel/data_parallel.py", line 146, in forward
    "them on device: {}".format(self.src_device_obj, t.device))
RuntimeError: module must have its parameters and buffers on device cuda:0 (device_ids[0]) but found one of them on device: cuda:3`
<!-- A clear and concise description of what you expected to happen. -->

## Environment

* OS: ubuntu16.04LTS
* Python version:                                                      3.7.5
* PyTorch version:                                                     1.2.0
* PyTorch Transformers version (or branch):             2.1.1
* Using GPU ?                                                           4-way 2080ti
* Distributed of parallel setup ?                                cuda10.0 cudnn 7.6.4
* Any other relevant information:

## Additional context
thank you.

Comments:
This problem comes out from multiple GPUs usage. The error you have reported says that you have parameters or the buffers of the model in **two different locations**.  
Said this, it's probably related to #1504 issue. Reading comments in the #1504 issue, i saw that @h-sugi suggests 4 days ago to modify the source code in `run_**.py` like this:

BEFORE: `device = torch.device("cuda" if torch.cuda.is_available() and not args.no_cuda else "cpu")`
AFTER: `device = torch.device("cuda:0" if torch.cuda.is_available() and not args.no_cuda else "cpu").`

Moreover, the person who opened the issue @ahotrod says this: _Have had many successful SQuAD fine-tuning runs on PyTorch 1.2.0 with Pytorch-Transformers 1.2.0, maybe even Transformers 2.0.0, and Apex 0.1. New environment built with the latest versions (Pytorch 1.3.0, Transformers 2.1.1) spawns data parallel related error above_

Please, keep us updated on this topic!

> ##  Bug
> Model I am using (Bert, XLNet....): Bert
> 
> Language I am using the model on (English, Chinese....): English
> 
> The problem arise when using:
> 
> * [ ]  the official example scripts: (give details)  : transformers/examples/run_glue.py
> * [ ]  my own modified scripts: (give details)
> 
> The tasks I am working on is:
> 
> * [ ]  an official GLUE/SQUaD task: (give the name) :  MRPC
> * [ ]  my own task or dataset: (give details)
> 
> ## To Reproduce
> Steps to reproduce the behavior:
> 
> 
> I've tested using
> python -m pytest -sv ./transformers/tests/
> python -m pytest -sv ./examples/
> and it works fine without couple of tesks.
> 
> 
> after test, i downloaded glue datafile via
> https://gist.github.com/W4ngatang/60c2bdb54d156a41194446737ce03e2e
> and tried run_glue.py
> 
> pip install -r ./examples/requirements.txt
> export GLUE_DIR=/path/to/glue
> export TASK_NAME=MRPC
> 
> 
> python ./examples/run_glue.py 
> --model_type bert 
> --model_name_or_path bert-base-uncased 
> --task_name $TASK_NAME 
> --do_train 
> --do_eval 
> --do_lower_case 
> --data_dir $GLUE_DIR/$TASK_NAME 
> --max_seq_length 128 
> --per_gpu_eval_batch_size=8 
> --per_gpu_train_batch_size=8 
> --learning_rate 2e-5 
> --num_train_epochs 3.0 
> --output_dir /tmp/$TASK_NAME/
> 
> and i got this error.
> 
> `11/11/2019 21:10:50 - INFO - __main__ - Total optimization steps = 345 Epoch: 0%| | 0/3 [00:00<?, ?it/sTraceback (most recent call last): | 0/115 [00:00<?, ?it/s] File "./examples/run_glue.py", line 552, in <module> main() File "./examples/run_glue.py", line 503, in main global_step, tr_loss = train(args, train_dataset, model, tokenizer) File "./examples/run_glue.py", line 146, in train outputs = model(**inputs) File "/home/insublee/anaconda3/envs/py_torch4/lib/python3.7/site-packages/torch/nn/modules/module.py", line 541, in __call__ result = self.forward(*input, **kwargs) File "/home/insublee/anaconda3/envs/py_torch4/lib/python3.7/site-packages/torch/nn/parallel/data_parallel.py", line 146, in forward "them on device: {}".format(self.src_device_obj, t.device)) RuntimeError: module must have its parameters and buffers on device cuda:0 (device_ids[0]) but found one of them on device: cuda:3`
> 
> ## Environment
> * OS: ubuntu16.04LTS
> * Python version:                                                      3.7.5
> * PyTorch version:                                                     1.2.0
> * PyTorch Transformers version (or branch):             2.1.1
> * Using GPU ?                                                           4-way 2080ti
> * Distributed of parallel setup ?                                cuda10.0 cudnn 7.6.4
> * Any other relevant information:
> 
> ## Additional context
> thank you.

thanks a lot. it works!!!! :)

@TheEdoardo93  After the change of `cuda` to `cuda:0`, will we still have multiple GPU usage for the jobs?

As stated in the official [docs](https://pytorch.org/tutorials/beginner/blitz/data_parallel_tutorial.html), if you use `torch.device('cuda:0')` you will use **only** a GPU. If you want to use **multiple** GPUs, you can easily run your operations on multiple GPUs by making your model run parallelly using DataParallel: `model = nn.DataParallel(model)`

You can read more information [here](https://discuss.pytorch.org/t/run-pytorch-on-multiple-gpus/20932/38) and [here](https://pytorch.org/tutorials/beginner/former_torchies/parallelism_tutorial.html).

> @TheEdoardo93 After the change of `cuda` to `cuda:0`, will we still have multiple GPU usage for the jobs?

@TheEdoardo93 I tested `run_glue.py` on a multi-gpu machine. Even, after changing `"cuda"` to `"cuda:0"` in this line https://github.com/huggingface/transformers/blob/b0ee7c7df3d49a819c4d6cef977214bd91f5c075/examples/run_glue.py#L425 , the training job will still use both GPUs relying on `torch.nn.DataParallel`. It means that `torch.nn.DataParallel` is smart enough even if you are defining `torch.device` to be `cuda:0`, if there are several gpus available, it will use all of them. 

> @TheEdoardo93 I tested `run_glue.py` on a multi-gpu machine. Even, after changing `"cuda"` to `"cuda:0"` in this line
> 
> https://github.com/huggingface/transformers/blob/b0ee7c7df3d49a819c4d6cef977214bd91f5c075/examples/run_glue.py#L425
> 
> , the training job will still use both GPUs relying on `torch.nn.DataParallel`. It means that `torch.nn.DataParallel` is smart enough even if you are defining `torch.device` to be `cuda:0`, if there are several gpus available, it will use all of them.

manually set 'args.n_gpu = 1' works for me

Just a simple `os.environ['CUDA_VISIBLE_DEVICES'] = 'GPU_NUM'` at the beginning of the script should work.

> > @TheEdoardo93 I tested `run_glue.py` on a multi-gpu machine. Even, after changing `"cuda"` to `"cuda:0"` in this line
> > https://github.com/huggingface/transformers/blob/b0ee7c7df3d49a819c4d6cef977214bd91f5c075/examples/run_glue.py#L425
> > 
> > , the training job will still use both GPUs relying on `torch.nn.DataParallel`. It means that `torch.nn.DataParallel` is smart enough even if you are defining `torch.device` to be `cuda:0`, if there are several gpus available, it will use all of them.
> 
> manually set 'args.n_gpu = 1' works for me

but then you are not able to use more than 1 gpu, right?

> @TheEdoardo93 I tested `run_glue.py` on a multi-gpu machine. Even, after changing `"cuda"` to `"cuda:0"` in this line
> 
> https://github.com/huggingface/transformers/blob/b0ee7c7df3d49a819c4d6cef977214bd91f5c075/examples/run_glue.py#L425
> 
> , the training job will still use both GPUs relying on `torch.nn.DataParallel`. It means that `torch.nn.DataParallel` is smart enough even if you are defining `torch.device` to be `cuda:0`, if there are several gpus available, it will use all of them.

I have trid but it doesn't work for me.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.