BART run run_train.sh RuntimeError: expected device cuda:0 but got device cpu
Author: chenbingxiayu
Pull request: no
Labels: name: wontfix

# 🐛 Bug

## Information

Model I am using (BART):

Language I am using the model on CNN dailymail (English, Chinese ...):

The problem arises when using:
* [* ] the official example scripts: (give details below)
text summarization of bart, run the script run_train.sh, following the guideline
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [*] an official GLUE/SQUaD task: (give the name)
the text summarization task, using cnn dailymail dataset.
* [ ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. run the script run_train.sh in examples/summarization/bart/run_train.sh
2. got the error information RuntimeError: expected device cuda:0 but got device cpu


<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->

## Environment info
<!-- You can run the command `python transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version: I pip install transformers two days ago, I am not sure the version.
- Platform: ubuntu 1604
- Python version: 3.6.7
- PyTorch version (GPU?):  1.4.0
- Tensorflow version (GPU?):
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no

Traceback:
Traceback (most recent call last):
  File "run_bart_sum.py", line 166, in <module>
    trainer = generic_train(model, args)
  File "******/transformer_base.py", line 304, in generic_train
    trainer.fit(model)
  File "******/anaconda3/lib/python3.6/site-packages/pytorch_lightning/trainer/trainer.py", line 676, in fit
    mp.spawn(self.ddp_train, nprocs=self.num_gpus, args=(model,))
  File "******/anaconda3/lib/python3.6/site-packages/torch/multiprocessing/spawn.py", line 171, in spawn
    while not spawn_context.join():
  File "******/anaconda3/lib/python3.6/site-packages/torch/multiprocessing/spawn.py", line 118, in join
    raise Exception(msg)
Exception: 

-- Process 0 terminated with the following error:
Traceback (most recent call last):
  File "******/anaconda3/lib/python3.6/site-packages/torch/multiprocessing/spawn.py", line 19, in _wrap
    fn(i, *args)
  File "******/anaconda3/lib/python3.6/site-packages/pytorch_lightning/trainer/distrib_data_parallel.py", line 341, in ddp_train
    self.run_pretrain_routine(model)
  File "******/anaconda3/lib/python3.6/site-packages/pytorch_lightning/trainer/trainer.py", line 924, in run_pretrain_routine
    False)
  File "******/anaconda3/lib/python3.6/site-packages/pytorch_lightning/trainer/evaluation_loop.py", line 263, in _evaluate
    output = self.evaluation_forward(model, batch, batch_idx, dataloader_idx, test_mode)
  File "******/anaconda3/lib/python3.6/site-packages/pytorch_lightning/trainer/evaluation_loop.py", line 418, in evaluation_forward
    output = model(*args)
  File "******/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "******/anaconda3/lib/python3.6/site-packages/pytorch_lightning/overrides/data_parallel.py", line 96, in forward
    output = self.module.validation_step(*inputs[0], **kwargs[0])
  File "******/run_bart_sum.py", line 58, in validation_step
    loss = self._step(batch)
  File "******/run_bart_sum.py", line 44, in _step
    lm_labels=lm_labels.cuda(),
  File "******/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "******/run_bart_sum.py", line 32, in forward
    lm_labels=lm_labels,
  File "******/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "******/anaconda3/lib/python3.6/site-packages/transformers/modeling_bart.py", line 925, in forward
    decoder_cached_states=decoder_cached_states,
  File "******/anaconda3/lib/python3.6/site-packages/transformers/modeling_bart.py", line 844, in forward
    decoder_cached_states=decoder_cached_states,
  File "******/anaconda3/lib/python3.6/site-packages/transformers/modeling_bart.py", line 499, in forward
    need_attn_weights=self.output_attentions,
  File "******/anaconda3/lib/python3.6/site-packages/transformers/modeling_bart.py", line 372, in forward
    attn_mask=attention_mask,
  File "******/anaconda3/lib/python3.6/site-packages/transformers/modeling_bart.py", line 629, in forward
    attn_weights = attn_weights.view(bsz, self.num_heads, tgt_len, src_len) + attn_mask
RuntimeError: expected device cuda:0 but got device cpu


Comments:
I meet the same bug when i use bart as an embedding layer.
Have u solve the problem?

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


> I meet the same bug when i use bart as an embedding layer.
> Have u solve the problem?

any luck on this?  seeing the same "RuntimeError: expected device cuda:0 but got device cpu"