Bart CUDA not working
Author: marmg
Pull request: no

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): BART - bart-large

Language I am using the model on (English, Chinese ...): English

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [x] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. Load model
2. Tokenize text
3. Send model and tensor to cuda
4. Forward the model

```python 
from transformers import BartConfig, BartTokenizer, BartForMaskedLM, BartModel

configuration = BartConfig()
tokenizer_class = BartTokenizer
model_class = BartForMaskedLM(configuration)

tokenizer = tokenizer_class.from_pretrained('bart-large')
model = model_class.from_pretrained('bart-large')

model.eval()
model.to('cuda')

tokens = tokenizer.encode("Text example to test natural language generation with bart.")
input_ids = torch.tensor([tokens]) 
input_ids = input_ids.to('cuda')

with torch.no_grad():
    last_hidden_states = model(input_ids)[0]
    
print("Len tokens:", len(tokens))
print("Shape last hidden states:", last_hidden_states.shape)
```

This code raises the following error:
```python
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-24-3bba66ed6aeb> in <module>
      3 
      4 with torch.no_grad():
----> 5     last_hidden_states = model(input_ids)[0]
      6 
      7 print("Len tokens:", len(tokens))

~/projects/p130-nlg/.conda/envs/p130-nlg-env-kgpubart/lib/python3.6/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    530             result = self._slow_forward(*input, **kwargs)
    531         else:
--> 532             result = self.forward(*input, **kwargs)
    533         for hook in self._forward_hooks.values():
    534             hook_result = hook(self, input, result)

~/projects/p130-nlg/.conda/envs/p130-nlg-env-kgpubart/lib/python3.6/site-packages/transformers/modeling_bart.py in forward(self, input_ids, attention_mask, encoder_outputs, decoder_input_ids, decoder_attention_mask, decoder_cached_states, lm_labels, **unused)
    923             encoder_outputs=encoder_outputs,
    924             decoder_attention_mask=decoder_attention_mask,
--> 925             decoder_cached_states=decoder_cached_states,
    926         )
    927         lm_logits = self.lm_head.forward(outputs[0])

~/projects/p130-nlg/.conda/envs/p130-nlg-env-kgpubart/lib/python3.6/site-packages/transformers/modeling_bart.py in forward(self, input_ids, attention_mask, decoder_input_ids, encoder_outputs, decoder_attention_mask, decoder_cached_states)
    842             attention_mask,
    843             decoder_attn_mask,
--> 844             decoder_cached_states=decoder_cached_states,
    845         )
    846         # Attention and hidden_states will be [] or None if they aren't needed

~/projects/p130-nlg/.conda/envs/p130-nlg-env-kgpubart/lib/python3.6/site-packages/transformers/modeling_bart.py in forward(self, input_ids, encoder_hidden_states, encoder_padding_mask, combined_mask, decoder_cached_states, **unused)
    497                 decoder_cached_states=layer_state,
    498                 attention_mask=combined_mask,
--> 499                 need_attn_weights=self.output_attentions,
    500             )
    501             if self.output_past:

~/projects/p130-nlg/.conda/envs/p130-nlg-env-kgpubart/lib/python3.6/site-packages/transformers/modeling_bart.py in forward(self, x, encoder_hidden_states, encoder_attn_mask, decoder_cached_states, attention_mask, need_attn_weights)
    370             decoder_cached_states=decoder_cached_states,
    371             need_weights=need_attn_weights,
--> 372             attn_mask=attention_mask,
    373         )
    374         x = F.dropout(x, p=self.dropout, training=self.training)

~/projects/p130-nlg/.conda/envs/p130-nlg-env-kgpubart/lib/python3.6/site-packages/transformers/modeling_bart.py in forward(self, query, key, value, key_padding_mask, decoder_cached_states, need_weights, static_kv, attn_mask)
    627 
    628         if attn_mask is not None:
--> 629             attn_weights = attn_weights.view(bsz, self.num_heads, tgt_len, src_len) + attn_mask
    630             attn_weights = attn_weights.view(bsz * self.num_heads, tgt_len, src_len)
    631 

RuntimeError: expected device cuda:0 but got device cpu
```

But I have tested this code with another examples (like GPT-2) and it works.

## Expected behavior

I would expect to get the tensor size, as with another models I have tested this code.

## Environment info
<!-- You can run the command `python transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version: 2.5.1
- Platform: Linux
- Python version: 3.6.7
- PyTorch version (GPU?): 1.4.0. GPU: Yes
- Tensorflow version (GPU?): No used
- Using GPU in script?: Trying to
- Using distributed or parallel set-up in script?: No


Comments:
I just merged a PR and your example works for me. Would you mind seeing if it is still broken in your system? Thanks for posting! 
(note that the fix won't be in v 2.5.1 if you pip installed). 


Reopen if still broken!