Incompatible tensor type when running BART on TPU
Author: marton-avrios
Pull request: no

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): `facebook/bart-large`

Language I am using the model on (English, Chinese ...): English

The problem arises when using:
* [x] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [x] an official GLUE/SQUaD task: XSUM
* [ ] my own task or dataset: (give details below)

## To reproduce

1. Setup a Google VM with the XLA image and configure it to use TPUs
2. Follow the instrcutions in `seq2seq` example for downloading XSUM
3. Then run
```
export PYTHONPATH="../":"${PYTHONPATH}"

python finetune.py \
    --learning_rate=3e-5 \
    --gpus 0 \
    --n_tpu_cores 8 \
    --do_train \
    --do_predict \
    --n_val 1000 \
    --val_check_interval 0.1 \
    --data_dir ${PWD}/xsum \
    --train_batch_size=1 \
    --eval_batch_size=1 \
    --output_dir=xsum_results \
    --num_train_epochs 1 \
    --model_name_or_path facebook/bart-large
```
...and you get something like
```
Exception in device=TPU:5: Attempted to call `variable.set_data(tensor)`, but `variable` and `tensor` have incompatible tensor type.
Traceback (most recent call last):
  File "/anaconda3/envs/myenv/lib/python3.6/site-packages/torch_xla/distributed/xla_multiprocessing.py", line 119, in _start_fn
    fn(gindex, *args)
  File "/anaconda3/envs/myenv/lib/python3.6/site-packages/pytorch_lightning/trainer/distrib_parts.py", line 222, in tpu_train
    self.run_pretrain_routine(model)
  File "/anaconda3/envs/myenv/lib/python3.6/site-packages/pytorch_lightning/trainer/trainer.py", line 1196, in run_pretrain_routine
    False)
  File "/anaconda3/envs/myenv/lib/python3.6/site-packages/pytorch_lightning/trainer/evaluation_loop.py", line 293, in _evaluate
    output = self.evaluation_forward(model, batch, batch_idx, dataloader_idx, test_mode)
  File "/anaconda3/envs/myenv/lib/python3.6/site-packages/pytorch_lightning/trainer/evaluation_loop.py", line 470, in evaluation_forward
    output = model.validation_step(*args)
  File "/home/martongyorgy/transformers/examples/seq2seq/finetune.py", line 145, in validation_step
    return self._generative_step(batch)
  File "/home/martongyorgy/transformers/examples/seq2seq/finetune.py", line 176, in _generative_step
    decoder_start_token_id=self.decoder_start_token_id,
  File "/anaconda3/envs/myenv/lib/python3.6/site-packages/torch/autograd/grad_mode.py", line 15, in decorate_context
    return func(*args, **kwargs)
  File "/anaconda3/envs/myenv/lib/python3.6/site-packages/transformers/generation_utils.py", line 248, in generate
    if self.get_output_embeddings() is None:
  File "/anaconda3/envs/myenv/lib/python3.6/site-packages/transformers/modeling_bart.py", line 1113, in get_output_embeddings
    return _make_linear_from_emb(self.model.shared)  # make it on the fly
  File "/anaconda3/envs/myenv/lib/python3.6/site-packages/transformers/modeling_bart.py", line 190, in _make_linear_from_emb
    lin_layer.weight.data = emb.weight.data
RuntimeError: Attempted to call `variable.set_data(tensor)`, but `variable` and `tensor` have incompatible tensor type.
```
## Environment info
```
- `transformers` version: 3.0.2
- Platform: Linux-4.9.0-12-amd64-x86_64-with-debian-9.12
- Python version: 3.6.10
- PyTorch version (GPU?): 1.5.0a0+ab660ae (False)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: TPU setup following Google Cloud tutorial for PyTorch
```


Comments:
Let's consolidate the discussion to #5895 .
Definitely an issue!