Pytorch T5 does not run on GPU
Author: nreimers
Pull request: no
Labels: name: wontfix

## 🐛 Bug

When I try to run T5 from the latest transformers version (and also from the most recent git version) on the GPU, I get the following error:

```
Traceback (most recent call last):
  File "T5_example.py", line 32, in <module>
    outputs = model(input_ids=input_ids)
  File "/home/reimers/anaconda3/envs/sbert/lib/python3.7/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/reimers/sbert/transformers/src/transformers/modeling_t5.py", line 780, in forward
    
  File "/home/reimers/anaconda3/envs/sbert/lib/python3.7/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/reimers/sbert/transformers/src/transformers/modeling_t5.py", line 616, in forward
    encoder_decoder_position_bias=encoder_decoder_position_bias,
  File "/home/reimers/anaconda3/envs/sbert/lib/python3.7/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/reimers/sbert/transformers/src/transformers/modeling_t5.py", line 422, in forward
    self_attention_outputs = self.layer[0](
  File "/home/reimers/anaconda3/envs/sbert/lib/python3.7/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/reimers/sbert/transformers/src/transformers/modeling_t5.py", line 373, in forward
    attention_output = self.SelfAttention(
  File "/home/reimers/anaconda3/envs/sbert/lib/python3.7/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/reimers/sbert/transformers/src/transformers/modeling_t5.py", line 338, in forward
    raise ValueError("No position_bias provided and no weights to compute position_bias")
  File "/home/reimers/sbert/transformers/src/transformers/modeling_t5.py", line 289, in compute_bias
   values = self.relative_attention_bias(rp_bucket)
  File "/home/reimers/anaconda3/envs/sbert/lib/python3.7/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/reimers/anaconda3/envs/sbert/lib/python3.7/site-packages/torch/nn/modules/sparse.py", line 114, in forward
    self.norm_type, self.scale_grad_by_freq, self.sparse)
  File "/home/reimers/anaconda3/envs/sbert/lib/python3.7/site-packages/torch/nn/functional.py", line 1484, in embedding
    return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
RuntimeError: Expected object of device type cuda but got device type cpu for argument #3 'index' in call to _th_index_select
``` 

This is the example code to reproduce the problem:
``` 
from transformers import T5Model, T5Tokenizer
import torch

tokenizer = T5Tokenizer.from_pretrained('t5-small')
model = T5Model.from_pretrained('t5-small')
model = model.to('cuda')
input_ids = torch.tensor(tokenizer.encode("Hello, my dog is cute"), device='cuda').unsqueeze(0) 
outputs = model(input_ids=input_ids)
last_hidden_states = outputs[0]
```

The error is in the file modeling_t5.py at line 284-289:
```
rp_bucket = self._relative_position_bucket(
            relative_position,  # shape (qlen, klen)
            bidirectional=not self.is_decoder,
            num_buckets=self.relative_attention_num_buckets,
        )
values = self.relative_attention_bias(rp_bucket)  # shape (qlen, klen, num_heads)
```

rp_bucket is a tensor on the CPU, which causes the above error.

If I move rp_bucket to the GPU, the code works correctly on the GPU:
```
rp_bucket = self._relative_position_bucket(
            relative_position,  # shape (qlen, klen)
            bidirectional=not self.is_decoder,
            num_buckets=self.relative_attention_num_buckets,
        )
 rp_bucket = rp_bucket.to('cuda') #Dirty quick fix
 values = self.relative_attention_bias(rp_bucket)  # shape (qlen, klen, num_heads)
```

I'm not sure why rp_bucket is on the CPU.

Comments:
I can also confirm that T5 runs on CPU but not GPU -- thanks for the hack fix, will use that until GPU tensor is fixed. 

Hi,
I was planning to run some examples with T5 on GPU. Is this already been fixed on GPU ?

@mohammedayub44, in v2.5.0 it works without any issue, I guess yes.

```
print (last_hidden_states)
tensor([[[ 9.2098e-02,  1.1048e-01,  2.6714e-02,  ...,  1.2918e-02,
           6.1260e-05,  9.5352e-02],
         [ 8.7042e-02,  8.3914e-02,  6.9337e-02,  ..., -3.9229e-02,
           3.3525e-04,  1.4291e-01],
         [ 9.6290e-02, -4.8915e-03,  5.5687e-02,  ..., -1.0703e-01,
           6.4940e-04, -2.1393e-01],
         [-3.0119e-03,  1.1048e-01,  3.0696e-03,  ..., -5.1768e-02,
           3.5166e-04,  1.5510e-01],
         [-6.3620e-02,  5.4474e-02, -1.8415e-02,  ..., -8.4559e-02,
           6.1696e-04,  5.8805e-02],
         [-6.0232e-02,  1.3885e-01,  7.9865e-03,  ..., -4.9981e-02,
           4.3370e-04,  4.4865e-02]]], device='cuda:0', grad_fn=<MulBackward0>)
```

Great I'll check it out. Thanks. 

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


Has this been corrected? I'm on version 2.8.0 on a GCP AI Platform Notebook using the PyTorch:1.4 image and I'm still getting this error. 

`
cuda0 = torch.device('cuda:0')
tokenized_text = tokenizer.encode(t5_prepared_Text, return_tensors="pt", max_length=512).to(cuda0)

summary_ids = model.generate(tokenized_text,
                             num_beams=2,
                             no_repeat_ngram_size=2,
                             min_length=50,
                             max_length=100,
                             early_stopping=True, )

output = tokenizer.decode(summary_ids[0], skip_special_tokens=True)

RuntimeError: Expected object of device type cuda but got device type cpu for argument #1 'self' in call to _th_index_select
`

I can confirm also having this issue on 2.10.0

```
RuntimeError: Expected object of device type cuda but got device type cpu for argument #3 'index' in call to _th_index_select
```
```
    model.to(DEVICE)
    model.train()
    input_ids.to(DEVICE)
    lm_labels.to(DEVICE)
    loss = model(input_ids=input_ids, lm_labels=lm_labels)[0]
    loss.backward()
```


This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.