Generate function does not work with GPU
Author: contribcode
Pull request: no
Labels: name: wontfix

Hello,

I want to use generate function with single GPU. Specifically, I fine tuned a GPT-2 model (on GPU) and subsequently, I want to generate text with it. 

When I run this 

```
input_ids.to(device)
sample_output = model.generate(
    input_ids, 
    do_sample=True, 
    max_length=150,
    top_k=50,
    top_p=0.92
)
```
I get this error

`RuntimeError: Input, output and indices must be on the current device`

When i move the model and the input `.to('cpu')`, it works.

Comments:
Hey @contribcode,

could you please provide a complete code snippet that would allow us to reproduce the error? Thanks!

@patrickvonplaten thank you for your response. 

```python
import torch
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
from transformers import GPT2LMHeadModel
from transformers import GPT2TokenizerFast
tokenizer = GPT2TokenizerFast.from_pretrained("gpt2")

special_tokens_dict = {'additional_special_tokens': ['[spanstarts]','[spanends]']}
num_added_toks = tokenizer.add_special_tokens(special_tokens_dict)
model = GPT2LMHeadModel.from_pretrained("gpt2", pad_token_id=tokenizer.eos_token_id)
model.resize_token_embeddings(len(tokenizer))
model.to(device)
```
I have posted the snippets about model definition, since I think the rest of the code does not affect the `generate` method. After this I just train the model. 

Regards.

Hey @contribcode, 

I can execute the above code snippet without error -> could you include the part of the code where the error arises?

Hey @patrickvonplaten, 

the part of the code where the error occurs is the code in the original post

```
input_ids = tokenizer.encode(train_texts['text'].iloc[0], return_tensors='pt')
model.to('cuda')
input_ids.to('cuda')
sample_output = model.generate(
    input_ids, 
    do_sample=True, 
    max_length=150,
    top_k=50,
    top_p=0.92
)
```

and I get the error 

`RuntimeError: Input, output and indices must be on the current device`

As I mentioned in the original post, if I move the model and input_ids `to('cpu')`, it works.

I also wanted to ask about parameters (`top_k`, `top_p`), are these values common for the generate function?

Could you please post a single code snippet that I can copy/paste into a terminal and run to reproduce the error? The above snippet is not executable because `train_texts['text'].iloc[0]` is not defined. It's impossible to help you if we are not able to reproduce the error, I'm afraid.

`train_texts['text'].iloc[0]` is just some text, for example: This man is a joke.

The whole code snippet is 

```python
import torch
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
from transformers import GPT2LMHeadModel
from transformers import GPT2TokenizerFast
tokenizer = GPT2TokenizerFast.from_pretrained("gpt2")

special_tokens_dict = {'additional_special_tokens': ['[spanstarts]','[spanends]']}
num_added_toks = tokenizer.add_special_tokens(special_tokens_dict)
model = GPT2LMHeadModel.from_pretrained("gpt2", pad_token_id=tokenizer.eos_token_id)
model.resize_token_embeddings(len(tokenizer))
model.to(device)

input_ids = tokenizer.encode('This man is a joke.', return_tensors='pt')
input_ids.to(device)
sample_output = model.generate(
    input_ids, 
    do_sample=True, 
    max_length=150,
    top_k=50,
    top_p=0.92
)
```

and the error is what I mentioned earlier.

This issue has been automatically marked as stale and been closed because it has not had recent activity. Thank you for your contributions.

If you think this still needs to be addressed please comment on this thread.

I have the same issue with Transformers 4.4.2 and PyTorch 1.8.0

UPD: also tried Transformers 4.5.1, the issue is still here

Very nice @cloveranon !

Now, do you know how/if GPUs can accelerate generation? Can this work in batches?

Maybe, @patrickvonplaten can shed some light.