Bug Report - Tokenizer Issue with Tensor Device Assignment in transformers/pipelines/text_generation.py
Author: ericzhou571
Pull request: no

### System Info

```
- `transformers` version: 4.29.0.dev0
- Platform: Linux-5.4.0-110-generic-x86_64-with-glibc2.10
- Python version: 3.8.13
- Huggingface_hub version: 0.14.1
- Safetensors version: 0.3.0
- PyTorch version (GPU?): 2.0.0+cu117 (True)
- Tensorflow version (GPU?): 2.9.3 (True)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>
```


### Who can help?

@ArthurZucker @Narsil

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

code example of https://huggingface.co/tiiuae/falcon-7b

from transformers import AutoTokenizer, AutoModelForCausalLM
import transformers
import torch

model = "tiiuae/falcon-7b"

```python
tokenizer = AutoTokenizer.from_pretrained(model)
pipeline = transformers.pipeline(
    "text-generation",
    model=model,
    tokenizer=tokenizer,
    torch_dtype=torch.bfloat16,
    trust_remote_code=True,
    device=torch.device(0),
)
sequences = pipeline(
   "Girafatron is obsessed with giraffes, the most glorious animal on the face of this Earth. Giraftron believes all other animals are irrelevant when compared to the glorious majesty of the giraffe.\nDaniel: Hello, Girafatron!\nGirafatron:",
    max_length=200,
    do_sample=True,
    top_k=10,
    num_return_sequences=1,
    eos_token_id=tokenizer.eos_token_id,
)
for seq in sequences:
    print(f"Result: {seq['generated_text']}")
```


### Expected behavior

every  thing should work well on gpu:0.

### More Detail

Dear Transformers GitHub team,

I hope this message finds you well. I would like to report a bug that I have identified in the code block provided below:
https://github.com/huggingface/transformers/blob/118e9810687dd713b6be07af79e80eeb1d916908/src/transformers/pipelines/text_generation.py#L203-L263

```python
def preprocess(self, prompt_text, prefix="", handle_long_generation=None, **generate_kwargs):
    inputs = self.tokenizer(
        prefix + prompt_text, padding=False, add_special_tokens=False, return_tensors=self.framework
    )
    inputs["prompt_text"] = prompt_text

    # Rest of the code...
```

Problem description:
The bug occurs in the `preprocess` method of the code block above. It seems that after tokenizing the `prompt_text`, the resulting tensor is not automatically moved to the same device where the model is located. This behavior causes an error when attempting to use the GPU for computation, specifically resulting in the following error message:

```
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu! (when checking argument for argument index in method wrapper_CUDA__index_select)
```

Expected behavior:
Ideally, the `preprocess` method should ensure that the tensor generated by the tokenizer is moved to the same device as the model before further processing or generation takes place. This would prevent any device mismatch errors when using the GPU for computations.

Possible solution:
To resolve this issue, I suggest modifying the `preprocess` method to include a device assignment step for the tokenized tensor. By using the `to` method, the tensor can be explicitly moved to the device where the model is located. Here's an example of how this could be implemented:

```python
inputs = self.tokenizer(
    prefix + prompt_text, padding=False, add_special_tokens=False, return_tensors=self.framework
)
inputs["input_ids"] = inputs["input_ids"].to(self.model.device)
if "attention_mask" in inputs:
    inputs["attention_mask"] = inputs["attention_mask"].to(self.model.device)
```

By adding these lines of code, the tensor and its attention mask (if applicable) will be correctly assigned to the same device as the model.

I hope this information helps in resolving the issue. Please let me know if you need any further clarification or assistance. Thank you for your attention to this matter.

Best regards,
Wenrui


Comments:
The `pipeline` will handle the device if you pass it to it with the `device` kwarg.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.