RuntimeError: "topk_cpu" not implemented for 'Half'
Author: MarvinLong
Pull request: no

### System Info

transformers            4.27.1
WSL2 with Ubuntu 20.04
GPU: 4090
CUDA VERSION: 11.8

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

**When I infer the model with 8bit for bloom 7b1**
```
model =  AutoModelForCausalLM.from_pretrained(model_path, device_map = "auto", load_in_8bit=True)
tokenizer = AutoTokenizer.from_pretrained(model_path)
input_ids = tokenizer(inputs, return_tensors="pt").input_ids
outputs = model.generate(input_ids, max_new_tokens=300, do_sample = True, top_k = 30, top_p = 0.85, temperature = 0.35, repetition_penalty=1.2)

```

### Expected behavior

```
/home/miniconda3/lib/python3.10/site-packages/transformers/generation/utils.py:1374: UserWarning: You are calling .generate() with the `input_ids` being on a device type different than your model's device. `input_ids` is on cpu, whereas the model is on cuda. You may experience unexpected behaviors or slower generation. Please make sure that you have put `input_ids` to the correct device by calling for example input_ids = input_ids.to('cuda') before running `.generate()`.
  warnings.warn(
Traceback (most recent call last):
  File "/mnt/d/project/aigc/belle/test_infer_int8.py", line 13, in <module>
    outputs = model.generate(input_ids, max_new_tokens=300, do_sample = True, top_k = 30, top_p = 0.85, temperature = 0.35, repetition_penalty=1.2)
  File "/home/miniconda3/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/home/miniconda3/lib/python3.10/site-packages/transformers/generation/utils.py", line 1452, in generate
    return self.sample(
  File "/home/miniconda3/lib/python3.10/site-packages/transformers/generation/utils.py", line 2482, in sample
    next_token_scores = logits_warper(input_ids, next_token_scores)
  File "/home/miniconda3/lib/python3.10/site-packages/transformers/generation/logits_process.py", line 92, in __call__
    scores = processor(input_ids, scores)
  File "/home/miniconda3/lib/python3.10/site-packages/transformers/generation/logits_process.py", line 302, in __call__
    indices_to_remove = scores < torch.topk(scores, top_k)[0][..., -1, None]
RuntimeError: "topk_cpu" not implemented for 'Half'
```

Comments:
cc @younesbelkada 

Hello @MarvinLong 
The reason behind that issue is that you forgot to pass `input_ids` on the same device as the model (here GPU)
The script:
```python
from transformers import AutoModelForCausalLM, AutoTokenizer

model =  AutoModelForCausalLM.from_pretrained(model_path, device_map = "auto", load_in_8bit=True)
tokenizer = AutoTokenizer.from_pretrained(model_path)
input_ids = tokenizer(inputs, return_tensors="pt").input_ids
outputs = model.generate(input_ids.to(0), max_new_tokens=300, do_sample = True, top_k = 30, top_p = 0.85, temperature = 0.35, repetition_penalty=1.2)
```
Should work
Also, a warning has been trigged to warn you about this !: 
```
/home/miniconda3/lib/python3.10/site-packages/transformers/generation/utils.py:1374: UserWarning: You are calling .generate() with the `input_ids` being on a device type different than your model's device. `input_ids` is on cpu, whereas the model is on cuda. You may experience unexpected behaviors or slower generation. Please make sure that you have put `input_ids` to the correct device by calling for example input_ids = input_ids.to('cuda') before running `.generate()`.
```
Thanks! 

For more context (in case you are interested):
This is because `device_map="auto"` will load the model using `accelerate`. Loading a model with `accelerate` will place several "forward hooks" to it. That will apply some post-processing to the input such as placing the output of the model on the same device as the input.
Here on the snippet you have shared, the input is placed on CPU, and you are loading the model in 8bit that will produce half-precision logits under the hood. In addition to that you are calling a sampling strategy that will involve calling some functions from pytorch such as [topk](https://pytorch.org/docs/stable/generated/torch.topk.html) on these logits that are not supported on CPU in half-precision, hence the error.

@younesbelkada 
Thanks a lot, this works