int8 with device_map doesn't work well in generation
Author: fnshwi
Pull request: no

**Versions**
`transformers==4.26.0`
`accelerate==0.17.1`

I was trying to do inference with the `.generate()` method using a 7B OPT model. When using fp16 in an 8-gpu(80G A100) node, I figure out that fitting each whole model into a GPU with independent process achieves the best speed. But when trying to use int8, I can't use `auto` in device_map(it will shard model into different gpus, which I don't want) and I'll have to design a device_map like this:
`{
  "model.decoder.embed_tokens": 1, 
  "lm_head": 1,
  "model.decoder.embed_positions": 1,
  "model.decoder.final_layer_norm": 1,
  "model.decoder.layers.0": 1,
  "model.decoder.layers.1": 1,
  "model.decoder.layers.2": 1,
  "model.decoder.layers.3": 1,
  "model.decoder.layers.4": 1,
  "model.decoder.layers.5": 1,
   ...
}`
I found the speed is rather slow compared to fp16 in generation. 
How can I correctly use int8 in a multi-process way(one independent process for a GPU) in an 8-card node?

Comments:
cc @younesbelkada Also if you want your whole model on one GPU, you shouldn't use the `device_map` argument but just place your model on that GPU.

@fnshwi you can run the following: 
```python
import torch

...
device_map = {"":torch.cuda.current_device()}
```

and use this device map instead of `"auto"`. the `"":device` means that you want to fit your entire model on that device
Related and similar issue: https://github.com/huggingface/transformers/issues/21736

Also if you run multiple processes, you can use a trick that [we use in `trl`](https://github.com/lvwerra/trl/blob/main/trl/models/modeling_base.py#L246-L258):

```python
from accelerate import Accelerator

current_device = Accelerator().process_index
device_map = {"":current_device}
```

@younesbelkada Thanks for your help, the device_map works well. But I still have the inference speed issue, it's much slower than fp16, the code is:
```
device_map = {"": torch.cuda.current_device()}
model = OPTForCausalLM.from_pretrained(ckpt_dir, load_in_8bit=True, device_map=device_map)
input_id = torch.tensor(input_id, device=device)
gen = model.generate(input_ids,
                     min_length=1,
                     max_length=512,
                     do_sample=True,
                     temperature=1.0,
                     num_return_sequences=topk)
```
I manually send input_id to device, is that correct?

Thanks for the feedback, yes you need to manually send the input ids to the correct device
How much slower is the int8 model compared to fp16?

@younesbelkada generating 130 cases in fp16 costs ~20mins, but int8 costs more than 1h  


Hi i'm having the same problem, as i'm having to convert the input to device every time it makes int8 way more slow than regular fp16

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

@fnshwi Hi, have you fixed the speed issue?

hi @sunhs 
for faster inference, i would advise you to use 4-bit models instead: https://huggingface.co/docs/transformers/quantization#4-bit