Having "RuntimeError: expected scalar type Half but found Char"  on LLaMa-2 inference stage
Author: kenchanLOL
Pull request: no

### System Info

Working on jupyter notebook of a docker Linux instance of A100 , Ubuntu, x86_64
![image](https://github.com/huggingface/transformers/assets/55791584/67b33111-748a-49a4-aa28-2f2009884c4d)


### Who can help?

@sgugger @muellerzr

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

I encounter the error while I was trying to run a 8-bit quantized LLaMA-2-70B model on two 40GB GPU of A100 .
![WhatsApp Image 2023-07-27 at 18 07 56](https://github.com/huggingface/transformers/assets/55791584/8d23bab6-25ea-486e-b921-39ad2ad17af1)
To reproduce the issue:
1. load model with local path
```
tokenizer = AutoTokenizer.from_pretrained(model_path)
model = AutoModelForCausalLM.from_pretrained(model_path, load_in_8bit = True, device_map = "auto")
```
2. run inference code
```
question = f""" 
Human: xxxxxxxxxxxxx
Assistant:
"""
question = tokenizer(question, return_tensors = "pt")
question = question.to(0)
output = model.generate(question["input_ids"], max_new_tokens = 120)
```
---
# Investigations & Attempts to solve this bug
## TLDR;
**disabling tensor parallelism by setting ```pretraining_tp``` = ```1``` in config.json or your config object**

I notice that the error was raised from the part which is recently update for supporting llama-2 and to be specific, it's the implementation of the Grouped-Query Attention (GQA) architecture. 
![image](https://github.com/huggingface/transformers/assets/55791584/7e490446-910a-4ecb-a25c-ceebca8410c8)

Further looking into the code, i think the bug is caused by missing a dtype handling part for the new Grouped-Query Attention (GQA) architecture while F.linear is being used instead of a forward call. It is because by setting the ```load_in_8bit``` arguement as True, the ```nn.Linear``` layer will be replaced by a equivalent ```bnb.nn.Linear8bitLt```. The dtype of hidden_states and query_slices[i] are float16 and int64 respectively. Usually, the forward function of Linear8bitLt will handle. However, it doesn't work the same way in the F.linear and thus it raises this error as two tensor have different dtypes. 

Apart from disabling tensor parallelism by setting pretraining_tp = 1 in model config while loading model(os it will use the llama-1 part), I made a small workaround to prove my hypothesis by aligning the dtype of hidden_states and query_slices[i], I try to manually dequantize the Linear8bitLt by adding a small code snippet like this
```
        key_value_slicing = (self.num_key_value_heads * self.head_dim) // self.pretraining_tp
        if isinstance(self.q_proj, bnb.nn.Linear8bitLt):
            q_w = self.q_proj.weight
            q_w = (q_w.CB * q_w.SCB.unsqueeze(1) / 127 ).to(torch.float16)
        query_slices= q_w.split((self.num_heads*self.head_dim)//self.pretraining tp, dim = 0)
        if isinstance(self. k_proj, bnb.nn.Linear8bitLt):
            k_w = self.k_proj.weight
            k_w = (k_w.CB * k_w.SCB.unsqueeze(1) / 127 ).to(torch.float16)
        key_slices = k_w.split(key_value_slicing, dim = 0)
        if isinstance(self. v_proj, bnb.nn.Linear8bitLt):
            v_w = self.v_proj.weight
            v_w = (v_w.CB * v_w.SCB.unsqueeze(1) / 127 ).to(torch.float16)
        value_slices = v_w.split(key_value_slicing, dim = 0)
        #query_sLices = sett,q pro].weightTsplit((self.num heads * self.head dim) // selT.pretrainlng tp, dim=G)
        #key_slices = self.K proj.weight.split(key value slicing, dim=0)                              ~
        #value_slices = seirTv pro].weight.split(key value slicing, dim=0)
```
similar code have to be added in:

-  line 202 (LLamaMLP forward function -> gate_proj, up_porj, down_proj
-  line 293 (LlamaAttention forward function -> q_proj, k_proj, v_proj)
-  line 364 (LlamaAttention forward function -> o_proj)

After using this workaround , I was able to run the model and get result as expected. However, the quality of generated text deteriorate significantly while the length of text increase.


### Expected behavior

Generate a completion that answer the human input

Comments:
cc @younesbelkada 

Hmm I really think we should set that value (`config.pretraining_tp`) to 1 (at least when the model is quantized) for all models as it can introduce unexpected behaviour to users. We saw it introduced bugs with PEFT (that users currently overcome by forcing `config.pretraining_tp` to be equal to 1) and now with quantization.
I also don't think this is the right fix as de-quantizing the layers like that on the fly can introduce a lot of rounding errors. Not sure also how this will work with nested quantization in 4bit. TLDR; I think that it will be too much of a pain for a little gain - I am pretty sure the generation quality will remain pretty much the same if `pretraining_tp` is equal to 1 (from my experience with bloom).
This will certainly create issues with the new quantization technique that is going to be added here: https://github.com/huggingface/transformers/pull/25062 and we can't patch the linear layer like that for each case (bnb 4bit, bnb 8bit, GPTQ).
@sgugger @ArthurZucker what do you think about forcing `config.pretraining_tp` to be equal to 1 at least for the quantized models? 

It is 1 on all checkpoints online and the provided code does not change it.

I checked and found that my config.json is not the most updated version. 
The latest version of config online is having ``` config.pretraining_tp``` as ```1``` Thanks for the reply
![image](https://github.com/huggingface/transformers/assets/55791584/53e61640-f382-44e5-9efa-999850812c6f)


@sgugger sorry for the confusion, I thought all models still had `pretraining_tp > 1`. 
@kenchanLOL thanks for confirming, setting that value to 1 should fix your issue I believe!