Why tokens / second is more on Float32 than Float16
Author: Anindyadeep
Pull request: no

### System Info

```
- `transformers` version: 4.34.1
- Platform: Linux-5.4.0-169-generic-x86_64-with-glibc2.31
- Python version: 3.9.16
- Huggingface_hub version: 0.17.3
- Safetensors version: 0.4.1
- Accelerate version: 0.26.1
- Accelerate config:    not found
- PyTorch version (GPU?): 2.0.0+cu117 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: YES
- Using distributed or parallel set-up in script?: NO
```

### Who can help?

@ArthurZucker, @younesbelkada 

### Information

- [X] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```python
import time 
import torch 
import numpy as np
from transformers import AutoModelForCausalLM, AutoTokenizer, GPTQConfig

@torch.inference_mode()
def benchmark_gptq(dtype, prompt = "Hello, what do you know about transformers", repetitions = 10):
    model_path = "./models/llama-2-7b-autogptq"
    quantization_config = GPTQConfig(
        bits=4, 
        group_size=128, 
        desc_act=False, use_exllama=False, 
        use_cuda_fp16=True if dtype == torch.float16 else False 
    )    
    
    model = AutoModelForCausalLM.from_pretrained(
        model_path,
        quantization_config=quantization_config,
        torch_dtype=dtype,
        device_map='cuda:0'
    )
    
    tokenizer = AutoTokenizer.from_pretrained(model_path)
    tokenized_input = tokenizer.encode(prompt, return_tensors='pt').to('cuda:0')
    results = [] 
    
    print("STARTING TO BENCHMARK FOR: ", "fp16" if dtype==torch.float16 else "fp32", "\n")
    for i in range(repetitions):
        print("....")
        start = time.time()
        output = model.generate(input_ids = tokenized_input, max_new_tokens = 100).detach().cpu().numpy()
        delta = time.time() - start
        results.append(
            len(output[0]) / delta
        )
        
    return np.mean(results)
    


if __name__ == '__main__':
    print("FP-16: ", benchmark_gptq(dtype=torch.float16))
    print("FP-32: ", benchmark_gptq(dtype=torch.float32))
```

### Expected behavior

This was the output:

```
FP-16:  39.50591397818114
FP-32:  49.23083222100881
```

Please note: The metric which is used here is: `tokens/sec`

But the expected should be the reversed right?

Comments:
cc @SunMarc , not 100% sure here but maybe this is expected with a potential silent fp32 -> fp16 downcasting overhead

@Anindyadeep - thanks for raising this issue! Could you also share the standard deviation between the runs? 

We only perform downcasting when there is no dtype specified. GPTQ model can indeed run with fp32 weight_dtype. However, we expect it to be slower than fp16. 

> @Anindyadeep - thanks for raising this issue! Could you also share the standard deviation between the runs?

Sure, here: 

```
Results in Mean, Std

FP-16:  (40.10957833368466, 1.3878278677006122)
FP-32:  (48.72477229543234, 0.28218654623962147)
```

Hi, I saw some patterns in the normal hugging face transformers inference, that with fp16 > fp32 when context size increased to 512. However this is not applying for AutoGPTQ

Here are the results (updated, for 512 context length)

```
FP-16: 30.42 ± 0.40
FP-32: 42.01 ± 1.03
```

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.