[`t5`] Fix T5 inference in `float16` + `bnb` error
Author: younesbelkada
Pull request: yes

# What does this PR do?

Currently on the `main` branch, the inference of `t5` is broken in half-precision. 
With the introduction of `_keep_in_fp32_modules` attributes in https://github.com/huggingface/transformers/pull/20683 , `wo` layers needs to be upcasted in `float32` for more accurate inference. 
It appears that in the aforementioned PR, we forgot to apply the same fix in `T5DenseActDense` layers, leading into a broken inference API when running inference in fp16 for models that uses `T5DenseActDense` layers instead of  `T5DenseGatedActDense` layers. This can be reproduced for example with `t5-small`:
```
from transformers import T5ForConditionalGeneration, T5Tokenizer
import torch

model_id = "t5-small"

model = T5ForConditionalGeneration.from_pretrained(model_id, device_map="auto", torch_dtype=torch.float16)

tokenizer = T5Tokenizer.from_pretrained(model_id)
input_tokens = tokenizer.encode("Translate the following in German: My name is Younes.", return_tensors="pt").to("cuda")

output = model.generate(input_tokens, max_length=32, num_beams=4, early_stopping=True)
print(tokenizer.decode(output[0], skip_special_tokens=True))
```
This is not the case for `flan` family since `flan-t5` uses `T5DenseGatedActDense` that has been correctly fixed in https://github.com/huggingface/transformers/pull/20760

This PR adds also a fix for 8-bit models. If a user wants to run inference in 8-bit without `_keep_in_fp32_modules` for backward compatibility reasons:
```
from transformers import T5ForConditionalGeneration, T5Tokenizer
import torch

T5ForConditionalGeneration._keep_in_fp32_modules = None
model_id = "google/flan-t5-small"

model = T5ForConditionalGeneration.from_pretrained(model_id, device_map="auto", torch_dtype=torch.float16)

tokenizer = T5Tokenizer.from_pretrained(model_id)
input_tokens = tokenizer.encode("Translate the following in German: My name is Younes.", return_tensors="pt").to("cuda")

output = model.generate(input_tokens, max_length=32, num_beams=4, early_stopping=True)
print(tokenizer.decode(output[0], skip_special_tokens=True))
```
They face an issue that is hard to interpret:
```
"addmm_cuda" not implemented for 'Char'
```
See for example: https://github.com/TimDettmers/bitsandbytes/issues/111#issuecomment-1368952450 

This is because the `hidden_states` are converted in `int8` [here](https://github.com/huggingface/transformers/blob/main/src/transformers/models/t5/modeling_t5.py#L313-L314), leading to entering a linear layer with an 8bit input, which leads to the error. Therefore one should cast to `self.wo.weight.dtype` only if `dtype != torch.int8`. (pointed out also [here](https://github.com/TimDettmers/bitsandbytes/issues/111#issuecomment-1402113167) )

This PR applies also `make fix-copies` introducing the fix to other architectures too - happy to revert that since this issue is only relevant for `T5` (LongT5 etc does not have `_keep_in_fp32_modules`)

This PR also tests everything, making sure this will never happen again!

cc @sgugger 

Comments:
_The documentation is not available anymore as the PR was closed or merged._