Cannot get the model weight of T5 INT8 model with Transformers 4.26.1
Author: XuhuiRen
Pull request: no

### System Info

- `transformers` version: 4.26.1
- Platform: Linux-3.10.0-862.el7.x86_64-x86_64-with-glibc2.17
- Python version: 3.8.13
- Huggingface_hub version: 0.12.1
- PyTorch version (GPU?): 1.13.0+cpu
- Tensorflow version (GPU?): not installed (No)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <No>

### Who can help?

@ArthurZucker @younesbelkada @sgu

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

The code is shown as follows:
```python
  import torch
  import transformers
  from datasets import load_dataset
  model_name = 't5-small'
  model_fp32 = transformers.AutoModelForSeq2SeqLM.from_pretrained(
      model_name,
  )
  model_int8 = torch.ao.quantization.quantize_dynamic(
      model_fp32,
      {torch.nn.Linear},
      dtype=torch.qint8)
  def get_example_inputs(model_name, dataset_name='sst2'):
      tokenizer = transformers.AutoTokenizer.from_pretrained(model_name)
      dataset = load_dataset(dataset_name, split='validation')
      text = dataset[0]['text'] if dataset_name=='lambada' else dataset[0]['sentence']
      input = tokenizer(text, padding='max_length', max_length=195, return_tensors='pt')
      example_inputs = input['input_ids'][0].to('cpu').unsqueeze(0)
      return example_inputs
  example_inputs = get_example_inputs(model_name, dataset_name='lambada')
  output = model_int8.generate(example_inputs)
  print(output)
```
The error message is shown as follows,
[Issue Report.txt](https://github.com/huggingface/transformers/files/10905558/Issue.Report.txt)


### Expected behavior

This example's expected behavior is quantizing the FP32 T5 model into INT8 format using INT8 model to generate the output.
This code could make a success with the previous version Transformers 4.26.0. After the recent updates, This code cannot run normally anymore. 

We found the error is result from the fix of another issue: https://github.com/huggingface/transformers/issues/20287. That is probably because that you have used weight from "self.wo". But that weight becomes a function in the INT8 module. It should be used by "self.wo.weight()". Please reconsider the previous fix for that issue to make it compatible.

Comments:
Cool that this seems to have been fixed for you! Could you tell us what the solution was? (for futur reference and other users who might stumble upon the smae problem)

@ArthurZucker Hi, Arthur, we still did not fix this issue. Could you please have a check for this issue? We just modify the previous description to make the problem more clear for you to solve. 

Hey! Is there a reason why you are not using `load_in_8bit = True`? If you install the `bits_and_bytes` library, getting the 8bit quantized version of the model is as easy as the following:
```python 
  import transformers
  from datasets import load_dataset
  model_name = 't5-small'
  model = transformers.AutoModelForSeq2SeqLM.from_pretrained(model_name,load_in_8bit = True, device_map="auto")
```

Hello @XuhuiRen ,

Your script worked fine on the  `main` branch of `transformers`

```python
import torch
import transformers
from datasets import load_dataset
model_name = 't5-small'
model_fp32 = transformers.AutoModelForSeq2SeqLM.from_pretrained(
    model_name,
)
model_int8 = torch.ao.quantization.quantize_dynamic(
    model_fp32,
    {torch.nn.Linear},
    dtype=torch.qint8
)

output = model_int8.generate(torch.LongTensor([[0, 1, 2, 3]]))
print(output)
```

Can you try it with:
```
pip install git+https://github.com/huggingface/transformers.git
```

For more context, the PR: https://github.com/huggingface/transformers/pull/21843 solved your issue

Hi, @ArthurZucker and @younesbelkada really thanks for your reply. Your solution is works for my issue.