Unable to see the weight files after quantization
Author: pradeepdev-1995
Pull request: no

### System Info

- `transformers` version: 4.23.1
- Platform: Linux-5.10.133+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.15
- Huggingface_hub version: 0.10.1
- PyTorch version (GPU?): 1.12.1+cu113 (False)
- Tensorflow version (GPU?): 2.9.2 (False)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

I have tried the following code for dynamic quantization
```
import torch
import os
from transformers import AutoConfig, AutoModel
model = AutoModel.from_pretrained("bert-base-uncased")
model_quantized = torch.quantization.quantize_dynamic(model, {torch.nn.Linear}, dtype=torch.qint8)  
quantized_output_dir = "quantized/"
if not os.path.exists(quantized_output_dir):
    os.makedirs(quantized_output_dir)
    model_quantized.save_pretrained(quantized_output_dir)
```
After the execution, I could see that there is a new folder named quantized created in the directory which contains only the ```config.json``` file.
contents are as follows
```
{
  "_name_or_path": "bert-base-uncased",
  "architectures": [
    "BertModel"
  ],
  "attention_probs_dropout_prob": 0.1,
  "classifier_dropout": null,
  "gradient_checkpointing": false,
  "hidden_act": "gelu",
  "hidden_dropout_prob": 0.1,
  "hidden_size": 768,
  "initializer_range": 0.02,
  "intermediate_size": 3072,
  "layer_norm_eps": 1e-12,
  "max_position_embeddings": 512,
  "model_type": "bert",
  "num_attention_heads": 12,
  "num_hidden_layers": 12,
  "pad_token_id": 0,
  "position_embedding_type": "absolute",
  "torch_dtype": "float32",
  "transformers_version": "4.23.1",
  "type_vocab_size": 2,
  "use_cache": true,
  "vocab_size": 30522
}
```
I can't see any other .bin or .wt files after quantization. Why it is so?

### Expected behavior

The model should be quantized and save the new quantized weight files in the provided folder along with the config.json file

Comments:
Maybe of interest to @michaelbenayoun :)

Hi @pradeepdev-1995 ,
You don't get this issue first ?
```
AttributeError: 'torch.dtype' object has no attribute 'numel'
```

Yes. @michaelbenayoun 
But after rerun in second time in the google colab,
It worked without error.
But only config file is there.

Yes, I observe the same thing. In any case, I do not think this will work because you have dtypes in your state dict, which is not handled correctly by `save_pretrained` for now.

@michaelbenayoun
Got it. So how can i do dynamic quantization on a model and save it in local for future use?
Please share the code snippet if possible 

This should work:
```python
import torch
import os
from transformers import AutoConfig, AutoModel
model = AutoModel.from_pretrained("bert-base-uncased")
model_quantized = torch.quantization.quantize_dynamic(
    model, {torch.nn.Linear}, dtype=torch.qint8
)  
quantized_output_dir = "quantized/"
if not os.path.exists(quantized_output_dir):
    os.makedirs(quantized_output_dir)
    model_quantized.config.save_pretrained(quantized_output_dir)
    torch.save(model_quantized.state_dict(), "quantized/pytorch_model.bin")
```

But note that you will not be able to restore your model afterwards, at least with a `from_pretrained`. You will need to:

1. Load the model, either with the pre-trained weights, or random ones
2. Convert the model to its dynamically quantized version
3. Do: `model.load_state_dict(torch.load(path_to_the_state_dict))`

You have other ways of saving your model:

- You can `jit.trace` / `jit.script` it
- You can use another approach, such as [quantization with ONNX Runtime with Optimum](https://huggingface.co/docs/optimum/onnxruntime/usage_guides/quantization)

@michaelbenayoun Thank you very much for the comments.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Hi, 

I tried this, but when I checked the `config.json`, it's showing float16. Do I need to worry about it or can I ignore it?