Can't save quantized models
Author: lamashnikov
Pull request: no
Labels: name: Quantization

name: bug

### System Info


- `transformers` version: 4.44.2
- Platform: Linux-6.8.0-40-generic-x86_64-with-glibc2.35
- Python version: 3.10.12
- Huggingface_hub version: 0.24.6
- Safetensors version: 0.4.4
- Accelerate version: not installed
- Accelerate config: not found
- PyTorch version (GPU?): 2.4.0+cu121 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: no
- Using GPU in script?: no
- GPU type: NVIDIA T600 Laptop GPU


### Who can help?

@ArthurZucker 

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [x] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

model = SentenceTransformer('sentence-transformers/paraphrase-MiniLM-L6-v2', device="cpu")
model._modules["1"].pooling_mode_mean_tokens = False
model._modules["1"].pooling_mode_cls_token = True
qconfig_dict = {
Embedding : float_qparams_weight_only_qconfig,
Linear: default_dynamic_qconfig
}
q_model = quantize_dynamic(model, qconfig_dict)
//work well uppon this point, model work well, it take a lot less space, great succes ! 
q_model.save_pretrained("lamashnikov/cls-quantitized-paraphrase-MiniLM-L6-v2")
//problem happen here


procedure was described here:
 https://github.com/UKPLab/sentence-transformers/blob/master/examples/training/distillation/model_quantization.py


### Expected behavior

not that: 

Traceback (most recent call last):
File "/home/censored/perso/./my_project.py", line 190, in
main()
File "/home/censored/perso/./my_project.py", line 171, in main
q_model.save_pretrained("lamashnikov/cls-quantitized-paraphrase-MiniLM-L6-v2")
File "/home/censored/.local/lib/python3.10/site-packages/sentence_transformers/SentenceTransformer.py", line 1072, in save_pretrained
self.save(
File "/home/sencored/.local/lib/python3.10/site-packages/sentence_transformers/SentenceTransformer.py", line 1037, in save
module.save(model_path, safe_serialization=safe_serialization)
File "/home/censored/.local/lib/python3.10/site-packages/sentence_transformers/models/Transformer.py", line 180, in save
self.auto_model.save_pretrained(output_path, safe_serialization=safe_serialization)
File "/home/censored/.local/lib/python3.10/site-packages/transformers/modeling_utils.py", line 2698, in save_pretrained
shared_names, disjoint_names = _find_disjoint(shared_ptrs.values(), state_dict)
File "/home/censored/.local/lib/python3.10/site-packages/transformers/modeling_utils.py", line 650, in _find_disjoint
areas.append((tensor.data_ptr(), _end_ptr(tensor), name))
AttributeError: 'torch.dtype' object has no attribute 'data_ptr'

Comments:
I'm able to save it to pickle format and restore it, but i wanted to save it to the hub so it's kind of annoying.

I can't either save it to safetensor format

`save_file(q_model.state_dict(), 'model.safetensors')`


File "/home/censored/.local/lib/python3.10/site-packages/safetensors/torch.py", line 286, in save_file
serialize_file(_flatten(tensors), filename, metadata=metadata)
File "/home/censored/.local/lib/python3.10/site-packages/safetensors/torch.py", line 470, in _flatten
raise ValueError(f"Key {k} is invalid, expected torch.Tensor but received {type(v)}")
ValueError: Key 0.auto_model.embeddings.word_embeddings._packed_params.dtype is invalid, expected torch.Tensor but received <class 'torch.dtype'>

After digging a little it seems that quantized models have a lot more information than """"normal"""" models, including some meta-data such as the type , and safe-tensors serialization wasn't expecting such meta-data. Seems that the name of the tensors is also changing from quantized to not quantized

quantized state dict:
q_model state dict:
<class 'torch.dtype'> 0.auto_model.embeddings.word_embeddings._packed_params.dtype
<class 'torch.Tensor'> 0.auto_model.embeddings.word_embeddings._packed_params._packed_weight
<class 'torch.dtype'> 0.auto_model.embeddings.position_embeddings._packed_params.dtype
<class 'torch.Tensor'> 0.auto_model.embeddings.position_embeddings._packed_params._packed_weight
<class 'torch.dtype'> 0.auto_model.embeddings.token_type_embeddings._packed_params.dtype
<class 'torch.Tensor'> 0.auto_model.embeddings.token_type_embeddings._packed_params._packed_weight
<class 'torch.Tensor'> 0.auto_model.embeddings.LayerNorm.weight
<class 'torch.Tensor'> 0.auto_model.embeddings.LayerNorm.bias
<class 'torch.Tensor'> 0.auto_model.encoder.layer.0.attention.self.query.scale
<class 'torch.Tensor'> 0.auto_model.encoder.layer.0.attention.self.query.zero_point
<class 'torch.dtype'> 0.auto_model.encoder.layer.0.attention.self.query._packed_params.dtype
<class 'tuple'> 0.auto_model.encoder.layer.0.attention.self.query._packed_params._packed_params
<class 'torch.Tensor'> 0.auto_model.encoder.layer.0.attention.self.key.scale
<class 'torch.Tensor'> 0.auto_model.encoder.layer.0.attention.self.key.zero_point
<class 'torch.dtype'> 0.auto_model.encoder.layer.0.attention.self.key._packed_params.dtype
<class 'tuple'> 0.auto_model.encoder.layer.0.attention.self.key._packed_params._packed_params
<class 'torch.Tensor'> 0.auto_model.encoder.layer.0.attention.self.value.scale
<class 'torch.Tensor'> 0.auto_model.encoder.layer.0.attention.self.value.zero_point
<class 'torch.dtype'> 0.auto_model.encoder.layer.0.attention.self.value._packed_params.dtype
<class 'tuple'> 0.auto_model.encoder.layer.0.attention.self.value._packed_params._packed_params
<class 'torch.Tensor'> 0.auto_model.encoder.layer.0.attention.output.dense.scale
<class 'torch.Tensor'> 0.auto_model.encoder.layer.0.attention.output.dense.zero_point
<class 'torch.dtype'> 0.auto_model.encoder.layer.0.attention.output.dense._packed_params.dtype
<class 'tuple'> 0.auto_model.encoder.layer.0.attention.output.dense._packed_params._packed_params
<class 'torch.Tensor'> 0.auto_model.encoder.layer.0.attention.output.LayerNorm.weight
<class 'torch.Tensor'> 0.auto_model.encoder.layer.0.attention.output.LayerNorm.bias
<class 'torch.Tensor'> 0.auto_model.encoder.layer.0.intermediate.dense.scale
<class 'torch.Tensor'> 0.auto_model.encoder.layer.0.intermediate.dense.zero_point
<class 'torch.dtype'> 0.auto_model.encoder.layer.0.intermediate.dense._packed_params.dtype
<class 'tuple'> 0.auto_model.encoder.layer.0.intermediate.dense._packed_params._packed_params
<class 'torch.Tensor'> 0.auto_model.encoder.layer.0.output.dense.scale
<class 'torch.Tensor'> 0.auto_model.encoder.layer.0.output.dense.zero_point
<class 'torch.dtype'> 0.auto_model.encoder.layer.0.output.dense._packed_params.dtype
<class 'tuple'> 0.auto_model.encoder.layer.0.output.dense._packed_params._packed_params
<class 'torch.Tensor'> 0.auto_model.encoder.layer.0.output.LayerNorm.weight
<class 'torch.Tensor'> 0.auto_model.encoder.layer.0.output.LayerNorm.bias
<class 'torch.Tensor'> 0.auto_model.encoder.layer.1.attention.self.query.scale
<class 'torch.Tensor'> 0.auto_model.encoder.layer.1.attention.self.query.zero_point
<class 'torch.dtype'> 0.auto_model.encoder.layer.1.attention.self.query._packed_params.dtype
<class 'tuple'> 0.auto_model.encoder.layer.1.attention.self.query._packed_params._packed_params
<class 'torch.Tensor'> 0.auto_model.encoder.layer.1.attention.self.key.scale
<class 'torch.Tensor'> 0.auto_model.encoder.layer.1.attention.self.key.zero_point
<class 'torch.dtype'> 0.auto_model.encoder.layer.1.attention.self.key._packed_params.dtype
<class 'tuple'> 0.auto_model.encoder.layer.1.attention.self.key._packed_params._packed_params
<class 'torch.Tensor'> 0.auto_model.encoder.layer.1.attention.self.value.scale
<class 'torch.Tensor'> 0.auto_model.encoder.layer.1.attention.self.value.zero_point
<class 'torch.dtype'> 0.auto_model.encoder.layer.1.attention.self.value._packed_params.dtype
<class 'tuple'> 0.auto_model.encoder.layer.1.attention.self.value._packed_params._packed_params
<class 'torch.Tensor'> 0.auto_model.encoder.layer.1.attention.output.dense.scale
<class 'torch.Tensor'> 0.auto_model.encoder.layer.1.attention.output.dense.zero_point
<class 'torch.dtype'> 0.auto_model.encoder.layer.1.attention.output.dense._packed_params.dtype
<class 'tuple'> 0.auto_model.encoder.layer.1.attention.output.dense._packed_params._packed_params
<class 'torch.Tensor'> 0.auto_model.encoder.layer.1.attention.output.LayerNorm.weight
<class 'torch.Tensor'> 0.auto_model.encoder.layer.1.attention.output.LayerNorm.bias
<class 'torch.Tensor'> 0.auto_model.encoder.layer.1.intermediate.dense.scale
<class 'torch.Tensor'> 0.auto_model.encoder.layer.1.intermediate.dense.zero_point
<class 'torch.dtype'> 0.auto_model.encoder.layer.1.intermediate.dense._packed_params.dtype
<class 'tuple'> 0.auto_model.encoder.layer.1.intermediate.dense._packed_params._packed_params
<class 'torch.Tensor'> 0.auto_model.encoder.layer.1.output.dense.scale
<class 'torch.Tensor'> 0.auto_model.encoder.layer.1.output.dense.zero_point
<class 'torch.dtype'> 0.auto_model.encoder.layer.1.output.dense._packed_params.dtype
<class 'tuple'> 0.auto_model.encoder.layer.1.output.dense._packed_params._packed_params
<class 'torch.Tensor'> 0.auto_model.encoder.layer.1.output.LayerNorm.weight
<class 'torch.Tensor'> 0.auto_model.encoder.layer.1.output.LayerNorm.bias
<class 'torch.Tensor'> 0.auto_model.encoder.layer.2.attention.self.query.scale
<class 'torch.Tensor'> 0.auto_model.encoder.layer.2.attention.self.query.zero_point
<class 'torch.dtype'> 0.auto_model.encoder.layer.2.attention.self.query._packed_params.dtype
<class 'tuple'> 0.auto_model.encoder.layer.2.attention.self.query._packed_params._packed_params
<class 'torch.Tensor'> 0.auto_model.encoder.layer.2.attention.self.key.scale
<class 'torch.Tensor'> 0.auto_model.encoder.layer.2.attention.self.key.zero_point
<class 'torch.dtype'> 0.auto_model.encoder.layer.2.attention.self.key._packed_params.dtype
<class 'tuple'> 0.auto_model.encoder.layer.2.attention.self.key._packed_params._packed_params
<class 'torch.Tensor'> 0.auto_model.encoder.layer.2.attention.self.value.scale
<class 'torch.Tensor'> 0.auto_model.encoder.layer.2.attention.self.value.zero_point
<class 'torch.dtype'> 0.auto_model.encoder.layer.2.attention.self.value._packed_params.dtype
<class 'tuple'> 0.auto_model.encoder.layer.2.attention.self.value._packed_params._packed_params
<class 'torch.Tensor'> 0.auto_model.encoder.layer.2.attention.output.dense.scale
<class 'torch.Tensor'> 0.auto_model.encoder.layer.2.attention.output.dense.zero_point
<class 'torch.dtype'> 0.auto_model.encoder.layer.2.attention.output.dense._packed_params.dtype
<class 'tuple'> 0.auto_model.encoder.layer.2.attention.output.dense._packed_params._packed_params
<class 'torch.Tensor'> 0.auto_model.encoder.layer.2.attention.output.LayerNorm.weight
<class 'torch.Tensor'> 0.auto_model.encoder.layer.2.attention.output.LayerNorm.bias
<class 'torch.Tensor'> 0.auto_model.encoder.layer.2.intermediate.dense.scale
<class 'torch.Tensor'> 0.auto_model.encoder.layer.2.intermediate.dense.zero_point
<class 'torch.dtype'> 0.auto_model.encoder.layer.2.intermediate.dense._packed_params.dtype
<class 'tuple'> 0.auto_model.encoder.layer.2.intermediate.dense._packed_params._packed_params
<class 'torch.Tensor'> 0.auto_model.encoder.layer.2.output.dense.scale
<class 'torch.Tensor'> 0.auto_model.encoder.layer.2.output.dense.zero_point
<class 'torch.dtype'> 0.auto_model.encoder.layer.2.output.dense._packed_params.dtype
<class 'tuple'> 0.auto_model.encoder.layer.2.output.dense._packed_params._packed_params
<class 'torch.Tensor'> 0.auto_model.encoder.layer.2.output.LayerNorm.weight
<class 'torch.Tensor'> 0.auto_model.encoder.layer.2.output.LayerNorm.bias
<class 'torch.Tensor'> 0.auto_model.encoder.layer.3.attention.self.query.scale
<class 'torch.Tensor'> 0.auto_model.encoder.layer.3.attention.self.query.zero_point
<class 'torch.dtype'> 0.auto_model.encoder.layer.3.attention.self.query._packed_params.dtype
<class 'tuple'> 0.auto_model.encoder.layer.3.attention.self.query._packed_params._packed_params
<class 'torch.Tensor'> 0.auto_model.encoder.layer.3.attention.self.key.scale
<class 'torch.Tensor'> 0.auto_model.encoder.layer.3.attention.self.key.zero_point
<class 'torch.dtype'> 0.auto_model.encoder.layer.3.attention.self.key._packed_params.dtype
<class 'tuple'> 0.auto_model.encoder.layer.3.attention.self.key._packed_params._packed_params
<class 'torch.Tensor'> 0.auto_model.encoder.layer.3.attention.self.value.scale
<class 'torch.Tensor'> 0.auto_model.encoder.layer.3.attention.self.value.zero_point
<class 'torch.dtype'> 0.auto_model.encoder.layer.3.attention.self.value._packed_params.dtype
<class 'tuple'> 0.auto_model.encoder.layer.3.attention.self.value._packed_params._packed_params
<class 'torch.Tensor'> 0.auto_model.encoder.layer.3.attention.output.dense.scale
<class 'torch.Tensor'> 0.auto_model.encoder.layer.3.attention.output.dense.zero_point
<class 'torch.dtype'> 0.auto_model.encoder.layer.3.attention.output.dense._packed_params.dtype
<class 'tuple'> 0.auto_model.encoder.layer.3.attention.output.dense._packed_params._packed_params
<class 'torch.Tensor'> 0.auto_model.encoder.layer.3.attention.output.LayerNorm.weight
<class 'torch.Tensor'> 0.auto_model.encoder.layer.3.attention.output.LayerNorm.bias
<class 'torch.Tensor'> 0.auto_model.encoder.layer.3.intermediate.dense.scale
<class 'torch.Tensor'> 0.auto_model.encoder.layer.3.intermediate.dense.zero_point
<class 'torch.dtype'> 0.auto_model.encoder.layer.3.intermediate.dense._packed_params.dtype
<class 'tuple'> 0.auto_model.encoder.layer.3.intermediate.dense._packed_params._packed_params
<class 'torch.Tensor'> 0.auto_model.encoder.layer.3.output.dense.scale
<class 'torch.Tensor'> 0.auto_model.encoder.layer.3.output.dense.zero_point
<class 'torch.dtype'> 0.auto_model.encoder.layer.3.output.dense._packed_params.dtype
<class 'tuple'> 0.auto_model.encoder.layer.3.output.dense._packed_params._packed_params
<class 'torch.Tensor'> 0.auto_model.encoder.layer.3.output.LayerNorm.weight
<class 'torch.Tensor'> 0.auto_model.encoder.layer.3.output.LayerNorm.bias
<class 'torch.Tensor'> 0.auto_model.encoder.layer.4.attention.self.query.scale
<class 'torch.Tensor'> 0.auto_model.encoder.layer.4.attention.self.query.zero_point
<class 'torch.dtype'> 0.auto_model.encoder.layer.4.attention.self.query._packed_params.dtype
<class 'tuple'> 0.auto_model.encoder.layer.4.attention.self.query._packed_params._packed_params
<class 'torch.Tensor'> 0.auto_model.encoder.layer.4.attention.self.key.scale
<class 'torch.Tensor'> 0.auto_model.encoder.layer.4.attention.self.key.zero_point
<class 'torch.dtype'> 0.auto_model.encoder.layer.4.attention.self.key._packed_params.dtype
<class 'tuple'> 0.auto_model.encoder.layer.4.attention.self.key._packed_params._packed_params
<class 'torch.Tensor'> 0.auto_model.encoder.layer.4.attention.self.value.scale
<class 'torch.Tensor'> 0.auto_model.encoder.layer.4.attention.self.value.zero_point
<class 'torch.dtype'> 0.auto_model.encoder.layer.4.attention.self.value._packed_params.dtype
<class 'tuple'> 0.auto_model.encoder.layer.4.attention.self.value._packed_params._packed_params
<class 'torch.Tensor'> 0.auto_model.encoder.layer.4.attention.output.dense.scale
<class 'torch.Tensor'> 0.auto_model.encoder.layer.4.attention.output.dense.zero_point
<class 'torch.dtype'> 0.auto_model.encoder.layer.4.attention.output.dense._packed_params.dtype
<class 'tuple'> 0.auto_model.encoder.layer.4.attention.output.dense._packed_params._packed_params
<class 'torch.Tensor'> 0.auto_model.encoder.layer.4.attention.output.LayerNorm.weight
<class 'torch.Tensor'> 0.auto_model.encoder.layer.4.attention.output.LayerNorm.bias
<class 'torch.Tensor'> 0.auto_model.encoder.layer.4.intermediate.dense.scale
<class 'torch.Tensor'> 0.auto_model.encoder.layer.4.intermediate.dense.zero_point
<class 'torch.dtype'> 0.auto_model.encoder.layer.4.intermediate.dense._packed_params.dtype
<class 'tuple'> 0.auto_model.encoder.layer.4.intermediate.dense._packed_params._packed_params
<class 'torch.Tensor'> 0.auto_model.encoder.layer.4.output.dense.scale
<class 'torch.Tensor'> 0.auto_model.encoder.layer.4.output.dense.zero_point
<class 'torch.dtype'> 0.auto_model.encoder.layer.4.output.dense._packed_params.dtype
<class 'tuple'> 0.auto_model.encoder.layer.4.output.dense._packed_params._packed_params
<class 'torch.Tensor'> 0.auto_model.encoder.layer.4.output.LayerNorm.weight
<class 'torch.Tensor'> 0.auto_model.encoder.layer.4.output.LayerNorm.bias
<class 'torch.Tensor'> 0.auto_model.encoder.layer.5.attention.self.query.scale
<class 'torch.Tensor'> 0.auto_model.encoder.layer.5.attention.self.query.zero_point
<class 'torch.dtype'> 0.auto_model.encoder.layer.5.attention.self.query._packed_params.dtype
<class 'tuple'> 0.auto_model.encoder.layer.5.attention.self.query._packed_params._packed_params
<class 'torch.Tensor'> 0.auto_model.encoder.layer.5.attention.self.key.scale
<class 'torch.Tensor'> 0.auto_model.encoder.layer.5.attention.self.key.zero_point
<class 'torch.dtype'> 0.auto_model.encoder.layer.5.attention.self.key._packed_params.dtype
<class 'tuple'> 0.auto_model.encoder.layer.5.attention.self.key._packed_params._packed_params
<class 'torch.Tensor'> 0.auto_model.encoder.layer.5.attention.self.value.scale
<class 'torch.Tensor'> 0.auto_model.encoder.layer.5.attention.self.value.zero_point
<class 'torch.dtype'> 0.auto_model.encoder.layer.5.attention.self.value._packed_params.dtype
<class 'tuple'> 0.auto_model.encoder.layer.5.attention.self.value._packed_params._packed_params
<class 'torch.Tensor'> 0.auto_model.encoder.layer.5.attention.output.dense.scale
<class 'torch.Tensor'> 0.auto_model.encoder.layer.5.attention.output.dense.zero_point
<class 'torch.dtype'> 0.auto_model.encoder.layer.5.attention.output.dense._packed_params.dtype
<class 'tuple'> 0.auto_model.encoder.layer.5.attention.output.dense._packed_params._packed_params
<class 'torch.Tensor'> 0.auto_model.encoder.layer.5.attention.output.LayerNorm.weight
<class 'torch.Tensor'> 0.auto_model.encoder.layer.5.attention.output.LayerNorm.bias
<class 'torch.Tensor'> 0.auto_model.encoder.layer.5.intermediate.dense.scale
<class 'torch.Tensor'> 0.auto_model.encoder.layer.5.intermediate.dense.zero_point
<class 'torch.dtype'> 0.auto_model.encoder.layer.5.intermediate.dense._packed_params.dtype
<class 'tuple'> 0.auto_model.encoder.layer.5.intermediate.dense._packed_params._packed_params
<class 'torch.Tensor'> 0.auto_model.encoder.layer.5.output.dense.scale
<class 'torch.Tensor'> 0.auto_model.encoder.layer.5.output.dense.zero_point
<class 'torch.dtype'> 0.auto_model.encoder.layer.5.output.dense._packed_params.dtype
<class 'tuple'> 0.auto_model.encoder.layer.5.output.dense._packed_params._packed_params
<class 'torch.Tensor'> 0.auto_model.encoder.layer.5.output.LayerNorm.weight
<class 'torch.Tensor'> 0.auto_model.encoder.layer.5.output.LayerNorm.bias
<class 'torch.Tensor'> 0.auto_model.pooler.dense.scale
<class 'torch.Tensor'> 0.auto_model.pooler.dense.zero_point
<class 'torch.dtype'> 0.auto_model.pooler.dense._packed_params.dtype
<class 'tuple'> 0.auto_model.pooler.dense._packed_params._packed_params

not quantized state dict:

model state dict:
<class 'torch.Tensor'> 0.auto_model.embeddings.word_embeddings.weight
<class 'torch.Tensor'> 0.auto_model.embeddings.position_embeddings.weight
<class 'torch.Tensor'> 0.auto_model.embeddings.token_type_embeddings.weight
<class 'torch.Tensor'> 0.auto_model.embeddings.LayerNorm.weight
<class 'torch.Tensor'> 0.auto_model.embeddings.LayerNorm.bias
<class 'torch.Tensor'> 0.auto_model.encoder.layer.0.attention.self.query.weight
<class 'torch.Tensor'> 0.auto_model.encoder.layer.0.attention.self.query.bias
<class 'torch.Tensor'> 0.auto_model.encoder.layer.0.attention.self.key.weight
<class 'torch.Tensor'> 0.auto_model.encoder.layer.0.attention.self.key.bias
<class 'torch.Tensor'> 0.auto_model.encoder.layer.0.attention.self.value.weight
<class 'torch.Tensor'> 0.auto_model.encoder.layer.0.attention.self.value.bias
<class 'torch.Tensor'> 0.auto_model.encoder.layer.0.attention.output.dense.weight
<class 'torch.Tensor'> 0.auto_model.encoder.layer.0.attention.output.dense.bias
<class 'torch.Tensor'> 0.auto_model.encoder.layer.0.attention.output.LayerNorm.weight
<class 'torch.Tensor'> 0.auto_model.encoder.layer.0.attention.output.LayerNorm.bias
<class 'torch.Tensor'> 0.auto_model.encoder.layer.0.intermediate.dense.weight
<class 'torch.Tensor'> 0.auto_model.encoder.layer.0.intermediate.dense.bias
<class 'torch.Tensor'> 0.auto_model.encoder.layer.0.output.dense.weight
<class 'torch.Tensor'> 0.auto_model.encoder.layer.0.output.dense.bias
<class 'torch.Tensor'> 0.auto_model.encoder.layer.0.output.LayerNorm.weight
<class 'torch.Tensor'> 0.auto_model.encoder.layer.0.output.LayerNorm.bias
<class 'torch.Tensor'> 0.auto_model.encoder.layer.1.attention.self.query.weight
<class 'torch.Tensor'> 0.auto_model.encoder.layer.1.attention.self.query.bias
<class 'torch.Tensor'> 0.auto_model.encoder.layer.1.attention.self.key.weight
<class 'torch.Tensor'> 0.auto_model.encoder.layer.1.attention.self.key.bias
<class 'torch.Tensor'> 0.auto_model.encoder.layer.1.attention.self.value.weight
<class 'torch.Tensor'> 0.auto_model.encoder.layer.1.attention.self.value.bias
<class 'torch.Tensor'> 0.auto_model.encoder.layer.1.attention.output.dense.weight
<class 'torch.Tensor'> 0.auto_model.encoder.layer.1.attention.output.dense.bias
<class 'torch.Tensor'> 0.auto_model.encoder.layer.1.attention.output.LayerNorm.weight
<class 'torch.Tensor'> 0.auto_model.encoder.layer.1.attention.output.LayerNorm.bias
<class 'torch.Tensor'> 0.auto_model.encoder.layer.1.intermediate.dense.weight
<class 'torch.Tensor'> 0.auto_model.encoder.layer.1.intermediate.dense.bias
<class 'torch.Tensor'> 0.auto_model.encoder.layer.1.output.dense.weight
<class 'torch.Tensor'> 0.auto_model.encoder.layer.1.output.dense.bias
<class 'torch.Tensor'> 0.auto_model.encoder.layer.1.output.LayerNorm.weight
<class 'torch.Tensor'> 0.auto_model.encoder.layer.1.output.LayerNorm.bias
<class 'torch.Tensor'> 0.auto_model.encoder.layer.2.attention.self.query.weight
<class 'torch.Tensor'> 0.auto_model.encoder.layer.2.attention.self.query.bias
<class 'torch.Tensor'> 0.auto_model.encoder.layer.2.attention.self.key.weight
<class 'torch.Tensor'> 0.auto_model.encoder.layer.2.attention.self.key.bias
<class 'torch.Tensor'> 0.auto_model.encoder.layer.2.attention.self.value.weight
<class 'torch.Tensor'> 0.auto_model.encoder.layer.2.attention.self.value.bias
<class 'torch.Tensor'> 0.auto_model.encoder.layer.2.attention.output.dense.weight
<class 'torch.Tensor'> 0.auto_model.encoder.layer.2.attention.output.dense.bias
<class 'torch.Tensor'> 0.auto_model.encoder.layer.2.attention.output.LayerNorm.weight
<class 'torch.Tensor'> 0.auto_model.encoder.layer.2.attention.output.LayerNorm.bias
<class 'torch.Tensor'> 0.auto_model.encoder.layer.2.intermediate.dense.weight
<class 'torch.Tensor'> 0.auto_model.encoder.layer.2.intermediate.dense.bias
<class 'torch.Tensor'> 0.auto_model.encoder.layer.2.output.dense.weight
<class 'torch.Tensor'> 0.auto_model.encoder.layer.2.output.dense.bias
<class 'torch.Tensor'> 0.auto_model.encoder.layer.2.output.LayerNorm.weight
<class 'torch.Tensor'> 0.auto_model.encoder.layer.2.output.LayerNorm.bias
<class 'torch.Tensor'> 0.auto_model.encoder.layer.3.attention.self.query.weight
<class 'torch.Tensor'> 0.auto_model.encoder.layer.3.attention.self.query.bias
<class 'torch.Tensor'> 0.auto_model.encoder.layer.3.attention.self.key.weight
<class 'torch.Tensor'> 0.auto_model.encoder.layer.3.attention.self.key.bias
<class 'torch.Tensor'> 0.auto_model.encoder.layer.3.attention.self.value.weight
<class 'torch.Tensor'> 0.auto_model.encoder.layer.3.attention.self.value.bias
<class 'torch.Tensor'> 0.auto_model.encoder.layer.3.attention.output.dense.weight
<class 'torch.Tensor'> 0.auto_model.encoder.layer.3.attention.output.dense.bias
<class 'torch.Tensor'> 0.auto_model.encoder.layer.3.attention.output.LayerNorm.weight
<class 'torch.Tensor'> 0.auto_model.encoder.layer.3.attention.output.LayerNorm.bias
<class 'torch.Tensor'> 0.auto_model.encoder.layer.3.intermediate.dense.weight
<class 'torch.Tensor'> 0.auto_model.encoder.layer.3.intermediate.dense.bias
<class 'torch.Tensor'> 0.auto_model.encoder.layer.3.output.dense.weight
<class 'torch.Tensor'> 0.auto_model.encoder.layer.3.output.dense.bias
<class 'torch.Tensor'> 0.auto_model.encoder.layer.3.output.LayerNorm.weight
<class 'torch.Tensor'> 0.auto_model.encoder.layer.3.output.LayerNorm.bias
<class 'torch.Tensor'> 0.auto_model.encoder.layer.4.attention.self.query.weight
<class 'torch.Tensor'> 0.auto_model.encoder.layer.4.attention.self.query.bias
<class 'torch.Tensor'> 0.auto_model.encoder.layer.4.attention.self.key.weight
<class 'torch.Tensor'> 0.auto_model.encoder.layer.4.attention.self.key.bias
<class 'torch.Tensor'> 0.auto_model.encoder.layer.4.attention.self.value.weight
<class 'torch.Tensor'> 0.auto_model.encoder.layer.4.attention.self.value.bias
<class 'torch.Tensor'> 0.auto_model.encoder.layer.4.attention.output.dense.weight
<class 'torch.Tensor'> 0.auto_model.encoder.layer.4.attention.output.dense.bias
<class 'torch.Tensor'> 0.auto_model.encoder.layer.4.attention.output.LayerNorm.weight
<class 'torch.Tensor'> 0.auto_model.encoder.layer.4.attention.output.LayerNorm.bias
<class 'torch.Tensor'> 0.auto_model.encoder.layer.4.intermediate.dense.weight
<class 'torch.Tensor'> 0.auto_model.encoder.layer.4.intermediate.dense.bias
<class 'torch.Tensor'> 0.auto_model.encoder.layer.4.output.dense.weight
<class 'torch.Tensor'> 0.auto_model.encoder.layer.4.output.dense.bias
<class 'torch.Tensor'> 0.auto_model.encoder.layer.4.output.LayerNorm.weight
<class 'torch.Tensor'> 0.auto_model.encoder.layer.4.output.LayerNorm.bias
<class 'torch.Tensor'> 0.auto_model.encoder.layer.5.attention.self.query.weight
<class 'torch.Tensor'> 0.auto_model.encoder.layer.5.attention.self.query.bias
<class 'torch.Tensor'> 0.auto_model.encoder.layer.5.attention.self.key.weight
<class 'torch.Tensor'> 0.auto_model.encoder.layer.5.attention.self.key.bias
<class 'torch.Tensor'> 0.auto_model.encoder.layer.5.attention.self.value.weight
<class 'torch.Tensor'> 0.auto_model.encoder.layer.5.attention.self.value.bias
<class 'torch.Tensor'> 0.auto_model.encoder.layer.5.attention.output.dense.weight
<class 'torch.Tensor'> 0.auto_model.encoder.layer.5.attention.output.dense.bias
<class 'torch.Tensor'> 0.auto_model.encoder.layer.5.attention.output.LayerNorm.weight
<class 'torch.Tensor'> 0.auto_model.encoder.layer.5.attention.output.LayerNorm.bias
<class 'torch.Tensor'> 0.auto_model.encoder.layer.5.intermediate.dense.weight
<class 'torch.Tensor'> 0.auto_model.encoder.layer.5.intermediate.dense.bias
<class 'torch.Tensor'> 0.auto_model.encoder.layer.5.output.dense.weight
<class 'torch.Tensor'> 0.auto_model.encoder.layer.5.output.dense.bias
<class 'torch.Tensor'> 0.auto_model.encoder.layer.5.output.LayerNorm.weight
<class 'torch.Tensor'> 0.auto_model.encoder.layer.5.output.LayerNorm.bias
<class 'torch.Tensor'> 0.auto_model.pooler.dense.weight
<class 'torch.Tensor'> 0.auto_model.pooler.dense.bias


The place where it explode (expecting a tensor, got meta-data), in "transformers/modeling_utils.py", line 650,

    for name in shared:
        tensor = state_dict[name]
        areas.append((tensor.data_ptr(), _end_ptr(tensor), name))
    areas.sort()


Thanks for the issue! cc @SunMarc 

Hey @lamashnikov, thanks for the issue. This happens because you can only save tensors with safetensors, hence the issue. For now, I would just suggest you to save with `safe_serialization=False`. You can still upload it to the hub. 

Hey @SunMarc ! thank you for your response! 

model = SentenceTransformer('sentence-transformers/paraphrase-MiniLM-L6-v2', device="cpu")
model._modules["1"].pooling_mode_mean_tokens = False
model._modules["1"].pooling_mode_cls_token = True
qconfig_dict = {
Embedding : float_qparams_weight_only_qconfig,
Linear: default_dynamic_qconfig
}
q_model = quantize_dynamic(model, qconfig_dict)
q_model.save_pretrained("lamashnikov/cls-quantitized-paraphrase-MiniLM-L6-v2", safe_serialization=False)


 File "/home/censored/perso/./licence_checker.py", line 182, in main
    q_model.save_pretrained("lamashnikov/cls-quantitized-paraphrase-MiniLM-L6-v2", safe_serialization=False)
  File "/home/censored/.local/lib/python3.10/site-packages/sentence_transformers/SentenceTransformer.py", line 1072, in save_pretrained
    self.save(
  File "/home/censored/.local/lib/python3.10/site-packages/sentence_transformers/SentenceTransformer.py", line 1037, in save
    module.save(model_path, safe_serialization=safe_serialization)
  File "/home/censored/.local/lib/python3.10/site-packages/sentence_transformers/models/Transformer.py", line 180, in save
    self.auto_model.save_pretrained(output_path, safe_serialization=safe_serialization)
  File "/home/censored/.local/lib/python3.10/site-packages/transformers/modeling_utils.py", line 2736, in save_pretrained
    state_dict_split = split_torch_state_dict_into_shards(
  File "/home/censored/.local/lib/python3.10/site-packages/huggingface_hub/serialization/_torch.py", line 330, in split_torch_state_dict_into_shards
    return split_state_dict_into_shards_factory(
  File "/home/censored/.local/lib/python3.10/site-packages/huggingface_hub/serialization/_base.py", line 108, in split_state_dict_into_shards_factory
    storage_id = get_storage_id(tensor)
  File "/home/censored/.local/lib/python3.10/site-packages/huggingface_hub/serialization/_torch.py", line 350, in get_torch_storage_id
    if tensor.device.type == "xla" and is_torch_tpu_available():
AttributeError: 'torch.dtype' object has no attribute 'device'


It doesn't seems to work either with safe_serialization=False

Hmm indeed since this function only allows for tensors. We should probably allow non tensors here and let torch.save or satentensors.save to trigger an error if we are trying to save non tensors values. Would you like to give it a go @lamashnikov ? Otherwise, I will have a look later ! 

Yes i would! But I had a rought day today so i might check it tomorrow if it's ok for you (I'm in Paris Timezone)

Hi @SunMarc 
I've been there so far: 

I modified split_state_dict_into_shard ( i did that crappy, just to test and do a proof of concept) to accept tuples and torch.dtypes. I know that this function have to be backend agnostic and type check should be done in _torch.py


    for key, tensor in state_dict.items():
        # when bnb serialization is used the weights in the state dict can be strings
        # check: https://github.com/huggingface/transformers/pull/24416 for more details
        if isinstance(tensor, str):
            logger.info("Skipping tensor %s as it is a string (bnb serialization)", key)
            continue
        if isinstance(tensor, torch.dtype):
            logger.info("tensor type info %s", key)
            current_shard[key] = tensor
            continue
        if isinstance(tensor, tuple):
            logger.info("packed tuple %s", key)
            storage_id = get_storage_id(tensor[1])
            tensor_size = get_storage_size(tensor[0]) + get_storage_size(tensor[1])
        else:
            storage_id = get_storage_id(tensor)
            tensor_size = get_storage_size(tensor)
         [....]
        # Add the tensor to the current shard
        current_shard[key] = tensor
        current_shard_size += tensor_size
        total_size += tensor_size


The save of the model is now happening without issues, and i get a pytorch.bin of the same size of the quantized model. However, i now have an issue when loading the model. 


Some weights of the model checkpoint at lamashnikov/cls-quantitized-paraphrase-MiniLM-L6-v2 were not used when initializing BertModel: ['embeddings.position_embeddings._packed_params._packed_weight', 'embeddings.position_embeddings._packed_params.dtype', 'embeddings.token_type_embeddings._packed_params._packed_weight', 'embeddings.token_type_embeddings._packed_params.dtype', 'embeddings.word_embeddings._packed_params._packed_weight', 'embeddings.word_embeddings._packed_params.dtype', 'encoder.layer.0.attention.output.dense._packed_params._packed_params', 'encoder.layer.0.attention.output.dense._packed_params.dtype', 'encoder.layer.0.attention.output.dense.scale', 'encoder.layer.0.attention.output.dense.zero_point', 'encoder.layer.0.attention.self.key._packed_params._packed_params', 'encoder.layer.0.attention.self.key._packed_params.dtype', 'encoder.layer.0.attention.self.key.scale', 'encoder.layer.0.attention.self.key.zero_point', 'encoder.layer.0.attention.self.query._packed_params._packed_params', 'encoder.layer.0.attention.self.query._packed_params.dtype', 'encoder.layer.0.attention.self.query.scale', 'encoder.layer.0.attention.self.query.zero_point', 'encoder.layer.0.attention.self.value._packed_params._packed_params', 'encoder.layer.0.attention.self.value._packed_params.dtype', 'encoder.layer.0.attention.self.value.scale', 'encoder.layer.0.attention.self.value.zero_point', 'encoder.layer.0.intermediate.dense._packed_params._packed_params', 'encoder.layer.0.intermediate.dense._packed_params.dtype', 'encoder.layer.0.intermediate.dense.scale', 'encoder.layer.0.intermediate.dense.zero_point', 'encoder.layer.0.output.dense._packed_params._packed_params', 'encoder.layer.0.output.dense._packed_params.dtype', 'encoder.layer.0.output.dense.scale', 'encoder.layer.0.output.dense.zero_point', 'encoder.layer.1.attention.output.dense._packed_params._packed_params', 'encoder.layer.1.attention.output.dense._packed_params.dtype', 'encoder.layer.1.attention.output.dense.scale', 'encoder.layer.1.attention.output.dense.zero_point', 'encoder.layer.1.attention.self.key._packed_params._packed_params', 'encoder.layer.1.attention.self.key._packed_params.dtype', 'encoder.layer.1.attention.self.key.scale', 'encoder.layer.1.attention.self.key.zero_point', 'encoder.layer.1.attention.self.query._packed_params._packed_params', 'encoder.layer.1.attention.self.query._packed_params.dtype', 'encoder.layer.1.attention.self.query.scale', 'encoder.layer.1.attention.self.query.zero_point', 'encoder.layer.1.attention.self.value._packed_params._packed_params', 'encoder.layer.1.attention.self.value._packed_params.dtype', 'encoder.layer.1.attention.self.value.scale', 'encoder.layer.1.attention.self.value.zero_point', 'encoder.layer.1.intermediate.dense._packed_params._packed_params', 'encoder.layer.1.intermediate.dense._packed_params.dtype', 'encoder.layer.1.intermediate.dense.scale', 'encoder.layer.1.intermediate.dense.zero_point', 'encoder.layer.1.output.dense._packed_params._packed_params', 'encoder.layer.1.output.dense._packed_params.dtype', 'encoder.layer.1.output.dense.scale', 'encoder.layer.1.output.dense.zero_point', 'encoder.layer.2.attention.output.dense._packed_params._packed_params', 'encoder.layer.2.attention.output.dense._packed_params.dtype', 'encoder.layer.2.attention.output.dense.scale', 'encoder.layer.2.attention.output.dense.zero_point', 'encoder.layer.2.attention.self.key._packed_params._packed_params', 'encoder.layer.2.attention.self.key._packed_params.dtype', 'encoder.layer.2.attention.self.key.scale', 'encoder.layer.2.attention.self.key.zero_point', 'encoder.layer.2.attention.self.query._packed_params._packed_params', 'encoder.layer.2.attention.self.query._packed_params.dtype', 'encoder.layer.2.attention.self.query.scale', 'encoder.layer.2.attention.self.query.zero_point', 'encoder.layer.2.attention.self.value._packed_params._packed_params', 'encoder.layer.2.attention.self.value._packed_params.dtype', 'encoder.layer.2.attention.self.value.scale', 'encoder.layer.2.attention.self.value.zero_point', 'encoder.layer.2.intermediate.dense._packed_params._packed_params', 'encoder.layer.2.intermediate.dense._packed_params.dtype', 'encoder.layer.2.intermediate.dense.scale', 'encoder.layer.2.intermediate.dense.zero_point', 'encoder.layer.2.output.dense._packed_params._packed_params', 'encoder.layer.2.output.dense._packed_params.dtype', 'encoder.layer.2.output.dense.scale', 'encoder.layer.2.output.dense.zero_point', 'encoder.layer.3.attention.output.dense._packed_params._packed_params', 'encoder.layer.3.attention.output.dense._packed_params.dtype', 'encoder.layer.3.attention.output.dense.scale', 'encoder.layer.3.attention.output.dense.zero_point', 'encoder.layer.3.attention.self.key._packed_params._packed_params', 'encoder.layer.3.attention.self.key._packed_params.dtype', 'encoder.layer.3.attention.self.key.scale', 'encoder.layer.3.attention.self.key.zero_point', 'encoder.layer.3.attention.self.query._packed_params._packed_params', 'encoder.layer.3.attention.self.query._packed_params.dtype', 'encoder.layer.3.attention.self.query.scale', 'encoder.layer.3.attention.self.query.zero_point', 'encoder.layer.3.attention.self.value._packed_params._packed_params', 'encoder.layer.3.attention.self.value._packed_params.dtype', 'encoder.layer.3.attention.self.value.scale', 'encoder.layer.3.attention.self.value.zero_point', 'encoder.layer.3.intermediate.dense._packed_params._packed_params', 'encoder.layer.3.intermediate.dense._packed_params.dtype', 'encoder.layer.3.intermediate.dense.scale', 'encoder.layer.3.intermediate.dense.zero_point', 'encoder.layer.3.output.dense._packed_params._packed_params', 'encoder.layer.3.output.dense._packed_params.dtype', 'encoder.layer.3.output.dense.scale', 'encoder.layer.3.output.dense.zero_point', 'encoder.layer.4.attention.output.dense._packed_params._packed_params', 'encoder.layer.4.attention.output.dense._packed_params.dtype', 'encoder.layer.4.attention.output.dense.scale', 'encoder.layer.4.attention.output.dense.zero_point', 'encoder.layer.4.attention.self.key._packed_params._packed_params', 'encoder.layer.4.attention.self.key._packed_params.dtype', 'encoder.layer.4.attention.self.key.scale', 'encoder.layer.4.attention.self.key.zero_point', 'encoder.layer.4.attention.self.query._packed_params._packed_params', 'encoder.layer.4.attention.self.query._packed_params.dtype', 'encoder.layer.4.attention.self.query.scale', 'encoder.layer.4.attention.self.query.zero_point', 'encoder.layer.4.attention.self.value._packed_params._packed_params', 'encoder.layer.4.attention.self.value._packed_params.dtype', 'encoder.layer.4.attention.self.value.scale', 'encoder.layer.4.attention.self.value.zero_point', 'encoder.layer.4.intermediate.dense._packed_params._packed_params', 'encoder.layer.4.intermediate.dense._packed_params.dtype', 'encoder.layer.4.intermediate.dense.scale', 'encoder.layer.4.intermediate.dense.zero_point', 'encoder.layer.4.output.dense._packed_params._packed_params', 'encoder.layer.4.output.dense._packed_params.dtype', 'encoder.layer.4.output.dense.scale', 'encoder.layer.4.output.dense.zero_point', 'encoder.layer.5.attention.output.dense._packed_params._packed_params', 'encoder.layer.5.attention.output.dense._packed_params.dtype', 'encoder.layer.5.attention.output.dense.scale', 'encoder.layer.5.attention.output.dense.zero_point', 'encoder.layer.5.attention.self.key._packed_params._packed_params', 'encoder.layer.5.attention.self.key._packed_params.dtype', 'encoder.layer.5.attention.self.key.scale', 'encoder.layer.5.attention.self.key.zero_point', 'encoder.layer.5.attention.self.query._packed_params._packed_params', 'encoder.layer.5.attention.self.query._packed_params.dtype', 'encoder.layer.5.attention.self.query.scale', 'encoder.layer.5.attention.self.query.zero_point', 'encoder.layer.5.attention.self.value._packed_params._packed_params', 'encoder.layer.5.attention.self.value._packed_params.dtype', 'encoder.layer.5.attention.self.value.scale', 'encoder.layer.5.attention.self.value.zero_point', 'encoder.layer.5.intermediate.dense._packed_params._packed_params', 'encoder.layer.5.intermediate.dense._packed_params.dtype', 'encoder.layer.5.intermediate.dense.scale', 'encoder.layer.5.intermediate.dense.zero_point', 'encoder.layer.5.output.dense._packed_params._packed_params', 'encoder.layer.5.output.dense._packed_params.dtype', 'encoder.layer.5.output.dense.scale', 'encoder.layer.5.output.dense.zero_point', 'pooler.dense._packed_params._packed_params', 'pooler.dense._packed_params.dtype', 'pooler.dense.scale', 'pooler.dense.zero_point']
- This IS expected if you are initializing BertModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing BertModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
Some weights of BertModel were not initialized from the model checkpoint at lamashnikov/cls-quantitized-paraphrase-MiniLM-L6-v2 and are newly initialized: ['embeddings.position_embeddings.weight', 'embeddings.token_type_embeddings.weight', 'embeddings.word_embeddings.weight', 'encoder.layer.0.attention.output.dense.bias', 'encoder.layer.0.attention.output.dense.weight', 'encoder.layer.0.attention.self.key.bias', 'encoder.layer.0.attention.self.key.weight', 'encoder.layer.0.attention.self.query.bias', 'encoder.layer.0.attention.self.query.weight', 'encoder.layer.0.attention.self.value.bias', 'encoder.layer.0.attention.self.value.weight', 'encoder.layer.0.intermediate.dense.bias', 'encoder.layer.0.intermediate.dense.weight', 'encoder.layer.0.output.dense.bias', 'encoder.layer.0.output.dense.weight', 'encoder.layer.1.attention.output.dense.bias', 'encoder.layer.1.attention.output.dense.weight', 'encoder.layer.1.attention.self.key.bias', 'encoder.layer.1.attention.self.key.weight', 'encoder.layer.1.attention.self.query.bias', 'encoder.layer.1.attention.self.query.weight', 'encoder.layer.1.attention.self.value.bias', 'encoder.layer.1.attention.self.value.weight', 'encoder.layer.1.intermediate.dense.bias', 'encoder.layer.1.intermediate.dense.weight', 'encoder.layer.1.output.dense.bias', 'encoder.layer.1.output.dense.weight', 'encoder.layer.2.attention.output.dense.bias', 'encoder.layer.2.attention.output.dense.weight', 'encoder.layer.2.attention.self.key.bias', 'encoder.layer.2.attention.self.key.weight', 'encoder.layer.2.attention.self.query.bias', 'encoder.layer.2.attention.self.query.weight', 'encoder.layer.2.attention.self.value.bias', 'encoder.layer.2.attention.self.value.weight', 'encoder.layer.2.intermediate.dense.bias', 'encoder.layer.2.intermediate.dense.weight', 'encoder.layer.2.output.dense.bias', 'encoder.layer.2.output.dense.weight', 'encoder.layer.3.attention.output.dense.bias', 'encoder.layer.3.attention.output.dense.weight', 'encoder.layer.3.attention.self.key.bias', 'encoder.layer.3.attention.self.key.weight', 'encoder.layer.3.attention.self.query.bias', 'encoder.layer.3.attention.self.query.weight', 'encoder.layer.3.attention.self.value.bias', 'encoder.layer.3.attention.self.value.weight', 'encoder.layer.3.intermediate.dense.bias', 'encoder.layer.3.intermediate.dense.weight', 'encoder.layer.3.output.dense.bias', 'encoder.layer.3.output.dense.weight', 'encoder.layer.4.attention.output.dense.bias', 'encoder.layer.4.attention.output.dense.weight', 'encoder.layer.4.attention.self.key.bias', 'encoder.layer.4.attention.self.key.weight', 'encoder.layer.4.attention.self.query.bias', 'encoder.layer.4.attention.self.query.weight', 'encoder.layer.4.attention.self.value.bias', 'encoder.layer.4.attention.self.value.weight', 'encoder.layer.4.intermediate.dense.bias', 'encoder.layer.4.intermediate.dense.weight', 'encoder.layer.4.output.dense.bias', 'encoder.layer.4.output.dense.weight', 'encoder.layer.5.attention.output.dense.bias', 'encoder.layer.5.attention.output.dense.weight', 'encoder.layer.5.attention.self.key.bias', 'encoder.layer.5.attention.self.key.weight', 'encoder.layer.5.attention.self.query.bias', 'encoder.layer.5.attention.self.query.weight', 'encoder.layer.5.attention.self.value.bias', 'encoder.layer.5.attention.self.value.weight', 'encoder.layer.5.intermediate.dense.bias', 'encoder.layer.5.intermediate.dense.weight', 'encoder.layer.5.output.dense.bias', 'encoder.layer.5.output.dense.weight', 'pooler.dense.bias', 'pooler.dense.weight']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.


It seems that the old architecture (without packed tensors) is forced from the loading process wich ignore my packed tensors and load randoms one unpacked instead 

Hi!
I'm having the exact same issue.

To be more precise, I am trying to push a quantized model to the HF hub using the [`PyTorchModelHubMixin`](https://huggingface.co/docs/huggingface_hub/package_reference/mixins#huggingface_hub.PyTorchModelHubMixin). The model itself is a simple ResNet and was quantized using [`torch.quantization.quantize_dynamic`](https://pytorch.org/docs/stable/generated/torch.ao.quantization.quantize_dynamic.html).

 I have the same error message about keys in the state dict that are not expected by safetensors when calling [`model.push_to_hub()`](https://huggingface.co/docs/huggingface_hub/package_reference/mixins#huggingface_hub.ModelHubMixin.push_to_hub):
 ```bash
File "/my_folder/code/hf_hub.py", line 97, in upload_model_reuse
    model.push_to_hub(
  File "/my_folder/code/.pixi/envs/default/lib/python3.12/site-packages/huggingface_hub/utils/_validators.py", line 114, in _inner_fn
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/my_folder/code/.pixi/envs/default/lib/python3.12/site-packages/huggingface_hub/hub_mixin.py", line 693, in push_to_hub
    self.save_pretrained(saved_path, config=config, model_card_kwargs=model_card_kwargs)
  File "/my_folder/code/.pixi/envs/default/lib/python3.12/site-packages/huggingface_hub/hub_mixin.py", line 425, in save_pretrained
    self._save_pretrained(save_directory)
  File "/my_folder/code/.pixi/envs/default/lib/python3.12/site-packages/huggingface_hub/hub_mixin.py", line 771, in _save_pretrained
    save_model_as_safetensor(model_to_save, str(save_directory / constants.SAFETENSORS_SINGLE_FILE))
  File "/my_folder/code/.pixi/envs/default/lib/python3.12/site-packages/safetensors/torch.py", line 155, in save_model
    to_removes = _remove_duplicate_names(state_dict)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/my_folder/code/.pixi/envs/default/lib/python3.12/site-packages/safetensors/torch.py", line 97, in _remove_duplicate_names
    shareds = _find_shared_tensors(state_dict)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/my_folder/code/.pixi/envs/default/lib/python3.12/site-packages/safetensors/torch.py", line 72, in _find_shared_tensors
    if v.device != torch.device("meta") and storage_ptr(v) != 0 and storage_size(v) != 0:
       ^^^^^^^^
AttributeError: 'torch.dtype' object has no attribute 'device'
 ```

@lamashnikov were you able to find a workaround?
@SunMarc Do you have any update on this?

Thanks!

> It seems that the old architecture (without packed tensors) is forced from the loading process which ignore my packed tensors and load randoms one unpacked instead

Could you share a reproducer of the error ? 
However, generally, we can't guarantee that serialization works for a quantization scheme that wasn't added in transformers HfQuantizer ! Quantization usually modifies the layers, hence the need for an integration. If there is enough traction, I can look into adding torch.quantization ! 

If you are using torch, we have recently integrated torchao [quantizer](https://huggingface.co/docs/transformers/en/quantization/torchao) and serialization is also supported ! Maybe it can replace the api you are using ? @grodino @lamashnikov 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Hi @SunMarc, 

Sorry for this late answer. 
As you suggested, I played a bit with [quantizer](https://huggingface.co/docs/transformers/en/quantization/torchao) but I finally decided to not upload the quantized model and quantize on the fly for now.

Best,
Augustin

Awesome, thanks for letting me know. Closing this issue !