SegFormer Quantized Model (Int8) not loading weights properly.
Author: TimbusCalin
Pull request: no

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.16.2
- Platform: Windows 11/Ubuntu 20.04
- Python version: 3.7
- PyTorch version (GPU?): 1.10.2+cu113
- Tensorflow version (GPU?): -
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?:  No

### Who can help

Models:

@NielsRogge.

Library:

- Vision: @NielsRogge, @sgugger

Model I am using (SegFormer):

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [X] my own modified scripts: (give details below)

The tasks I am working on is:
* [X] my own task or dataset: (give details below)



Steps to reproduce the behavior:

I use SegFormer on a custom dataset.

SegFormer works on the custom dataset (3 classes, IoU @0.77, prediction looks great, done just like in the tutorial from @NielsRogge).

What I want to do, is to perform dynamic post-training quantization on the SegFormer model.

1. `model_quantized = quantize_dynamic(model, {nn.Linear}, inplace=False, dtype=torch.qint8)`
2.  `model_quantized.save_pretrained("segformer_quantized.pt")`

Here comes the problem:

VARIANT 1:
```
feature_extractor_inference = SegformerFeatureExtractor()
model = SegformerForSemanticSegmentation.from_pretrained("segformer_quantized.pt").to(device)
```

VARIANT 2:

```
feature_extractor_inference = SegformerFeatureExtractor.from_pretrained("segformer_quantized.pt/config.json")
model = SegformerForSemanticSegmentation.from_pretrained("segformer_quantized.pt").to(device)
```

Both variants seems not to properly load the quantized weights, hence blank predictions.

How can this problem be solved? 

Thank you in advance,

@NielsRogge

```
Some weights of the model checkpoint at nvidia/mit-b0 were not used when initializing SegformerForSemanticSegmentation: ['classifier.weight', 'classifier.bias']
- This IS expected if you are initializing SegformerForSemanticSegmentation from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing SegformerForSemanticSegmentation from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
Some weights of SegformerForSemanticSegmentation were not initialized from the model checkpoint at nvidia/mit-b0 and are newly initialized: ['decode_head.linear_fuse.weight', 'decode_head.linear_c.3.proj.bias', 'decode_head.batch_norm.bias', 'decode_head.linear_c.0.proj.bias', 'decode_head.classifier.bias', 'decode_head.batch_norm.weight', 'decode_head.linear_c.0.proj.weight', 'decode_head.linear_c.2.proj.weight', 'decode_head.linear_c.1.proj.bias', 'decode_head.linear_c.2.proj.bias', 'decode_head.batch_norm.running_var', 'decode_head.batch_norm.running_mean', 'decode_head.linear_c.1.proj.weight', 'decode_head.classifier.weight', 'decode_head.batch_norm.num_batches_tracked', 'decode_head.linear_c.3.proj.weight']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
Some weights of the model checkpoint at segformed_lanes_quantized.pt were not used when initializing SegformerModel: ['segformer.encoder.block.3.1.attention.self.value.zero_point', 'segformer.encoder.block.2.1.mlp.dense2._packed_params._packed_params', 'segformer.encoder.block.0.0.attention.self.query._packed_params.dtype', 'segformer.encoder.block.0.0.mlp.dense1.zero_point', 'segformer.encoder.block.0.0.attention.self.query._packed_params._packed_params', 'segformer.encoder.block.2.0.attention.self.key.zero_point', 'segformer.encoder.block.0.0.attention.self.value.zero_point', 'decode_head.linear_c.3.proj._packed_params._packed_params', 'segformer.encoder.block.1.0.mlp.dense1._packed_params._packed_params', 'segformer.encoder.block.2.1.mlp.dense2.zero_point', 'segformer.encoder.block.1.0.mlp.dense2._packed_params._packed_params', 'segformer.encoder.block.2.1.attention.self.key._packed_params.dtype', 'segformer.encoder.block.3.0.attention.self.key._packed_params._packed_params', 'segformer.encoder.block.0.0.attention.self.key._packed_params.dtype', 'decode_head.batch_norm.running_var', 'segformer.encoder.block.0.0.mlp.dense1._packed_params.dtype', 'segformer.encoder.block.1.0.attention.output.dense.zero_point', 'decode_head.linear_c.0.proj.scale', 'segformer.encoder.block.2.1.attention.self.value.scale', 'segformer.encoder.block.2.0.attention.self.key.scale', 'segformer.encoder.block.2.0.mlp.dense2.zero_point', 'segformer.encoder.block.2.1.attention.self.query.scale', 'segformer.encoder.block.1.0.mlp.dense2._packed_params.dtype', 'segformer.encoder.block.0.1.attention.self.value._packed_params._packed_params', 'segformer.encoder.block.3.1.attention.self.value._packed_params._packed_params', 'segformer.encoder.block.1.0.attention.self.query.zero_point', 'segformer.encoder.block.2.0.attention.self.key._packed_params._packed_params', 'segformer.encoder.block.1.1.attention.self.key._packed_params._packed_params', 'segformer.encoder.block.1.0.attention.self.value._packed_params._packed_params', 'segformer.encoder.block.3.0.attention.self.query._packed_params._packed_params', 'segformer.encoder.block.0.1.attention.self.key.scale', 'segformer.encoder.block.3.0.attention.self.key.scale', 'segformer.encoder.block.0.1.mlp.dense2.scale', 'decode_head.linear_c.1.proj._packed_params.dtype', 'segformer.encoder.block.2.0.mlp.dense1.zero_point', 'segformer.encoder.block.3.0.mlp.dense2._packed_params.dtype', 'segformer.encoder.block.1.1.mlp.dense1.scale', 'segformer.encoder.block.1.0.mlp.dense2.scale', 'segformer.encoder.block.1.1.attention.self.value._packed_params.dtype', 'segformer.encoder.block.3.1.attention.self.value._packed_params.dtype', 'segformer.encoder.block.3.1.attention.output.dense.scale', 'segformer.encoder.block.3.1.attention.self.query.scale', 'decode_head.batch_norm.bias', 'decode_head.linear_c.2.proj._packed_params.dtype', 'segformer.encoder.block.3.0.attention.output.dense._packed_params.dtype', 'segformer.encoder.block.2.0.attention.output.dense.scale', 'segformer.encoder.block.0.0.mlp.dense2.zero_point', 'segformer.encoder.block.2.1.mlp.dense1.zero_point', 'segformer.encoder.block.2.0.mlp.dense1._packed_params.dtype', 'segformer.encoder.block.3.0.attention.output.dense._packed_params._packed_params', 'segformer.encoder.block.1.0.mlp.dense1.zero_point', 'segformer.encoder.block.1.0.mlp.dense1._packed_params.dtype', 'segformer.encoder.block.3.1.attention.self.query._packed_params._packed_params', 'segformer.encoder.block.0.0.attention.self.value.scale', 'segformer.encoder.block.3.1.mlp.dense1._packed_params.dtype', 'segformer.encoder.block.1.0.attention.self.key._packed_params._packed_params', 'segformer.encoder.block.2.0.attention.self.query.scale', 'segformer.encoder.block.1.1.mlp.dense1._packed_params.dtype', 'segformer.encoder.block.1.1.mlp.dense1.zero_point', 'segformer.encoder.block.0.0.mlp.dense2.scale', 'segformer.encoder.block.0.0.attention.output.dense.zero_point', 'segformer.encoder.block.0.1.attention.self.query._packed_params.dtype', 'segformer.encoder.block.3.0.mlp.dense1._packed_params.dtype', 'segformer.encoder.block.0.0.attention.self.value._packed_params._packed_params', 'segformer.encoder.block.2.1.attention.self.key.zero_point', 'decode_head.linear_c.1.proj.zero_point', 'decode_head.linear_c.1.proj.scale', 'segformer.encoder.block.2.1.mlp.dense1.scale', 'segformer.encoder.block.3.0.attention.self.query.zero_point', 'segformer.encoder.block.2.0.attention.self.query.zero_point', 'decode_head.linear_c.0.proj.zero_point', 'segformer.encoder.block.3.1.attention.self.query._packed_params.dtype', 'segformer.encoder.block.1.1.attention.self.query.zero_point', 'segformer.encoder.block.1.1.attention.self.key.zero_point', 'segformer.encoder.block.1.0.attention.self.key.zero_point', 'segformer.encoder.block.0.1.attention.self.key._packed_params.dtype', 'segformer.encoder.block.3.0.attention.output.dense.scale', 'segformer.encoder.block.0.1.mlp.dense2._packed_params._packed_params', 'segformer.encoder.block.1.1.attention.output.dense.zero_point', 'segformer.encoder.block.0.0.mlp.dense1.scale', 'segformer.encoder.block.2.1.mlp.dense2.scale', 'segformer.encoder.block.3.0.attention.self.value.scale', 'segformer.encoder.block.1.0.mlp.dense1.scale', 'decode_head.linear_c.2.proj.scale', 'segformer.encoder.block.2.1.attention.self.value._packed_params.dtype', 'segformer.encoder.block.3.0.attention.self.value._packed_params.dtype', 'segformer.encoder.block.2.0.attention.self.query._packed_params.dtype', 'segformer.encoder.block.3.0.mlp.dense2.scale', 'segformer.encoder.block.3.1.attention.self.key.scale', 'segformer.encoder.block.0.0.attention.self.query.zero_point', 'segformer.encoder.block.3.0.attention.self.query._packed_params.dtype', 'segformer.encoder.block.2.0.mlp.dense2._packed_params._packed_params', 'segformer.encoder.block.3.0.mlp.dense2._packed_params._packed_params', 'segformer.encoder.block.2.0.mlp.dense1._packed_params._packed_params', 'segformer.encoder.block.3.1.attention.output.dense._packed_params.dtype', 'segformer.encoder.block.1.0.attention.self.key.scale', 'segformer.encoder.block.3.1.mlp.dense1.zero_point', 'segformer.encoder.block.3.1.mlp.dense1._packed_params._packed_params', 'segformer.encoder.block.3.0.mlp.dense2.zero_point', 'segformer.encoder.block.2.1.attention.output.dense._packed_params.dtype', 'segformer.encoder.block.0.1.attention.self.query.zero_point', 'segformer.encoder.block.3.0.mlp.dense1.scale', 'segformer.encoder.block.3.0.attention.self.value._packed_params._packed_params', 'segformer.encoder.block.3.0.attention.self.value.zero_point', 'segformer.encoder.block.2.0.attention.self.key._packed_params.dtype', 'segformer.encoder.block.0.1.mlp.dense1.zero_point', 'segformer.encoder.block.3.1.mlp.dense2._packed_params._packed_params', 'segformer.encoder.block.3.1.mlp.dense2.zero_point', 'decode_head.linear_c.2.proj.zero_point', 'segformer.encoder.block.1.1.attention.self.query._packed_params.dtype', 'segformer.encoder.block.0.1.attention.output.dense._packed_params.dtype', 'segformer.encoder.block.0.1.attention.output.dense.zero_point', 'segformer.encoder.block.1.1.attention.self.value.scale', 'decode_head.linear_c.0.proj._packed_params.dtype', 'segformer.encoder.block.0.1.attention.self.key.zero_point', 'segformer.encoder.block.0.1.attention.output.dense._packed_params._packed_params', 'segformer.encoder.block.0.1.attention.self.query._packed_params._packed_params', 'segformer.encoder.block.1.0.attention.output.dense._packed_params.dtype', 'segformer.encoder.block.2.1.attention.self.query._packed_params.dtype', 'segformer.encoder.block.0.0.mlp.dense2._packed_params.dtype', 'segformer.encoder.block.2.1.attention.self.value._packed_params._packed_params', 'segformer.encoder.block.0.1.attention.self.value.zero_point', 'segformer.encoder.block.2.0.mlp.dense2._packed_params.dtype', 'segformer.encoder.block.0.1.mlp.dense2.zero_point', 'decode_head.batch_norm.num_batches_tracked', 'segformer.encoder.block.0.1.attention.self.query.scale', 'segformer.encoder.block.1.1.mlp.dense2.zero_point', 'segformer.encoder.block.1.1.attention.self.key._packed_params.dtype', 'segformer.encoder.block.3.1.attention.self.key._packed_params._packed_params', 'segformer.encoder.block.0.1.mlp.dense2._packed_params.dtype', 'decode_head.linear_c.3.proj.scale', 'segformer.encoder.block.2.1.attention.output.dense.zero_point', 'segformer.encoder.block.0.1.mlp.dense1._packed_params.dtype', 'segformer.encoder.block.0.0.attention.self.value._packed_params.dtype', 'segformer.encoder.block.1.0.attention.self.value.scale', 'segformer.encoder.block.1.1.attention.output.dense._packed_params._packed_params', 'segformer.encoder.block.0.1.attention.self.key._packed_params._packed_params', 'decode_head.batch_norm.running_mean', 'segformer.encoder.block.2.1.mlp.dense1._packed_params.dtype', 'segformer.encoder.block.1.0.attention.self.key._packed_params.dtype', 'segformer.encoder.block.1.1.attention.self.query._packed_params._packed_params', 'segformer.encoder.block.1.0.attention.output.dense._packed_params._packed_params', 'segformer.encoder.block.1.1.attention.output.dense.scale', 'segformer.encoder.block.2.1.attention.self.query._packed_params._packed_params', 'segformer.encoder.block.1.1.attention.output.dense._packed_params.dtype', 'segformer.encoder.block.2.0.mlp.dense2.scale', 'segformer.encoder.block.2.1.attention.output.dense._packed_params._packed_params', 'segformer.encoder.block.2.1.attention.self.key.scale', 'segformer.encoder.block.3.1.attention.self.key._packed_params.dtype', 'segformer.encoder.block.3.0.attention.output.dense.zero_point', 'decode_head.linear_c.3.proj.zero_point', 'segformer.encoder.block.1.1.mlp.dense2._packed_params._packed_params', 'segformer.encoder.block.0.1.attention.self.value.scale', 'segformer.encoder.block.0.1.attention.self.value._packed_params.dtype', 'segformer.encoder.block.1.0.attention.self.query._packed_params.dtype', 'segformer.encoder.block.1.1.attention.self.key.scale', 'segformer.encoder.block.1.0.attention.output.dense.scale', 'segformer.encoder.block.1.0.mlp.dense2.zero_point', 'segformer.encoder.block.3.1.attention.self.key.zero_point', 'segformer.encoder.block.0.1.mlp.dense1._packed_params._packed_params', 'segformer.encoder.block.2.0.attention.output.dense._packed_params._packed_params', 'segformer.encoder.block.1.1.mlp.dense2._packed_params.dtype', 'segformer.encoder.block.3.1.mlp.dense1.scale', 'segformer.encoder.block.1.0.attention.self.query._packed_params._packed_params', 'segformer.encoder.block.0.0.attention.output.dense._packed_params._packed_params', 'segformer.encoder.block.0.0.attention.self.key.zero_point', 'segformer.encoder.block.0.0.mlp.dense2._packed_params._packed_params', 'segformer.encoder.block.3.1.mlp.dense2._packed_params.dtype', 'segformer.encoder.block.2.0.attention.self.value.zero_point', 'segformer.encoder.block.3.1.attention.self.query.zero_point', 'segformer.encoder.block.0.0.attention.self.key._packed_params._packed_params', 'segformer.encoder.block.3.0.attention.self.key._packed_params.dtype', 'segformer.encoder.block.1.0.attention.self.query.scale', 'segformer.encoder.block.2.1.attention.output.dense.scale', 'segformer.encoder.block.1.0.attention.self.value.zero_point', 'decode_head.linear_c.1.proj._packed_params._packed_params', 'segformer.encoder.block.3.1.attention.output.dense.zero_point', 'segformer.encoder.block.1.1.attention.self.query.scale', 'decode_head.linear_fuse.weight', 'decode_head.classifier.bias', 'segformer.encoder.block.2.1.attention.self.query.zero_point', 'segformer.encoder.block.1.0.attention.self.value._packed_params.dtype', 'segformer.encoder.block.1.1.mlp.dense1._packed_params._packed_params', 'segformer.encoder.block.0.1.attention.output.dense.scale', 'segformer.encoder.block.2.1.attention.self.value.zero_point', 'decode_head.linear_c.0.proj._packed_params._packed_params', 'segformer.encoder.block.0.0.attention.self.key.scale', 'segformer.encoder.block.0.0.mlp.dense1._packed_params._packed_params', 'segformer.encoder.block.1.1.attention.self.value.zero_point', 'segformer.encoder.block.0.1.mlp.dense1.scale', 'segformer.encoder.block.3.0.attention.self.key.zero_point', 'decode_head.linear_c.3.proj._packed_params.dtype', 'segformer.encoder.block.2.0.attention.output.dense._packed_params.dtype', 'segformer.encoder.block.3.1.mlp.dense2.scale', 'decode_head.classifier.weight', 'segformer.encoder.block.2.0.attention.self.value._packed_params._packed_params', 'segformer.encoder.block.3.1.attention.self.value.scale', 'segformer.encoder.block.3.0.mlp.dense1._packed_params._packed_params', 'segformer.encoder.block.2.0.attention.self.query._packed_params._packed_params', 'segformer.encoder.block.2.1.attention.self.key._packed_params._packed_params', 'segformer.encoder.block.2.0.attention.self.value._packed_params.dtype', 'segformer.encoder.block.2.1.mlp.dense2._packed_params.dtype', 'decode_head.linear_c.2.proj._packed_params._packed_params', 'segformer.encoder.block.0.0.attention.output.dense.scale', 'segformer.encoder.block.2.0.attention.self.value.scale', 'segformer.encoder.block.3.0.mlp.dense1.zero_point', 'segformer.encoder.block.1.1.attention.self.value._packed_params._packed_params', 'decode_head.batch_norm.weight', 'segformer.encoder.block.3.1.attention.output.dense._packed_params._packed_params', 'segformer.encoder.block.2.0.mlp.dense1.scale', 'segformer.encoder.block.3.0.attention.self.query.scale', 'segformer.encoder.block.0.0.attention.self.query.scale', 'segformer.encoder.block.2.0.attention.output.dense.zero_point', 'segformer.encoder.block.1.1.mlp.dense2.scale', 'segformer.encoder.block.2.1.mlp.dense1._packed_params._packed_params', 'segformer.encoder.block.0.0.attention.output.dense._packed_params.dtype']
- This IS expected if you are initializing SegformerModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing SegformerModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
Some weights of SegformerModel were not initialized from the model checkpoint at segformed_lanes_quantized.pt and are newly initialized: ['segformer.encoder.block.1.1.mlp.dense1.weight', 'segformer.encoder.block.2.1.mlp.dense2.bias', 'segformer.encoder.block.3.0.attention.self.key.weight', 'segformer.encoder.block.1.1.mlp.dense1.bias', 'segformer.encoder.block.1.0.attention.self.key.bias', 'segformer.encoder.block.3.1.attention.output.dense.bias', 'segformer.encoder.block.0.1.mlp.dense1.bias', 'segformer.encoder.block.3.1.attention.self.value.weight', 'segformer.encoder.block.3.0.attention.self.query.weight', 'segformer.encoder.block.0.0.attention.self.key.bias', 'segformer.encoder.block.2.0.attention.self.query.weight', 'segformer.encoder.block.3.1.mlp.dense2.weight', 'segformer.encoder.block.1.1.attention.self.query.bias', 'segformer.encoder.block.2.0.attention.self.value.weight', 'segformer.encoder.block.3.1.attention.self.key.bias', 'segformer.encoder.block.2.1.mlp.dense2.weight', 'segformer.encoder.block.2.1.attention.self.value.weight', 'segformer.encoder.block.3.1.attention.self.query.weight', 'segformer.encoder.block.2.0.attention.self.query.bias', 'segformer.encoder.block.1.1.attention.output.dense.weight', 'segformer.encoder.block.3.0.mlp.dense1.weight', 'segformer.encoder.block.3.0.attention.self.value.weight', 'segformer.encoder.block.3.0.mlp.dense2.weight', 'segformer.encoder.block.0.0.mlp.dense2.bias', 'segformer.encoder.block.0.1.mlp.dense2.bias', 'segformer.encoder.block.2.0.attention.self.key.bias', 'segformer.encoder.block.2.0.mlp.dense1.weight', 'segformer.encoder.block.1.0.attention.self.query.bias', 'segformer.encoder.block.2.1.attention.self.key.weight', 'segformer.encoder.block.0.0.attention.self.key.weight', 'segformer.encoder.block.0.0.mlp.dense1.weight', 'segformer.encoder.block.0.1.attention.output.dense.bias', 'segformer.encoder.block.0.1.attention.self.query.weight', 'segformer.encoder.block.3.1.mlp.dense2.bias', 'segformer.encoder.block.1.0.mlp.dense1.bias', 'segformer.encoder.block.2.0.mlp.dense1.bias', 'segformer.encoder.block.0.0.mlp.dense1.bias', 'segformer.encoder.block.2.0.attention.output.dense.bias', 'segformer.encoder.block.1.1.mlp.dense2.weight', 'segformer.encoder.block.2.0.attention.output.dense.weight', 'segformer.encoder.block.3.1.attention.output.dense.weight', 'segformer.encoder.block.1.1.attention.self.value.weight', 'segformer.encoder.block.2.0.attention.self.value.bias', 'segformer.encoder.block.1.0.mlp.dense1.weight', 'segformer.encoder.block.0.0.mlp.dense2.weight', 'segformer.encoder.block.0.1.attention.self.query.bias', 'segformer.encoder.block.2.1.mlp.dense1.bias', 'segformer.encoder.block.0.0.attention.self.query.bias', 'segformer.encoder.block.3.0.attention.output.dense.weight', 'segformer.encoder.block.1.0.mlp.dense2.weight', 'segformer.encoder.block.1.0.attention.output.dense.bias', 'segformer.encoder.block.0.1.attention.self.key.bias', 'segformer.encoder.block.3.0.attention.self.key.bias', 'segformer.encoder.block.3.1.attention.self.query.bias', 'segformer.encoder.block.0.0.attention.self.value.weight', 'segformer.encoder.block.3.1.mlp.dense1.weight', 'segformer.encoder.block.3.0.mlp.dense1.bias', 'segformer.encoder.block.2.1.attention.self.key.bias', 'segformer.encoder.block.2.1.attention.self.query.bias', 'segformer.encoder.block.1.1.attention.self.key.bias', 'segformer.encoder.block.3.0.attention.self.query.bias', 'segformer.encoder.block.2.1.attention.self.query.weight', 'segformer.encoder.block.2.1.attention.output.dense.weight', 'segformer.encoder.block.2.0.mlp.dense2.weight', 'segformer.encoder.block.1.1.mlp.dense2.bias', 'segformer.encoder.block.1.0.attention.self.query.weight', 'segformer.encoder.block.3.0.attention.self.value.bias', 'segformer.encoder.block.0.0.attention.output.dense.weight', 'segformer.encoder.block.2.1.attention.self.value.bias', 'segformer.encoder.block.0.1.mlp.dense1.weight', 'segformer.encoder.block.1.0.attention.self.value.bias', 'segformer.encoder.block.2.1.attention.output.dense.bias', 'segformer.encoder.block.0.1.attention.output.dense.weight', 'segformer.encoder.block.1.0.attention.output.dense.weight', 'segformer.encoder.block.1.0.mlp.dense2.bias', 'segformer.encoder.block.2.0.attention.self.key.weight', 'segformer.encoder.block.0.1.attention.self.key.weight', 'segformer.encoder.block.3.0.mlp.dense2.bias', 'segformer.encoder.block.1.0.attention.self.value.weight', 'segformer.encoder.block.0.0.attention.self.value.bias', 'segformer.encoder.block.0.1.attention.self.value.bias', 'segformer.encoder.block.2.0.mlp.dense2.bias', 'segformer.encoder.block.1.1.attention.output.dense.bias', 'segformer.encoder.block.3.1.attention.self.value.bias', 'segformer.encoder.block.1.0.attention.self.key.weight', 'segformer.encoder.block.0.1.mlp.dense2.weight', 'segformer.encoder.block.3.1.attention.self.key.weight', 'segformer.encoder.block.3.0.attention.output.dense.bias', 'segformer.encoder.block.1.1.attention.self.value.bias', 'segformer.encoder.block.1.1.attention.self.key.weight', 'segformer.encoder.block.0.1.attention.self.value.weight', 'segformer.encoder.block.3.1.mlp.dense1.bias', 'segformer.encoder.block.0.0.attention.self.query.weight', 'segformer.encoder.block.0.0.attention.output.dense.bias', 'segformer.encoder.block.1.1.attention.self.query.weight', 'segformer.encoder.block.2.1.mlp.dense1.weight']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
```



Comments:
Quantized (only background predicted) : 
![quantized](https://user-images.githubusercontent.com/23454613/158686845-804e594a-b874-43d1-a52c-d8b3f87f4d11.png)



Initial model:
![normal](https://user-images.githubusercontent.com/23454613/158686978-01e29eae-8efd-4b71-897a-6802c7a773e7.png)



Hi,

Cool to see you're interested in quantizing one of the models I've implemented!

I feel like this could be interesting to support model quantization of vision models, cc @lewtun. We recently added ONNX support for the Vision Transformer (ViT), but things like dynamic post-training quantization are very interesting as well.

To be honest I'm not an expert on that topic, hopefully @lewtun can bring some knowledge here.

Thank you very much for the fast response (also for the awesome contributions), looking forward to finding out a solution to this :D 

Hey @TimbusCalin thanks for opening this issue. In general, you cannot load quantized models using the `from_pretrained()` method because quantization changes the model's `state_dict`. You can find a work around in this [forum post](https://discuss.huggingface.co/t/pegasus-model-weights-compression-pruning/6381/9?u=lewtun), but it does require one to load a dummy model.

We're actually working on making is easy to apply various quantization techniques in our `optimum` [library](https://github.com/huggingface/optimum), so feel free to check that out in case it's relevant to your use case

@NielsRogge @lewtun  thank you guys for the prompt responses and suggestions given!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.