metadata
language: en
license: apache-2.0
tags: text-classfication, int8, PostTrainingStatic
datasets:
- sst2
INT8 DistilBERT base uncased finetuned SST-2 (Post-training static quantization)
This is an INT8 PyTorch model quantized by intel/nlp-toolkit using provider: Intel® Neural Compressor. The original fp32 model comes from the fine-tuned model distilbert-base-uncased-finetuned-sst-2-english
Test result below comes from Amazon Web Services c6i.xlarge (intel ice lake: 4 vCPUs, 8g Memory) instance.
| int8 | fp32 | |
|---|---|---|
| Throughput (samples/sec) | 47.554 | 23.046 |
| Accuracy(f1-score) | 0.9037 | 0.9106 |
| Model size (MB) | 66 | 255 |
Load with nlp-toolkit:
from nlp_toolkit import OptimizedModel
int8_model = OptimizedModel.from_pretrained(
'Intel/distilbert-base-uncased-finetuned-sst-2-english-int8-static',
)
Notes:
- The INT8 model has better performance than the FP32 model when the CPU is fully occupied. Otherwise, there will be the illusion that INT8 is inferior to FP32.