bconsolvo commited on
Commit
c023e8a
1 Parent(s): ac67851

Update model card according to Hugging Face model card standards

Browse files
Files changed (1) hide show
  1. README.md +88 -39
README.md CHANGED
@@ -6,58 +6,107 @@ tags:
6
  - int8
7
  - Intel® Neural Compressor
8
  - PostTrainingStatic
9
- datasets:
10
  - sst2
11
- metrics:
12
- - accuracy
 
 
 
 
 
 
 
 
 
 
 
 
 
13
  ---
14
 
15
- # INT8 DistilBERT base uncased finetuned SST-2
16
 
17
- ## Post-training static quantization
 
18
 
19
- ### PyTorch
 
 
 
 
 
 
 
 
20
 
21
- This is an INT8 PyTorch model quantized with [huggingface/optimum-intel](https://github.com/huggingface/optimum-intel) through the usage of [Intel® Neural Compressor](https://github.com/intel/neural-compressor).
22
-
23
- The original fp32 model comes from the fine-tuned model [distilbert-base-uncased-finetuned-sst-2-english](https://huggingface.co/distilbert-base-uncased-finetuned-sst-2-english).
24
-
25
- The calibration dataloader is the train dataloader. The default calibration sampling size 100 isn't divisible exactly by batch size 8, so
26
- the real sampling size is 104.
27
-
28
- #### Test result
29
-
30
- | |INT8|FP32|
31
- |---|:---:|:---:|
32
- | **Accuracy (eval-accuracy)** |0.9037|0.9106|
33
- | **Model size (MB)** |65|255|
34
-
35
- #### Load with optimum:
36
 
 
37
  ```python
38
  from optimum.intel.neural_compressor.quantization import IncQuantizedModelForSequenceClassification
39
  int8_model = IncQuantizedModelForSequenceClassification.from_pretrained(
40
- 'Intel/distilbert-base-uncased-finetuned-sst-2-english-int8-static',
41
- )
42
  ```
43
 
44
- ### ONNX
45
-
46
- This is an INT8 ONNX model quantized with [Intel® Neural Compressor](https://github.com/intel/neural-compressor).
47
-
48
- The original fp32 model comes from the fine-tuned model [distilbert-base-uncased-finetuned-sst-2-english](https://huggingface.co/distilbert-base-uncased-finetuned-sst-2-english).
49
-
50
-
51
- #### Test result
52
-
53
- | |INT8|FP32|
54
- |---|:---:|:---:|
55
- | **Accuracy (eval-f1)** |0.9060|0.9106|
56
- | **Model size (MB)** |80|256|
57
-
58
  #### Load ONNX model:
59
-
60
  ```python
61
  from optimum.onnxruntime import ORTModelForSequenceClassification
62
- model = ORTModelForSequenceClassification.from_pretrained('Intel/distilbert-base-uncased-finetuned-sst-2-english-int8-static')
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
63
  ```
 
6
  - int8
7
  - Intel® Neural Compressor
8
  - PostTrainingStatic
9
+ datasets:
10
  - sst2
11
+ model-index:
12
+ - name: distilbert-base-uncased-finetuned-sst-2-english-int8-static
13
+ results:
14
+ - task:
15
+ type: sentiment-classification
16
+ name: Sentiment Classification
17
+ dataset:
18
+ type: sst2
19
+ name: Stanford Sentiment Treebank
20
+ metrics:
21
+ - type: accuracy
22
+ value: 90.37
23
+ name: accuracy
24
+ config: accuracy
25
+ verified: false
26
  ---
27
 
28
+ ## Model Details: INT8 DistilBERT base uncased finetuned SST-2
29
 
30
+ This model is an INT8 model fine-tuned and quantized from the original FP32 model [distilbert-base-uncased-finetuned-sst-2-english](https://huggingface.co/distilbert-base-uncased-finetuned-sst-2-english).
31
+ The same model is provided in two different formats: PyTorch and ONNX.
32
 
33
+ | Model Detail | Description |
34
+ | ----------- | ----------- |
35
+ | Model Authors - Company | Intel |
36
+ | Date | March 29, 2022 for PyTorch model & February 3, 2023 for ONNX model |
37
+ | Version | 1 |
38
+ | Type | NLP DistilBERT (INT8) - Sentiment Classification (+/-) |
39
+ | Paper or Other Resources | [https://github.com/huggingface/optimum-intel](https://github.com/huggingface/optimum-intel) |
40
+ | License | Apache 2.0 |
41
+ | Questions or Comments | [Community Tab](https://huggingface.co/Intel/distilbert-base-uncased-finetuned-sst-2-english-int8-static/discussions) and [Intel Developers Discord](https://discord.gg/rv2Gp55UJQ) |
42
 
43
+ | Intended Use | Description |
44
+ | ----------- | ----------- |
45
+ | Primary intended uses | Inference for sentiment classification (classifying whether a statement is positive or negative) |
46
+ | Primary intended users | Anyone |
47
+ | Out-of-scope uses | This model is already fine-tuned and quantized to INT8. It is not suitable for further fine-tuning in this form. To fine-tune your own model, you can start with [distilbert-base-uncased-finetuned-sst-2-english](https://huggingface.co/distilbert-base-uncased-finetuned-sst-2-english). |
 
 
 
 
 
 
 
 
 
 
48
 
49
+ #### Load PyTorch model with Optimum
50
  ```python
51
  from optimum.intel.neural_compressor.quantization import IncQuantizedModelForSequenceClassification
52
  int8_model = IncQuantizedModelForSequenceClassification.from_pretrained(
53
+ 'Intel/distilbert-base-uncased-finetuned-sst-2-english-int8-static')
 
54
  ```
55
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
56
  #### Load ONNX model:
 
57
  ```python
58
  from optimum.onnxruntime import ORTModelForSequenceClassification
59
+ model = ORTModelForSequenceClassification.from_pretrained(
60
+ 'Intel/distilbert-base-uncased-finetuned-sst-2-english-int8-static')
61
+ ```
62
+
63
+ | Factors | Description |
64
+ | ----------- | ----------- |
65
+ | Groups | Movie reviewers from the internet |
66
+ | Instrumentation | Text movie single-sentence reviews taken from 4 authors. More information can be found in the original paper by [Pang and Lee (2005)](https://arxiv.org/abs/cs/0506075) |
67
+ | Environment | - |
68
+ | Card Prompts | Model deployment on alternate hardware and software can change model performance |
69
+
70
+ | Metrics | Description |
71
+ | ----------- | ----------- |
72
+ | Model performance measures | Accuracy |
73
+ | Decision thresholds | - |
74
+ | Approaches to uncertainty and variability | - |
75
+
76
+ | | PyTorch INT8 | ONNX INT8 | FP32 |
77
+ |---|---|---|---|
78
+ | **Accuracy (eval-accuracy)** |0.9037|0.9060|0.9106|
79
+ | **Model Size (MB)** |65|80|255|
80
+
81
+ | Training and Evaluation Data | Description |
82
+ | ----------- | ----------- |
83
+ | Datasets | The dataset can be found here: [datasets/sst2](https://huggingface.co/datasets/sst2). There dataset has a total of 215,154 unique phrases, annotated by 3 human judges. |
84
+ | Motivation | Dataset was chosen to showcase the benefits of quantization on an NLP classification task with the [Optimum Intel](https://github.com/huggingface/optimum-intel) and [Intel® Neural Compressor](https://github.com/intel/neural-compressor) |
85
+ | Preprocessing | The calibration dataloader is the train dataloader. The default calibration sampling size 100 isn't divisible exactly by batch size 8, so the real sampling size is 104.|
86
+
87
+ | Quantitative Analyses | Description |
88
+ | ----------- | ----------- |
89
+ | Unitary results | The model was only evaluated on accuracy. There is no available comparison between evaluation factors. |
90
+ | Intersectional results | There is no available comparison between the intersection of evaluated factors. |
91
+
92
+ | Ethical Considerations | Description |
93
+ | ----------- | ----------- |
94
+ | Data | The data that make up the model are movie reviews from authors on the internet. |
95
+ | Human life | The model is not intended to inform decisions central to human life or flourishing. It is an aggregated set of movie reviews from the internet. |
96
+ | Mitigations | No additional risk mitigation strategies were considered during model development. |
97
+ | Risks and harms | The data are biased toward the particular reviewers' opinions and the judges (labelers) of the data. The extent of the risks involved by using the model were considered but remain unknown.|
98
+ | Use cases | - |
99
+
100
+ | Caveats and Recommendations |
101
+ | ----------- |
102
+ | There are no additional caveats or recommendations for this model. |
103
+
104
+ # BibTeX Entry and Citation Info
105
+ ```
106
+ @misc{distilbert-base-uncased-finetuned-sst-2-english-int8-static
107
+ author = {Xin He, Yu Wenz},
108
+ title = {distilbert-base-uncased-finetuned-sst-2-english-int8-static},
109
+ year = {2022},
110
+ url = {https://huggingface.co/Intel/distilbert-base-uncased-finetuned-sst-2-english-int8-static},
111
+ }
112
  ```