Create README.md

README.md

@@ -0,0 +1,108 @@
+---
+language:
+- en
+tags:
+- generated_from_trainer
+datasets:
+- glue
+metrics:
+- accuracy
+model-index:
+- name: yujiepan/bert-base-uncased-sst2-int8-unstructured80
+  results:
+  - task:
+      name: Text Classification
+      type: text-classification
+    dataset:
+      name: GLUE SST2
+      type: glue
+      config: sst2
+      split: validation
+      args: sst2
+    metrics:
+    - name: Accuracy
+      type: accuracy
+      value: 0.91284
+pipeline_tag: text-classification
+---
+
+<!-- This model card has been generated automatically according to the information the Trainer had access to. You
+should probably proofread and complete it, then remove this comment. -->
+
+# Joint magnitude pruning, quantization and distillation on BERT-base/SST-2
+
+This model conducts unstructured magnitude pruning, quantization and distillation at the same time on BERT-base when finetuning on the GLUE SST2 dataset.
+It achieves the following results on the evaluation set:
+- Torch accuracy: 0.9128
+- OpenVINO IR accuracy: 0.9128
+- Sparsity in transformer block linear layers: 0.80
+
+## Setup
+
+```
+conda install pytorch torchvision torchaudio pytorch-cuda=11.6 -c pytorch -c nvidia
+pip install optimum[openvino,nncf]==1.7.0
+# TODO
+pip install wandb # optional
+```
+
+## NNCF config
+
+See `nncf_config.json` in this repo.
+
+
+## Run
+
+We use one card for training.
+
+```
+NNCFCFG=/path/to/nncf/config
+python run_glue.py \
+--lr_scheduler_type cosine_with_restarts \
+--cosine_cycle_ratios 11,6 \
+--cosine_cycle_decays 1,1 \
+--save_best_model_after_epoch -1 \
+--save_best_model_after_sparsity 0.7999 \
+--model_name_or_path textattack/bert-base-uncased-SST-2 \
+--teacher_model_or_path yoshitomo-matsubara/bert-large-uncased-sst2 \
+--distillation_temperature 2 \
+--task_name sst2 \
+--nncf_compression_config $NNCFCFG \
+--distillation_weight 0.95 \
+--output_dir /tmp/bert-base-uncased-sst2-int8-unstructured80-17epoch \
+--run_name bert-base-uncased-sst2-int8-unstructured80-17epoch \
+--overwrite_output_dir \
+--do_train \
+--do_eval \
+--max_seq_length 128 \
+--per_device_train_batch_size 32 \
+--per_device_eval_batch_size 32 \
+--learning_rate 5e-05 \
+--optim adamw_torch \
+--num_train_epochs 17 \
+--logging_steps 1 \
+--evaluation_strategy steps \
+--eval_steps 250 \
+--save_strategy steps \
+--save_steps 250 \
+--save_total_limit 1 \
+--fp16 \
+--seed 1
+```
+
+The best model checkpoint is stored in the `best_model` folder. Here we only upload that checkpoint folder together with some config files.
+
+
+## inference
+
+https://gist.github.com/yujiepan-work/c38dc4e56c7a9d803c42988f7b7d260a
+
+
+### Framework versions
+
+- Transformers 4.26.0
+- Pytorch 1.13.1+cu116
+- Datasets 2.8.0
+- Tokenizers 0.13.2
+
+For a full description of the environment, please refer to `pip-requirements.txt` and `conda-requirements.txt`.