ekurtic commited on
Commit
c5a81ae
1 Parent(s): ae8113d

Initial commit

Browse files
README.md ADDED
@@ -0,0 +1,72 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # BERT-tiny model finetuned with M-FAC
2
+
3
+ This model is finetuned on MNLI dataset with state-of-the-art second-order optimizer M-FAC.
4
+ Check NeurIPS 2021 paper for more details on M-FAC: [https://arxiv.org/pdf/2107.03356.pdf](https://arxiv.org/pdf/2107.03356.pdf).
5
+
6
+ ## Finetuning setup
7
+
8
+ For fair comparison against default Adam baseline, we finetune the model in the same framework as described here [https://github.com/huggingface/transformers/tree/master/examples/pytorch/text-classification](https://github.com/huggingface/transformers/tree/master/examples/pytorch/text-classification) and just swap Adam optimizer with M-FAC.
9
+ Hyperparameters used by M-FAC optimizer:
10
+
11
+ ```bash
12
+ learning rate = 1e-4
13
+ number of gradients = 1024
14
+ dampening = 1e-6
15
+ ```
16
+
17
+ ## Results
18
+
19
+ We share the best model out of 5 runs with the following score on MNLI validation set:
20
+
21
+ ```bash
22
+ matched_accuracy = 69.55
23
+ mismatched_accuracy = 70.58
24
+ ```
25
+
26
+ Mean and standard deviation for 5 runs on MNLI validation set:
27
+ | | Matched Accuracy | Mismatched Accuracy |
28
+ |:----:|:-----------:|:----------:|
29
+ | Adam | 65.36 ± 0.13 | 66.78 ± 0.15 |
30
+ | M-FAC | 68.28 ± 3.29 | 68.98 ± 3.05 |
31
+
32
+ Results can be reproduced by adding M-FAC optimizer code in [https://github.com/huggingface/transformers/blob/master/examples/pytorch/text-classification/run_glue.py](https://github.com/huggingface/transformers/blob/master/examples/pytorch/text-classification/run_glue.py) and running the following bash script:
33
+
34
+ ```bash
35
+ CUDA_VISIBLE_DEVICES=0 python run_glue.py \
36
+ --seed 42 \
37
+ --model_name_or_path prajjwal1/bert-tiny \
38
+ --task_name mnli \
39
+ --do_train \
40
+ --do_eval \
41
+ --max_seq_length 128 \
42
+ --per_device_train_batch_size 32 \
43
+ --learning_rate 1e-4 \
44
+ --num_train_epochs 5 \
45
+ --output_dir out_dir/ \
46
+ --optim MFAC \
47
+ --optim_args '{"lr": 1e-4, "num_grads": 1024, "damp": 1e-6}'
48
+ ```
49
+
50
+ We believe these results could be improved with modest tuning of hyperparameters: `per_device_train_batch_size`, `learning_rate`, `num_train_epochs`, `num_grads` and `damp`. For the sake of fair comparison and a robust default setup we use the same hyperparameters across all models (`bert-tiny`, `bert-mini`) and all datasets (SQuAD version 2 and GLUE).
51
+
52
+ ## BibTeX entry and citation info
53
+
54
+ ```bibtex
55
+ @article{DBLP:journals/corr/abs-2107-03356,
56
+ author = {Elias Frantar and
57
+ Eldar Kurtic and
58
+ Dan Alistarh},
59
+ title = {Efficient Matrix-Free Approximations of Second-Order Information,
60
+ with Applications to Pruning and Optimization},
61
+ journal = {CoRR},
62
+ volume = {abs/2107.03356},
63
+ year = {2021},
64
+ url = {https://arxiv.org/abs/2107.03356},
65
+ eprinttype = {arXiv},
66
+ eprint = {2107.03356},
67
+ timestamp = {Tue, 20 Jul 2021 15:08:33 +0200},
68
+ biburl = {https://dblp.org/rec/journals/corr/abs-2107-03356.bib},
69
+ bibsource = {dblp computer science bibliography, https://dblp.org}
70
+ }
71
+
72
+ ```
config.json ADDED
@@ -0,0 +1,38 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "_name_or_path": "prajjwal1/bert-tiny",
3
+ "architectures": [
4
+ "BertForSequenceClassification"
5
+ ],
6
+ "attention_probs_dropout_prob": 0.1,
7
+ "classifier_dropout": null,
8
+ "finetuning_task": "mnli",
9
+ "gradient_checkpointing": false,
10
+ "hidden_act": "gelu",
11
+ "hidden_dropout_prob": 0.1,
12
+ "hidden_size": 128,
13
+ "id2label": {
14
+ "0": "LABEL_0",
15
+ "1": "LABEL_1",
16
+ "2": "LABEL_2"
17
+ },
18
+ "initializer_range": 0.02,
19
+ "intermediate_size": 512,
20
+ "label2id": {
21
+ "LABEL_0": 0,
22
+ "LABEL_1": 1,
23
+ "LABEL_2": 2
24
+ },
25
+ "layer_norm_eps": 1e-12,
26
+ "max_position_embeddings": 512,
27
+ "model_type": "bert",
28
+ "num_attention_heads": 2,
29
+ "num_hidden_layers": 2,
30
+ "pad_token_id": 0,
31
+ "position_embedding_type": "absolute",
32
+ "problem_type": "single_label_classification",
33
+ "torch_dtype": "float32",
34
+ "transformers_version": "4.10.0.dev0",
35
+ "type_vocab_size": 2,
36
+ "use_cache": true,
37
+ "vocab_size": 30522
38
+ }
pytorch_model.bin ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:4efd952b1d6113e552e8da4dd5ae678a023578c604a43710148d077999965689
3
+ size 17565095
special_tokens_map.json ADDED
@@ -0,0 +1 @@
 
 
1
+ {"unk_token": "[UNK]", "sep_token": "[SEP]", "pad_token": "[PAD]", "cls_token": "[CLS]", "mask_token": "[MASK]"}
tokenizer_config.json ADDED
@@ -0,0 +1 @@
 
 
1
+ {"do_lower_case": true, "unk_token": "[UNK]", "sep_token": "[SEP]", "pad_token": "[PAD]", "cls_token": "[CLS]", "mask_token": "[MASK]", "tokenize_chinese_chars": true, "strip_accents": null, "special_tokens_map_file": null, "name_or_path": "prajjwal1/bert-tiny", "do_basic_tokenize": true, "never_split": null, "tokenizer_class": "BertTokenizer"}
vocab.txt ADDED
The diff for this file is too large to render. See raw diff