New version (distilbert-base to bert-base; attention matrices imitation)

README.md

@@ -1,14 +1,18 @@
 # TinyBERT_L-4_H-312_v2 English Sentence Encoder
 
-This is distilled from the `distilbert-base-nli-stsb-mean-tokens` pre-trained model from [Sentence-Transformers](https://sbert.net/).
+This is distilled from the `bert-base-nli-stsb-mean-tokens` pre-trained model from [Sentence-Transformers](https://sbert.net/).
 
 The embedding vector is obtained by mean/average pooling of the last layer's hidden states.
 
+Update 20210325: Added the attention matrices imitation objective as in the TinyBERT paper, and the distill target has been changed from `distilbert-base-nli-stsb-mean-tokens` to `bert-base-nli-stsb-mean-tokens` (they have almost the same STSb performance).
+
 ## Model Comparison
 
 We compute cosine similarity scores of the embeddings of the sentence pair to get the spearman correlation on the STS benchmark (bigger is better):
 
 |                                      | Dev   | Test  |
 | ------------------------------------ | ----- | ----- |
+| bert-base-nli-stsb-mean-tokens       | .8704 | .8505 |
 | distilbert-base-nli-stsb-mean-tokens | .8667 | .8516 |
 | TinyBERT_L-4_H-312_v2-distill-AllNLI | .8587 | .8283 |
+| TinyBERT_L-4_H (20210325)            | .8551 | .8341 |

pytorch_model.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:55cf95c30a8ab86ae85cb790b49781ff8398d664fd7be6f5815ce5aa0e0dd5f6
-size 57432312
+oid sha256:6cab2a4a2fb96d79e17a634bf921aeddd6fe297287ff6ba6464dee261f7e6226
+size 57432311