SlowGuess/NAACL_2021_OCR huggingface dataset:
Table 10: Results on Weibo.
Model	P	R	F1
BERT+CRF (teacher)	74.30	68.38	71.22
Vanilla BiLSTM+CRF	64.08	43.54	51.85
+ TE	65.67	47.13	54.87
+ TE +TOKEN-EM	60.79	47.85	53.55
+ TE +TOKEN-POS	67.02	45.69	54.34
+ TE + SEQ	66.90	49.87	57.14
+ TE + DA	72.73	66.99	69.74
+ TE + DA +TOKEN-EM	71.04	68.66	69.83
+ TE + DA +TOKEN-POS	72.80	67.22	69.90
+ TE + DA + SEQ (student)	74.29	69.14	71.62
Table 11: Results on OntoNotes 4.0.
Model	P	R	F1
BERT+CRF (teacher)	81.87	82.91	82.39
Vanilla BiLSTM+CRF	66.97	62.02	64.4
+ TE	72.62	67.15	69.78
+ TE +TOKEN-EM	72.13	68.85	70.46
+ TE +TOKEN-POS	72.18	70.22	71.19
+ TE + SEQ	75.34	69.55	72.33
+ TE + DA	76.06	74.80	75.43
+ TE + DA +TOKEN-EM	75.21	72.84	74.52
+ TE + DA +TOKEN-POS	75.99	73.80	74.88
+ TE + DA + SEQ (student)	77.94	74.23	76.05
Table 12: Results on CoNLL-2003.
Model	P	R	F1
BERT+CRF (teacher)	91.46	92.61	92.03
Vanilla BiLSTM+CRF	80.67	79.5	80.08
+ TE	88.43	88.27	88.35
+ TE +TOKEN-EM	87.70	89.23	88.46
+ TE +TOKEN-POS	88.38	88.95	88.66
+ TE + SEQ	89.29	89.79	89.69
+ TE + DA	90.20	91.18	90.69
+ TE + DA +TOKEN-EM	90.18	90.80	90.49
+ TE + DA +TOKEN-POS	90.70	91.27	90.98
+ TE + DA + SEQ (student)	90.85	91.50	91.17
Table 9: Comparison of F1 scores on four transformer layers as student model. The abbreviations herein is as same as Table 1, except BP means BERT pretrained parameters are used. Noting that, we simply choose the first four transformer layers from BERT model to initialize the student model, however, other reasonable strategies as in (Sun et al., 2019b; Jiao et al., 2019) will be studied in our future work.
Models	Datasets
English Datasets	Chinese Datasets
English	Chinese	CoNLL-2003	OntoNotes 5.0	MSRA	Weibo	OntoNotes 4.0
BERT+CRF (teacher)	Vanilla BERT+CRF	NA	NA	94.80	67.33	79.16
Glyce+BERT	NA	NA	95.54	67.60	80.62
BERT+CRF (teacher)	92.03	89.92	95.91	71.22	82.39
+ BP (no distillation)	89.01	88.16	93.68	64.09	75.07
+ BP + TOKEN-EM	88.53	88.49	94.47	65.99	75.59
+ BP + TOKEN-POS	89.21	88.46	94.55	65.10	75.35
+ BP + SEQ	89.81	88.60	94.62	66.42	77.53
+ BP + DA (no distillation)	90.86	88.59	94.83	70.11	79.17
+ BP + DA + TOKEN-EM	90.83	88.67	94.69	70.09	79.35
+ BP + DA + TOKEN-POS	91.04	88.78	94.84	69.73	79.31
+ BP + DA + SEQ (student)	91.21	89.06	95.07	70.65	79.91