SlowGuess/ACL_2020_OCR huggingface dataset:
BERT. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 833-844, Hong Kong, China. Association for Computational Linguistics.
Ruochen Xu and Yiming Yang. 2017. Cross-lingual distillation for text classification. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1415–1425, Vancouver, Canada. Association for Computational Linguistics.
Jie Yang, Shuailong Liang, and Yue Zhang. 2018. Design challenges and misconceptions in neural sequence labeling. In Proceedings of the 27th International Conference on Computational Linguistics, pages 3879-3889, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. 2019. Xlnet: Generalized autoregressive pretraining for language understanding. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32, pages 5753-5763. Curran Associates, Inc.
Daniel Zeman and Jan Hajic, editors. 2018. Proceedings of the CoNLL 2018 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies. Association for Computational Linguistics, Brussels, Belgium.
In this appendix, we use ISO 639-1 codes<sup>3</sup> to represent each language for simplification.
A.1 Zero-shot Transfer
Table 9, 10 shows performance of zero-shot transfer on the NER and POS tagging datasets. Our Posterior approach outperforms Baseline in 24 out of 28 languages on NER and 20 out of 24 languages on POS tagging.
3https://en.wikipedia.org/wiki/List_of_ISO_639-1_codes
ar	be	ca	cs	da	el	eo	et	fi	gl
(1): TEACHER	14.77	26.96	57.75	57.16	65.19	45.70	35.81	49.66	55.61	63.73
(2): BASELINE	27.72	64.64	55.78	65.40	68.33	60.76	37.94	59.54	63.41	64.83
(3): EMISSION	26.92	63.75	55.30	64.27	68.09	59.86	37.28	59.23	63.68	64.99
(4): POSTERIOR	27.83	64.62	56.82	65.69	69.08	60.66	38.44	60.47	64.03	65.07
(5): POSTERIOR+TOP-K	28.31	64.54	56.34	65.80	69.08	61.33	38.14	60.16	63.62	65.12
Δ: (4)-(1)	13.06	37.66	-0.93	8.53	3.89	14.96	2.63	10.81	8.42	1.34
Δ: (5)-(1)	13.54	37.58	-1.41	8.64	3.89	15.63	2.33	10.50	8.01	1.39
Δ: (4)-(2)	0.11	-0.02	1.04	0.29	0.76	-0.10	0.49	0.93	0.61	0.24
Δ: (5)-(2)	0.60	-0.10	0.55	0.40	0.75	0.57	0.20	0.62	0.20	0.29
Δ: (4)-(3)	0.91	0.88	1.53	1.42	1.00	0.80	1.16	1.24	0.35	0.08
Δ: (5)-(3)	1.40	0.79	1.04	1.53	0.99	1.47	0.86	0.93	-0.06	0.13
hr	hu	hy	kk	ko	lt	ms	no	pl	pt
(1): TEACHER	50.53	52.49	21.55	22.82	26.88	45.35	24.09	62.76	56.53	51.77
(2): BASELINE	60.19	62.75	32.32	35.85	35.56	52.31	24.76	67.38	69.31	52.10
(3): EMISSION	59.79	61.37	30.69	31.63	35.26	51.95	25.07	67.49	69.07	52.30
(4): POSTERIOR	61.10	63.34	32.80	37.38	36.19	52.75	25.42	68.58	70.27	53.51
(5): POSTERIOR+TOP-K	60.58	63.21	32.57	34.10	36.70	52.83	25.14	67.51	69.90	53.53
Δ: (4)-(1)	10.57	10.85	11.25	14.56	9.31	7.40	1.33	5.82	13.74	1.74
Δ: (5)-(1)	10.05	10.72	11.02	11.28	9.82	7.48	1.05	4.75	13.37	1.76
Δ: (4)-(2)	0.91	0.60	0.49	1.53	0.63	0.44	0.66	1.20	0.96	1.42
Δ: (5)-(2)	0.40	0.46	0.25	-1.75	1.14	0.53	0.38	0.13	0.58	1.44
Δ: (4)-(3)	1.31	1.97	2.12	5.75	0.93	0.80	0.35	1.09	1.20	1.22
Δ: (5)-(3)	0.79	1.83	1.88	2.47	1.44	0.88	0.07	0.03	0.83	1.23
ro	ru	sk	sv	tr	uk	vi	zh	Avg.
(1): TEACHER	34.96	21.91	52.84	70.44	45.98	25.04	30.05	3.40	41.85
(2): BASELINE	36.46	28.68	60.44	68.91	57.14	49.19	33.38	28.94	50.86
(3): EMISSION	36.20	28.63	60.08	69.48	56.29	46.23	33.27	27.25	50.19
(4): POSTERIOR	37.06	29.07	61.09	68.23	57.88	48.76	33.64	30.15	51.43
(5): POSTERIOR+TOP-K	36.33	29.05	60.78	69.30	57.68	46.82	33.28	30.04	51.14
Δ: (4)-(1)	2.10	7.16	8.25	-2.21	11.90	23.72	3.59	26.75	9.58
Δ: (5)-(1)	1.37	7.14	7.94	-1.14	11.70	21.78	3.23	26.64	9.29
Δ: (4)-(2)	0.60	0.40	0.65	-0.68	0.74	-0.44	0.26	1.21	0.57
Δ: (5)-(2)	-0.13	0.37	0.34	0.39	0.54	-2.38	-0.10	1.11	0.28
Δ: (4)-(3)	0.86	0.45	1.01	-1.25	1.59	2.53	0.37	2.90	1.23
Δ: (5)-(3)	0.13	0.42	0.70	-0.18	1.39	0.58	0.02	2.79	0.94
Table 9: F1 scores of zero-shot transfer on the WikiAnn NER datasets.  $\Delta$  represents the difference of F1 score.
ar	bg	ca	cs	da	de	es	eu	fi
(1): TEACHER	47.85	48.24	80.04	51.62	53.79	44.35	81.03	44.29	51.50
(2): BASELINE	80.82	88.59	89.95	87.55	88.35	87.70	91.32	69.62	80.06
(3): EMISSION	80.85	88.62	90.00	87.56	88.47	87.89	91.27	69.68	80.10
(4): POSTERIOR	80.95	88.26	89.77	87.50	88.68	87.79	91.48	70.03	80.52
(5): POSTERIOR+TOP-K	80.77	88.30	89.77	87.46	88.58	87.84	91.29	70.17	80.38
Δ: (4)-(1)	33.10	40.02	9.73	35.88	34.89	43.44	10.45	25.74	29.02
Δ: (5)-(1)	32.92	40.06	9.73	35.84	34.79	43.49	10.26	25.88	28.88
Δ: (4)-(2)	0.12	-0.33	-0.18	-0.05	0.33	0.09	0.15	0.41	0.47
Δ: (5)-(2)	-0.05	-0.30	-0.18	-0.09	0.23	0.14	-0.03	0.55	0.32
Δ: (4)-(3)	0.09	-0.36	-0.24	-0.06	0.21	-0.10	0.20	0.34	0.42
Δ: (5)-(3)	-0.08	-0.33	-0.23	-0.10	0.11	-0.05	0.02	0.49	0.28
hi	hr	it	ko	nl	no	pl	pt	ro
(1): TEACHER	33.09	69.40	79.33	37.90	40.02	50.86	48.68	77.66	70.45
(2): BASELINE	76.41	88.28	93.66	58.47	87.30	88.84	85.26	93.38	86.20
(3): EMISSION	76.15	88.17	93.74	58.65	87.32	88.94	85.27	93.49	86.15
(4): POSTERIOR	76.64	88.46	93.70	59.09	87.19	88.91	85.31	93.42	86.33
(5): POSTERIOR+TOP-K	76.44	88.34	93.83	58.85	87.20	88.83	85.60	93.15	86.57
Δ: (4)-(1)	43.55	19.06	14.37	21.19	47.17	38.05	36.63	15.76	15.88
Δ: (5)-(1)	43.35	18.94	14.50	20.95	47.18	37.97	36.92	15.49	16.12
Δ: (4)-(2)	0.23	0.18	0.03	0.62	-0.11	0.07	0.05	0.03	0.13
Δ: (5)-(2)	0.03	0.06	0.17	0.38	-0.10	0.00	0.34	-0.23	0.36
Δ: (4)-(3)	0.50	0.29	-0.05	0.45	-0.13	-0.03	0.04	-0.07	0.18
Δ: (5)-(3)	0.30	0.18	0.09	0.21	-0.11	-0.10	0.33	-0.34	0.41
ru	sk	sr	sv	tr	zh	Avg.
(1): TEACHER	50.81	56.09	70.04	50.63	54.93	51.55	56.01
(2): BASELINE	88.15	87.67	89.70	89.73	71.49	70.24	84.11
(3): EMISSION	88.10	87.73	89.60	89.91	71.68	70.72	84.17
(4): POSTERIOR	88.22	87.83	89.95	89.96	71.93	70.93	84.28
(5): POSTERIOR+TOP-K	88.10	87.84	89.92	89.69	71.99	70.74	84.24
Δ: (4)-(1)	37.41	31.74	19.91	39.33	17.00	19.38	28.28
Δ: (5)-(1)	37.29	31.75	19.88	39.06	17.06	19.19	28.23
Δ: (4)-(2)	0.07	0.16	0.25	0.23	0.44	0.69	0.17
Δ: (5)-(2)	-0.05	0.18	0.22	-0.04	0.50	0.50	0.12
Δ: (4)-(3)	0.12	0.10	0.35	0.05	0.24	0.21	0.12
Δ: (5)-(3)	0.00	0.11	0.32	-0.21	0.31	0.02	0.07
Table 10: F1 scores of zero-shot transfer on the UD POS tagging datasets.  $\Delta$  represents the difference of F1 score.