SlowGuess/NAACL_2021_OCR huggingface dataset:
We conduct experiments on mBERT (Devlin et al., 2019), which covers 104 languages with most Wikipedia articles. For a language that is not pretrained with mBERT, we train its subword tokenizer with the task data. Further, we combine the vocabulary from the newly trained tokenizer with the original mBERT vocabulary. A similar
Method	tel	fa
(1)	target only	75.00	73.86
(2)	JT	75.13	74.81
MetaXL	77.36	76.69
Table 8: F1 for sentiment analysis on mBERT on two settings using (1) only the target language data; (2) target language data along with 10k examples of English.
approach has been adopted in (Artetxe et al., 2020). Table 12 and Table 8 present results for NER and SA respectively where we finetune the tasks on mBERT. Note that the languages of SA are both covered by mBERT and XLM-R, while the languages of NER are not. Table 13 show MetaXL results on mBERT with various sizes of source data.
Nevertheless, our method consistently brings gains on both tasks. We observe an average of 2 F1 points improvement on NER and 2.0 F1 points improvement on SA. It shows that the improvement brought by our method is consistent across different language models.
Source	Method	qu	cdo	ilo	xmf	mhr	mi	tk	gn	average
(1)	-	target only	57.14	37.72	61.32	59.07	55.17	76.27	55.56	48.89	56.39
(2)	10k en	JT	71.49	50.21	76.19	73.39	66.36	89.34	66.04	67.89	70.11
MetaXL	72.57	57.02	81.55	65.56	70.18	90.64	66.98	68.54	71.63
(3)	20k en	JT	73.19	53.93	88.78	71.49	62.56	90.80	68.29	69.44	72.31
MetaXL	73.04	55.17	85.99	73.09	70.97	89.21	66.02	73.39	73.36
Table 9: Experiment results for NER on XLM-R across three settings where we, (1) only use the target language data; (2) use target language data along with 10k examples of English; (3) use target language data along with 20k examples of English. JT stands for joint training
Layer	Method	qu	cdo	ilo	xmf	mhr	mi	tk	gn	average
-	JT	66.1	55.83	80.77	69.32	71.11	82.29	61.61	65.44	69.06
L0	MetaXL	70.43	54.76	77.14	66.09	68.72	89.53	63.59	69.86	70.02
L6	MetaXL	65.53	56.67	78.5	72.0	68.75	88.05	65.73	66.96	70.27
L0,12	MetaXL	69.83	53.97	69.44	69.26	66.96	89.41	67.92	65.18	69.00
Table 10: Experiment results for NER on XLM-R with RTN placed across multiple layer settings. (All with 5k English examples)
Layer	Method	qu	cdo	ilo	xmf	mhr	mi	tk	gn	average
-	JT	66.10	55.83	80.77	69.32	71.11	82.29	61.61	65.44	69.06
L0	JT w/ RTN	50.81	45.67	60.09	58.91	63.83	81.71	65.37	52.02	59.80
L12	JT w/ RTN	64.41	50.2	73.83	63.87	68.7	85.88	71.92	58.6	67.18
Table 11: Experiment results for NER on XLM-R, Joint Training (JT) with RTN. (All with 5k English examples)
Source	Method	qu	cdo	ilo	xmf	mhr	mi	tk	gn	average
(1)	-	target	58.44	26.77	63.39	32.06	53.66	82.90	52.53	46.01	51.97
(2)	English	JT	60.25	35.29	73.06	43.45	60.17	86.29	60.09	57.80	59.55
MetaXL	63.76	38.63	76.36	45.14	60.63	88.96	64.81	54.13	61.55
Table 12: NER results on mBERT where we use 5k English examples as auxiliary data and place RTN after 12th layer.
NER (average)	SA (tel)	SA (fa)
# en	JT	MetaXL	Δ	# en	JT	MetaXL	Δ	# en	JT	MetaXL	Δ
5k	59.55	61.55	+2.00	100	75.12	77.36	+2.24	100	74.25	75.78	+1.53
10k	62.36	63.66	+1.30	1k	74.76	76.39	+1.63	1k	74.71	75.58	+0.87
20k	62.39	63.38	+0.99	5k	74.07	78.15	+4.08	5k	74.81	76.69	+1.88
Table 13: F1 on various source language transfer data sizes on mBERT. # en denotes the number of English examples used for transfer.  $\Delta$  denotes the improvement of MetaXL over the joint training baseline.