SlowGuess/NAACL_2022_OCR huggingface dataset:
Cross-lingual Name Entity Recognition CoNLL and WikiAnn are common xNER benchmarks. All of those datasets are annotated with 4 types of entities, namely PER, LOC, ORG and MISC in BIO tagging schema following (Pan et al., 2017). Of note, CoNLL and WikiAnn both contain four language test sets: Spanish, Dutch, English, German. Besides, WikiAnn additional consists of another two language test sets: Hindi, Arabic
E Detailed Results on cross-lingual sequence labeling tasks
E.1 Results on MLQA
Table 5 shows results of our models in each language.
E.2 Results on XQUAD
Table 6 presents results of our models in each language on XQUAD.
E.3 Results on CoNLL
Table 7 presents results of our models in each language on CoNLL.
E.4 Results on WikiAnn
Table 8 presents results of our models in each language on CoNLL.
F Few-shot results on xMRC datasets
We present our model performance with different number of training examples:
Models	en	es	de	ar	hi	vi	Avg.
Ours(XLM-R)	69.14/47.1	58.66/37.79	65.43/48.80	78.67/65.23	69.39/49.31	62.72/42.07	67.34/49.11
Ours(Info-XLM)	69.94/48.61	61.66/40.79	67.03/50.79	79.98/66.73	70.90/52.31	64.72/45.57	69.07/50.81
Table 5: The performance of our models on MLQA datasets.
Models	en	es	de	ar	hi	vi	Avg.
Ours(XLM-R)	80.2/66.5	76.0/60.01	75.7/61.62	68.97/50.93	68.11/50.67	74.12/52.52	76.73/60.87
Ours(Info-XLM)	80.98/68.61	77.46/61.35	74.53/61.19	71.61/54.28	70.40/52.63	73.72/51.69	78.22/61.99
Table 6: The performance of our models on XQUAD datasets.
$\{64,128,256,512,1024\}$  shown in Table 9
G Pre-training data for CLISM
In this component, we further conduct extensive experiments to explore the relation between pretraining data and model performances. As shown in Figure 5, we present the resulting model performances on xMRC datasets with different number of pre-training data:  $\{1M,2M,4M,8M,12M\}$ . Obviously, ours achieve best results given 1 million pre-training data. With the amount of pre-training data increased, the performances drop gradually due to overfitting. The results prove the effectiveness of our proposed pre-training paradigm, that is, our methods could optimize the xPLMs sufficiently even if with limited pre-training data.
Models	en	es	nl	de	Avg.
Ours(XLM-R)	91.1	78.3	78.0	70.5	80.63
Ours(Info-XLM)	91	77.6	80	71.9	81.13
Table 7: The performance of our models on CoNLL datasets.
Models	en	es	de	ar	hi	nl	Avg.
Ours(XLM-R)	81.88	77.7	80	74.33	57	68.95	73.31
Ours(Info-XLM)	82.5	71.7	80.5	76.3	63	69.5	73.87
Table 8: The performance of our models on WikiAnn datasets.
Models	MLQA	XQUAD
64 examples
XLM-R	6/1.5	8/2
Info-XLM	7/3.2	8.7/5.1
Ours (XLM-R)	14/7.2	15.5/8.2
128 examples
XLM-R	10/3.5	12/4
Info-XLM	11.4/4.5	15.7/7
Ours (XLM-R)	26.4/15.9	29.5/19.7
256 examples
XLM-R	18.1/9.8	20.4/10.4
Info-XLM	19.8/10.5	26.2/13.4
Ours (XLM-R)	39.3/25.8	44.2/29.0
512 examples
XLM-R	26.8/15.5	33.35/19.1
Info-XLM	29.7/18.9	36.0/21.0
Ours (XLM-R)	50.8/35.9	53.94/37.1
1024 examples
XLM-R	38.8/23.0	43.25/28.03
Info-XLM	42.7/24.7	48.23/30.2
Ours (XLM-R)	57.8/42.4	61.25/44.92
Table 9: Model performances across xMRC datasets when the number of training examples is  $\{64,128,256,512,1024\}$
(a) MLQA
Figure 5: Model performance with different pre-training data on MLQA and XQUAD datasets.
(b) XQUAD