SlowGuess/NAACL_2022_OCR huggingface dataset:
In Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing, pages 1070-1079.
Anders Søgaard. 2011. Data point selection for cross-language adaptation of dependency parsers. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 682-686.
Yan Song, Prescott Klassen, Fei Xia, and Chunyu Kit. 2012. Entropy-based training data selection for domain adaptation. In Proceedings of COLING 2012: Posters, pages 1191-1200, Mumbai, India. The COLING 2012 Organizing Committee.
Andreas Stolcke. 2002. Srilm - an extensible language modeling toolkit. In INTERSPEECH. ISCA.
Hung-Yu Tseng, Hsin-Ying Lee, Jia-Bin Huang, and Ming-Hsuan Yang. 2020. Cross-domain few-shot classification via learned feature-wise transformation. arXiv preprint arXiv:2001.08735.
Laurens Van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-sne. Journal of machine learning research, 9(11).
Shijie Wu and Mark Dredze. 2019. Beto, bentz, becas: The surprising cross-lingual effectiveness of BERT. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 833–844, Hong Kong, China. Association for Computational Linguistics.
Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2020. mt5: A massively multilingual pre-trained text-to-text transformer. arXiv preprint arXiv:2010.11934.
Donggeun Yoo and In So Kweon. 2019. Learning loss for active learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 93-102.
We report the number of sentences in both training and test data in Table 9 and 10. POS task lower number of training data relative to NER task for most of the languages. XNLI task has enormous amount of training data compared to POS and NER.
Language	POS	NER
Train	Test	Language Group	Train	Test	Language Group
EN	11732	15039	-	19632	10000	-
BG	8736	1116	C1	16235	10000	C1
DE	149249	56354	C1	18515	10000	C1
ES	14092	1278	C1	17817	10000	C1
FI	14979	8233	C1	18933	10000	C1
FR	-	-	-	18109	10000	C1
SV	3167	3000	C1	14495	10000	C1
TR	7745	9619	C1	18433	10000	C1
EL	1637	456	C1	15908	10000	C1
EU	5383	1799	C2	8089	10000	C1
HI	13291	2000	C2	3948	1000	C1
KO	22947	5563	C2	18796	10000	C2
RU	3841	3601	C1	18795	10000	C1
VI	-	-	-	16066	10000	C1
AR	5956	2040	C2	17703	10000	C2
HE	5174	491	C2	18329	10000	C2
JA	7025	2172	C2	19012	10000	C2
UR	3892	535	C2	13043	1000	C2
ZH	3995	2451	C2	18310	10000	C2
TH	-	-	-	17683	10000	C3
Table 9: We report the statistics of training and test data available in each language for our experiments.
Language	Train	Test	XLMR Group	mBERT Group
AR	392403	5010	C1	C1
BG	392335	5010	C1	C1
DE	392440	5010	C1	C1
EL	392331	5010	C1	C1
EN	392568	5010	C1	C1
ES	392405	5010	C1	C1
FR	392405	5010	C1	C1
HI	392356	5010	C1	C2
RU	392318	5010	C1	C1
SW	391819	5010	C1	C2
TH	392480	5010	C1	C2
TR	392177	5010	C1	C1
UR	388826	5010	C1	C2
VI	392416	5010	C1	C1
ZH	392251	5010	C1	C1
Table 10: We report the statistics of training and test data available in each language for XNLI.
B Study of  $\lambda$  and  $\gamma$  parameters on few-shot transfer
We conduct experiments using the following set of values for  $\lambda \in \{0, 0.5, 1\}$  for NER and POS tasks. We have reported the results in Table 14 and 15. We find the parameter  $\lambda = 1$  to be providing highest performance on average for PE, while  $\lambda = 0.5$  show better performance for  $C_2$  language group when  $k = 10$ . For LE methods, we observe  $\lambda = 0.5$  provides the highest gains in POS tasks. However for NER task, LE method any  $\lambda$  parameter provides best gains on average. The gains start diminishing with higher  $\lambda$  values in general, but for  $C_2$  language group,  $\lambda = 0.5$  provides best gains for smaller values of  $k$ .
In Table 13, we observe that increasing the value  $\gamma$  beyond 1 hurts the performance for mBERT.  $\gamma = 3$  provides higher gains in few cases for XLM-R. But overall, we consider  $\gamma = 1$  to provide consistent gains across models.
C Qualitative Analysis of Samples
We have compared sentences selected using RAND and LE for POS task in Table 11. The comparison of examples from XNLI task selected using RAND and GE is shown in Table 12.
Translated Text
RAND	Study: Domestic violence in the United States affects 25% of women and 7.5 of menThe Lebanese authorities, led by the Command of the Emergency Force, contacted the Command and asked them to move to prevent and prevent the violations of Israel, which is carrying out works inside the Lebanese territories, including especially unloading sand or paving slopes, according to what was announced by an official source.
LE	It was also agreed to participate in the Maqama cultural festivals held on both sides by the two sides and to exchange and exchange musical, folk bands, artists, painters, playwrights, and others.Arafat is also scheduled to visit Kuala Lumpur, Jakarta, and Tokyo, according to Palestinian sources.
Table 11: Sample sentences in AR from POS task using RAND and LE ( $\lambda = 0.5$ ) methods for  $k = 10$  using XLM-R. The tokens are highlighted having ground truth class labels that are mispredicted using the zero-shot model. In case for AR, we noticed following class labeled are wrongly predicted: Other, Interjection, Particle, Adjective, Determiner, Pronoun and Adverb. LE ( $\gamma = 0.5$ ) select sentences containing these class labels more frequently than RAND.
Translated Premise	Translated Hypothesis
RAND	The 2000 census is the most important and provides valuable information.
The guarantee is a really poor tree guarantee.
This drama isn’t as interesting as nice celebrity meat, according to all the major gossip sites.
I am a Californian.
They would try to kill him like a pack of savages.
Egyptians travel to surrounding countries to visit casinos.	Census Monitoring Board to oversee the 2000 decennial census.
Yes I know the guy at wolfe told us they cut the tree warranty like six months or less.
Who cares what David Hare did to Arthur Schnitzler’s la rende when there’s celebrity meat to be suffered?
Oh yes yes yes i’m from i’m from up around i’m a new yorker myself.
They would come down on him like a pack of blood thirsty wolves.
Egyptian nationals are not allowed to gamble so casinos are only open to foreign guests over the age of 21 (you will be asked for ID).
GE	People from France are usually very boring.
The French are proud, dynamic patriots.
They like to sit in fancy places.
That’s because they’re fancy seats and all.
If they apologize for forgetting their name, the awkwardness will all go away.
If you can’t do better, just apologize for forgetting her name.	For a people so proud of their identity, the French are a rich mix.
For a people so proud of their identity, the French are a rich mix.
‘Cause they’re fancy places and stuff.
‘Cause they’re fancy places and stuff.
If nothing is done to alleviate the situation, you can say bluntly, I’m so sorry I’ve forgotten her name.
If nothing is done to alleviate the situation, you can say, bluntly, I’m so sorry I can’t remember her name.
Table 12: Sample sentences in DE from XNLI task using RAND and GE  $(\gamma = 1)$  methods for  $k = 10$  using XLM-R. We observe that GE  $(\gamma = 1)$  select two examples having similar context but different labels.
Method	S = 400k	S = 40k
10	100	500	1k	5k	10k	10	100	500	1k	5k	10k
mBERT	RAND	-0.5	-0.9	-0.2	0.6	2.4	3.4	-1.2	-0.7	0.0	0.9	2.8	3.8
DCE	0.0	-0.1	-0.1	0.5	2.8	3.9	0.2	-0.3	0.2	0.7	3.6	4.3
PE (λ = 1)	-0.4	-1.0	-0.3	0.4	2.6	3.4	-	-	-	-	-	-
GE	-0.1	-0.9	-0.5	0.2	2.6	3.1	-	-	-	-	-	-
GE (γ = 1)	0.1	-0.1	0.7	1.4	3.3	4.2	0.3	0.7	0.6	1.9	3.9	4.6
GE (γ = 2)	-0.3	-0.7	-0.1	1.3	3.1	4.1	-	-	-	-	-	-
GE (γ = 3)	-0.6	-0.2	0.2	1.3	3.3	4.1	-	-	-	-	-	-
mBERT	RAND	-0.1	-0.2	0.0	0.2	1.2	1.5	-0.7	-0.6	-0.6	0.1	1.1	1.5
DCE	0.2	0.7	0.5	0.7	1.6	2.1	-0.4	-0.9	-0.2	0.1	1.5	1.9
PE (λ = 1) -0.4	-0.0	0.0	0.3	1.3	1.5	-	-	-	-	-	-
GE	-0.3	-0.1	0.2	0.4	1.5	2.6	-	-	-	-	-	-
GE (γ = 1)	0.1	0.5	0.6	1.2	1.9	2.1	-0.1	0.5	-0.1	0.6	1.6	2.0
GE (γ = 2)	0.4	0.3	0.7	1.3	2.1	2.3	-	-	-	-	-	-
GE (γ = 3)	0.7	0.3	1.1	1.0	2.1	2.2	-	-	-	-	-	-
Table 13: Few-shot performance on XNLI tasks with varying number of target language examples  $k$  using EN as the pivot language. We have reported the  $\Delta$  delta between few-shot and zero-shot performance averaged across all languages.  $S$  denotes the size of the pivot-language corpus.
Method	k=10	k=50	k=100	k=500	k=1000
△C1	△C2	△C3	△C1	△C2	△C3	△C1	△C2	△C3	△C1	△C2	△C3	△C1	△C2	△C3
mBERT	RAND	2.9	9.9	0.5	6.4	15.8	1.3	7.7	17.4	1.3	12.1	26.9	18.6	14.0	30.4	31.2
DCE	1.8	8.4	4.0	5.1	12.8	2.8	5.2	11.8	3.3	9.9	23.0	18.8	12.3	28.5	29.2
PE (λ=0)	3.9	12.0	1.6	6.3	16.7	0.4	7.6	17.8	0.3	12.2	27.1	18.3	13.7	30.0	32.5
PE (λ=0.5)	4.4	13.2	0.2	7.6	18.1	1.0	8.2	18.8	0.0	12.5	27.8	18.5	14.2	30.3	32.1
PE (λ=1)	3.1	10.0	5.7	6.8	14.5	4.7	7.5	16.1	3.9	12.4	24.5	19.8	14.2	27.4	27.4
LE	5.5	11.3	2.5	7.4	18.4	1.1	8.9	18.9	0.7	13.0	27.6	18.9	14.9	30.6	31.0
LE (λ=0)	5.6	11.0	0.7	8.4	18.3	0.1	8.7	18.4	-0.0	12.9	26.9	15.1	14.8	30.0	29.8
LE (λ=0.5)	4.4	13.9	0.4	7.6	17.3	-0.1	8.6	19.1	-0.4	12.7	27.5	12.9	15.0	30.0	27.9
XLM-R	RAND	1.4	8.0	0.3	6.7	15.3	0.4	7.8	16.8	1.5	12.8	26.1	20.7	14.6	29.4	27.7
DCE	-3.8	0.5	2.4	3.4	10.0	0.9	4.3	10.5	-0.2	10.5	23.8	19.2	13.2	27.8	26.3
PE (λ=0)	1.1	8.5	1.5	6.2	14.7	7.3	7.8	15.6	6.4	13.0	25.3	22.1	14.7	29.0	28.1
PE (λ=0.5)	2.5	10.9	-1.2	7.6	15.4	2.3	8.6	17.0	-1.0	13.0	25.4	20.1	14.9	28.7	28.8
PE (λ=1)	4.4	8.6	3.6	7.0	16.5	1.2	7.9	17.8	0.5	12.3	26.1	19.0	14.2	29.9	28.4
LE	3.0	8.1	5.7	7.9	15.7	3.4	9.0	16.7	2.4	13.0	26.1	18.2	14.5	29.1	23.4
LE (λ=0)	2.4	8.2	1.4	7.4	16.0	5.0	8.5	16.9	4.0	13.0	26.1	16.5	14.5	29.3	23.2
LE (λ=0.5)	2.4	11.0	2.8	7.6	16.6	5.2	8.8	16.0	2.5	12.6	25.9	16.3	14.4	28.6	22.8
Table 14: Few-shot cross-lingual transfer performance on NER tasks with varying number of target language examples  $k$  using EN as the pivot language. We have reported the  $\triangle$  delta between few-shot and zero-shot performance averaged across the languages in each category  ${C}_{1}$  and  ${C}_{2}$  ,and  ${C}_{3}$  .
Method	k=10	k=50	k=100	k=500	k=1000
△C1	△C2	△C1	△C2	△C1	△C2	△C1	△C2	△C1	△C2
mBERT	Rand	4.1	22.6	6.7	27.5	7.3	28.0	11.9	46.5	12.5	48.4
DCE	2.3	18.7	5.2	24.3	6.0	25.9	11.8	46.1	12.4	48.3
PE (λ=0)	4.9	22.5	6.9	27.7	7.4	28.1	10.2	38.8	10.9	40.5
PE (λ=0.5)	5.2	22.5	7.0	28.0	7.5	28.3	10.4	38.9	11.2	40.5
PE (λ=1)	4.4	23.4	7.0	27.8	7.4	28.1	10.5	38.8	11.2	40.4
LE	3.9	20.1	6.3	26.3	7.1	26.9	12.1	46.9	12.8	48.7
LE (λ=0)	4.5	21.9	6.8	27.3	7.5	27.9	12.1	47.0	12.0	48.7
LE (λ=0.5)	4.1	23.3	6.8	28.2	7.5	28.6	11.3	47.0	11.5	48.8
XLM-R	RAND	3.1	24.6	5.2	28.5	5.6	28.8	9.5	42.3	9.8	44.6
DCE	1.8	22.1	3.7	26.0	4.5	27.2	9.4	42.0	9.8	44.5
PE (λ=0)	4.0	24.7	5.6	28.6	5.9	29.0	8.5	39.4	8.9	41.3
PE (λ=0.5)	4.1	25.1	5.9	28.8	6.2	29.2	8.8	39.4	9.1	41.2
PE (λ=1)	3.4	24.7	5.6	29.1	6.1	29.2	8.8	39.2	9.1	41.1
LE	2.9	22.1	5.5	27.6	6.2	28.5	9.6	42.6	9.9	44.9
LE (λ=0)	3.1	24.9	5.6	28.6	6.1	28.8	9.5	42.7	9.3	44.9
LE (λ=0.5)	3.5	25.2	5.8	29.2	6.4	29.2	9.0	42.8	8.9	44.9
Table 15: Few-shot cross-lingual transfer performance on POS tasks with varying number of target language examples  $k$  using EN as the pivot language. We have reported the  $\bigtriangleup$  delta between few-shot and zero-shot performance averaged across the languages in each category  ${C}_{1}$  and  ${C}_{2}$  .