SlowGuess/NAACL_2022_OCR huggingface dataset:
Table 3: Few-shot cross-lingual transfer performance on POS tasks with varying number of target language examples  $k$  using EN as the pivot language. We have reported the  $\Delta$  delta between few-shot and zero-shot performance averaged across the languages in each category  ${C}_{1}$  and  ${C}_{2}$  .
and  $\gamma$  for data selection. We perform experiments for  $\lambda \in \{0, 0.5, 1\}$  and  $\gamma \in \{1, 2, 3\}$ , which are reported in Appendix B. In Table 2 and 3, we are reporting the best setups for PE and LE, where we observe the highest gains for NER and POS tasks, respectively. The methods based on PE and LE consistently outperform the baseline RAND and DCE for all values of  $k$  on POS task, and most of the cases in the case of NER. DCE performs worse compared to RAND for all the languages from groups  $C_1$  and  $C_2$ . In general, the gains obtained through PE and LE compared to RAND are higher for  $C_1$  than  $C_2$ . Similarly, the proposed approaches are more useful compared to RAND for small values  $k$ , and the advantage of our sampling strategies diminishes as  $k$  approaches to 1000. For TH  $(\in C_3)$ , due to deficient zero-shot transfer performance in
NER, the gains are not consistent across models. However, all three approaches outperform RAND for small values of  $k$ .
Method	10	100	500	1k	5k	10k
mBERT	RAND	-0.5	-0.9	-0.2	0.6	2.4	3.4
DCE	0.0	-0.1	-0.1	0.5	2.8	3.9
PE (λ = 1)	-0.4	-1.0	-0.3	0.4	2.6	3.4
GE	-0.1	-0.9	-0.5	0.2	2.6	3.1
GE (γ = 1)	0.1	-0.1	0.7	1.4	3.3	4.2
XLM-R	RAND	-0.1	-0.2	0.0	0.2	1.2	1.5
DCE	0.2	0.7	0.5	0.7	1.6	2.1
PE (λ = 1)	-0.4	-0.0	0.0	0.3	1.3	1.5
GE	-0.3	-0.1	0.2	0.4	1.5	2.6
GE (γ = 1)	0.1	0.5	0.6	1.2	1.9	2.1
Table 4: Averaged few-shot performance on XNLI tasks with varying number of target language examples  $k$  using EN as the pivot language.
For XNLI, the averaged deltas across all languages are reported in Table 4. As DCE requires a sentence to train n-gram language model, hence we represent a sentence in XNLI by joining the hypothesis and the premise of an instance with a separator (-). The few-shot improvements are less pronounced than the sequence labeling tasks; noticeable gains start after seeing  $k = 500$  target-language examples. As the size of the target-language corpus in XNLI is enormous compared to POS and NER, we also evaluated the methods for  $k = 10000$ . Surprisingly, GE  $(\gamma = 1)$  and DCE outperforms RAND. As DCE selects examples in batches of 10, it selects examples having similar contexts similar to GE  $(\gamma = 1)$ , which benefits the few-shot learning. Since GE  $(\gamma = 1)$  also includes uncertainty sampling, it outperforms DCE for most of the values of  $k$ . Due to the large corpus size of XNLI, diversity becomes crucial during sampling.
We observe low few-shot gains for PE as it does not induce diversity. To measure the impact of pivot size, we trained a zero-shot model with 40k EN examples and observe similar trends for both DCE and GE (see Table 13 in Appendix).
Method	XNLI	NER	POS
C1(24)	C2(4)	C1(24)	C2(12)	C1(16)	C2(16)
PE	1	1	8	1	1	7
LE	-	-	15	5	3	10
GE	18	2	-	-	-	-
4.1 Effect of  $\lambda$  and  $\gamma$  parameters
In Appendix B, we have provided detailed results by varying  $\lambda$  and  $\gamma$ . For LE, a higher value of  $\lambda$  is required for the POS task due to the higher number of class labels than NER. The number of classes is 18 for POS and 7 for the NER task. Due to the higher number of class labels, the norm of loss embedding distribution has a higher tail. Hence, a higher value of  $\lambda$  is required for POS. We limited the value of  $\lambda$  to 0.5 as beyond that, very few examples were left for selection.
We incorporate  $\gamma$  parameter to include examples similar in context. As sentences with similar context will also have similar class labels in the case of POS and NER tasks, further decreasing the diversity in samples. Hence, we only consider experimenting with  $\gamma$  for XNLI. We observe that  $\gamma = 1$  provides the best performance on average, suggesting that having two samples of similar contexts provides better few-shot learning.
4.2 Statistical Significance Test
We perform a pairwise  $t$ -Test for measuring the statistical significance of the proposed methods against the RAND baseline. We perform  $t$ -test for each language using both mBERT and XLM-R and have reported the number of languages having  $p$ -value less than the critical point (which is 0.1 in our case) for each language group. We have considered the following methods in our tests: GE ( $\gamma = 1$ ) for XNLI, LE for NER, LE ( $\lambda = 0.5$ ) for POS and PE ( $\lambda = 1$ ) for all the tasks.
In Table 5, we notice that for XNLI, GE provides significant gains than RAND for 18 out of 24 cases
from  $C_1$  group, and 2 out of 4 cases from  $C_2$  group. For NER, the gains are significant for 20 out of 32 cases while using LE, but only 9 cases have significant gains using PE. For POS, we observe LE provide significant gains for cases compared to PE. We can conclude that the embedding-based methods provide better gains than uncertainty-based methods for most languages.
Table 5: Pairwise  $t$ -Test is performed using the proposed methods against RAND. We have reported the number of languages in each group having significance level of 0.1 using both XLM-R and mBERT models.
Method	10	50	100	500	1000
mBERT	RAND	3.9	8.8	10.5	18.6	21.2
DCE	1.8	5.5	6.5	15.3	19.2
LE	5.0	11.0	12.2	19.8	22.0
XLM-R	RAND	7.3	15.3	17.1	26.5	29.3
DCE	-0.8	8.6	10.5	22.9	26.9
LE	8.7	15.9	17.1	26.1	29.1
Table 6:  $\triangle$  Delta between Few-shot and zero-shot performance on NER tasks using ZH as the pivot language, averaged across all languages.
5 Impact of Pivot Language
We conduct few-shot experiments considering ZH as the pivot language to validate the effectiveness of our method across different pivots. The delta between the gains using ZH as the pivot have been reported in Table 6 on the NER task. The delta has been averaged across all the languages. LE provides consistent gains over RAND, and gains saturate beyond 500 examples.
5.1 Embedding Visualization
We visualize the loss embeddings for DE language using t-SNE (Van der Maaten and Hinton, 2008) in Figure 1. Most of the samples using RAND  $(\nabla)$  tend to have lower norm of loss embedding, which may not be ideal for few-shot learning. We notice that examples having lower norm of loss embedding are clustered together and highlighted with ocean colour. Hence, samples selected via LE  $(\times)$  are more likely to have higher norm or higher uncertainty estimates. It is also evident that the samples from LE (cluster centre) will have higher diversity than RAND for few-shot learning.
5.2 Qualitative Analysis of Samples
We have compared sentences selected using RAND and LE for the NER task in Table 7. Random sampling has no constraints due to which it may select examples having very few entities which might not improve the few-shot performance. Since LE uses loss as the measure of uncertainty, it selects
Raw Text	Translated Text
RAND	Er beschäftigt sich dört hauptsächlich mit dem AuswärftigenAmt und der SPD. In der weiblichen Hauptrolle ist Elżbieta Czyzewska zu sehen. Aus Asien in den Nordwesten, wo die Erfahrung Lange verharte, dann an die Ostkūste gelangte, um erst rund drei Jahrtausende später den Westen zu erreichen. Er lebt in Fernwald.	There he mainly deals with the Foreign Office and the SPD. Elżbieta Czyzewska stars in the female lead. From Asia to the northwest, where the invention remained for a long time, then reached the east coast, only to reach the west some three millennia later. He lives in Fernwald.
LE	Auch unter Trainer Joachim Löw behietet er den Posten des Managers. Maria de Lourdes Ruivo da Silva Matos Pintasilgo. Weiterleitung Atlantic Coast Hockey League. Chuck Weyant und Al Herman gingen mit Dunn-Rennwagen an den Start, wobei der 13.	He also retained the post of manager under coach Joachim Löw. Maria de Lourdes Ruivo da Silva Matos Pintasilgo. Forwarding Atlantic Coast Hockey League. Chuck Weyant and Al Herman competed in Dunn racers, with 13th.
Figure 1: The t-SNE visualization of loss embedding using the complete DE language NER corpus with XLM-R. The clusters of loss embedding are highlighted with  $\times$ , while samples from RAND are highlighted with  $\triangledown$ .
sentences with a higher number of entities often miss-classified by the zero-shot model. Hence, LE will most probably improve the few-shot performance compared to RAND. Similarly, we observe in Table 8 that for POS, LE selects sentences containing class labels that are often incorrectly tagged by the zero-shot model. In the case of XNLI, we found that the GE-based method selects more competing examples (similar hypotheses for different premises leading to different labels), which effectively can enhance the model capability. Selected examples using different methods for POS and XNLI tasks can be found in Appendix C.
6.1 Cross-lingual Transfer
In recent years, several pre-trained multilingual language models have been proposed including mBERT (Devlin et al., 2019), XLM (Conneau and Lample, 2019), XLM-R (Conneau et al., 2020), mBART (Liu et al., 2020), mT5 (Xue et al., 2020) and ERNIE-M (Ouyang et al., 2020) for cross
Table 7: Samples in DE language from NER task using RAND and LE methods for  $k = 10$  using XLM-R. Highlighted tokens are entities. We observe that LE tends to pick examples containing more entities than RAND.
Language	RAND	LE
ar	30.15%	35.21%
eu	16.53%	20.17%
he	34.11%	34.77%
hi	28.25%	31.70%
ko	31.65%	34.64%
ja	86.34%	86.44%
ur	20.29%	20.26%
zh	73.27	77.52
Table 8: Percentage of tokens from sentences sampled using RAND and  $\mathrm{LE}(\lambda = 0.5)$  for  $k = 10$  from POS task. We calculated the percentage of tokens from class labels that are mispredicted using the zero-shot model for more than  $40\%$  on the whole language corpus. We observe that the LE method selects sentences containing more incorrect class labels without access to the ground truth labels. The languages from  $C_1$  group are not considered as the gains are not relatively low.
lingual transfer. Pires et al. (2019) show mBERT to have good zero-shot performance on NER and POS tagging tasks and attributed the effectiveness of transfer to the typological similarity between the languages. In contrast, several works (Karthikeyan et al., 2019; Wu and Dredze, 2019) have shown that cross-lingual transfer does not depend on subword vocabulary overlap and joint training across languages. Lauscher et al. (2020) empirically demonstrate that both pre-training corpora sizes and linguistic similarity are strongly correlated with the zero-shot transfer. Target languages with smaller pretraining corpora or higher linguistic dissimilarity with the pivot language have a low zero-shot transfer. Furthermore, they have shown that the gap can be reduced significantly by fine-tuning with a small number of target-language examples. Nooralahzadeh et al. (2020) study the cross-lingual transfer in meta-learning setting and demonstrate improvement in zero-shot and few-shot settings. While (Lauscher et al., 2020; Nooralahzadeh et al., 2020) focus on reducing zero-shot transfer gap us
ing few-shot learning, in this work, we explore the data selection methods to get better cross-lingual transfer than the often used random sampling.
6.2 Training Data Selection
The problem of training data selection has been extensively studied for several NLP tasks, with the most notable ones from area of Machine Translation systems where target-domain data is limited and large non-domain-specific data is available. The task is to pick sentences that are closer to the target domain and also penalize the sentences which are out-of-domain. Moore and Lewis (2010) and Axelrod et al. (2011) address this problem by ranking sentences using the cross-entropy of target-domain-specific and non-domain-specific  $n$ -gram language models. Dara et al. (2014) employ an extension of the cross-entropy difference by including a vocabulary saturation filter which removes selection of very similar sentences. Song et al. (2012) have shown the effectiveness of cross-entropy selecting in-domain data for word segmentation and POS tagging tasks. We also use an extension of cross-entropy for selecting training data from the target language corpus for effective few-shot transfer using multilingual transformer models and compare with the proposed methods.
6.3 Active Learning
Active Learning has been widely used to reduce the amount of labeling to learn good models, (Yoo and Kweon, 2019; Fu et al., 2013). Uncertainty sampling methods have been commonly used in AL, where the most uncertain samples are selected for labeling. Various metrics have defined uncertainty using least confidence, sample margin, and predictive entropy. On the other hand, diversity sampling methods (Sener and Savarese, 2018; Gissin and Shalev-Shwartz, 2019) select examples which can act as a surrogate for the entire dataset. Chaudhary et al. (2019) used AL-based approaches to select entity spans for labeling in a cross-lingual transfer learning setting. However, this work was limited to only two languages. Our work focuses on data selection for cross-lingual transfer on a large and diverse set of target languages.
This work explored various data sampling strategies for few-shot learning for two sequence labeling and a semantic tasks on 20 target languages.
Our study shows that the embedding-based strategies, LE and GE, consistently outperform random sampling baseline across languages and sample sizes. Some of the salient observations are as follows. On NER and POS tasks, languages of the group  $C_2$  show significant improvements in few-shot performance, suggesting that the gains from few-shot learning are strongly correlated to the zero-shot transfer gap. LE and GE-based data selection methods show consistent gains over the RAND strategy for each target language group, but these gains saturate as the sample size,  $k$ , increases beyond 500. The saturation occurs due to the relatively smaller target-language corpus size (varies between 5k and 20k for NER and POS, respectively) effectively reducing the diversity in the total sample. LE provides better few-shot performance than PE in terms of statistical significance. DCE only performs better than RAND for Thai. As DCE does not use any form of information from the fine-tuned model and if the target-language corpus size is small, it fails to select novel target language examples any better than RAND. However, in TH, for which zero-shot performance is close to zero, DCE selects the highly representative and diverse training examples for small values of  $k$ . The trends for XNLI are different from that of the sequence labeling tasks. GE and DCE outperform all other methods, with gains increasing with the value of  $k$ , which suggests that the size of the target-language corpus is crucial for data selection. XNLI has about 400k examples in each target-language corpus, much larger than that of NER and POS, signifying the importance of diversity sampling.
Based on our observations, we recommend the LE-based sampling strategy for data selection for cross-lingual few-shot transfer for sequence labeling tasks and GE-based sampling for classification tasks. While the optimal parameter setting for the LE sampling algorithm varies across tasks, we recommend the vanilla LE method without any parameter for most of the tasks. For tasks having higher number of class labels, we recommend using LE variant with  $\lambda$  such as 0 or 0.5.
In future the work can be extended to other high-level tasks such as cross-lingual QA and Machine translation. We would also like to extend this work in a reinforcement learning (Liu et al., 2019) or meta-learning (Tseng et al., 2020) framework, where the parameters can be automatically learnt for various tasks and settings.
Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. 2020. On the cross-lingual transferability of monolingual representations. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4623–4637, Online. Association for Computational Linguistics.
David Arthur and Sergei Vassilvitskii. 2006. k-means++: The advantages of careful seeding. Technical report, Stanford.
Jordan T Ash, Chicheng Zhang, Akshay Krishnamurthy, John Langford, and Alekh Agarwal. 2019. Deep batch active learning by diverse, uncertain gradient lower bounds. In International Conference on Learning Representations.
Amittai Axelrod, Xiaodong He, and Jianfeng Gao. 2011. Domain adaptation via pseudo in-domain data selection. In Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, pages 355-362, Edinburgh, Scotland, UK. Association for Computational Linguistics.
John Blitzer, Mark Dredze, and Fernando Pereira. 2007. Biographies, bollywood, boom-boxes and blenders: Domain adaptation for sentiment classification. In Proceedings of the 45th annual meeting of the association of computational linguistics, pages 440-447.
Aditi Chaudhary, Jiateng Xie, Zaid Sheikh, Graham Neubig, and Jaime G Carbonell. 2019. A little annotation does a lot of good: A study in bootstrapping low-resource named entity recognizers. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5164-5174.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Édouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8440-8451.
Alexis Conneau and Guillaume Lample. 2019. Crosslingual language model pretraining. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc.
Alexis Conneau, Rudy Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger Schwenk, and Veselin Stoyanov. 2018. Xnli: Evaluating crosslingual sentence representations. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2475-2485.
Sandipan Dandapat, Priyanka Biswas, Monojit Choudhury, and Kalika Bali. 2009. Complex linguistic annotation - no easy way out! a case from Bangla and Hindi POS labeling tasks. In Proceedings of
the Third Linguistic Annotation Workshop (LAW III), pages 10-18, Suntec, Singapore. Association for Computational Linguistics.
Aswarth Abhilash Dara, Josef van Genabith, Qun Liu, John Judge, and Antonio Toral. 2014. Active learning for post-editing based incrementally retrained mt. In Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics, volume 2: Short Papers, pages 185-189.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Karen Fort. 2016. Collaborative annotation for reliable natural language processing: Technical and sociological aspects.
Yifan Fu, Xingquan Zhu, and Bin Li. 2013. A survey on instance selection for active learning. Knowledge and information systems, 35(2):249-283.
Daniel Gissin and Shai Shalev-Shwartz. 2019. Discriminative active learning. arXiv preprint arXiv:1907.06347.
Pratik Joshi, Sebastin Santy, Amar Budhiraja, Kalika Bali, and Monojit Choudhury. 2020. The state and fate of linguistic diversity and inclusion in the NLP world. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6282-6293, Online. Association for Computational Linguistics.
K Karthikeyan, Zihan Wang, Stephen Mayhew, and Dan Roth. 2019. Cross-lingual ability of multilingual bert: An empirical study. In International Conference on Learning Representations.
Anne Lauscher, Vinit Ravishankar, Ivan Vulic, and Goran Glavaš. 2020. From zero to hero: On the limitations of zero-shot language transfer with multilingual transformers. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4483-4499.
Miaofeng Liu, Yan Song, Hongbin Zou, and Tong Zhang. 2019. Reinforced training data selection for domain adaptation. In Proceedings of the 57th annual meeting of the association for computational linguistics, pages 1957-1968.
Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. 2020. Multilingual denoising pretraining for neural machine translation. Transactions of the Association for Computational Linguistics, 8:726-742.
Robert C. Moore and William Lewis. 2010. Intelligent selection of language model training data. In Proceedings of the ACL 2010 Conference Short Papers, pages 220-224, Uppsala, Sweden. Association for Computational Linguistics.
Joakim Nivre, Marie-Catherine de Marneffe, Filip Ginter, Yoav Goldberg, Jan Hajic, Christopher D. Manning, Ryan McDonald, Slav Petrov, Sampo Pyysalo, Natalia Silveira, Reut Tsarfaty, and Daniel Zeman. 2016. Universal Dependencies v1: A multilingual treebank collection. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC'16), pages 1659-1666, Portoroz, Slovenia. European Language Resources Association (ELRA).
Farhad Nooralahzadeh, Giannis Bekoulis, Johannes Bjerva, and Isabelle Augenstein. 2020. Zero-shot cross-lingual transfer with meta learning. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4547-4562, Online. Association for Computational Linguistics.
Xuan Ouyang, Shuohuan Wang, Chao Pang, Yu Sun, Hao Tian, Hua Wu, and Haifeng Wang. 2020. Ernie-m: Enhanced multilingual representation by aligning cross-lingual semantics with monolingual corpora. arXiv preprint arXiv:2012.15674.
Telmo Pires, Eva Schlinger, and Dan Garrette. 2019. How multilingual is multilingual BERT? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4996-5001, Florence, Italy. Association for Computational Linguistics.
Afshin Rahimi, Yuan Li, and Trevor Cohn. 2019. Massively multilingual transfer for NER. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 151-164, Florence, Italy. Association for Computational Linguistics.
Nils Reimers and Iryna Gurevych. 2020. Making monolingual sentence embeddings multilingual using knowledge distillation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4512-4525.
Marta Sabou, Kalina Bontcheva, and Arno Scharl. 2012. Crowdsourcing research opportunities: Lessons from natural language processing. In Proceedings of the 12th International Conference on Knowledge Management and Knowledge Technologies, i-KNOW '12, New York, NY, USA. Association for Computing Machinery.
Ozan Sener and Silvio Savarese. 2018. Active learning for convolutional neural networks: A core-set approach. In International Conference on Learning Representations.
Burr Settles and Mark Craven. 2008. An analysis of active learning strategies for sequence labeling tasks.