SlowGuess/NAACL_2022_OCR huggingface dataset:
Table 1: Main results of all methods with their standard deviation (sd.) of 5 repetitive experiments for all tasks with  $K \in 1,5,10$ . Scores are averaged by all target languages. Best scores are bold. Cells are colored by performance difference over zero-shot baseline: +3 or more, +0 to +3, -0 to -3, -3 or more. ord-FS+dev: ordinary few-shot learning that fine-tunes on one target language each time with development set; ord-FS: the ord-FS+dev method without development set; mix-FT: mixed fine-tuning on concatenated target examples together; naive-mix-train: naively training both source and all target examples together; gradient-mix-train: utilizing stochastic gradient surgery during the naive-mix-train.
NER	POS	TyDiQA	XNLI
lang.	Δ F1 (%)	lang.	Δ F1 (%)	lang.	Δ F1 (%)	lang.	Δ Acc. (%)
pa	17.60	wo	3.82	bn	12.27	sw	2.36
zh	15.24	mr	3.51	te	11.14	ur	1.95
ar	14.14	hi	2.60	sw	10.58	ru	1.68
vi	13.22	tr	2.18	ar	9.45	fr	0.91
hi	12.68	fi	1.55	fi	9.05	zh	0.78
Table 2: Top-5 languages that achieve the highest improvement by using gradient-mix-train methods compared to ord-FS on all tasks in 5-shot learning. Most languages are distant from English.
tasks with various  $K$ , and importantly, overcomes the serious overfitting on the TyDiQA task and highly boosts the performance (e.g., from 48.73% of mix-FT to 57.03% of naive-mix-train in 1-shot learning). Furthermore, applying stochastic gradient surgery on mixed training (gradient-mix-train) achieves the best performance on all tasks with all settings of  $K$  and outperforms ord-FS by a significant margin, such as up to 7.36% averaged absolute improvement on NER in 5-shot learning. On the other hand, the gap between our methods and ord-FS in POS is smaller than in NER (the same type of task). The reason could be that the strong POS task baseline has already left less room for further improvement.
5.1 Which Language Benefits Most?
Table 1 shows the strong effectiveness of gradient-mix-train in improving the overall performance of each task. Here, we are interested in taking a closer look at the results of specific languages and investi
gating which language benefits most. Take 5-shot learning as an example. Table 2 illustrates the top-5 languages which boost most by using gradient-mixtrain over ord-FS in all tasks $^8$ , where the improvement is up to  $17.60\%$  absolute F1 scores for  $pa$  in the NER task. Most of the languages in the top-5 list are linguistically distant from English. We hypothesize that for such distant languages, the model has difficulty in learning the target training instances by abruptly shifting to the target domain. For closely related languages, the model is able to extrapolate the target-specific knowledge whose priors are close to English so that the model is less sensitive to these few target training examples than distant languages. However, gradient-mix-train is able to smoothly learn the distribution of source domain and extrapolate (distant) target domains by mixed training and gradient-level optimization.
5.2 Visualization of Gradient De-Conflicting
We take the NER task as an example to analyze the gradient de-conflicting of stochastic gradient surgery since it covers the most languages among all tasks. In Figure 1, we use a symmetric heatmap to visualize pair-wise gradient similarities, averaged by all 5 checkpoints in 5-shot learning. Note that languages in the figure are adjacent to other languages in the same linguistic language family. The gradient of English is calculated by the randomly picked 100 batches on average, and gradients of
For the languages that benefit the least, gradient-mix-train still yields large gains over the baseline on NER and TyDiQA. We discuss this further in Appendix D.
(a) Gradient similarity across languages without gradient surgery
(b) Gradient similarity across languages with gradient surgery
Figure 1: Gradient similarities across 48 languages in the NER task with 5 shots. Deeper colors represent higher cosine similarities. Conflicting gradients are directly marked as while cells in the heatmap. The similarities are highly improved after stochastic gradient surgery. The gradients are averaged from 5 checkpoints.
the other target languages are calculated by their 5 training instances. To highlight the conflicting gradients across languages, we directly mark the cells with negative similarities as pure white color.
Figure 1a shows the gradient similarities of the naive-mix-train model. As expected, gradient similarities of many language pairs are conflicting (white color cells), and gradients of most languages are approximately orthogonal, where their similarities are close to 0. It is worth mentioning that gradients similarities between English and most languages are conflicting. In comparison, in Figure 1b, we illustrates the gradient similarities of gradient-mix-train, and the gradient similarities between English and most of the target languages are
positive. Moreover, gradients of most target language pairs have higher similarities (deeper colors), which also verifies the correctness of our statement that target languages utilize English as a pivot language to de-conflict and even improve their similarities. The only two exceptions are  $th$  and  $ja$ , the two hardest task in NER, whose F1 in zero-shot learning is only  $1.02\%$  and  $18.31\%$ . Their similarities with other languages are negative but positive between themselves. However, gradient-mix-train still achieve impressive improvement on  $th$  ( $\Delta = 3.13\%$ ) and  $ja$  ( $\Delta = 5.40\%$ ) compared to naive-mix-train (see the full results in Appendix C). We also notice the clustering by membership closeness in the linguistic family, along with the diag-
Performance of 5-Shot NER with Different Language Sets
(a) Performance on various subsets of languages in NER
(b) Performance on various subsets of languages in TyDiQA
Figure 2: Performance of gradient-mix-train on different sets of languages compared to ord-FS for (a) NER and (b) TyDiQA. gradient-{all,subset,single} represents training on all/subset/single languages by using graident-mix-train.
nal of gradient similarity matrix, e.g., Indo-Aryan  $(bn,gu,hi,mr,pa,ur)$ . Moreover, some language families are positive correlated, e.g., Slavic  $(bg,pl,$  ru,uk) and Austronesian  $(id,jv,ms)$ .
Figure 3: Dev F1 scores of ord-FS+dev in TyDiQA. 6 out of 8 target languages overfit quickly, where they achieve the best performance at the first epoch.
5.3 Mixed Training with One Single Language
In some cases, people are only interested in one target language and do not have resources for other languages. Hence, we further explore the effectiveness of gradient-mix-train in one target language case. We conduct experiments on the NER and TyDiQA tasks that show larger gaps among different methods than other two tasks. Considering the high expense of training the source data from scratch for every target language, we run experiments on subsets of languages for each task. For the NER task, we test on 8 languages: ar, hi, my, pa, which use different scripts from English, and hu, nl, fr, tr, which share the same script with English. Figure 2a shows the results for NER. Gradient-mix-train with only one single language is labeled as gradient-single in the figure (blue, the second bar). We can focus on comparing ord-FS (green,
<sup>9</sup>We reduce  $\alpha$  for NER to 0.1 due to only one language considered.
the first bar). We notice that gradient-single still outperforms ord-FS by a large margin for 4 non-Latin-script languages (e.g.,  $14.29\%$  improvement for ar). In comparison, their gap becomes smaller when it comes to 4 Latin-script languages (e.g.,  $1.78\%$  improvement for nl). Numeric results are shown in Appendix E. For the TyDiQA task, We pick 5 languages: ar, fi, id, sw, te. We note that gradient-single still highly boosts the performance compared to ord-FS.
5.4 Do the Same Scripts Help?
Continuing the previous discussions in Section 5.3, we add a new baseline, gradient-all (red, the last bar in Figure 2), which uses gradient-mix-train method with all languages (original settings). Interestingly, gradient-all outperforms gradient-single on all selected languages except for  $ar$  in NER, and a similar phenomenon also happens in TyDiQA. Note that  $ar$  is the only language that uses Arabic script in TyDiQA and only shares the same script with  $yo$  and  $kk$  among 48 languages in NER. It brings a question that do small examples of other languages which use the same scripts help in few-shot learning? Hence, we move our experiments further on using gradient-mix-train with subsets of languages. We still consider the languages used in Section 5.3. Note that these languages are carefully selected. In NER, only  $my$  and  $pa$  share the same script (Brahmic) among 4 distant languages, and  $hu$ ,  $nl$ ,  $fr$ ,  $tr$  share the Latin script from different language families. In TyDiQA, only  $fi$ ,  $id$  and  $sw$  use the same script (Latin). We train 4 similar languages and 4 distant languages in NER, respectively. For TyDiQA, we train all 5 languages. The results of mixed training on subset of languages is denoted as gradient-subset $^{10}$  (pink, the third bar) in Figure 2. As expected, gradient-subset achieves better performance than gradient-single on all similar languages and on  $my$  among distant languages in the NER task. As for other languages using distinct scripts, their performance slightly degenerates compared to gradient-single. A similar discussion also holds for the high-level TyDiQA task, but gaps between gradient-single and gradient-subset are smaller. In conclusion, to pursue the best performance, we recommend using gradient-mix-train with languages that share the same script or only a single language that uses a distinct script.
5.5 Escaping from Overfitting
The overfitting causes the significant degeneration of ord-FS performance in TydiQA. Figure 3 shows that 6 out of 8 target languages achieve the best dev score at the first epoch and decrease significantly afterwards. However, the phenomenon of degeneration is imperceptible in other tasks because only a few languages hit the same overfitting issue, e.g.,  $6.38\%$  of languages achieve the best score at the first epoch in 1-shot learning for NER, and none of them has the issue in 10-shot learning. Different from two-step methods, one of the biggest benefits of gradient-mix-train is the perfect fit for only using the source dev set to avoid overfitting (for model selection) because training and dev steps use the same (dominant) language. Thus, although gradient-mix-train can also be further improved by using unrealistic target dev sets, the gaps are smaller compared to ord-FS (Appendix F).
We study the deficiencies of target-adapting in few-shot cross-lingual transfer and propose a mixed training method with gradient-level optimization. Our best model achieves state-of-the-art on four diverse NLP tasks with all values of  $K$ . Moreover, we are the first to use a single model to train all target languages and find that languages can benefit from others that share the same scripts. We also show the effectiveness of our method compared to target-adapting in a single target language case, and the gaps are still significant. Finally, we propose only using source dev set in few-shot settings and show that our method is development-free for targets and also able to escape from overfitting issues.
We thank anonymous reviewers for their valuable comments. We thank Kelly Marchisio for her helpful suggestions. This work was supported in part by IARPA BETTER (#2019-19051600005). The views and conclusions contained in this work are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, or endorsements of ODNI, IARPA, or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for governmental purposes notwithstanding any copyright annotation therein.
${}^{10}\alpha$  is 0.4 for NER to ensure that each language has the same chance of explosion as gradient-single during training.
Mikel Artetxe, Gorka Labaka, and Eneko Agirre. 2016. Learning principled bilingual mappings of word embeddings while preserving monolingual invariance. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2289-2294, Austin, Texas. Association for Computational Linguistics.
Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. 2020. On the Cross-lingual Transferability of Monolingual Representations. In Proceedings of ACL 2020.
Mikel Artetxe and Holger Schwenk. 2019. Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyond. Transactions of the Association for Computational Linguistics, 7:597-610.
Emily M Bender. 2011. On achieving and evaluating language-independence in nlp. Linguistic Issues in Language Technology, 6(3):1-26.
Zhao Chen, Vijay Badrinarayanan, Chen-Yu Lee, and Andrew Rabinovich. 2018. Gradnorm: Gradient normalization for adaptive loss balancing in deep multitask networks. In International Conference on Machine Learning.
Jonathan H. Clark, Eunsol Choi, Michael Collins, Dan Garrette, Tom Kwiatkowski, Vitaly Nikolaev, and Jennimaria Palomaki. 2020. TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languages. In Transactions of the Association of Computational Linguistics.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettle-moyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8440-8451, Online. Association for Computational Linguistics.
Alexis Conneau, Guillaume Lample, Marc'Aurelio Ranzato, Ludovic Denoyer, and Hervé Jégou. 2018a. Word translation without parallel data. In International Conference on Learning Representations.
Alexis Conneau, Rudy Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger Schwenk, and Veselin Stoyanov. 2018b. XNLI: Evaluating cross-lingual sentence representations. In Proceedings of EMNLP 2018, pages 2475–2485.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages
4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Li Fei-Fei, Rob Fergus, and Pietro Perona. 2006. One-shot learning of object categories. IEEE transactions on pattern analysis and machine intelligence, 28(4):594-611.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. 2017. Model-agnostic meta-learning for fast adaptation of deep networks. In Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 1126-1135. PMLR.
Michael A. Hedderich, David Adelani, Dawei Zhu, Jesujoba Alabi, Udia Markus, and Dietrich Klakow. 2020. Transfer learning and distant supervision for multilingual transformer models: A study on African languages. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2580-2591, Online. Association for Computational Linguistics.
Tsung-Yuan Hsu, Chi-Liang Liu, and Hung-yi Lee. 2019. Zero-shot reading comprehension by crosslingual transfer learning with multi-lingual language representation model. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5933-5940, Hong Kong, China. Association for Computational Linguistics.
Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan First, and Melvin Johnson. 2020. Xtreme: A massively multilingual multi-task benchmark for evaluating cross-lingual generalization. CoRR, abs/2003.11080.
Katharina Kann, Kyunghyun Cho, and Samuel R. Bowman. 2019. Towards realistic practices in low-resource natural language processing: The development set. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3342-3349, Hong Kong, China. Association for Computational Linguistics.
Anne Lauscher, Vinit Ravishankar, Ivan Vulic, and Goran Glavaš. 2020. From zero to hero: On the limitations of zero-shot language transfer with multilingual Transformers. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4483-4499, Online. Association for Computational Linguistics.
Yaobo Liang, Nan Duan, Yeyun Gong, Ning Wu, Fenfei Guo, Weizhen Qi, Ming Gong, Linjun Shou, Daxin Jiang, Guihong Cao, Xiaodong Fan, Ruofei Zhang, Rahul Agrawal, Edward Cui, Sining Wei, Taroon Bharti, Ying Qiao, Jiun-Hung Chen, Winnie Wu, Shuguang Liu, Fan Yang, Daniel Campos, Rangan Majumder, and Ming Zhou. 2020. XGLUE: A new
benchmark dataset for cross-lingual pre-training, understanding and generation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6008-6018, Online. Association for Computational Linguistics.
Fuli Luo, Wei Wang, Jiahao Liu, Yijia Liu, Bin Bi, Songfang Huang, Fei Huang, and Luo Si. 2021. VECO: Variable and flexible cross-lingual pre-training for language understanding and generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 3980–3994, Online. Association for Computational Linguistics.
Kaushal Kumar Maurya, Maunendra Sankar Desarkar, Yoshinobu Kano, and Kumari Deepshikha. 2021. ZmbART: An unsupervised cross-lingual transfer framework for language generation. In Findings of the Association for Computational Linguistics: ACLIJCNLP 2021, pages 2804–2818, Online. Association for Computational Linguistics.
Michael McCloskey and Neal J Cohen. 1989. Catastrophic interference in connectionist networks: The sequential learning problem. In *Psychology of learning and motivation*, volume 24, pages 109-165. Elsevier.
Tomas Mikolov, Quoc V Le, and Ilya Sutskever. 2013. Exploiting similarities among languages for machine translation. arXiv preprint arXiv:1309.4168.
Joakim Nivre, Mitchell Abrams, Željko Agić, Lars Ahrenberg, Lene Antonsen, Maria Jesus Aranzabe, Gashaw Arutie, Masayuki Asahara, Luma Ateyah, Mohammed Attia, et al. 2018. Universal dependencies 2.2.
Sinno Jialin Pan and Qiang Yang. 2009. A survey on transfer learning. IEEE Transactions on knowledge and data engineering, 22(10):1345-1359.
Xiaoman Pan, Boliang Zhang, Jonathan May, Joel Nothman, Kevin Knight, and Heng Ji. 2017. Cross-lingual name tagging and linking for 282 languages. In Proceedings of ACL 2017, pages 1946–1958.
Telmo Pires, Eva Schlinger, and Dan Garrette. 2019. How multilingual is multilingual BERT? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4996-5001, Florence, Italy. Association for Computational Linguistics.
Sebastian Ruder, Noah Constant, Jan Botha, Aditya Siddhant, Orhan Firat, Jinlan Fu, Pengfei Liu, Junjie Hu, Dan Garrette, Graham Neubig, and Melvin Johnson. 2021. XTREME-R: Towards more challenging and nuanced multilingual evaluation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 10215-10245, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Sebastian Ruder, Ivan Vulic, and Anders Søgaard. 2019. A survey of cross-lingual word embedding models. Journal of Artificial Intelligence Research, 65:569-631.
Ozan Sener and Vladlen Koltun. 2018. Multi-task learning as multi-objective optimization. In NeurIPS.
Jake Snell, Kevin Swersky, and Richard Zemel. 2017. Prototypical networks for few-shot learning. Advances in Neural Information Processing Systems, 30:4077-4087.
Ivan Vulic, Goran Glavaš, Roi Reichart, and Anna Korhonen. 2019. Do we really need fully unsupervised cross-lingual embeddings? In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4407-4418, Hong Kong, China. Association for Computational Linguistics.
Xinyi Wang, Ankur Bapna, Melvin Johnson, and Orhan Firat. 2021a. Gradient-guided loss masking for neural machine translation. arXiv: Computation and Language.
Xinyi Wang, Yulia Tsvetkov, and Graham Neubig. 2020. Balancing training for multilingual neural machine translation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8526-8537, Online. Association for Computational Linguistics.
Yuxuan Wang, Wanxiang Che, Jiang Guo, Yijia Liu, and Ting Liu. 2019. Cross-lingual BERT transformation for zero-shot dependency parsing. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5721-5727, Hong Kong, China. Association for Computational Linguistics.
Zirui Wang, Yulia Tsvetkov, Orhan Firat, and Yuan Cao. 2021b. Gradient vaccine: Investigating and improving multi-task optimization in massively multilingual models. In International Conference on Learning Representations.
Shijie Wu and Mark Dredze. 2019. Beto, bentz, becas: The surprising cross-lingual effectiveness of BERT. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 833–844, Hong Kong, China. Association for Computational Linguistics.
Haoran Xu, Seth Ebner, Mahsa Yarmohammadi, Aaron Steven White, Benjamin Van Durme, and Kenton Murray. 2021. Gradual fine-tuning for low-resource domain adaptation. In Proceedings of the Second Workshop on Domain Adaptation for NLP, pages 214-221, Kyiv, Ukraine. Association for Computational Linguistics.