SlowGuess/NAACL_2021_OCR huggingface dataset:
cant gains to the joint training baseline  $(JT)$  over using target language data only (target only), as in the NER task. In addition, MetaXL still outperforms joint training by around 0.9 and 1.6 F1 score on Telugu and Persian. These results support our hypothesis that MetaXL can transfer representations from other languages more effectively. That, in turn, contributes to the performance gain on the target task.
4.2 Source Language Data Size
To evaluate how MetaXL performs with different sizes of source language data, we perform experiments on varying the size of source data. For NER, we experiment with 5k, 10k, and 20k source examples. For SA, we test on 1k, 3k and 5k  $^4$  source examples.
As observed from Table 4, MetaXL delivers consistent gains as the size of source data increases over the joint training model (except on fa when using 5k auxiliary data). However, the marginal gain decreases as the source data size increases on NER. We also note that MetaXL continues to improve even when joint training leads to a minor performance drop for SA.
4.3 Placement of Representation Transformation Network
Previous works (Jawahar et al., 2019; Tenney et al., 2019) have observed that lower and intermediate layers encode surface-level and syntactic information, whereas top layers are more semantically focused. These findings suggest that the placement of the representation transformation network can potentially affect the effectiveness of transfer. To
4No significant gains were observed for any of the models when going beyond 5K examples.
5Please refer to Appendix C for full results.
NER (average)	SA (tel)	SA (fa)
# en	JT	MetaXL	Δ	# en	JT	MetaXL	Δ	# en	JT	MetaXL	Δ
5k	69.06	71.13	+2.07	1k	88.68	90.53	+1.85	1k	85.51	87.14	+1.63
10k	70.11	71.63	+1.52	3k	87.13	87.23	+0.10	3k	82.88	86.19	+3.31
20k	72.31	73.36	+1.05	5k	84.91	85.71	+0.80	5k	86.34	85.63	-0.71
Table 4: F1 on various source language transfer data sizes. # en denotes the number of English examples used for transfer.  $\Delta$  denotes the improvement of MetaXL over the joint training baseline. RTN is placed after 12th layer.
Method	NER	SA
Average	tel	fa
JT	69.06	88.68	85.51
MetaXL L0	70.02	89.52	85.41
MetaXL L6	70.27	86.00	85.80
MetaXL L12	71.13	90.53	87.14
MetaXL L0,12	69.00	84.85	86.64
this end, we conducted experiments with representation transformation networks placed at various depths of the Transformer model.
Specifically, we experiment with placing the representation transformation network after the 0th (embedding layer), 6th and 12th layer (denoted by L0, L6, L12). We also experiment with placing two identical representation transformation networks after both the 0th and 12th layers.
As observed from Table 5, transformations at the 12th layer are consistently effective, suggesting that transformation at a higher and more abstract level results in better transfer for both tasks. Transferring from lower layers leads to fewer gains for SA, coinciding with the fact that SA is more reliant on global semantic information. Transferring at multiple layers does not necessarily lead to higher performance, possibly because it results in increased instability in the bi-level optimization procedure.
4.4 Joint Training with Representation Transformation Networks
There are two major differences between MetaXL and joint training: (1) source language data un
Table 5: F1 when placing the transfer component at different positions on XLM-R. Under this setting, we use 5k English data for NER and 1K English data for SA. L stands for layer.
Layer	Method	NER Average	SA
tel	fa
-	JT	69.06	88.68	85.51
L0	JT w/ RTN	59.80	63.95	72.32
MetaXL	70.02	89.52	85.41
L12	JT w/ RTN	67.18	83.75	70.40
MetaXL	71.13	90.53	87.14
Table 6: F1 when joint training with and without the representation transformation network in XLM-R. In this setting, we use 5k English examples for NER and 1k English examples for SA. NER results are aggregated over 8 target languages. Bold denotes that MetaXL outperforms both JT and JT w/ RTN baselines.
dergoes transformation via an augmented representation transformation network; (2) we adopt a bi-level optimization procedure to update the base model and the representation transformation network. To verify that the performance gain from MetaXL is not attributed to increased model capacity, we conduct experiments on joint training using the representation transformation network. Specifically, the forward pass remains the same as MetaXL, whereas the backward optimization employs the standard stochastic gradient descent algorithm. We conduct experiments on placing the representation transformation network after the 0th layer or 12th layer and present results in Table 6<sup>7</sup>.
Interestingly, joint training with the representation transformation network deteriorates the model performance compared to vanilla joint training. Transferring after the 0th layer is even more detrimental than the 12th layer. This finding shows that Transformer models are rather delicate to subtle architectural changes. In contrast, MetaXL breaks the restriction, pushing the performance higher for both layer settings.
$^6$ Please refer to Appendix B.2 for full results.
7Please refer to Appendix B.3 for full results.
Figure 3: PCA visualization of token-level representations of Quechua and English from the joint training mBERT model on NER. With MetaXL, the Hausdorff distance drops from 0.60 to 0.53 and the F1 score increases from 60.25 to 63.76.
4.5 Analysis of Transformed Representations
To verify that MetaXL does bring the source and target language spaces closer, we qualitatively and quantitatively demonstrate the representation shift with transformation. In particular, we collect representations of both the source and target languages from the joint training and the MetaXL models, with mBERT as the multilingual encoder, and present the 2-component PCA visualization in Figure 1 for SA and Figure 3 for NER. SA models are trained on Telugu paired with 5k English examples, and NER models are trained on Quechua paired with 5k English. From the figures, MetaXL merges the representations from two languages for SA, but the phenomenon is not as evident for NER.
Singh et al. (2019) quantitatively analyze mBERT representations with canonical correlation analysis (CCA). However, CCA does not suit our case as we do not have access to semantically aligned data for various languages. Thus we adopt Hausdorff distance as a metric that has been widely used in vision and NLP tasks (Huttenlocher et al., 1993; Dubuisson and Jain, 1994; Patra et al., 2019) to measure the distance between two distinct datasets. Informally, the Hausdorff distance measures the average proximity of data representations in the source language to the nearest ones in the target language, and vice versa. Given a set of representations of the source language  $S = \{s_1, s_2, \ldots, s_m\}$  and a set of representations of the target language  $T = \{t_1, t_2, \ldots, t_n\}$ , we compute the Hausdorff distance as follows:
$$
\max  \left\{\max  _ {s \in \mathcal {S}} \min  _ {t \in \mathcal {T}} d (s, t), \max  _ {t \in \mathcal {T}} \min  _ {s \in \mathcal {S}} d (s, t) \right\} \tag {5}
$$
where cosine distance is used as as the inner distance, i.e.,
$$
d (s, t) \triangleq 1 - \cos (s, t) \tag {6}
$$
For SA, we observe a drastic drop of Hausdorff distance from 0.57 to 0.20 and a substantial performance improvement of around 4 F1 score. For NER, we observe a minor decline of Hausdorff distance from 0.60 to 0.53 as the representations are obtained at the token level, leading to a significant performance gain of 3 F1 score. For NER, we observe a correlation of 0.4 between performance improvement and the reduction in representation distance. Both qualitative visualization and quantitative metrics confirm our hypothesis that MetaXL performs more effective transfer by bringing the representations from different languages closer.
4.6 Additional Results on High-resource Languages
fr	es	ru	zh
JT	76.50	72.87	71.14	60.62
MetaXL	72.43	70.38	71.08	58.81
Table 7: F1 on mBERT rich languages in a simulated low-resource setting.
Despite our experiments so far on extremely low-resource languages, given by few labeled data for fine-tuning and limited or no unlabeled data for pretraining, MetaXL is generally applicable to all languages. To better understand the scope of applying MetaXL to languages with varying resources, we perform experiments on 4 target languages that do not belong to our extremely low-resource category for the NER task, namely, Spanish (es), French (fr), Italian (it), Russian (ru) and Chinese (zh). These languages are typically considered high-resource with 20k labeled examples in the WikiAnn datasets and large amount of unlabeled data consumed by mBERT for pre-training. We use only 100 examples for all target languages to mimic the low-resource setting and use 5k English examples for transfer.
As shown in Table 7, we found slight performance drop using MetaXL for these high-resource languages. We conjecture that these languages have been learned quite well with the mBERT model during the pre-training phase, therefore, leaving less scope for effective representation transformation in
the low-resource setup. Nonetheless, this can be remedied with a back-off strategy by further finetuning the resulting model from MetaXL on the concatenated data from both source and target languages to match the performance of JT training. As high-resource languages are out of the scope of this paper, we leave further analysis and understanding of these scenarios for future work.
Unifying Language Spaces MetaXL in essence brings the source and target representations closer. Previous works have shown that learning invariant representations across languages leads to better transfer. On the representation level, adversarial training is widely adopted to filter away language-related information (Xie et al., 2017; Chen et al., 2018). One the form level, Xia et al. (2019) show that replacing words in a source language with the correspondence in the target language brings significant gains in low-resource machine translation.
Adapters Adapter networks are designed to encode task (Houlsby et al., 2019; Stickland and Murray, 2019; Pfeiffer et al., 2020a), domain (Bapna and First, 2019) and language (Pfeiffer et al., 2020c) specific information to efficiently share parameters across settings. Though RTN in MetaXL is similar to adapter networks in architecture, in contrast to adapter networks, it plays a more explicit role in transforming representations across languages to bridge the representation gap. More importantly, MetaXL trains the representation transformation network in a meta-learning based paradigm, significantly different from how adapters are trained.
Meta Learning MetaXL falls into the category of meta learning for its goal to learn to transform under the guidance of the target task. Related techniques have been used in Finn et al. (2017), which aims to learn a good initialization that generalizes well to multiple tasks and is further extended to low-resource machine translation (Gu et al., 2018) and low-resource natural language understanding tasks (Dou et al., 2019). The bi-level optimization procedure is widely adopted spanning across neural architecture search (Liu et al., 2019), instance re-weighting (Ren et al., 2018; Shu et al., 2019), learning from pseudo labels (Pham et al., 2020) and mitigating negative inference in multilingual systems (Wang et al., 2020). MetaXL is the first to
meta learn a network that explicitly transforms representations for cross-lingual transfer on extremely low-resource languages.
In this paper, we study cross-lingual transfer learning for extremely low-resource languages without large-scale monolingual corpora for pre-training or sufficient annotated data for fine-tuning. To allow for effective transfer from resource-rich source languages and mitigate the representation gap between multilingual pre-trained representations, we propose MetaXL to learn to transform representations from source languages that best benefits a given task on the target language. Empirical evaluations on cross-lingual sentiment analysis and named entity recognition tasks demonstrate the effectiveness of our approach. Further analysis on the learned transformations verify that MetaXL indeed brings the representations of both source and target languages closer, thereby, explaining the performance gains. For future work, exploring transfer from multiple source languages to further improve the performance and investigating the placement of multiple representation transformation networks on multiple layers of the pre-trained models are both interesting directions to pursue.
We thank the anonymous reviewers for their constructive feedback, and Wei Wang for valuable discussions.
This work addresses cross-lingual transfer learning onto extremely low-resource languages, which is a less studied area in NLP community. We expect that progress and findings presented in this paper could advocate awareness of advancing NLP for extremely low-resource languages and help improve information access for such under-represented language communities.
The proposed method is somewhat compute- intensive as it requires approximating second-order gradients for updating the meta-parameters. This might impose negative impact on carbon footprint from training the described models. Future work on developing more efficient meta-learning optimization methods and accelerating meta-learning training procedure might help alleviate this concern.
Oliver Adams, Adam Makarucha, Graham Neubig, Steven Bird, and Trevor Cohn. 2017. Cross-lingual word embeddings for low-resource language modeling. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 937-947.
Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. 2019. On the cross-lingual transferability of monolingual representations. arXiv preprint arXiv:1910.11856.
Mikel Artetxe, Sebastian Ruder, Dani Yogatama, Gorka Labaka, and Eneko Agirre. 2020. A call for more rigor in unsupervised cross-lingual learning. arXiv preprint arXiv:2004.14958.
Ankur Bapna and Orhan First. 2019. Simple, scalable adaptation for neural machine translation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 1538-1548, Hong Kong, China. Association for Computational Linguistics.
Xilun Chen, Yu Sun, Ben Athiwaratkun, Claire Cardie, and Kilian Weinberger. 2018. Adversarial deep averaging networks for cross-lingual sentiment classification. Transactions of the Association for Computational Linguistics, 6:557-570.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Édouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8440–8451.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Zi-Yi Dou, Keyi Yu, and Antonios Anastasopoulos. 2019. Investigating meta-learning algorithms for low-resource natural language understanding tasks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the
9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 1192-1197, Hong Kong, China. Association for Computational Linguistics.
M-P Dubuisson and Anil K Jain. 1994. A modified Hausdorff distance for object matching. In Proceedings of 12th international conference on pattern recognition, volume 1, pages 566-568. IEEE.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. 2017. Model-agnostic meta-learning for fast adaptation of deep networks. In Proceedings of the 34th International Conference on Machine Learning-Volume 70, pages 1126-1135.
Rama Rohit Reddy Gangula and Radhika Mamidi. 2018. Resource creation towards automated sentiment analysis in telugu (a low resource language) and integrating multiple domain sources to enhance sentiment prediction. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018).
Jiatao Gu, Yong Wang, Yun Chen, Victor O. K. Li, and Kyunghyun Cho. 2018. Meta-learning for low-resource neural machine translation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3622-3631, Brussels, Belgium. Association for Computational Linguistics.
Julia Hirschberg and Christopher D Manning. 2015. Advances in natural language processing. Science, 349(6245):261-266.
Pedram Hosseini, Ali Ahmadian Ramaki, Hassan Maleki, Mansoureh Anvari, and Seyed Abolghasem Mirroshandel. 2018. Sentipers: A sentiment analysis corpus for persian. arXiv preprint arXiv:1801.07737.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for nlp. In International Conference on Machine Learning, pages 2790-2799.
Daniel P Huttenlocher, Gregory A. Klanderman, and William J Rucklidge. 1993. Comparing images using the hausdorff distance. IEEE Transactions on pattern analysis and machine intelligence, 15(9):850-863.
Ganesh Jawahar, Benoit Sagot, and Djamé Seddah. 2019. What does bert learn about the structure of language? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3651-3657.
Phillip Keung, Yichao Lu, György Szarvas, and Noah A. Smith. 2020. The multilingual Amazon reviews corpus. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4563-4568, Online. Association for Computational Linguistics.
Yu-Hsiang Lin, Chian-Yu Chen, Jean Lee, Zirui Li, Yuyan Zhang, Mengzhou Xia, Shruti Rijhwani, Junxian He, Zhisong Zhang, Xuezhe Ma, Antonios Anastasopoulos, Patrick Littell, and Graham Neubig. 2019. Choosing transfer languages for cross-lingual learning. In The 57th Annual Meeting of the Association for Computational Linguistics (ACL), Florence, Italy.
Hanxiao Liu, Karen Simonyan, and Yiming Yang. 2019. DARTS: Differentiable architecture search. In International Conference on Learning Representations.
Xiaoman Pan, Boliang Zhang, Jonathan May, Joel Nothman, Kevin Knight, and Heng Ji. 2017. Cross-lingual name tagging and linking for 282 languages. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1946–1958.
Barun Patra, Joel Ruben Antony Moniz, Sarthak Garg, Matthew R. Gormley, and Graham Neubig. 2019. Bilingual lexicon induction with semi-supervision in non-isometric embedding spaces. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 184-193, Florence, Italy. Association for Computational Linguistics.
Jonas Pfeiffer, Aishwarya Kamath, Andreas Rückle, Kyunghyun Cho, and Iryna Gurevych. 2020a. Adapterfusion: Non-destructive task composition for transfer learning. arXiv preprint arXiv:2005.00247.
Jonas Pfeiffer, Andreas Rückle, Clifton Poth, Aishwarya Kamath, Ivan Vulic, Sebastian Ruder, Kyunghyun Cho, and Iryna Gurevych. 2020b. AdapterHub: A framework for adapting transformers. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 46-54, Online. Association for Computational Linguistics.
Jonas Pfeiffer, Ivan Vulić, Iryna Gurevych, and Sebastian Ruder. 2020c. MAD-X: An Adapter-Based Framework for Multi-Task Cross-Linguual Transfer. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7654–7673, Online. Association for Computational Linguistics.
Hieu Pham, Qizhe Xie, Zihang Dai, and Quoc V Le. 2020. Meta pseudo labels. arXiv preprint arXiv:2003.10580.
Afshin Rahimi, Yuan Li, and Trevor Cohn. 2019. Massively multilingual transfer for ner. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 151-164.
Mengye Ren, Wenyuan Zeng, Bin Yang, and Raquel Urtasun. 2018. Learning to reweight examples for robust deep learning. In International Conference on Machine Learning, pages 4334-4343.
Jun Shu, Qi Xie, Lixuan Yi, Qian Zhao, Sanping Zhou, Zongben Xu, and Deyu Meng. 2019. Meta-weight-net: Learning an explicit mapping for sample weighting. In Advances in Neural Information Processing Systems, pages 1919-1930.
Jasdeep Singh, Bryan McCann, Richard Socher, and Caiming Xiong. 2019. Bert is not an interlingua and the bias of tokenization. In Proceedings of the 2nd Workshop on Deep Learning Approaches for Low-Resource NLP (DeepLo 2019), pages 47-55.
Asa Cooper Stickland and Iain Murray. 2019. Bert and pals: Projected attention layers for efficient adaptation in multi-task learning. In International Conference on Machine Learning, pages 5986-5995.
Ian Tenney, Dipanjan Das, and Ellie Pavlick. 2019. Bert rediscovers the classical nlp pipeline. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4593-4601.
Zirui Wang, Zachary C. Lipton, and Yulia Tsvetkov. 2020. On negative interference in multilingual models: Findings and a meta-learning treatment. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4438-4450, Online. Association for Computational Linguistics.
Shijie Wu and Mark Dredze. 2019. Beto, bentz, becas: The surprising cross-lingual effectiveness of bert. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 833-844.
Mengzhou Xia, Xiang Kong, Antonios Anastasopoulos, and Graham Neubig. 2019. Generalized data augmentation for low-resource translation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5786-5796.
Qizhe Xie, Zihang Dai, Yulun Du, Eduard Hovy, and Graham Neubig. 2017. Controllable invariance through adversarial feature learning. In Advances in Neural Information Processing Systems, pages 585-596.
Guoqing Zheng, Ahmed Hassan Awadallah, and Susan Dumais. 2021. Meta label correction for noisy label learning. In Proceedings of the 35th AAAI Conference on Artificial Intelligence.
Shuyan Zhou, Shruti Rijhwani, and Graham Neubig. 2019. Towards zero-resource cross-lingual entity linking. arXiv preprint arXiv:1909.13180.
We use a maximum sequence length of 200 and 256 for NER and AS respectively. We use a bottleneck dimension of  $r = 384$  and  $r = 192$  for the representation transformation network, same as Pfeiffer et al. (2020c). During the bi-level optimization process, we adopt a learning rate of 3e-05 for training the main architecture and tuned the learning rate on 3e-5, 1e-6 and 1e-7 for training the representation transformation network. We use a batch size of 16 for NER and 12 for AS, and train 20 epochs for each experiment on both tasks. We use a single NVIDIA Tesla V100 with a 32G memory size for each experiment. For each language, we pick the best model according to the validation performance after each epoch.
B Detailed Results on Each Language
B.1 Source Data Size
The full results of using 10k and 20k English examples as transfer data are presented in Table 9.
B.2 Placement of RTN
The full results of placing the representation transformation network at different layers are presented in Table 10.
B.3 Joint Training w/ RTN
The full results of joint training with the representation transformation network are presented in Table 11.
C Additional Results on mBERT