SlowGuess/NAACL_2021_OCR huggingface dataset:
Mengzhou Xia\* Guoqing Zheng\* Subhabrata Mukherjee\* Milad Shokouhi\* Graham Neubig\* Ahmed Hassan Awadallah
$^{\S}$ Princeton University  $\dagger$ Carnegie Mellon University  $\ddagger$ Microsoft Research mengzhou@princeton.edu, {zheng, submukhe, milads} @ microsoft.com gneubig@cs.cmu.edu, hassanam@microsoft.com
The combination of multilingual pre-trained representations and cross-lingual transfer learning is one of the most effective methods for building functional NLP systems for low-resource languages. However, for extremely low-resource languages without large-scale monolingual corpora for pre-training or sufficient annotated data for fine-tuning, transfer learning remains an under-studied and challenging task. Moreover, recent work shows that multilingual representations are surprisingly disjoint across languages (Singh et al., 2019), bringing additional challenges for transfer onto extremely low-resource languages. In this paper, we propose MetaXL, a meta-learning based framework that learns to transform representations judiciously from auxiliary languages to a target one and brings their representation spaces closer for effective transfer. Extensive experiments on real-world low-resource languages - without access to large-scale monolingual corpora or large amounts of labeled data - for tasks like cross-lingual sentiment analysis and named entity recognition show the effectiveness of our approach. Code for MetaXL is publicly available at github.com/microsoft/MetaXL.
Recent advances in multilingual pre-trained representations have enabled success on a wide range of natural language processing (NLP) tasks for many languages. However, these techniques may not readily transfer onto extremely low-resource languages, where: (1) large-scale monolingual corpora are not available for pre-training and (2) sufficient labeled data is lacking for effective finetuning for downstream tasks. For example, multilingual BERT (mBERT) (Devlin et al., 2018) is pre-trained on 104 languages with many articles on
(a)
Figure 1: First two principal components of sequence representations (corresponding to [CLS] tokens) of Telugu and English examples from a jointly fine-tuned mBERT and a MetaXL model for the task of sentiment analysis. MetaXL pushes the source (EN) and target (TEL) representations closer to realize a more effective transfer. The Hausdorff distance between the source and target representations drops from 0.57 to 0.20 with F1 score improvement from 74.07 to 78.15.
Wikipedia and XLM-R (Conneau et al., 2020) is pre-trained on 100 languages with CommonCrawl Corpora. However, these models still leave behind more than 200 languages with few articles available in Wikipedia, not to mention the 6,700 or so languages with no Wikipedia text at all (Artetxe et al., 2020). Cross-lingual transfer learning for these extremely low-resource languages is essential for better information access but under-studied in practice (Hirschberg and Manning, 2015). Recent work on cross-lingual transfer learning using pre-trained representations mainly focuses on transferring across languages that are already covered by existing representations (Wu and Dredze, 2019). In contrast, existing work on transferring to languages without significant monolingual resources tends to be more sparse and typically focuses on specific tasks such as language modeling (Adams et al., 2017) or entity linking (Zhou et al., 2019).
Building NLP systems in these settings is challenging for several reasons. First, a lack of sufficient annotated data in the target language prevents effective fine-tuning. Second, multilingual
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 499-511 June 6-11, 2021. ©2021 Association for Computational Linguistics
pre-trained representations are not directly transferable due to language disparities. Though recent work on cross-lingual transfer mitigates this challenge, it still requires a sizeable monolingual corpus to train token embeddings (Artetxe et al., 2019). As noted, these corpora are difficult to obtain for many languages (Artetxe et al., 2020).
Additionally, recent work (Singh et al., 2019) shows that contextualized representations of different languages do not always reside in the same space but are rather partitioned into clusters in multilingual models. This representation gap between languages suggests that joint training with combined multilingual data may lead to sub-optimal transfer across languages. This problem is further exacerbated by the, often large, lexical and syntactic differences between languages with existing pre-trained representations and the extremely low-resource ones. Figure 1(a) provides a visualization of one such example of the disjoint representations of a resource-rich auxiliary language (English) and resource-scarce target language (Telugu).
We propose a meta-learning based method, MetaXL, to bridge this representation gap and allow for effective cross-lingual transfer to extremely low-resource languages. MetaXL learns to transform representations from auxiliary languages in a way that maximally facilitates transfer to the target language. Concretely, our meta-learning objective encourages transformations that increase the alignment between the gradients of the source-language set with those of a target-language set. Figure 1(b) shows that MetaXL successfully brings representations from seemingly distant languages closer for more effective transfer.
We evaluate our method on two tasks: named entity recognition (NER) and sentiment analysis (SA). Extensive experiments on 8 low-resource languages for NER and 2 low-resource languages for SA show that MetaXL significantly improves over strong baselines by an average of 2.1 and 1.3 F1 score with XLM-R as the multilingual encoder.
2 Meta Representation Transformation
2.1 Background and Problem Definition
The standard practice in cross-lingual transfer learning is to fine-tune a pre-trained multilingual language model  $f_{\theta}$  parameterized by  $\theta$ , (e.g. XLM-R and mBERT) with data from one or more auxiliary
languages  ${}^{1}$  and then apply it to the target language. This is widely adopted in the zero-shot transfer setup where no annotated data is available in the target language. The practice is still applicable in the few-shot setting, in which case a small amount of annotated data in the target language is available.
In this work, we focus on cross-lingual transfer for extremely low-resource languages where only a small amount of unlabeled data and task-specific annotated data are available. That includes languages that are not covered by multilingual language models like XLM-R (e.g., Maori or Turkmen), or low-resource languages that are covered but with many orders of magnitude less data for pre-training (e.g., Telegu or Persian). We assume the only target-language resource we have access to is a small amount of task-specific labeled data.
More formally, given: (1) a limited amount of annotated task data in the target language, denoted as  $\mathcal{D}_t = \{(x_t^{(i)},y_t^{(i)});i\in [1,N]\}$ , (2) a larger amount of annotated data from one or more source language(s), denoted as  $\mathcal{D}_s = \{(x_s^{(j)},y_s^{(j)});j\in [1,M]\}$  where  $M\gg N$  and (3) a pre-trained model  $f_{\theta}$ , which is not necessarily trained on any monolingual data from the target language - our goal is to adapt the model to maximize the performance on the target language.
When some target language labeled data is available for fine-tuning, a standard practice is to jointly fine-tune (JT) the multilingual language model using a concatenation of the labeled data from both the source and target languages  $\mathcal{D}_s$  and  $\mathcal{D}_t$ . The representation gap (Singh et al., 2019) between the source language and target language in a jointly trained model brings additional challenges, which motivates our proposed method.
2.2 Representation Transformation
The key idea of our approach is to explicitly learn to transform source language representations, such that when training with these transformed representations, the parameter updates benefit performance on the target language the most. On top of an existing multilingual pre-trained model, we introduce an additional network, which we call the representation transformation network to model this transformation explicitly.
The representation transformation network models a function  $g_{\phi}:\mathbb{R}^{d}\to \mathbb{R}^{d}$ , where  $d$  is the di
<sup>1</sup>We also refer to auxiliary languages as source languages as opposed to target languages.
Figure 2: Overview of MetaXL. For illustration, only two Transformer layers are shown for XLM-R, and the representation transformation network is placed after the first Transformer layer. ① source language data passes through the first Transformer layer, through the current representation transformation network, and finally through the remaining layers to compute a training loss with the corresponding source labels. ② The training loss is backpropagated onto all parameters, but only parameters of XLM-R are updated. The updated weights of XLM-R are a function of the current representation transformation network due to gradient dependency (highlighted by the light-purple background of the updated XLM-R). ③ A batch of target language data passes through the updated XLM-R and the meta loss is evaluated with the corresponding labels. ④ The meta loss is back-propagated into the representation transformation network, since the meta-loss is in effect a function of weights from that network, and only the representation transformation network is updated.
Repeat following steps until convergence:
mension of the representations. Conceptually, any network with proper input and output sizes is feasible. We opt to employ a two-layer feed-forward network, a rather simple architecture with the intention to avoid heavy parameter overhead on top of the pre-trained model. The input to the representation transformation network is representations from any layer of the pre-trained model. By denoting representations from layer  $i$  as  $h_i \in \mathbb{R}^d$ , we have a parameterized representation transformation network as follows:
$$
g _ {\phi} \left(h _ {i}\right) = w _ {2} ^ {T} \left(\operatorname {R e L U} \left(w _ {1} ^ {T} h _ {i} + b _ {1}\right)\right) + b _ {2} \tag {1}
$$
where  $\phi = \{w_1, w_2, b_1, b_2 | w_1 \in \mathbb{R}^{d \times r}, w_2 \in \mathbb{R}^{r \times d}, b_1 \in \mathbb{R}^r, b_2 \in \mathbb{R}^d\}$  is the set of parameters of the representation transformation network. In practice, we set  $r$  to be bottlenecked, i.e.  $r < d$ , so the representation transformation network first compresses the input representation and then projects back onto the original dimension of the input representation.
As shown in Figure 2, by assuming that the base model has  $N$  layers, a source example  $(x_{s},y_{s})\in D_{s}$  passes through the first  $i$  layers, then through the representation transformation network, finally through the last  $N - i$  layers of the base model. We denote the final logits of this batch as  $f(x_{s};\theta ,\phi)$
encoded by both the base model and the representation transformation network. In contrast, for a target example  $x_{t}, y_{t} \in D_{t}$ , we only pass it through the base model as usual, denoted as  $f(x_{t};\theta)$ .
Ideally, suppose that we have a representation transformation network that could properly transform representations from a source language to the target language. In that case, the source data can be almost equivalently seen as target data on a representation level. Unfortunately, we cannot train such a representation transformation network in a supervised manner without extensive parallel data.
Architecturally, the representation transformation network adopts a similar structure to existing works on language and task adapters for cross-lingual and multi-task transfer (Pfeiffer et al., 2020b), a simple down- and up-projection of input representations. Nevertheless, beyond network architecture, the goal and training procedure of the two approaches are significantly different. Adapters are typically trained to encode task or language-specific information by fixing the rest of the model and updating the parameters of the adapters only. Adapters allow training parameter-efficient models that could be flexibly adapted to multiple languages and tasks. While in our proposed method, we use the representation trans
Algorithm 1 Training procedure for MetaXL
Input: Input data from the target language  $D_{t}$  and the source language  $D_{s}$
1: Initialize base model parameters  $\theta$  with pretrained XLM-R weights, initialize parameters of the representation transformation network  $\phi$  randomly
2: while not converged do
3: Sample a source batch  $(x_{s},y_{s})$  from  $D_{s}$  and a target batch  $(x_{t},y_{t})$  from  $D_{t}$ ;
4: Update  $\theta \colon \theta^{(t + 1)} = \theta^{(t)} - \alpha \nabla_{\theta}\mathcal{L}(x_s;\theta^{(t)},\phi^{(t)})$
5: Update  $\phi \colon \phi^{(t + 1)} = \phi^{(t)} - \beta \nabla_{\phi}\mathcal{L}(x_t;\theta^{(t)} - \alpha \nabla_\theta \mathcal{L}(x_s;\theta^{(t)},\phi^{(t)}))$
6: end while
fer network at training time to adjust the training dynamics to maximally improve test-time performance on the target language. The optimization procedure and the function of the representation transformation network will be discussed in more detail in the next section.
The training of the representation transformation network conforms to the following principle: If the representation transformation network  $g_{\phi}$  effectively transforms the source language representations, such transformed representations  $f(x_{s};\phi ,\theta)$  should be more beneficial to the target task than the original representations  $f(x_{s};\theta)$ , such that the model achieves a smaller evaluation loss  $\mathcal{L}_{\mathcal{D}_t}$  on the target language. This objective can be formulated as a bi-level optimization problem:
$$
\min  _ {\phi} \mathcal {L} _ {\mathcal {D} _ {t}} \left(f \left(x _ {t}; \theta^ {*} (\phi)\right), y _ {t}\right) \tag {2}
$$
s.t.  $\theta^{*}(\phi) = \arg \min_{\theta}\mathcal{L}_{\mathcal{D}_{s}}(f(x_{s};\phi ,\theta),y_{s})$
where  $\mathcal{L}(\cdot)$  is the task loss function. In this bi-level optimization, the parameters  $\phi$  of the representation transformation network are the meta parameters, which are only used at training time and discarded at test time. Exact solutions require solving for the optimal  $\theta^{*}$  whenever  $\phi$  gets updated. This is computationally infeasible, particularly when the base model  $f$  is complex, such as a Transformer-based language model. Similar to existing work involving such optimization problems (Finn et al., 2017; Liu et al., 2019; Shu et al., 2019; Zheng et al., 2021), instead of solving the optimal  $\theta^{*}$  for any given  $\phi$ , we adopt a one-step stochastic gradient descent update for  $\theta$  as an estimate to the optimal base model for a given  $\phi$ :
$$
\theta^ {\prime} = \theta - \alpha \nabla_ {\theta} \mathcal {L} _ {\mathcal {D} _ {s}} (f (x _ {s}; \phi , \theta), y _ {s}) \tag {3}
$$
where  $\mathcal{L}_{\mathcal{D}_s}(x_s;\cdot)$  is the loss function of the lower problem in Equation 2 and  $\alpha$  is the corresponding
learning rate. Note that the resulting  $\theta^{\prime}$  is in effect a function of  $\phi$ . We then evaluate the updated weights  $\theta^{\prime}$  on data  $x_{t}$  from the target language for updating  $g_{\phi}$ :
$$
\phi^ {\prime} = \phi - \beta \nabla_ {\phi} \mathcal {L} _ {\mathcal {D} _ {t}} \left(f \left(x _ {t}; \theta^ {\prime}\right), y _ {t}\right) \tag {4}
$$
where  $\mathcal{L}_{\mathcal{D}_t}(x_t;\cdot)$  is the loss function of the upper problem in Equation 2 and  $\beta$  is its corresponding learning rate. Note that the meta-optimization is performed over the parameters of the representation transformation network  $g_{\phi}$  whereas the objective is calculated solely using the updated parameters of the main architecture  $\theta^{\prime}$ . By plugging Equation 3 into Equation 4, we can further expand the gradient term  $\nabla_{\phi}\mathcal{L}(f(x_t;\theta^{\prime}),y_t)$ . We omit  $f$  and  $y$  in the following derivative for simplicity.
$$
\begin{array}{l} \nabla_ {\phi} \mathcal {L} _ {\mathcal {D} _ {t}} (x _ {t}; \theta^ {\prime}) \\ = \nabla_ {\phi} \mathcal {L} _ {\mathcal {D} _ {t}} \left(x _ {t}; \theta - \alpha \nabla_ {\theta} \mathcal {L} _ {\mathcal {D} _ {s}} \left(x _ {s}; \theta , \phi\right)\right) \\ = - \alpha \nabla_ {\phi , \theta} ^ {2} \mathcal {L} _ {\mathcal {D} _ {s}} (x _ {s}; \theta , \phi) \nabla_ {\theta} \mathcal {L} _ {\mathcal {D} _ {t}} (x _ {t}; \theta^ {\prime}) \\ = - \alpha \nabla_ {\phi} \left(\nabla_ {\theta} \mathcal {L} _ {\mathcal {D} _ {s}} \left(x _ {s}; \theta , \phi\right) ^ {T} \nabla_ {\theta} \mathcal {L} _ {\mathcal {D} _ {t}} \left(x _ {t}; \theta^ {\prime}\right)\right) \\ \end{array}
$$
During training, we alternatively update  $\theta$  with Equation 3 and  $\phi$  with Equation 4 until convergence. We term our method MetaXL, for its nature to leverage Meta-learning for extremely low-resource cross(X)-Linguual transfer. Both Figure 2 and Algorithm 1 outline the procedure for training MetaXL.
We conduct experiments on two diverse tasks, namely, sequence labeling for Named Entity Recognition (NER) and sentence classification task for Sentiment Analysis (SA). For the NER task, we use the cross-lingual Wikiann dataset (Pan et al., 2017). For the sentiment analysis task, we use the English portion of Multilingual Amazon Reviews
Language	Code	Language Family	Related Language
Quechua	qu	Quechua	Spanish
Min Dong	cdo	Sino-Tibetan	Chinese
Ilocano	ilo	Austronesian	Indonesian
Mingrelian	xmf	Kartvelian	Georgian
Meadow Mari	mhr	Uralic	Russian
Maori	mi	Austronesian	Indonesian
Turkmen	tk	Turkic	Turkish
Guarani	gn	Tupian	Spanish
Table 1: Target language information on the NER task. The data set size of the these languages is 100.
Corpus (MARC) (Keung et al., 2020) as the high-resource language and product review datasets in two low-resource languages, Telugu and Persian (Gangula and Mamidi, 2018; Hosseini et al., 2018).
WikiAnn WikiAnn (Pan et al., 2017) is a multilingual NER dataset constructed with Wikipedia articles and anchor links. We use the train, development and test partitions provided in Rahimi et al. (2019). The dataset size ranges from 100 to 20k for different languages.
MARC The Multilingual Amazon Reviews Corpus (Keung et al., 2020) is a collection of Amazon product reviews for multilingual text classification. The dataset contains reviews in English, Japanese, German, French, Spanish, and Chinese with five-star ratings. Each language has 200k examples for training. Note that we only use its English dataset.
SentiPers SentiPers (Hosseini et al., 2018) is a sentiment corpus in Persian (fa) consisting of around 26k sentences of users' opinions for digital products. Each sentence has an assigned quantitative polarity from the set of  $\{-2, -1, 0, 1, 2\}$ .
Sentiraama Sentiraama (Gangula and Mamidi, 2018) is a sentiment analysis dataset in Telugu (tel), a language widely spoken in India. The dataset contains example reviews in total, labeled as either positive or negative.
Pre-processing For SA, we use SentiPers and Sentiraama as target language datasets and MARC as the source language dataset. To unify the label space, we curate MARC by assigning negative labels to reviews rated with 1 or 2 and positive labels to those rated with 4 or 5. We leave out neutral reviews rated with 3. For SentiPers, we assign negative labels to reviews rated with -1 and -2 and positive labels to those rated with 1 or 2. For SentiPers, though the dataset is relatively large, we
mimic the low-resource setting by manually constructing a train, development, and test set with 100, 1000, and 1000 examples through sampling. For Sentiraama, we manually split the dataset into train, development, and test subsets of 100, 103, and 100 examples.2
Base Model We use mBERT<sup>3</sup> (Devlin et al., 2018) and XLM-R (Conneau et al., 2020) as our base models, known as the state-of-the-art multilingual pre-trained model. However, our method is generally applicable to all types of Transformer-based language models.
Target Language For NER, we use the same 8 low-resource languages as Pfeiffer et al. (2020c), summarized in Table 1. These languages have only 100 examples in the WikiAnn dataset and are not included for pre-training XLM-R. For SA, Persian and Telugu are the target languages. For both tasks under any setting, we only use a fixed number of 100 examples for each target language.
Source Language The selection of source languages is crucial for transfer learning. We experiment with two choices source languages on NER: English and a related language to the target language. The related language was chosen based on LangRank (Lin et al., 2019), a tool for choosing transfer languages for cross-lingual learning. A list of related languages used for each target is shown in Table 1. In absence of training data that fit our related-language criteria for the low-resource target languages in SA, we use only English as the source language.
Tokenization For all languages, either pretrained with XLM-R or not, we use XLM-R's default tokenizer for tokenizing. We tried with the approach where we train subword tokenizers for unseen languages similar to Artetxe et al. (2020) but obtained worse results than using the XLM-R tokenizer as is, due to the extremely small scale of target language data. We conjecture that the subword vocabulary that XLM-R learns is also beneficial to encode languages on which it is not even
2Details of data splits can be found at github.com/ microsoft/MetaXL.
${}^{3}$  XLM-R as a base model leads to significantly better results for both baselines and MetaXL than mBERT, thus we mainly present results with XLM-R in the main text. Detailed results on mBERT can be found in Appendix C
Source	Method	qu	cdo	ilo	xmf	mhr	mi	tk	gn	average
(1)	-	target	57.14	37.72	61.32	59.07	55.17	76.27	55.56	48.89	56.39
(2)	English	JT	66.10	55.83	80.77	69.32	71.11	82.29	61.61	65.44	69.06
MetaXL	68.67	55.97	77.57	73.73	68.16	88.56	66.99	69.37	71.13
(3)	Related	JT	79.65	53.91	78.87	79.67	66.96	87.86	64.49	70.54	72.74
MetaXL	77.06	57.26	75.93	78.37	69.33	86.46	73.15	71.96	73.69
Table 2: F1 for NER across three settings where we, (1) only use the target language data; (2) use target language data along with 5k examples of English; (3) use the target language data along with 5k examples of a related language. JT stands for joint training and MetaXL stands for Meta Representation Transformation. We bold the numbers with a better average performance in each setting.
Method	tel	fa
(1)	target only	86.87	82.58
(2)	JT	88.68	85.51
MetaXL	89.52	87.14
Table 3: F1 for sentiment analysis on two settings using (1) only the target language data; (2) target language data along with 1k examples of English.
pre-trained on. We leave exploring the best tokenization strategy for leveraging pre-trained model on unseen language as future work.
NER We present results of NER in Table 2, where we use 5k examples from English or a related language as source data. When we only use the annotated data of the target language to fine-tune XLM-R (target), we observe that the performance varies significantly across languages, ranging from 37.7 to 76.3 F1 score. Jointly fine-tuning XLM-R with target and source data  $(JT)$  leads to a substantial average gain of around 12.6 F1 score. Using the same amount of data from a related language (instead of English) is more effective, showing an average improvement of 16.3 F1 score over using target data only. Our proposed method, MetaXL, consistently outperforms the joint training baselines, leading to a significant average gain of 2.07 and 0.95 F1 score when paired with English or related languages, respectively.
SA We present results on the task of SA in Table 3, where we use 1K examples from English as source language data. We find that auxiliary data from source languages brings less but still signifi