SlowGuess/NAACL_2021_OCR huggingface dataset:
Meryem M'hamdi $^{1*}$ , Doo Soon Kim $^{2}$ , Franck Dernoncourt $^{2}$ , Trung Bui $^{2}$ , Xiang Ren $^{1}$  and Jonathan May $^{1}$
{meryem, xiangren, jonmay} @isi.edu
{dkim, franck.dernoncourt, bui}  $@$  adobe.com
Multilingual models, such as M-BERT and XLM-R, have gained increasing popularity, due to their zero-shot cross-lingual transfer learning capabilities. However, their generalization ability is still inconsistent for typologically diverse languages and across different benchmarks. Recently, meta-learning has garnered attention as a promising technique for enhancing transfer learning under low-resource scenarios: particularly for cross-lingual transfer in Natural Language Understanding (NLU).
In this work, we propose X-METRA-ADA, a cross-lingual MEta-TRAnswer learning ADAptation approach for NLU. Our approach adapts MAML, an optimization-based meta-learning approach, to learn to adapt to new languages. We extensively evaluate our framework on two challenging cross-lingual NLU tasks: multilingual task-oriented dialog and typologically diverse question answering. We show that our approach outperforms naive fine-tuning, reaching competitive performance on both tasks for most languages. Our analysis reveals that X-METRA-ADA can leverage limited data for faster adaptation.
Cross-lingual transfer learning is a technique used to adapt a model trained on a downstream task in a source language to directly generalize to the task in new languages. It aims to come up with common cross-lingual representations and leverages them to bridge the divide between resources to make any NLP application scale to multiple languages. This is particularly useful for data-scarce scenarios, as it reduces the need for API calls implied by machine translation or costly task-specific annotation for new languages.
Figure 1: An overview of the X-METRA-ADA framework: we use English as the source and Spanish as the target language. The meta-train stage transfers from the source to the target languages, while the meta-adaptation further adapts the model to the target language. The application is few-shot if the test language is seen in any stage of X-METRA-ADA; or zero-shot if the test language is unseen.
Transformer-based contextualized embeddings and their multilingual counterparts such as M-BERT (Devlin et al., 2019) have become popular as off-the-shelf representations for cross-lingual transfer learning. While these multilingual representations exhibit some cross-lingual capability even for languages with low lexical overlap with English, the transfer quality is reduced for languages that exhibit different typological characteristics (Pires et al., 2019).
The generalization of such representations has been extensively evaluated on traditional tasks such as Part-of-Speech (POS) tagging, Named Entity Recognition (NER) and Cross-lingual Document Classification (CLDC) (Ahmad et al., 2019; Wu and Dredze, 2019; Bari et al., 2020a; Schwenk and Li, 2018), with ever-growing open community annotation efforts like Universal Dependencies (Nivre et al., 2020) and CoNLL shared tasks (Tjong Kim Sang, 2002; Tjong Kim Sang and De Meulder, 2003). On the other hand, cross-lingual Natural Language Understanding (NLU)
*Work was started while the first author was a research intern at Adobe.
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3617-3632 June 6-11, 2021. ©2021 Association for Computational Linguistics
tasks have gained less attention, with smaller benchmark datasets that cover a handful of languages and don't truly model linguistic variety (Conneau et al., 2018; Artetxe et al., 2020). Natural Language Understanding tasks are critical for dialog systems, as they make up an integral part of the dialog pipeline. Understanding and improving the mechanism behind cross-lingual transfer for natural language understanding in dialog systems require evaluations on more challenging and typologically diverse benchmarks.
Numerous approaches have attempted to build stronger cross-lingual representations on top of those multilingual models; however, most require parallel corpora (Wang et al., 2019; Lample and Conneau, 2019) and are biased towards high-resource and balanced setups. This fuels the need for a method that doesn't require explicit cross-lingual alignment for faster adaptation to low-resource setups.
Meta-learning, a method for "learning to learn", has found favor especially among the computer vision and speech recognition communities (Nichol et al., 2018; Triantafillou et al., 2020; Winata et al., 2020). Meta-learning has been used for machine translation (Gu et al., 2018), few-shot relation classification (Gao et al., 2019), and on a variety of GLUE tasks (Dou et al., 2019). Recently, Nooralahzadeh et al. (2020) apply the MAML (Finn et al., 2017) algorithm to cross-lingual transfer learning for XNLI (Conneau et al., 2018) and MLQA (Lewis et al., 2020), NLU tasks that are naturally biased towards machine translation-based solutions. Nooralahzadeh et al. are able to show improvement over strong multilingual models, including M-BERT. However, they mainly show the effects of meta-learning as a first step in a framework that relies on supervised finetuning, making it difficult to properly compare and contrast both approaches.
We study cross-lingual meta-transfer learning from a different perspective. We distinguish between meta-learning and fine-tuning and design systematic experiments to analyze the added value of meta-learning compared to naive fine-tuning. We also build our analysis in terms of more typologically diverse cross-lingual NLU tasks: Multilingual Task-Oriented Dialogue System (MTOD) (Schuster et al., 2019) and Typologically Diverse Question Answering (TyDiQA) (Clark et al., 2020). While XNLI is a clas
sification task, MTOD is a joint classification and sequence labelling task and is more typologically diverse. TyDiQA is not a classification task, but we show how meta-learning can be applied usefully to it. We also show greater performance improvements from meta-learning than fine-tuning on transfer between typologically diverse languages.
To the best of our knowledge, we are the first to conduct an extensive analysis applied to MTOD and TyDiQA to evaluate the quality of cross-lingual meta-transfer. Our contributions are three-fold:
- Proposing X-METRA-ADA, $^{1}$  a language-agnostic meta-learning framework (Figure 1), and extensively evaluating it.
- Applying X-METRA-ADA to two challenging cross-lingual and typologically diverse task-oriented dialog and QA tasks, which includes recipes for constructing appropriate meta-tasks (Section 2.3).
- Analyzing the importance of different components in cross-lingual transfer and the scalability of our approach across different k-shot and down-sampling configurations (Section 4.2).
We make use of optimization-based meta-learning on top of pre-trained models with two levels of adaptation to reduce the risk of over-fitting to the target language: (i) meta-training from the source language to the target language(s) (ii) meta-adaptation on the same target language(s) for more language-specific adaptation (Figure 1).
We apply our approach to two cross-lingual downstream tasks: MTOD (Section 2.1) and TyDiQA (Section 2.2). We start by describing the base architectures for both tasks, before explaining how they are incorporated into our meta-learning pipeline. Applying meta-learning to a task requires the construction of multiple 'pseudo-tasks', which are instantiated as pairs of datasets. We describe this construction for our downstream tasks in Section 2.3. Finally, we present our X-METRA-ADA algorithm (Section 2.4).
2.1 Multilingual Task-Oriented Dialog (MTOD)
Similar to the architecture in Castellucci et al. (2019), we model MTOD's intent classification and slot filling subtasks jointly. For that purpose, we
1We release our code at: github.com/ meryemmhamdi1/meta CROSS_nlu_qa.
Figure 2: Architecture of Base MTOD.
use a joint text classification and sequence labeling framework with feature representation based on Transformer (Vaswani et al., 2017). More specifically, given a multilingual pre-trained model, we use it to initialize the word-piece embeddings layer. Then, we add on top of it a text classifier to predict the intent from the [CLS] token representation and a sequence labeling layer in the form of a linear layer to predict the slot spans (in BIO annotation), as shown in Figure 2. We optimize parameters using the sum of both intent and CRF based slot losses.
2.2 Typologically Diverse Question Answering (TyDiQA)
Figure 3: Question Answering Base Model.
Inspired by Hu et al. (2020), we apply to TyDiQA the same architecture as the original BERT fine-tuning procedure for question answering on SQuAD (Devlin et al., 2019). Specifically, the input question (after prepending it with a [CLS] token) and the context are concatenated as a single packed sequence separated by a [SEP] token. Then, the embeddings of the context are fed to a linear layer plus a softmax to compute the probability that each token is the START or END of the answer. The whole architecture is fine-tuned by
optimizing for the joint loss over the START and END predictions. Any START and END positions that are outside of the scope of the context end up being truncated because of Transformer-based embeddings length limitations and are ignored during training. Figure 3 illustrates the architecture.
2.3 Psuedo-task Datasets
Meta-learning is distinguished from fine-tuning in that the former seeks an initialization point that is maximally useful to multiple downstream learning tasks, while the latter seeks to directly optimize a downstream 'child' task from the initialization point of a 'parent' task. To apply meta-learning to data scenarios that more closely fit fine-tuning, we construct multiple 'pseudo-tasks' by subsampling from parent and child task datasets. A pseudo-task is defined as a tuple  $T = (S, Q)$ , where each of  $S$  and  $Q$  are labeled samples. In the inner loops of meta-learning, the loss on  $Q$  from a model trained on  $S$  is used to adapt the initialization point (where  $Q$  and  $S$  are referred to as the query and support in meta-learning literature). Pseudo-tasks are constructed in such a way as to make them balanced and non-overlapping. We describe our approach for each task below.
2.3.1 MTOD Pseudo-task Construction
MTOD labeled data consists of a sentence from a dialogue along with a sentence-level intent label and subsequence slot labels. From the available data, we draw a number of task sets  $\mathcal{T}$ ; each  $T = (S,Q) \in \mathcal{T}$  consists of  $k$  intent and slot-labeled items per intent class in  $S$  and  $q$  items per class in  $Q$ . Although carefully arranged to have the same number of items per class per task in each of the support and the query sets, the same task splits are used for slot prediction as well. During meta-training and meta-adaptation, task batches are sampled randomly from  $\mathcal{T}$ .
2.3.2 QA Pseudo-task Construction
Unlike MTOD, QA is not a standard classification task with fixed classes; thus, it is not directly amenable to class distribution balancing across pseudo-task query and support sets. To construct pseudo-tasks for QA from the available (question, context, answer) span triplet data, we use the following procedure: We draw a task  $T = (S, Q)$ , by first randomly drawing  $q$  triplets, forming  $Q$ . For each triplet  $t$  in  $Q$ , we draw the  $k/q$  most similar triplets to  $t$  from the remaining available data,
thus forming  $S$ . For two triplets  $t_1, t_2$  we define similarity as  $\cos(f(t_1), f(t_2))$ , where  $f(.)$  is a representation of the concatenation of the triplet elements delimited by a space; we use a cross-lingual extension to SBERT's pre-trained model (Reimers and Gurevych, 2019, 2020).
2.3.3 Cross-lingual extension
In the original MAML (Finn et al., 2017), in every iteration we sample a task set  $\mathcal{T}$  from a single distribution  $\mathcal{D}$ , and the support and query sets in a single task  $T$  would be drawn from a common space. We distinguish between the distributions  $\mathcal{D}_{\mathrm{meta - train}}$  and  $\mathcal{D}_{\mathrm{meta - adapt}}$ , which correspond to the two levels of adaptation introduced in Section 2 and explained below in Section 2.4.
To enable cross-lingual transfer, we draw data for the support set of tasks in  $\mathcal{D}_{\mathrm{meta - train}}$  from task data in the high-resource base language (English, in our experiments). For the query set in  $\mathcal{D}_{\mathrm{meta - train}}$  and for both support and query sets in  $\mathcal{D}_{\mathrm{meta - adapt}}$ , we sample from task data in the language to be evaluated.
2.4 X-METRA-ADA Algorithm
Following the notation described in the above sections, we present our algorithm X-METRA-ADA, our adaptation of MAML to cross-lingual transfer learning in two stages. In each stage we use the procedure outlined in Algorithm 1. We start by sampling a batch of tasks from distribution  $\mathcal{D}$ . For every task  $T_{j} = (S_{j},Q_{j})$ , we update  $\theta_{j}$  over  $n$  steps using batches drawn from  $S_{j}$ . At the end of this inner loop, we compute the gradients with respect to the loss of  $\theta_{j}$  on  $Q_{j}$ . At the end of all tasks of each batch, we sum over all pre-computed gradients and update  $\theta$ , thus completing one outer loop. The difference between meta-train and meta-adapt stages comes down to the parameters and hyperparameters passed into Algorithm 1.
- Meta-train: This stage is similar to classical MAML. Task sets are sampled from  $\mathcal{D}_{\mathrm{meta - train}}$  which uses high-resource (typically English) data in support sets and low-resource data in the query sets. The input model  $\theta_B$  is typically a pretrained multilingual downstream base model, and we use hyperparameters  $n = 5, \alpha = 1\mathrm{e} - 3$  and  $\beta = 1\mathrm{e} - 2$  for MTOD and  $\alpha = \beta = 3\mathrm{e} - 5$  for QA.
Algorithm 1 X-METRA-ADA
Require: Task set distribution D, pre-trained learner B with parameters θB, meta-learner M with parameters (θ, α, β, n)
1: Initialize θ← θB
2: while not done do
3: Sample batch of tasks T = {T1, T2, ... Tb} ∼ D
4: for all Tj = (Sj, Qj) in T do
5: Initialize θj← θ
6: for t = 1...n do
7: Evaluate ∂Bθj/∂θj = ∇θjLStj(Bθj)
8: Update θj = θj - α ∂Bθj/∂θj
9: end for
10: Evaluate query loss LQjTj(Bθj) and save it for outer loop
11: end for
12: Update θ← θ - β∇θ ∑j=1b LQjTj(Bθj)
13: end while
- Meta-adapt: During this stage, we ensure the model knows how to learn from examples within the target language under a low-resource regime. Task sets are sampled from  $\mathcal{D}_{\mathrm{meta-adapt}}$ , which uses low-resource data in both support and query sets. The input model is the optimization resulting from meta-train, and we use hyperparameters  $n = 5, \alpha = 1\mathrm{e} - 3$  and  $\beta = 1\mathrm{e} - 2$  for MTOD and  $\alpha = \beta = 3\mathrm{e} - 5$  for QA.
For dialogue intent prediction, we use the Multilingual Task-Oriented Dialogue (MTOD) (Schuster et al., 2019) dataset. MTOD covers 3 languages (English, Spanish, and Thai), 3 domains (alarm, reminder, and weather), 12 intent types, and 11 slot types. $^{3}$  We train models with the English training data (Train) but for the other languages we use the provided development sets (Dev) to further our goals to analyze methods of few-shot transfer. We evaluate on the provided test sets. Moreover, we evaluate on an in-house dataset of 7 languages. $^{4}$
For QA, we use the Typologically Diverse QA (TyDiQA-GoldP) (Clark et al., 2020) dataset. TyDiQA is a typologically diverse question answering dataset covering 11 languages. Like Hu et al. (2020), we use a simplified version of the primary task. Specifically, we discard questions that don't have an answer and use only the gold passage as context, keeping only the short answer and its spans. This makes the task similar to XQuAD and MLQA,
Thus  $k$  is constrained to be a multiple of  $q$ .
3We follow the same pre-processing and evaluation as Liu et al. (2020).
4 More details are included in the Appendix C.
although unlike these tasks, the questions are written without looking at the answers and without machine translation. As with MTOD, we use the English training data as Train. Since development sets are not specified for MTOD, we instead reserve  $10\%$  of the training data in each of the other languages as Dev. We report on the provided test sets. Statistics of datasets for both tasks can be found in Appendix A.
In order to fairly and consistently evaluate our approach to few-shot transfer learning via meta-learning and to ablate components of the method, we design a series of experiments based on both internal and external baselines. Our internal baselines ablate the effect of the X-METRA-ADA algorithm vs. conventional fine-tuning from a model trained on a high-resource language by keeping the data sets used for training constant. As our specific data conditions are not reproduced in any externally reported results on these tasks, we instead compare to other reported results using English-only or entirely zero-shot training data.
Internal Evaluation We design the following fine-tuning/few-shot schemes:
- PRE: An initial model is fine-tuned on the Train split of English only and then evaluated on new languages with no further tuning or adaptation. This strawman baseline has exposure to English task data only.
- MONO: An initial model is fine-tuned on the Dev split of the target language. This baseline serves as a comparison for standard fine-tuning (FT, below), which shows the value of combining MONO and PRE.
-  $FT$ : We fine-tune the PRE model on the Dev split of the target language. This is a standard transfer learning approach that combines PRE and MONO.
-  $FT$  w/EN: Like FT, except both the Dev split of the target language and the Train split of English are used for fine-tuning. This is used for dataset equivalence with X-METRA-ADA (below).
- X-METRA: We use the PRE model as  $\theta_B$  for meta-train, the Train split from English to form support sets in  $D_{meta-train}$ , and all of the Dev split of the target language to form query sets in  $D_{meta-train}$ .
- X-METRA-ADA: We use the PRE model as  $\theta_{B}$  for meta-train, the Train split from English to form support sets in  $\mathcal{D}_{\mathrm{meta - train}}$ . For MTOD, we use  $75\%$  of the Dev split of the target language to form query sets in  $\mathcal{D}_{\mathrm{meta - train}}$ . We use the remaining  $25\%$  of the Dev split of the target language for both the support and query sets of  $\mathcal{D}_{\mathrm{meta - adapt}}$ . For QA, we use ratios of  $60\%$  for  $\mathcal{D}_{\mathrm{meta - train}}$  and  $40\%$  for  $\mathcal{D}_{\mathrm{meta - adapt}}$ .
All models are ultimately fine-tuned versions of BERT and all have access to the same task training data relevant for their variant. That is, X-METRA-ADA and PRE both see the same English Train data and MONO, FT, and X-METRA-ADA see the same target language Dev data. However, since X-METRA-ADA uses both Train and Dev to improve upon PRE, and FT only uses Dev, we make an apples-to-apples comparison, data-wise, by including FT w/EN experiments as well.
External Baselines We focus mainly on transfer learning baselines from contextualized embeddings for a coherent external comparison; supervised experiments on target language data such as those reported in Schuster et al. (2019) are inappropriate for comparison because they use much more in-language labeled data to train. The experiments we compare to are zero-shot in the sense that they are not trained directly on the language-specific task data. However, most of these external baselines involve some strong cross-lingual supervision either through cross-lingual alignment or mixed-language training. We also include machine translation baselines, which are often competitive and hard to beat. Our work, by contrast, uses no parallel language data or resources beyond pretrained multilingual language models, labeled English data, and few-shot labeled target language data. To the best of our knowledge, we are the first to explore cross-lingual meta-transfer learning for those benchmarks, so we only report on our X-METRA-ADA approach in addition to those baselines.
For MTOD, then, we focus on the following external baselines:
Cross-lingual alignment-based approaches: We use MCoVe, a multilingual version of contextualized word vectors with an autoencoder objective as reported by Schuster et al. (2019) in addition to M-BERT (Liu et al., 2020). We also include XLM trained on Translation Language Modeling (TLM) + Masked Language Modeling (MLM)
(Lample and Conneau, 2019) as enhanced by Transformer and mixed-training as reported by Liu et al. (2020).
- Mixed-language training approaches: We use M-BERT + Transformer + mixed training using data from the dialogue domain: from (a) human-based word selection  $(\underline{\mathrm{MLT}}_H)$  and (b) attention-based word selection  $(\underline{\mathrm{MLT}}_A)$ , both are reported by Liu et al. (2020).
- Translation-based approaches: We use the zero-shot version of MMTE, the massively multilingual translation encoder by Siddhant et al. (2020) fine-tuned on intent classification. We also include Translate Train (TTrain) (Schuster et al., 2019), which translates English training data into target languages to train on them in addition to the target language training data.
For TyDiQA-GoldP, out of the already mentioned baselines, we use M-BERT, XLM, MMTE, and TTrain (which unlike (Schuster et al., 2019) only translates English to the target language to train on it without data augmentation). In addition to that we also include XLM-R as reported by Hu et al. (2020).
We use M-BERT (bert-base-multilingual-cased) with 12 layers as initial models for MTOD and TyDiQA-GoldP in our internal evaluation. We use xlm-r-distilroberta-base-paraphrase-v1 model for computing similarities when constructing the QA meta-dataset (Section 2.3.2).
Our implementation of X-METRA-ADA from scratch uses learn2learn (Arnold et al., 2020) for differentiation and update rules in the inner loop. We use the first-order approximation option in learn2learn for updating the outer loop, also introduced in Finn et al. (2017). For each model, we run for 3 to 4 different random initializations (for some experiments like PRE for TyDiQA-GoldP we use only 2 seeds respectively) and report the average and standard deviation of the best model for the few-shot language for each run. We use training loss convergence as a criteria for stopping. For the FT and MONO baselines, we don't have the luxury of Dev performance, since those baselines use the
Dev dataset for training. The Dev set is chosen to simulate a low-resource setup. More details on the hyperparameters used can be found in Appendix B.
4.1 Zero-shot and Few-shot Cross-Lingual NLU and QA
Model	Spanish	Thai
Intent Acc	Slot F1	Intent Acc	Slot F1
External Baselines
MCoVe†	53.9	19.3	70.7	35.6
M-BERT‡	73.7	51.7	28.1	10.6
MLTh	82.9	74.9	53.8	26.1
MLTΔ	87.9	73.9	73.5	27.1
XLM‡	87.5	68.5	72.6	27.9
MMTE+	93.6	-	89.6	-
TTrain‡	85.4	72.9	95.9	55.4
Zero-shot Learning
PRE	70.2	38.2	45.4	12.5
Few-shot Learning
MONO	82.4 ±6.0	43.9 ±1.5	79.1 ±4.7	54.1 ±3.9
FT	90.7 ±0.3	67.6 ±1.3	78.9 ±0.2	66.0 ±2.1
FT w/EN	88.7 ±0.4	67.4 ±1.4	73.7 ±0.1	66.0 ±1.6
X-METRA	89.6 ±1.3	63.6 ±0.5	80.2 ±1.2	70.4 ±1.2
X-METRA-ADA	92.9 ±0.6	60.9 ±1.9	86.3 ±1.7	69.6 ±1.9