SlowGuess/NAACL_2022_OCR huggingface dataset:
hungyilee@ntu.edu.tw
Shang-Wen Li
Amazon AI*
shangwel@amazon.com
Ngoc Thang Vu
thangvu@ims.uni-stuttgart.de
Deep learning has been the mainstream technique in natural language processing (NLP) area. However, the techniques require many labeled data and are less generalizable across domains. Meta-learning is an arising field in machine learning studying approaches to learn better learning algorithms. Approaches aim at improving algorithms in various aspects, including data efficiency and generalizability. Efficacy of approaches has been shown in many NLP tasks, but there is no systematic survey of these approaches in NLP, which hinders more researchers from joining the field. Our goal with this survey paper is to offer researchers pointers to relevant meta-learning works in NLP and attract more attention from the NLP community to drive future innovation. This paper first introduces the general concepts of meta-learning and the common approaches. Then we summarize task construction settings and application of meta-learning for various NLP problems and review the development of meta-learning in NLP community.
Recently, deep learning (DL) based natural language processing (NLP) has been one of the research mainstreams and yields significant performance improvement in many NLP problems. However, DL models are data-hungry. The downside limits such models' application to different domains, languages, countries, or styles because collecting in-genre data for model training are costly.
To address the challenges, meta-learning techniques are gaining attention. Meta-learning, or Learning to Learn, aims to learn better learning algorithms, including better parameter initialization (Finn et al., 2017), optimization strategy (Andrychowicz et al., 2016; Ravi and Larochelle, 2017), network architecture (Zoph
and Le, 2017; Zoph et al., 2018; Pham et al., 2018a), distance metrics (Vinyals et al., 2016; Gao et al., 2019a; Sung et al., 2018), and beyond (Mishra et al., 2018). Meta-learning allows faster fine-tuning, converges to better performance, yields more generalizable models, and it achieves outstanding results for few-shot image classification (Triantafillou et al., 2020). The benefits alleviate the dependency of learning algorithms on labels and make model development more scalable. Image processing is one of the machine learning areas with abundant applications and established most of the examples in the previous survey papers on meta-learning (Hospedales et al., 2021; Huisman et al., 2021).
On the other hand, there are works showing benefits of meta-learning techniques in performance and data efficiency via applying meta-learning to NLP problems. Please refer to Tables 2 and 3 in the appendix for NLP applications improved by meta-learning. Tutorial (Lee et al., 2021b) and Workshop (Lee et al., 2021a) are organized at ACL 2021 to encourage exchange and collaboration among NLP researchers interested in these techniques. To facilitate more NLP researchers and practitioners benefiting from the advance of meta-learning and participating in the area, we provide a systematic survey of meta-learning to NLP problems in this paper. There is another survey paper on meta-learning in NLP (Yin, 2020). While Yin (2020) describes meta-learning methods in general, this paper focuses on the idea of making meta-learning successful when applied to NLP and provides a broader review of publications on NLP meta-learning. This paper is organized as below.
- A brief introduction of meta-learning backgrounds, general concepts, and algorithms in Section 2.
- Common settings for constructing meta-learning tasks in Section 3.
*Work done while working at Amazon Inc. The current affiliation is Meta AI.
Human Language Technologies, pages 666 - 684
- Adaptation of general meta-learning approaches to NLP problems in Section 4.
Meta-learning approaches for special topics, including knowledge distillation and life-long learning for NLP applications in Section 5.
Due to space constraints, we will not give too many detailed descriptions of general meta-learning techniques in this survey paper. For general concepts of meta-learning, we encourage readers to read the previous overview paper (Yin, 2020; Hospedales et al., 2021; Huisman et al., 2021).
2 Background Knowledge for Meta Learning
The goal of machine learning (ML) is to find a function  $f_{\theta}(x)$  parametrized by model parameters  $\theta$  for inference from training data. For machine translation (MT), the input  $x$  is a sentence, while  $f_{\theta}(x)$  is the translation of  $x$ ; for automatic speech recognitoin (ASR),  $x$  is an utterance, while  $f_{\theta}(x)$  is the transcription; In DL,  $\theta$  are the network parameters, or weights and biases of a network. To learn  $\theta$ , there is a loss function  $l(\theta; \mathcal{D})$ , where  $\mathcal{D}$  is a set of paired examples for training,
$$
\mathcal {D} = \left\{\left(x _ {1}, y _ {1}\right), \left(x _ {2}, y _ {2}\right), \dots , \left(x _ {K}, y _ {K}\right) \right\}, \tag {1}
$$
where  $x_{k}$  is function input,  $y_{k}$  is the ground truth, and  $K$  is the number of examples in  $\mathcal{D}$ . The loss function  $l(\theta ;\mathcal{D})$  is defined as below:
$$
l (\theta ; \mathcal {D}) = \sum_ {k = 1} ^ {K} d \left(f _ {\theta} \left(x _ {k}\right), y _ {k}\right). \tag {2}
$$
where  $d(f_{\theta}(x_k),y_k)$  is the "distance" between the function output  $f_{\theta}(x_k)$  and the ground truth  $y_{k}$ . For classification problem,  $d(.,.)$  can be cross-entropy; for regression, it can be L1/L2 distance. The following optimization problem is solved to find the optimal parameter set  $\theta^{*}$  for inference via minimizing the loss function  $l(\theta ;\mathcal{D})$ .
$$
\theta^ {*} = \arg \min  _ {\theta} l (\theta ; \mathcal {D}). \tag {3}
$$
In meta-learning, what we want to learn is a learning algorithm. The learning algorithm can also be considered as a function, denoted as  $F_{\phi}(.)$ . The input of  $F_{\phi}(.)$  is the training data, while the output of the function  $F_{\phi}(.)$  is the learned model parameters, or  $\theta^{*}$  in (3). The learning algorithm  $F_{\phi}(.)$  is parameterized by meta-parameters  $\phi$ , which is
what we want to learn in meta-learning. If  $F_{\phi}(.)$  represents gradient descent for deep network,  $\phi$  can be initial parameters, learning rate, network architecture, etc. Different meta-learning approaches focus on learning different components. For example, model-agnostic meta-learning (MAML) focuses on learning initial parameters (Finn et al., 2017), which will be further described in Section 4.1. Learning to Compare methods like Prototypical Network (Snell et al., 2017) in Section 4.2 learn the latent representation of the inputs and their distance metrics for comparison. Network architecture search (NAS) in Section 4.3 learns the network architecture (Zoph and Le, 2017; Zoph et al., 2018; Pham et al., 2018a).
To learn meta-parameters  $\phi$ , meta-training tasks  $\mathcal{T}_{train}$  are required.
$$
\mathcal {T} _ {\text {t r a i n}} = \left\{\mathcal {T} _ {1}, \mathcal {T} _ {2}, \dots , \mathcal {T} _ {N} \right\}, \tag {4}
$$
where  $\mathcal{T}_n$  is a task, and  $N$  is the number of tasks in  $\mathcal{T}_{train}$ . Usually, all the tasks belong to the same NLP problem; for example, all the  $\mathcal{T}_n$  are QA but from different corpora, but it is also possible that the tasks belong to various problems. Each task  $\mathcal{T}_n$  includes a support set  $S_n$  and a query set  $\mathcal{Q}_n$ . Both  $S_n$  and  $\mathcal{Q}_n$  are paired examples as  $\mathcal{D}$  in (1). The support set plays the role of training data in typical ML, while the query set can be understood as the testing data in typical ML. However, to not confuse the reader, we use the terms support and query sets in the context of meta-learning instead of training and testing sets.
In meta-learning, there is a loss function  $L(\phi; \mathcal{T}_{train})$ , which represents how "bad" a learning algohtm paramereized by  $\phi$  is on  $\mathcal{T}_{train}$ .  $L(\phi; \mathcal{T}_{train})$  is the performance over all the tasks in  $\mathcal{T}_{train}$ ,
$$
L (\phi ; \mathcal {T} _ {\text {t r a i n}}) = \sum_ {n = 1} ^ {N} l \left(\theta^ {n}; \mathcal {Q} _ {n}\right). \tag {5}
$$
The definition of the function  $l(\cdot)$  above is the same as in (2).  $l(\theta^n;\mathcal{Q}_n)$  for each task  $\mathcal{T}_n$  is obtained as below. For each task  $\mathcal{T}_n$  in  $\mathcal{T}_{train}$ , we use a support set  $S_{n}$  to learn a model by the learning algorithm  $F_{\phi}$ . The learned model is denoted as  $\theta^n$  where  $\theta^n = F_{\phi}(\mathcal{S}_n)$ . This procedure is equivalent to typical ML training. We called this step within-task training. Then  $\theta^n$  is evaluated on  $\mathcal{Q}_n$  to obtain  $l(\theta^n;\mathcal{Q}_n)$  in (5). We called this step within-task testing. One execution of within-task training and
followed by one execution of within-task testing is called an episode.
The optimization task below is solved to learn meta-parameters  $\phi$ .
$$
\phi^ {*} = \arg \min  _ {\phi} L (\phi ; \mathcal {T} _ {t r a i n}). \tag {6}
$$
If  $\phi$  is differentiable with respect to  $L(\phi; \mathcal{T}_{train})$ , then we can use gradient descent to learn meta-parameters; if not, we can use reinforcement learning algorithm or evolutionary algorithm. Solving (6) is called cross-task training in this paper, which usually involves running many episodes on meta-training tasks. To evaluate  $\phi^{*}$ , we need meta-testing tasks  $\mathcal{T}_{test}$ , tasks for evaluating algorithms parameterized by meta-parameters  $\phi^{*1}$ . We do cross-task testing on  $\mathcal{T}_{test}$ , that is, running an episode on each meta-testing task to evaluate algorithms parameterized by meta-parameters  $\phi^{*}$ .
In order to facilitate the reading of our paper, we summarize the most important terminologies and their meanings in Table 1 in the appendix.
3 Task Construction
In this section, we discuss different settings of constructing meta-training tasks  $\mathcal{T}_{train}$  and meta-testing tasks  $\mathcal{T}_{test}$ .
3.1 Cross-domain Transfer
A typical setting for constructing the tasks is based on domains (Qian and Yu, 2019; Yan et al., 2020; Li et al., 2020a; Park et al., 2021; Chen et al., 2020b; Huang et al., 2020a; Dai et al., 2020; Wang et al., 2021b; Dingliwal et al., 2021; Qian et al., 2021). In this setting, all the tasks, no matter belonging to  $\mathcal{T}_{train}$  or  $\mathcal{T}_{test}$ , are the same NLP problems. In each task  $\mathcal{T}_n$ , the support set  $S_n$  and the query set  $\mathcal{Q}_n$  are from the same domain, while different tasks contain the examples from different domains. In each task, the model is trained on the support set of a domain (usually having a small size) and evaluated on the query set in the same domain, which can be considered as domain adaptation. From the meta-training tasks  $\mathcal{T}_{train}$ , cross-task training finds meta-parameters  $\phi^*$  parameterizing the learning algorithm  $F_{\phi^*}$ . With a sufficient number of tasks in  $\mathcal{T}_{train}$ , cross-task training should find a suitable  $\phi^*$  for a wide range of domains, and thus also works
well on the tasks in  $\mathcal{T}_{test}$  containing the domains unseen during cross-task training. Hence, meta-learning can be considered as one way to improve domain adaptation. If the support set in each task includes only a few examples, the meta-learning has to find the meta-parameters  $\phi^{*}$  that can learn from a small support set and generalize well to the query set in the same domain. Therefore, meta-learning is considered one way to achieve few-shot learning.
The cross-domain setting is widespread. We only provide a few examples in this subsection. In MT, each meta-training task includes the documents from a specific domain (e.g., news, laws, etc.), while each meta-testing task also contains documents from one domain but not covered by the meta-training tasks (e.g., medical records) (Li et al., 2020a). For another example, both meta-training and meta-testing tasks are DST. The meta-training tasks include hotel booking, flight ticket booking, etc., while the testing task is taxi booking (Huang et al., 2020a; Wang et al., 2021b; Dingliwal et al., 2021). Domain has different meanings in different NLP problems. For example, in speech processing tasks, the domains can refer to accents (Winata et al., 2020b; Huang et al., 2021) or speakers (Klejch et al., 2019; Wu et al., 2021b; Huang et al., 2022).
3.2 Cross-lingual Transfer
If we consider different languages as different domains, then the cross-lingual transfer can be regarded as a special case of cross-domain transfer. Suppose each task contains the examples of an NLP problem from one language, and different tasks are in different languages. In this case, cross-task training finds meta-parameters  $\phi^{*}$  from the languages in  $\mathcal{T}_{train}$ , and cross-task testing evaluate the meta-parameters  $\phi^{*}$  on new languages in  $\mathcal{T}_{test}$ . This setting aims at finding the learning algorithm  $F_{\phi^{*}}(.)$  that works well on the NLP problem of any language given the support set of the language. Cross-language settings have been applied to NLI and QA in X-MAML (Nooralahzadeh et al., 2020), documentation classification (van der Heijden et al., 2021), dependency parsing (Langedijk et al., 2021), MT (Gu et al., 2018), and ASR (Hsu et al., 2020; Winata et al., 2020a; Chen et al., 2020d; Xiao et al., 2021).
For the meta-learning methods aiming at learning the initial parameters like MAML (will be intro
If the learning processing of  $\phi$  also involves some hyperparameter selection, then meta-validation tasks are needed, but in this paper, we ignore the discussion of meta-validation tasks for simplicity.
duced in Section 4.1), the network architecture used in all tasks must have the same network architecture. A unified network architecture across all tasks is not obvious in cross-lingual learning because the vocabularies in different tasks are different. Before multilingual pretrained models are available, unified word embeddings across languages are required. Gu et al. (2018) uses the universal lexical representation to overcome the input-output mismatch across different languages. Recently, by using multilingual pretrained models as encoders, such as M-BERT (Devlin et al., 2019) or XLMR (Conneau et al., 2020), all languages can share the same network architecture (Nooralahzadeh et al., 2020; van der Heijden et al., 2021).
3.3 Cross-problem Training
Here the meta-training and meta-testing tasks can come from different problems. For example, the meta-training tasks include MT and NLI, while the meta-testing tasks include QA and DST. The cross-problem setting is not usual, but there are still some examples. In Bansal et al. (2020a), the meta-training tasks are the GLUE benchmark tasks (Wang et al., 2018), while the meta-testing tasks are NLP problems, including entity typing, NLI, sentiment classification, and various other text classification tasks, not in the GLUE. All the meta-training and meta-testing tasks can be formulated as classification but with different classes. In Indurthi et al. (2020), the meta-training tasks are MT and ASR, while the meta-testing task is speech translation (ST). CrossFit is a benchmark corpus for this cross-problem setting (Ye et al., 2021).
The intrinsic challenge in the cross-problem setting is that different NLP problems may need very different meta-parameters in learning algorithms, so it may be challenging to find unified meta-parameters on the meta-training tasks that can generalize to meta-testing tasks. In addition, the meta-learning algorithms learning initial parameters such as MAML require all the tasks to have a unified network architecture. If different problems need different network architecture, then the original MAML cannot be used in the cross-problem setting. LEOPARD (Bansal et al., 2020a) and ProtoMAML (van der Heijden et al., 2021) are the MAML variants that can be used in the classification tasks with different class numbers. Both approaches use the data of a class to generate the class-specific head, so only the parameters of the
Figure 1: The task construction of cross-domain transfer in Section 3.1 and domain generalization in Section 3.4. Different colors represents data from different domains.
head parameter generation model are required. The head parameter generation model is shared across all classes, so the network architecture becomes class-number agnostic. On the other hand, recently, universal models for a wide range of NLP problems have been emerged (Raffel et al., 2019; Chen et al., 2021; Ao et al., 2021). We believe the development of the universal models will intrigue the cross-problem setting in meta-learning.
3.4 Domain Generalization
Traditional supervised learning assumes that the training and testing data have the same distribution. Domain shift refers to the problem that a model performs poorly when training data and testing data have very different statistics. Domain adaptation in Section 3.1 uses little domain-specific data to adapt the model<sup>2</sup>. On the other hand, domain generalization techniques attempt to alleviate the domain mismatch issue by producing models that generalize well to novel testing domains.
Meta-learning can also be used to realize domain generalization by learning an algorithm that can train from one domain but evaluate on the other. To simulate the domain generalization scenario, a set of meta-training tasks are constructed by sampling data from different domains as the support and query sets. With the meta-training tasks above, cross-task training will find the meta-parameters  $\phi^{*}$  that work well on the scenario where the training (support) and testing (query) examples are from different domains. Fig. 1 shows how to construct tasks for domain generalization and compares the construction with the cross-domain transfer setting. The setting has been used to improve the domain generalization for semantic parsing (Wang et al.,
2The domain-specific data are usually labelled, but unlabelled domain-specific data can be leveraged as well (Kouw and Loog, 2021), which is out of scope here.
2021a) and language generalization<sup>3</sup> for sentiment classification and relevance classification (Li et al., 2020c).
3.5 Task Augmentation
In meta-learning, it is critical to have a large number of diverse tasks in the meta-training tasks  $\mathcal{T}_{train}$  to find a set of meta-parameters  $\phi^{*}$  that can generalize well to the meta-testing tasks. However, considering the setting in the previous subsections, different tasks contain examples in various domains, language, or even NLP problems, so a large and diverse  $\mathcal{T}_{train}$  are often not available. In typical ML, data augmentation comes in handy when data is lacking. In meta-learning, augmenting tasks is similarly understood as data augmentation in ML. Data augmentation becomes task augmentation because the "training examples" in meta-learning are a collection of tasks. Task augmentation approaches in meta-learning can be categorized into two main directions: a) Inventing more tasks (without human labeling efforts) to increase the number and diversity of the meta-training tasks  $\mathcal{T}_{train}$ . b) Splitting training data from one single dataset into homogenous partitions that allow applying meta-learning techniques and therefore improve the performance. NLP-specific methods have been proposed in both categories.
Inventing more tasks The main question is how to construct a massive amount of tasks efficiently. There is already some general task augmentation approaches proposed for general meta-learning (Yao et al., 2021a; Ni et al., 2021; Rajendran et al., 2020; Yao et al., 2021b). Here we only focus on NLP-specific approaches. Inspired from the self-supervised learning, Bansal et al. (2020b) generates a large number of cloze tasks, which can be considered as multi-class classification tasks but obtained without labeling effort, to augment the meta-training tasks. Bansal et al. (2021) further explores the influence of unsupervised task distribution and creates task distributions that are inductive to better meta-training efficacy. The self-supervised generated tasks improve the performance on a wide range of different meta-testing tasks which are classification problems (Bansal et al., 2020b), and it even performs comparably with supervised meta-learning methods on FewRel 2.0 benchmark (Gao et al., 2019b) on 5-shot evaluation (Bansal et al., 2021).
Generating tasks from a monolithic corpus Many tasks can be constructed with one monolithic corpus (Huang et al. (2018); Guo et al. (2019); Wu et al. (2019); Jiang et al. (2019); Chien and Lieow (2019); Li et al. (2020b); MacLaughlin et al. (2020); Wang et al. (2020a); Pasunuru and Bansal (2020); Xu et al. (2021a); Murty et al. (2021)). First, the training set of the corpus is split into support partition,  $\mathcal{D}_s$ , and query partition,  $\mathcal{D}_q$ . Two subsets of examples are sampled from  $\mathcal{D}_s$  and  $\mathcal{D}_q$  as the support set,  $S$ , and query set,  $\mathcal{Q}$ , respectively. In each episode, model parameters  $\theta$  are updated with  $S$ , and then the losses are computed with the updated model and  $\mathcal{Q}$ . The meta-parameters  $\phi$  are then updated based on the losses, as the meta-learning framework introduced in Section 2. The test set of the corpus is used to build  $\mathcal{T}_{test}$  for evaluation. As compared to constructing  $\mathcal{T}_{train}$  from multiple relevant corpora, which are often not available, building  $\mathcal{T}_{train}$  with one corpus makes meta-learning methodology more applicable. Besides, results obtained from one corpus are more comparable with existing NLP studies. However, only using a single data stream makes the resulting models less generalizable to various attributes such as domains and languages.
How to sample the data points to form a task $^{4}$  is the key in such category. In NAS research in Section 4.3, the support and query sets are usually randomly sampled. Learning to Compare in Section 4.2 splits the data points of different classes in different tasks based on some predefined criteria. There are some NLP-specific ways to construct the tasks. In Huang et al. (2018), a relevance function is designed to sample the support set  $S$  based on its relevance to the query set  $\mathcal{Q}$ . In Guo et al. (2019), a retrieval model is used to retrieve the support set  $S$  from the whole dataset. DReCa (Murty et al., 2021) applies clustering on BERT representations to create tasks.
4 Meta-Learning for NLP Tasks
This section shows the most popular meta-learning methods for NLP and how they fit into NLP tasks. Due to space limitations, only the major trends are mentioned. Please refer to Table 2 and 3 in the appendix for a complete survey.
4 If a corpus includes data from different domains, and we sample the data in the same domain to create a task, then the setting here becomes cross-domain in Section 3.1.
3if a language is considered as a domain
4.1 Learning to Initialize
In typical DL, gradient descent is widely used to solve (3). Gradient descent starts from a set of initial parameters  $\theta^0$ , and then the parameters  $\theta$  are updated iteratively according to the directions of the gradient. There is a series of meta-learning approaches targeting at learning the initial parameters  $\theta^0$ . In these learn-to-init approaches, the metaparameters  $\phi$  to be learned are the initial parameters  $\theta^0$  for gradient descent, or  $\phi = \theta^0$ . MAML (Finn et al., 2017) and its first-order approximation, FO-MAML (Finn et al., 2017), Reptile (Nichol et al., 2018), etc., are the representative approaches of learn-to-init. We surveyed a large number of papers using MAML-based approaches to NLP applications in the last three years and summarized them in Table 4 in the appendix.