SlowGuess/NAACL_2022_OCR huggingface dataset:
Learning to Initialize v.s. Self-supervised Learning The learn-to-init approaches aim at learning a set of good initial parameters. On the other hand, self-supervised approaches like BERT also have the same target. There is a natural question: are they complementary? Based on the survey in Table 4 in the appendix, it is common to use the self-supervised models to "initialize" the meta-parameters  $\phi$  in learn-to-init approaches. To find the optimal  $\phi^{*}$  in (5), gradient descent is used as well, and thus the "initial parameters for initial parameters", or  $\phi^0$  is required. A self-supervised model usually serves the role of  $\phi^0$ , and the learn-to-init approaches further update  $\phi^0$  to find  $\phi^{*}$ .
Learn-to-init and self-supervised learning are complementary. The self-supervised objectives are different from the objective of the target NLP problem, so there is a "learning gap". On the other hand, learn-to-init approaches learn to achieve good performance on the query sets of the meta-training tasks, so it directly optimizes the objective of the NLP problems. The benefit of self-supervised learning is that it does not require labeled data, while labeling is still needed to prepare the examples in meta-training tasks.
Learning to Initialize v.s. Multi-task Learning Multi-task learning is another way to initialize model parameters, which usually serves as the baseline of learn-to-init in the literature. In multitask learning, all the labelled data from the metatraining tasks is put together to train a model. That is, all the support sets  $S_{n}$  and query sets  $\mathcal{Q}_n$  in the meta-training tasks  $T_{train}$  are put together as a
training set  $\mathcal{D}$ , and the loss (3) is optimized to find a parameter  $\theta^{*}$ . Then  $\theta^{*}$  is used as initial parameters for the meta-testing tasks.
Both multi-task learning and meta-learning leverage the examples in the meta-training tasks, but with different training criteria. Learn-to-init finds the initial parameters suitable to be updated by updating the model on the support sets and then evaluating it on the query sets. In contrast, multi-task learning does not consider that the initial parameters would be further updated at all during training. Therefore, in terms of performance, learn-to-init is usually shown to be better than multi-task learning (Dou et al., 2019; Chen et al., 2020b). On the other hand, in terms of training speed, meta-learning, which optimizes (5), is more computationally intensive than multi-task learning optimizing (3).
Three-stage Initialization Since learn-to-init, multi-task, self-supervised learning all have their pros and cons, they can be integrated to draw on the strong points of each other. A common way to integrate the three approaches is "three-stage initialization" as below. a) First, initialize a model by self-supervised learning, which leverages unlabeled data. Its objective is usually not directly related to the target NLP problem. b) Then, multitask learning is used to fine-tune the self-supervised model. The objective of multi-task learning is the target NLP problem but does not consider the update procedure in gradient descent. c) Finally, learn-to-init, which finds the initial parameters suitable for update, is used to fine-tune the multi-task model.
Learn-to-init is chosen to be the last stage because its training objective is closest to the target of looking for good initial parameters, but it is the most computationally intensive method, and thus it is only used to change the model a little bit. The three-stage initialization has been tested in several works (Nooralahzadeh et al., 2020; Wu et al., 2021b; van der Heijden et al., 2021; Langedijk et al., 2021), but it does not always improve the performance (Wu et al., 2021b; van der Heijden et al., 2021).
Challenges Learn-to-init is an essential paradigm for few-shot learning and usually achieves outstanding results in the few-shot learning benchmarks of image classification (Tri
antafillou et al., 2020). However, it has fallen short of yielding state-of-the-art results on NLP few-shot learning benchmarks (Ye et al., 2021; Chen et al., 2022; Bragg et al., 2021). For example, on the cross-task few-shot learning benchmark, CrossFit, simple multi-task learning outperforms existing learn-to-init in many cases (Ye et al., 2021). One possible reason is meta-learning methods are susceptible to hyper-parameters and even random seeds (Antoniou et al., 2019). Hence, it is difficult to obtain decent performance without exhaustively tuning hyperparameters. The research about developing more stable learn-to-init methods may lead to more practical real-world applications for the approaches. There is a study about stabilizing the cross-task training of learn-to-init methods by reducing the variance of gradients for NLP (Wang et al., 2021b).
4.2 Learning to Compare
Learning to Compare methods are widely applied to NLP tasks. Among many others, we find applications of Learning to Compare methods in text classification (Yu et al., 2018; Tan et al., 2019; Geng et al., 2019; Sun et al., 2019b; Geng et al., 2020), sequence labeling (Hou et al., 2020; Oguz and Vu, 2021), semantic relation classification (Ye and Ling, 2019; Chen et al., 2019a; Gao et al., 2019a; Ren et al., 2020), knowledge completion (Xiong et al., 2018; Wang et al., 2019b; Zhang et al., 2020; Sheng et al., 2020) and speech recognition (Lux and Vu, 2021) tasks.
Most of the proposed methods are based on Matching Network (Vinyals et al., 2016), Prototypical Network (Snell et al., 2017) and Relation Network (Sung et al., 2018), and extend these architectures in two aspects: a) how to embed text input in a vector space with/without context information, and b) how to compute the distance/similarity/relation between two inputs in this space. Since these questions have had deep roots in the computation linguistics research for many years (Schütze, 1992; Manning and Schutze, 1999), Learning to Compare methods is one of the most important methods among other meta-learning methods in the context of NLP despite their simplicity. Notably, to date, such family of methods is mainly applied to classification tasks.
4.3 Neural Network Architecture Search
Neural network architecture search (NAS) is another common meta-learning technique applied
to NLP including language modeling (WikiText-103 (Merit et al., 2017), PTB (Mikolov et al., 2010)), NER (CoNLL-2003 (Sang and De Meulder, 2003)), TC (GLUE (Wang et al., 2019a)), and MT (WMT'14 (Bojar et al., 2014)). As discussed in Section 3.5, these techniques are often trained/evaluated with a single, matched dataset, which is different from other meta-learning approaches.
Moreover, in contrast to conventional NAS methods that focus on learning the topology in an individual recurrent or convolutional cell, NAS methods have to be redesigned in order to make the search space suitable for NLP problems, where contextual information often plays an important role. Jiang et al. (2019) pioneers the application of NAS to NLP tasks beyond language modeling (NER in this case), and improves differentiable NAS by redesigning its search space for natural language processing. Li et al. (2020b) extends the search space of NAS to cover more RNN architectures and allow the exploring of intra- and inter-token connection to increase the expressibility of searched networks. As the popularity of pretrained language models (PLM) grows in NLP area, researchers also apply NAS to discover better topology for PLM such as BERT. Wang et al. (2020a) introduces Hardware-Aware Transformers (HAT) to search Transformer architecture optimized for inference speed and memory footprint in different hardware platforms. NAS-BERT (Xu et al., 2021b) and AdaBERT (Chen et al., 2020a) explores task-agnostic and task-dependent network compression techniques with NAS respectively. EfficientBERT (Dong et al., 2021) applies NAS to search for more efficient architecture of feed-forward network that is suitable for edge device deployment.
To show the efficacy of NAS, we summarize the performance of several state-of-the-art NAS approaches on GLUE benchmarks (Wang et al., 2019a) in Table 5 in the appendix. These approaches are applied to BERT to discover architectures with smaller sizes, faster inference speed, and better model accuracy. For comparison, performance from original and manually compressed BERT models is also presented. The results show that the BERT architecture improved by NAS yields performance competitive to BERT (c.f., 82.3 from EfficientBERT vs 82.5 from BERT) and is  $6.9\mathrm{x}$  smaller and  $4.4\mathrm{x}$  faster. The searched architecture also outperforms man
ually designed, parameter- and inference-efficient model (MobileBERT $_{\text{TINY}}$ ) at similar size and speed. These results suggest the efficacy of NAS in discovering more efficient network architectures. As NLP researchers continue to design even larger PLMs while the need of deployment on edge devices grows, we expect there will be increasing investment in innovating NAS techniques to make PLM networks more compact and accelerate inference.
Challenges The main bottleneck for NAS being widely applied is the prohibitive requirement in computation resources for architecture search. Approaches such as Efficient Neural Architecture Search (ENAS, Pham et al. (2018b)) and Flexible and Expressive Neural Architecture Search (FE-NAS, Pasunuru and Bansal (2020)) are proposed to improve the search efficiency. As PLMs usually have bulky sizes and slow training speed, search efficiency is even more critical when applying NAS to PLM. Weight-sharing techniques are often applied to accelerate searching (Wang et al., 2020a; Dong et al., 2021; Xu et al., 2021b).
4.4 Meta-learning for Data Selection
Multi-linguality, multi-task, and multi-label see many impacts on NLP problems due to the diversity of human languages. To learn models with balanced performance over attributes (e.g., languages, tasks, labels), a common approach is to weight the training examples for data selection to learn models with balanced performance over the attributes, and it is a natural assumption that meta-learning techniques derive more generalizable weighting than manually tuned hyperparameters. For example, Wu et al. (2019) add another gradient update step wrapping the conventional classifier update for training meta-parameters that controls the weight when aggregating losses from different labels to update classifier's parameters. In addition to gradient update, meta-learned weights are also applied directly to training examples for data selection to address the issue of noisy labeling. Shu et al. (2019) propose a technique to jointly learn a classifier and a weighting function, where a conventional gradient update for the classifier and a meta-learning update for the weighting is performed alternatively. The function weights examples to mitigate model overfitting towards biased training data caused by corrupted labels or class imbalance. Zheng et al. (2021) apply a similar framework but extend the weighting with
a label correction model. Both techniques show improvement over SOTA in text classification with biased training data.
Additionally, as the progress in the research of pre-training and transfer learning, there is a trend of leveraging datasets in multiple languages, domains, or tasks to jointly pre-train models to learn transferable knowledge. A meta-learned data selector can also help in this scenario by choosing examples that benefit model training and transferability. For instance, Wang et al. (2020b) investigate the common challenge of imbalanced training examples across languages in multilingual MT, which is conventionally addressed by tuning hyperparameters manually to up-sample languages with less resources. The authors propose Differentiable Data Selection (DDS) to parameterize the sampling strategies. DDS is trained with episodes and REINFORCE algorithm to optimize parameters of sampler and MT models in an alternating way for the MT models to converge with better performance across languages. Pham et al. (2021) formulate data sampling for multilingual MT as a problem of back-translation to generate examples of parallel utterances from unlabeled corpora in target language. The back-translation is jointly trained with MT models to improve translation result through better distribution of training examples and data augmentation. Tarunesh et al. (2021) further study knowledge transferring across tasks and languages. The authors combine Reptile and DDS to metalearn samplers with six different languages (en, hi, es, de, fr, and zh) and five different tasks (QA, NLI, paraphrase identification, POS tagging, and NER) and demonstrate competitive performance on XTREME multilingual benchmark dataset (Hu et al., 2020).
5 Meta-learning beyond Accuracy
In the previous sections, meta-learning is used to obtain better evaluation metrics for NLP applications. This section illustrates how meta-learning can improve NLP applications from more aspects beyond performance.
5.1 Learn to Knowledge Distillation
Knowledge distillation method was proposed in (Hinton et al., 2015). The main goal is to transfer knowledge from a so-called teacher model, e.g., a vast neural network trained with a lot of training data, to a more compact student model, e.g., a neu
ral network with much less trainable parameters. The main weaknesses of this method are as follows: a) the number of teacher models is fixed to one that could limit the power of the transferring process; b) the teacher model is not optimized for the transferring process and c) the teacher model is not aware of the student model during the transferring process. Meta-learning methods can be applied to partially fix these issues. The high-level idea is to increase the number of teacher models and the number of student models and consider each pair of a teacher model and a student model as a task in the meta-learning framework. By doing so, we can train a meta teacher model that works better than a single teacher model (Pan et al., 2020), and we can optimize the transferring process and force the teacher model to be aware of the student model (Zhou et al., 2022).
5.2 Learn to Life-long learning
This subsection discusses how to use meta-learning to improve lifelong learning (LLL) (Chen and Liu, 2018). The real world is changing and evolving from time to time, and therefore machines naturally need to update and adapt to the new data they receive. However, when a trained deep neural network is adapted to a new dataset with a different distribution, it often loses the knowledge previously acquired and performs the previous seen data worse than before. This phenomenon is called catastrophic forgetting (McCloskey and Cohen, 1989). There is a wide range of LLL approaches aiming for solving catastrophic forgetting (Parisi et al., 2019). Among them, the following directions apply meta-learning:
Meta-learning for Regularization-based LLL methods Regularization-based LLL methods aim to consolidate essential parameters in a model when adapting models with new data (Kirkpatrick et al., 2017; Zenke et al., 2017; Schwarz et al., 2018; Aljundi et al., 2018; Ehret et al., 2021). Metalearning targets "how to consolidate" and has some successful examples in NLP applications. KnowledgeEditor (De Cao et al., 2021) learns the parameter update strategies that can learn the new data and simultaneously retain the same predictions on the old data. KnowledgeEditor has been applied to fact
checking and QA. Editable Training (Sinitsin et al., 2020) employs learn-to-init approaches to find the set of initial parameters, ensuring that new knowledge can be learned after updates without harming the performance of old data. Editable Training empirically demonstrates the effectiveness on MT.
Meta-learning for Data-based LLL Methods The basic idea of data-based methods is to store a limited number of previously seen training examples in memory and then use them for empirical replay, that is, training on seen examples to recover knowledge learned (Sprechmann et al., 2018; de Masson d'Autume et al., 2019; Sun et al., 2019a) or to derive optimization constraints (Lopez-Paz and Ranzato, 2017; Li and Hoiem, 2017; Saha and Roy, 2021). A hurdle for data-based approaches is the need to store an unrealistically large number of training examples in memory to achieve good performance. To achieve sample efficiency, Obamuyide and Vlachos (2019a); Wang et al. (2020c); Wu et al. (2021a) uses meta-learning to learn a better adaptation algorithm that recovers the knowledge learned with a limited amount of previously seen data. Experiments on text classification and QA benchmarks validate the effectiveness of the framework, achieving state-of-the-art performance using only  $1\%$  of the memory size (Wang et al., 2020c).
This paper investigates how meta-learning is used in NLP applications. We review the task construction settings (Section 3), the commonly used methods including learning to initialize, learning to compare and neural architecture search (Section 4), and highlight research directions that go beyond improving performance (Section 5). We hope this paper will encourage more researchers in the NLP community to work on meta-learning.
Rahaf Aljundi, Francesca Babiloni, Mohamed Elhoseiny, Marcus Rohrbach, and Tinne Tuytelaars. 2018. Memory aware synapses: Learning what (not) to forget. In ECCV.
Marcin Andrychowicz, Misha Denil, Sergio Gomez, Matthew W. Hoffman, David Pfau, Tom Schaul, Brendan Shillingford, and Nando de Freitas. 2016. Learning to learn by gradient descent by gradient descent. In NIPS.
Antreas Antoniou, Harrison Edwards, and Amos Storkey. 2019. How to train your MAML. In International Conference on Learning Representations.
On the other hand, in meta-learning, usually, we assume stationary task distribution. Can we do meta-learning with evident distributional shift or when tasks arrive sequentially? There is also research along the direction (Finn et al., 2019; Yap et al., 2021), but out of the scope of this review paper.
Junyi Ao, Rui Wang, Long Zhou, Shujie Liu, Shuo Ren, Yu Wu, Tom Ko, Qing Li, Yu Zhang, Zhihua Wei, Yao Qian, Jinyu Li, and Furu Wei. 2021. SpeechT5: Unified-modal encoder-decoder pre-training for spoken language processing.
Cyprien de Masson d'Autume, Sebastian Ruder, Lingpeng Kong, and Dani Yogatama. 2019. Episodic memory in lifelong language learning. In NeurIPS.
Trapit Bansal, Karthick Prasad Gunasekaran, Tong Wang, Tsendsuren Munkhdalai, and Andrew McCallum. 2021. Diverse distributions of self-supervised tasks for meta-learning in NLP. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing.
Trapit Bansal, Rishikesh Jha, and Andrew McCallum. 2020a. Learning to few-shot learn across diverse natural language classification tasks. In Proceedings of the 28th International Conference on Computational Linguistics.
Trapit Bansal, Rishikesh Jha, Tsendsuren Munkhdalai, and Andrew McCallum. 2020b. Self-supervised meta-learning for few-shot natural language classification tasks. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP).
Ahmed Baruwa, Mjoeed Abisiga, Ibrahim Gbadegesin, and Afeez Fakunle. 2019. Leveraging end-to-end speech recognition with neural architecture search. In IJSER.
Ondrej Bojar, Christian Buck, Christian Federmann, Barry Haddow, Philipp Koehn, Johannes Leveling, Christof Monz, Pavel Pecina, Matt Post, Herve Saint-Amand, Radu Soricut, Lucia Specia, and Ales Tamchyna. 2014. Findings of the 2014 workshop on statistical machine translation. In Proceedings of the Ninth Workshop on Statistical Machine Translation.
Avishek Joey Bose, Ankit Jain, Piero Molino, and William L. Hamilton. 2020. Meta-graph: Few shot link prediction via meta learning.
Jonathan Bragg, Arman Cohan, Kyle Lo, and Iz Beltagy. 2021. FLEX: Unifying evaluation for few-shot nlp. In Advances in Neural Information Processing Systems.
Daoyuan Chen, Yaliang Li, Minghui Qiu, Zhen Wang, Bofang Li, Bolin Ding, Hongbo Deng, Jun Huang, Wei Lin, and Jingren Zhou. 2020a. Adabert: Task-adaptive bert compression with differentiable neural architecture search. In IJCAI.
Mingyang Chen, Wen Zhang, Wei Zhang, Qiang Chen, and Huajun Chen. 2019a. Meta relational learning for few-shot link prediction in knowledge graphs. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP).
Xilun Chen, Asish Ghoshal, Yashar Mehdad, Luke Zettlemoyer, and Sonal Gupta. 2020b. Low-resource domain adaptation for compositional task-oriented semantic parsing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP).
Yanda Chen, Ruiqi Zhong, Sheng Zha, George Karypis, and He He. 2022. Meta-learning via language model in-context tuning. ACL.
Yangbin Chen, Tom Ko, Lifeng Shang, Xiao Chen, Xin Jiang, and Qing Li. 2020c. An investigation of few-shot learning in spoken term classification. In INTERSPEECH.
Yi-Chen Chen, Po-Han Chi, Shu wen Yang, Kai-Wei Chang, Jheng hao Lin, Sung-Feng Huang, Da-Rong Liu, Chi-Liang Liu, Cheng-Kuang Lee, and Hung yi Lee. 2021. SpeechNet: A universal modularized model for speech processing tasks.
Yi-Chen Chen, Jui-Yang Hsu, Cheng-Kuang Lee, and Hung yi Lee. 2020d. DARTS-ASR: Differentiable architecture search for multilingual speech recognition and adaptation. In INTERSPEECH.
Yutian Chen, Yannis Assael, Brendan Shillingford, David Budden, Scott Reed, Heiga Zen, Quan Wang, Luis C. Cobo, Andrew Trask, Ben Laurie, Caglar Gulcehre, Aaron van den Oord, Oriol Vinyals, and Nando de Freitas. 2019b. Sample efficient adaptive text-to-speech. In ICLR.
Zhiyuan Chen and Bing Liu. 2018. Lifelong machine learning. Synthesis Lectures on Artificial Intelligence and Machine Learning, 12.
Jen-Tzung Chien and Wei Xiang Lieow. 2019. Meta learning for hyperparameter optimization in dialogue system. In INTERSPEECH.
Henry Conklin, Bailin Wang, Kenny Smith, and Ivan Titov. 2021. Meta-learning to compositionally generalize. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers).
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics.
Yinpei Dai, Hangyu Li, Chengguang Tang, Yongbin Li, Jian Sun, and Xiaodan Zhu. 2020. Learning low-resource end-to-end goal-oriented dialog for fast and reliable system deployment. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics.
Nicola De Cao, Wilker Aziz, and Ivan Titov. 2021. Editing factual knowledge in language models. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers).
Saket Dingliwal, Shuyang Gao, Sanchit Agarwal, Chien-Wei Lin, Tagyoung Chung, and Dilek Hakkani-Tur. 2021. Few shot dialogue state tracking using meta-learning. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume.
Chenhe Dong, Guangrun Wang, Hang Xu, Jiefeng Peng, Xiaozhe Ren, and Xiaodan Liang. 2021. EfficientBERT: Progressively searching multilayer perceptron via warm-up knowledge distillation. In EMNLP.