SlowGuess/NAACL_2022_OCR huggingface dataset:
Mengzhou Xia, Guoqing Zheng, Subhabrata Mukherjee, Milad Shokouhi, Graham Neubig, and Ahmed Hassan Awadal-lah. 2021. MetaXL: Meta representation transformation for low-resource cross-lingual learning. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies.
Yubei Xiao, Ke Gong, Pan Zhou, Guolin Zheng, Xiaodan Liang, and Liang Lin. 2021. Adversarial meta sampling for multilingual low-resource speech recognition. In AAAI.
Wenhan Xiong, Mo Yu, Shiyu Chang, Xiaoxiao Guo, and William Yang Wang. 2018. One-shot relational learning for knowledge graphs. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing.
Guangyue Xu, Parisa Kordjamshidi, and Joyce Chai. 2021a. Zero-shot compositional concept learning. In Proceedings of the First Workshop on Meta Learning and Its Applications to Natural Language Processing.
Jin Xu, Xu Tan, Renqian Luo, Kaitao Song, Jian Li, Tao Qin, and Tie-Yan Liu. 2021b. Nas-bert: task-agnostic and adaptive-size bert compression with neural architecture search. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining.
Weijia Xu, Batool Haider, Jason Krone, and Saab Mansour. 2021c. Soft layer selection with meta-learning for zero-shot cross-lingual transfer. In Proceedings of the First Workshop on Meta Learning and Its Applications to Natural Language Processing.
Ming Yan, Hao Zhang, Di Jin, and Joey Tianyi Zhou. 2020. Multi-source meta transfer for low resource multiple-choice question answering. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics.
Huaxiu Yao, Long-Kai Huang, Linjun Zhang, Ying Wei, Li Tian, James Zou, Junzhou Huang, and Zhenhui Li. 2021a. Improving generalization in meta-learning via task augmentation. In Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 11887-11897. PMLR.
Huaxiu Yao, Linjun Zhang, and Chelsea Finn. 2021b. Meta-learning with fewer tasks through task interpolation.
Pauching Yap, Hippolyt Ritter, and David Barber. 2021. Addressing catastrophic forgetting in few-shot problems. In ICML.
Qinyuan Ye, Bill Yuchen Lin, and Xiang Ren. 2021. CrossFit: A few-shot learning challenge for cross-task generalization in NLP. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing.
Zhi-Xiu Ye and Zhen-Hua Ling. 2019. Multi-level matching and aggregation network for few-shot relation classification. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics.
Wenpeng Yin. 2020. Meta-learning for few-shot natural language processing: A survey. arXiv preprint arXiv:2007.09604.
Mo Yu, Xiaoxiao Guo, Jinfeng Yi, Shiyu Chang, Saloni Potdar, Yu Cheng, Gerald Tesauro, Haoyu Wang, and Bowen Zhou. 2018. Diverse few-shot text classification with multiple metrics. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers).
Friedemann Zenke, Ben Poole, and Surya Ganguli. 2017. Continual learning through synaptic intelligence. In ICML.
Chuxu Zhang, Huaxiu Yao, Chao Huang, Meng Jiang, Zhenhui Li, and Nitesh V Chawla. 2020. Few-shot knowledge graph completion. In AAAI.
Guoqing Zheng, Ahmed H. Awadallah, and Susan Dumais. 2021. Meta label correction for noisy label learning. In AAAI.
Wangchunshu Zhou, Canwen Xu, and Julian McAuley. 2022. BERT learns to teach: Knowledge distillation with meta learning. ACL.
Barret Zoph and Quoc V. Le. 2017. Neural architecture search with reinforcement learning. In ICLR.
Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoc V. Le. 2018. Learning transferable architectures for scalable image recognition. In CVPR.
Table 1: Terminologies and their meanings.
Terminologies	Meaning
(NLP) Problem	a type of NLP problems like QA, POS, or MT
Model Parameter	parameters of models making inference for underlying problems
Meta-parameter	parameters of learning algorithms (e.g., model init, optimizers) that are shared across tasks
Support Set	a set of training examples for updating model parameters
Query Set	a set of testing examples for evaluating model parameters
Task	combination of one support set and one query set
Within-task Training	learning model parameter with support set
Within-task Testing	using query set to evaluate model parameters
Episode	one execution of within-task training and followed by one execution of within-task testing
Meta-training Tasks	tasks generated for learning meta-parameter
Meta-testing Tasks	tasks generated for evaluating algorithms parameterized by meta-parameter
Cross-task Training	learning meta-parameter, which usually involves running many episodes on meta-training tasks
Cross-task Testing	running an episode on each meta-testing task to evaluate algorithms parameterized by meta-parameter
Table 3: Continue of Table 2. Pham et al. (2021) learns a backtranslation model for data augmentation, so it is considered as SEL.
Work	Method	Application	Task construction
(Pham et al., 2021)	SEL	MT	Monolithic
(Guo et al., 2019)	INIT	PAR	Monolithic
(Huang et al., 2018)	INIT	PAR	Monolithic
(Langedijk et al., 2021)	INIT	PAR	Cross-lingual
(Chen et al., 2020b)	INIT	PAR	Cross-domain
(Wang et al., 2021a)	INIT	PAR	Domain Generalization
(Qian and Yu, 2019)	INIT	DG	Cross-domain
(Madotto et al., 2019)	INIT	DG	Cross-domain (each domain is one type of persona)
(Mi et al., 2019)	INIT	DG	Cross-domain
(Huang et al., 2020a)	INIT	DST	Cross-domain
(Dingliwal et al., 2021)	INIT	DST	Cross-domain
(Huang et al., 2020b)	INIT	DST	Cross-domain
(Dai et al., 2020)	INIT	DG	Cross-domain
(Qian et al., 2021)	INIT	DG	Cross-domain
(Chien and Lieow, 2019)	OPT	DG	Monolithic
(Hsu et al., 2020)	INIT	ASR	Cross-lingual
(Klejch et al., 2019)	INIT	ASR	Cross-domain (each domain refers to a speaker)
(Winata et al., 2020a)	INIT	ASR	Cross-lingual
(Winata et al., 2020b)	INIT	ASR	Cross-domain (each domain refers to a accent)
(Xiao et al., 2021)	INIT	ASR	Cross-lingual
(Klejch et al., 2018)	OPT	ASR	Cross-domain (each domain refers to a speaker)
(Chen et al., 2020d)	NAS	ASR	Cross-lingual
(Baruwa et al., 2019)	NAS	ASR	Monolithic
(Wu et al., 2021b)	INIT	SS	Cross-domain (each domain refers to a speaker)
(Huang et al., 2021)	INIT	SS	Cross-domain(each domain refers to a accent)
(Chen et al., 2020c)	INIT	KS	The tasks are separated by keyword sets.
(Parnami and Lee, 2020)	COM	KS	The tasks are separated by keyword sets.
(Huh et al., 2021)	COM	KS	The tasks are separated by keyword sets.
(Mazzawi et al., 2019)	NAS	KS	Monolithic
(Lux and Vu, 2021)	COM	KS	The tasks are separated by keyword sets.
(Serrà et al., 2019)	ALG	VC	Cross-domain (each domain refers to a speaker)
(Chen et al., 2019b)	ALG	VC	Cross-domain (each domain refers to a speaker)
(Huang et al., 2022)	INIT	VC	Cross-domain (each domain refers to a speaker)
(Tarunesh et al., 2021)	INIT, SEL	QA, TC, TAG	Cross-lingual, Cross-problem
(Eloff et al., 2019)	COM	MG	Monolithic
(Surís et al., 2019)	ALG	MG	Each task contains multiple examples of text-image pairs.
(Xu et al., 2021a)	COM	MG	Each task contains an image and a word set.
(De Cao et al., 2021)	OPT	TC, QA	Life-long learning
(Sinitsin et al., 2020)	INIT	MT	Life-long learning
(Wang et al., 2020c)	INIT	TC, QA	Life-long learning
(Wu et al., 2021a)	INIT	IE	Life-long learning
(Obamuyide and Vlachos, 2019a)	INIT	IE	Life-long learning
Table 4: Summary of learn-to-init variants. This table contains the following information. (1) Method: There are many variants in the learn-to-init family. The most representative one is MAML. Typical MAML (Finn et al., 2017) has large computation intensity, so the first-order approximations like FOMAML (Finn et al., 2017) and Reptile (Nichol et al., 2018) are widely used. DG-MAML (Li et al., 2018) is for domain generalization. Typical learn-to-init assumes that all the tasks use the same network architecture, but LEOPARD (Bansal et al., 2020a) and Proto(FO)MAML (Triantafillou et al., 2020) are proposed to overcome the limitation. (2) How to Initialize the Initialization: Learn-to-init approaches aim at learning the initial parameters. But where does the initialization of MAML come from? We found that using self-supervised pre-training as initialization is common. The table specifies the pre-trained models used to initialize the learn-to-init methods. '-' means the initial parameters are learned from random initialization or cannot tell based on the descriptions in the papers.
Work	Method	How to Initialize the Initialization
(Bansal et al., 2020a)	LEOPARD	BERT
(Li et al., 2020a)	MAML	Word Embedding
(Park et al., 2021)	MAML	XLM
(Gu et al., 2018)	FOMAML	Word Embedding
(Langedijk et al., 2021)	FOMAML	mBERT
(Chen et al., 2020b)	Reptile	BART
(Huang et al., 2020a)	MAML	BERT
(Wang et al., 2021b)	Propose a new method based on Reptile	Word Embedding
(Dingliwal et al., 2021)	Reptile	RoBERTa
(Qian and Yu, 2019)	MAML	Word Embedding
(Qian et al., 2021)	MAML	Word Embedding
(Madotto et al., 2019)	MAML	Word Embedding
(Dai et al., 2020)	MAML	-
(Hsu et al., 2020)	FOMAML	Multilingual ASR
(Xiao et al., 2021)	MAML/FOMAML/Reptile	-
(Winata et al., 2020b)	MAML	Pretrain by Supervised Learning
(Klejch et al., 2019)	FOMAML	-
(Huang et al., 2021)	MAML/FOMAML	-
(Indurthi et al., 2020)	FOMAML	-
(Winata et al., 2020a)	FOMAML	-
(Wu et al., 2021b)	MAML	Pretrain by Multi-task Learning
(Ke et al., 2021)	MAML	BERT
(Xia et al., 2021)	MetaXL	mBERT/XLM-R
(Dou et al., 2019)	MAML/FOMAML/Reptile	BERT
(Obamuyide and Vlachos, 2019b)	FOMAML	Word Embedding
(Lv et al., 2019)	MAML	-
(Holla et al., 2020)	FOMAML/Proto(FO)MAML	Word Embedding/ELMo/BERT
(Huang et al., 2020b)	MAML	Word Embedding
(Mi et al., 2019)	MAML	-
(Wang et al., 2021a)	DG-MAML	BERT
(Conklin et al., 2021)	DG-MAML	-
(M'hamdi et al., 2021)	MAML	mBERT
(Nooralahzadeh et al., 2020)	MAML	BERT/mBERT/XLM-R
(Garcia et al., 2021)	MAML	mBERT
(van der Heijden et al., 2021)	FOMAML/Reptile/Proto(FO)MAML	XLM-R
(Bansal et al., 2020b)	LEOPARD	BERT
(Murty et al., 2021)	FOMAML	BERT
(Hua et al., 2020)	Reptile	-
(Yan et al., 2020)	MAML	BERT/RoBERTa
(Wang et al., 2019b)	Reptile	-
(Bose et al., 2020)	Meta-Graph	-
Table 5: Performance of selected NAS approaches on the test set of GLUE benchmark.
Model	#Params	Latency	MNLI	QQP	QNLI	SST-2	MRPC	RTE	Avg
BERTBASE (Google)	108.9M	362ms	84.6	71.2	90.5	93.5	88.9	66.4	82.5
MobileBERTINY (Sun et al., 2020)	15.1M	96ms	81.5	68.9	89.5	91.7	87.9	65.1	80.8
AdaBERT (Chen et al., 2020a)	6.4-9.5M	12.4-28.5ms	81.6	70.7	86.8	91.8	85.1	64.4	80.1
EfficientBERT (Dong et al., 2021)	16M	103ms	83.0	71.2	90.6	92.3	88.9	67.8	82.3