SlowGuess/NAACL_2021_OCR huggingface dataset:
compare with an oracle model MAML-ORACLE which is trained over a mixture of dataset-based tasks as well as oracle reasoning categories. Finally, MAML-DRECA is our model which trains MAML over a mixture of the original dataset-based tasks as well as the augmented tasks from DRECA.
Evaluation. To control for variations across different support sets, we sample 5-10 random support sets for each test task. We finetune each of our models on these support sets and report means and  $95\%$  confidence intervals assuming the accuracies follow a Gaussian.
Training Details. We use first order MAML (FoMAML) for computational efficiency. We use BERT-base as provided in the transformers library (Wolf et al., 2019) as the parameterization for  $h_{\theta}$  and EMBED(); The meta-training inner loop optimization involves 10 gradient steps with Adam, with a support set of 2 examples (2-way 1-shot) for all except GLUE-SciTail where the support set size is 8 (2-way 4-shot). We experiment with 4-shot adaptation on GLUE-SciTail to match the evaluation setup from Bansal et al. (2020b). The mixing weight  $\lambda$  is set to 0.5 for all our experiments. More details can be found in Appendix A.2.2.
Results. We report results on the synthetic HANS-FEWSHOT setting in Table 4, where we find that DRECA improves over all baselines. In particular, we observe an improvement of  $+6.94$  over MULTITASK (FINETUNE + K-NN) and  $+4.3$  over MAML-BASE. Moreover, we observe that MAML-DRECA obtains a comparable accuracy as MAML-ORACLE.
Next, we report results on our 3 real NLI settings in Table 3. Again, we find that DRECA improves model performance across all 3 settings: MAML-DRECA improves over MAML-BASE by +2.5 points on COMBINEDNLI-QANLI, +2.7 points on COMBINEDNLI-RTE and +1.6 points on GLUE-SciTail. On GLUE-SciTail, we compare against SMLMT (Bansal et al., 2020b) and find that MAML-DRECA improves over it by 1.5 accuracy points. However, we note that the confidence intervals of these approaches overlap, and also that (Bansal et al., 2020a) consider the entire GLUE data to train the meta-learner whereas we only consider NLI datasets within GLUE.
Model	COMBINEDNLI-QANLI	COMBINEDNLI-RTE	GLUE-SciTail
MULTITASK (FINETUNE)	69.66 ± 0.39	65.47 ± 3.19	75.80 ± 2.58
MULTITASK (K-NN)	68.97 ± 1.26	63.69 ± 6.65	69.76 ± 3.74
MULTITASK (FINETUNE + K-NN)	67.38 ± 2.61	66.52 ± 5.48	76.44 ± 1.77
MAML-BASE	69.43 ± 0.81	72.61 ± 0.85	76.38 ± 1.25
SMLMT (Bansal et al., 2020b)	-	-	76.75 ± 2.08
MAML-DRECA	71.98 ± 0.79	75.36 ± 0.69	77.91 ± 1.60
Table 3: Results on NLI few-shot learning. We report the mean and  $95\%$  confidence intervals assuming accuracies follow a Gaussian. Bolded cells represent the best mean accuracy for the particular dataset. For all settings except GLUE-SciTail, we consider 2 way 1 shot adaptation. For GLUE-SciTail, we consider 2 way 4 shot adaptation. SMLMT numbers are taken directly from Bansal et al. (2020b).
Model	HANS-FEWSHOT
MULTITASK (FINETUNE)	80.76 ± 1.83
MULTITASK (K-NN)	70.35 ± 2.29
MULTITASK (FINETUNE + K-NN)	80.59 ± 1.63
MAML-BASE	82.64 ± 1.80
MAML-ORACLE	86.74 ± 1.06
MAML-DRECA	87.53 ± 2.38
Table 4: Results on HANS-FEWSHOT. We report the mean and  $95\%$  confidence intervals assuming accuracies follow a Gaussian.
Dataset	#Reasoning Categories	Cluster purity
HANS-FEWSHOT	10	85.6%
Table 5: Measuring cluster purity. Our model is effective at recovering underlying reasoning types.
7.1 Visualizing the geometry of finetuned BERT on HANS-FEWSHOT
We start by visualizing finetuned BERT embeddings used by DRECA for HANS-FEWSHOT. As mentioned earlier, HANS consists of 30 manually defined syntactic templates which can be grouped into 15 reasoning categories. Following the procedure for EMBED() (details in Appendix A.2.2), we finetune BERT (Devlin et al., 2019) for 5000 randomly chosen examples from HANS. To obtain a vector representation for each example  $x = (p,h)$ , we concatenate the vector at the [CLS] token, along with a mean pooled representation of the premise and hypothesis. We then use t-SNE (Maaten and Hinton, 2008) to project these representations onto 2 dimensions. Each point in Fig. 4 is colored with its corresponding reasoning category, and we can observe a clear clustering of examples according
Figure 4: t-SNE plot of BERT vectors after finetuning on HANS. We see distinct clusters corresponding to the various reasoning categories.
to their reasoning category.
7.2 Evaluating DRECA Cluster Purity
To understand if reasoning categories can be accurately recovered with our approach, we measure the purity of DRECA clusters for HANS-FEWSHOT where true reasoning categories are known. This is evaluated by computing the number of examples belonging to the majority reasoning type for each cluster and then dividing by the total number of examples. From Table 5, we observe high cluster purity which provides evidence that DRECA is able to recover true reasoning categories.
7.3 Distribution of linguistic phenomena across clusters
We seek to understand how different linguistic phenomena present in the overall population are distributed among various clusters. To perform this analysis, we focus on MultiNLI annotation tags from Williams et al. (2018). A subset of examples in MultiNLI are assigned tags based on the presence of certain keywords, e.g., time words like days of the week; quantifiers like every, each, some;
negation words like no, not, never. Additionally, certain tags are assigned based on the PTB (Marcus et al., 1993) parses of examples, e.g., presence or absence of adjectives/adverbs etc. For each annotation tag, we compute the fraction of examples labeled with that tag in each cluster. We visualize this for 10 annotation tags and indicate statistically significant deviations from the averages in Fig. 5. Statistical significance is measured with binomial testing with a Bonferroni correction to account for multiple testing.
For every annotation tag, we shade all clusters that contain a statistically significant deviation from the mean. For instance, there is a positive cluster with 2.5 fold enrichment in Negation tags compared to the average, and a negative cluster that contains over 4 times the population average of Negation (Hyp only) tags. Similarly, among Conditionals, we have positive clusters that contain 1.4 times the population average and a negative cluster containing half the population average. Interestingly, we find most positive clusters to be significantly impoverished in Adverb (Hyp only) tags, while most negative clusters are enriched in these tags. This analysis presents evidence that clusters used by DRECA localize linguistic phenomena to a small number of clusters.
Comparing with CACTUs. Our work is most similar to CACTUs from Hsu et al. (2019). Apart from differences in the modality considered (text vs images), we differ in the following ways. Conceptually, Hsu et al. (2019) consider a fully unsupervised meta-learning setting where no labels are provided and use cluster IDs to induce labels, while our goal is to produce additional tasks in a supervised meta-learning setting. Second, CACTUs tasks are constructed by directly applying k-means on the entire training dataset while we apply k-means separately on each label group and construct tasks by choosing a cluster from each label group, leading to tasks with uniform label distribution. Finally, while CACTUs uses constructed tasks directly, our work using them to augment the original task distribution.
Number of examples in support set. All evaluation in this work considers small support sets where number of examples per label range from 1-4. This setting is somewhat restrictive since in practice, one might be able to get a few hundred
(a) adj/adv-hyp-only-ptb
(c) Negation
(e) Conditionals
(g) Modals
Figure 5: Fraction of cluster examples belonging to each linguistic annotation tag. Cluster groups corresponding to entailment (non-entailment) are colored green (red). The fraction of examples in the overall population is marked with a dashed line. We observe that many clusters have a statistically significant oversampling / undersampling of certain tags (drawn with a lighter color), according to a binomial test with a p-value of 0.05 under a Bonferroni correction for multiple testing.
(b) Plural (Premise Only)
(d) Negation (Hyp only)
(f) Quantifiers
(i) Superlatives (Hyp only)
(h) Adverb (Hyp only)
(j) Belief Verbs
examples for the target domain. These moderately sized support sets could themselves be heterogeneous where adapting a single learner might be hard. In such cases, we can use a similar clustering approach to separate out the support set into homogeneous tasks and adapt a separate learner for each task. These learners could then be plugged into a mixture of experts framework for making predictions.
Using k-means to produce task refinements. While we are able to get sufficiently homogeneous clusters with k-means, we note one shortcoming with this approach. Any input has multiple attributes / factors of variations and it may be possible to create a clustering for each factor. The current k-means based approach doesn't model this since we only produce a single clustering of the data. For instance,  $x_{1} =$  The man was walking in the park  $\Rightarrow$  The man is not at home and  $x_{2} =$  He went with his friends to the mall  $\Rightarrow$  He is not at work can belong to the same cluster if the underlying metric is based on reasoning types. At the same time, it could also be clustered with  $x_{3} =$  The man was walking in the park  $\Rightarrow$  The woman is in the park if the distance metric is based on lexical similarity. A promising direction for future work is to explore these multi-clusterings based on the various factors of variation present in the training data.
Non-meta learning based few-shot adaptation. In this work, we use tools from meta-learning to directly optimize for few-shot behavior. While not directly comparable to us, there have been many recent approaches to few-shot adaptation for NLP that do not use meta-learning. Brown et al. (2020) show impressive few-shot adaptation in large language models through "in-context learning" which is presumably acquired only through its language modeling objective,. Schick and Schütze (2020) train multiple models on lexical variations of a small support set and use these to label additional unlabeled examples from the target domain. These "self-labeled" examples are used to train a second model which can then make predictions on query examples. Finally, Gao et al. (2020) explore incontext learning of smaller language models for few-shot adaptation. In particular, they introduce a pipeline to identify useful prompts for the target domain, along with informative labeled examples to comprehend as context for the LM.
Many papers point out fundamental challenges in creating systems that achieve human-like understanding of tasks like NLI. Here, we studied conditions under which systems can learn from extremely few samples. We believe that such systems would complement and enhance further study into more sophisticated challenges such as model extrapolation.
One of the main ingredients for successful application of meta-learning is a large number of high quality training tasks to sample learning episodes for the meta-learner. We observe that such a task distribution is usually not available for important NLP problems, leading to less desirable ad hoc attempts that treat entire datasets as tasks. In response, we propose DRECA as a simple and general purpose task-augmentation strategy. Our approach creates a refinement of the original set of tasks (entire datasets) that roughly correspond to linguistic phenomena present in the dataset. We show that training on a task distribution augmented with DRECA leads to consistent improvements on 4 NLI few-shot classification problems, matching other approaches that require additional unlabeled data and well as oracles that have access to the true task distribution.
We are grateful to Eric Mitchell, Robin Jia, Alex Tamkin, John Hewitt, Pratyusha Sharma and the anonymous reviewers for helpful comments. The authors would also like to thank other members of the Stanford NLP group for feedback on an early draft of the paper. This work has been partially supported by JD.com American Technologies Corporation ("JD") under the SAIL-JD AI Research Initiative and partially by Toyota Research Institute ("TRI"). This article solely reflects the opinions and conclusions of its authors and not JD, any entity associated with JD.com, TRI, or any other Toyota entity. Christopher Manning is a CIFAR Fellow.
11 Reproducibility
Code and model checkpoints will be available at https://github.com/MurtyShikhar/DReCA.
Trapit Bansal, Rishikesh Jha, and Andrew McCallum. 2020a. Learning to few-shot learn across diverse natural language classification tasks. In Proceedings of the 28th International Conference on Computational Linguistics (COLING).
Trapit Bansal, Rishikesh Jha, Tsendsuren Munkhdalai, and Andrew McCallum. 2020b. Self-supervised meta-learning for few-shot natural language classification tasks. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 522-534.
Yujia Bao, Menghua Wu, Shiyu Chang, and Regina Barzilay. 2020. Few-shot text classification with distributional signatures. In International Conference on Learning Representations.
Roy Bar-Haim, Ido Dagan, Bill Dolan, Lisa Ferro, Danilo Giampiccolo, Bernardo Magnini, and Idan Szpektor. 2006. The second PASCAL recognising textual entailment challenge. In Proceedings of the Second PASCAL Challenges Workshop on Recognising Textual Entailment, pages 1-9.
Samy Bengio, Yoshua Bengio, Jocelyn Cloutier, and Jan Gecsei. 1997. On the optimization of a synaptic learning rule. In Daniel S. Levine and Wesley R. Elsberry, editors, Optimality in Biological and Artificial Networks?, pages 265-287. Routledge.
Luisa Bentivogli, Ido Dagan, Hoa Trang Dang, Danilo Giampiccolo, and Bernardo Magnini. 2009. The fifth PASCAL recognizing textual entailment challenge. In Proc. Text Analysis Conference (TAC'09).
Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. In Empirical Methods in Natural Language Processing (EMNLP).
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Advances in Neural Information Processing Systems, volume 33, pages 1877-1901. Curran Associates, Inc.
Ido Dagan, Oren Glickman, and Bernardo Magnini. 2005. The PASCAL recognising textual entailment challenge. In Proceedings of the First International Conference on Machine Learning Challenges: Evaluating Predictive Uncertainty Visual Object Classification, and Recognizing Textual Entailment, MLCW'05, page 177-190, Berlin, Heidelberg. Springer-Verlag.
Dorottya Demszky, Kelvin Guu, and Percy Liang. 2018. Transforming question answering datasets into natural language inference datasets. *ArXiv*, abs/1809.02922.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Zi-Yi Dou, Keyi Yu, and Antonios Anastasopoulos. 2019. Investigating meta-learning algorithms for low-resource natural language understanding tasks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 1192-1197, Hong Kong, China. Association for Computational Linguistics.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. 2017. Model-agnostic meta-learning for fast adaptation of deep networks. Proceedings of Machine Learning Research, 70:1126-1135.
Tianyu Gao, Adam Fisch, and Danqi Chen. 2020. Making pre-trained language models better few-shot learners. arXiv preprint arXiv:2012.15723.
Ruiying Geng, Binhua Li, Yongbin Li, Xiaodan Zhu, Ping Jian, and Jian Sun. 2019. Induction networks for few-shot text classification. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3904-3913, Hong Kong, China. Association for Computational Linguistics.
Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and Bill Dolan. 2007. The third PASCAL recognizing textual entailment challenge. In Proceedings of the ACL-07 Workshop on Textual Entailment and Paraphrasing, pages 1-9.
Edward Grefenstette, Brandon Amos, Denis Yarats, Phu Mon Htut, Artem Molchanov, Franziska Meier, Douwe Kiela, Kyunghyun Cho, and Soumith Chintala. 2019. Generalized inner loop meta-learning. arXiv preprint arXiv:1910.01727.
Jiatao Gu, Yong Wang, Yun Chen, Victor O. K. Li, and Kyunghyun Cho. 2018. Meta-learning for low-resource neural machine translation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3622-3631, Brussels, Belgium. Association for Computational Linguistics.
Xu Han, Hao Zhu, Pengfei Yu, Ziyun Wang, Yuan Yao, Zhiyuan Liu, and Maosong Sun. 2018. FewRel: A large-scale supervised few-shot relation classification dataset with state-of-the-art evaluation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4803-4809, Brussels, Belgium. Association for Computational Linguistics.
Nithin Holla, Pushkar Mishra, Helen Yannakoudakis, and Ekaterina Shutova. 2020. Learning to learn to disambiguate: Meta-learning for few-shot word sense disambiguation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings, pages 4517-4533, Online. Association for Computational Linguistics.
Kyle Hsu, Sergey Levine, and Chelsea Finn. 2019. Unsupervised learning via meta-learning. In International Conference on Learning Representations.
Pratik Joshi, S. Aditya, Aalok Sathe, and M. Choudhury. 2020. TaxiNLI: Taking a ride up the NLU hill. ArXiv, abs/2009.14505.
Tushar Khot, Ashish Sabharwal, and Peter Clark. 2018. Scitail: A textual entailment dataset from science question answering. In AAAI 2018.
Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. 2017. RACE: Large-scale ReAding comprehension dataset from examinations. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 785-794, Copenhagen, Denmark. Association for Computational Linguistics.
L. V. D. Maaten and Geoffrey E. Hinton. 2008. Visualizing data using t-SNE. Journal of Machine Learning Research, 9:2579-2605.
Bill MacCartney and Christopher D. Manning. 2008. Modeling semantic containment and exclusion in natural language inference. In Proceedings of COLING, pages 521-528, Manchester, UK.
M. Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. 1993. Building a large annotated corpus of English: The Penn treebank. Comput. Linguistics, 19:313-330.
Tom McCoy, Ellie Pavlick, and Tal Linzen. 2019. Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3428-3448, Florence, Italy. Association for Computational Linguistics.
Adam Poliak, Aparajita Haldar, Rachel Rudinger, J. Edward Hu, Ellie Pavlick, Aaron Steven White, and Benjamin Van Durme. 2018. Collecting Diverse Natural Language Inference Problems for Sentence Representation Evaluation. In Empirical Methods in Natural Language Processing (EMNLP).
Janarthanan Rajendran, Alexander Irpan, and Eric Jang. 2020. Meta-learning requires meta-augmentation. In Advances in Neural Information Processing Systems, volume 33, pages 5705-5715. Curran Associates, Inc.
Kyle Richardson, H. Hu, L. Moss, and A. Sabharwal. 2020. Probing natural language inference models through semantic fragments. *ArXiv*, abs/1909.07521.
Adam Santoro, Sergey Bartunov, M. Botvinick, Daan Wierstra, and T. Lillicrap. 2016. Meta-learning with memory-augmented neural networks. In ICML.
Timo Schick and Hinrich Schütze. 2020. It's not just size that matters: Small language models are also few-shot learners. ArXiv 2009.07118.
Jurgen Schmidhuber. 1987. Evolutionary principles in self-referential learning. on learning how to learn. Diploma thesis, Technische Universität München, Germany.
Jake Snell, Kevin Swersky, and Richard Zemel. 2017. Prototypical networks for few-shot learning. In Advances in neural information processing systems, pages 4077-4087.
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, koryakavukcuoglu, and Daan Wierstra. 2016. Matching networks for one shot learning. In D. D. Lee, M. Sugiyama, U. V. Luxburg, I. Guyon, and R. Garnett, editors, Advances in Neural Information Processing Systems 29, pages 3630-3638. Curran Associates, Inc.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In International Conference on Learning Representations.
Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1112-1122. Association for Computational Linguistics.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2019. Huggingface's transformers: State-of-the-art natural language processing. ArXiv, abs/1910.03771.
Mingzhang Yin, George Tucker, Mingyuan Zhou, Sergey Levine, and Chelsea Finn. 2020. Metalearning without memorization. In International Conference on Learning Representations.
Mo Yu, Xiaoxiao Guo, Jinfeng Yi, Shiyu Chang, Saloni Potdar, Yu Cheng, Gerald Tesauro, Haoyu Wang, and Bowen Zhou. 2018. Diverse few-shot text classification with multiple metrics. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1206-1215, New Orleans, Louisiana. Association for Computational Linguistics.
A.1 2D Sine Wave Regression: Training Details
We use a two layer neural network with 40 dimensional hidden representations and ReLU nonlinearity as the parameterization of  $f$ . Following Finn et al. (2017), we take a single gradient step on the support set at meta-training time, and take 10 gradient steps at meta-test time. The MAML weights are optimized with Adam and the inner loop adaptation is done with SGD with a learning rate of 1e-2. For each outer loop update, we sample 5 tasks, and each episode consists of a support set of size 5, i.e., we consider 5 shot adaptation.
A.2 NLI Experiments
A.2.1 Dataset Generation Details
We describe details of how our datasets are generated below. Note that all our datasets are in English.
HANS-FEWSHOT. The reasoning categories we use are in Table 6. We randomly split these 15 reasoning categories in HANS into training and test tasks. For each task, we sample 500 examples split equally among entailment and not_entailment.